Thanks for the feedback and discussions from Subbu, Seunghyun and Jackie for the new table configuration field to enable peer segment download. In the end, we decided to
- Add a new optional string field *peerSegmentDownloadScheme* to the SegmentsValidationAndRetentionConfig in the TableConfig. The value can be *http* or *https*. The field will enable download of segments for both realtime and offline table segments from peer servers. In the beginning, only realtime table segments download will be supported. The design details can be found in this section of the cwiki doc <https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion#By-passingdeep-storerequirementforRealtimesegmentcompletion-EnablebesteffortsegmentuploadinSplitSegmentCommiteranddownloadsegmentfrompeerservers.>. I will send a PR soon to add this table config. Thanks, Ting On Tue, May 5, 2020 at 5:16 PM TING CHEN <tingc...@uber.com> wrote: > > As part of the proposal > <https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion> > to bypass deep store for segment completion, I plan to add a new optional > string field *peerSegmentDownloadScheme* to > the SegmentsValidationAndRetentionConfig in the TableConfig. The value can > be *http* or *https*. > > 1. SplitSegmentCommitter > > <https://github.com/apache/incubator-pinot/blob/31c55afdb6a40f98189308ce6292587ead9d0dec/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SplitSegmentCommitter.java> > will check this value. If it exists, the segment committer will be able to > finish segment commit successfully even if the upload to the segment store > fails. The committer will report a special marker to the controller about > the segment is available in peer servers. > 2. When Pinot servers fail to download segments from the segment > store, they can also check this field's value. If it exists, it can > download segments from peer servers using either HTTP and HTTPS segment > fetchers as configured. (related PR > <https://github.com/apache/incubator-pinot/pull/5336> in review for > how to discover such servers.) > > Note this is a table level config. We will test the new download behavior > in realtime tables in incremental fashion. Once fully proven, this config > can be upgraded to server level config. > > Please let me know if you have any questions on this. Thanks @mcvsubbu for > coming up with the idea and offline discussions. > > Ting Chen > >
