No, I mean the follwing: (1) Segment download URI in metadata will have either a valid URI (from deepstore) OR be empty. (2) In case it is empty, we construct the URIs of all selected peers and pass it to the segment fetcher (3) We add a new method to the SegmentFetcher interface that takes a list of URIs instead of a single URI (4) We modify the retry logic in the base class to pick a random one from the list (even if the list size is 1). (5) default implementation for the list could be to take a random URI (or the first one, or whatever) from the list and call the existing method of one URI
=Subbu On 2020/06/11 22:09:05, TING CHEN <[email protected]> wrote: > You mean multiple URIs in a segment's download url. No for this project. > > On Thu, Jun 11, 2020 at 2:59 PM kishore g <[email protected]> wrote: > > > +1 peer. > > > > unrelated to this - do we support multiple URI's? > > > > On Thu, Jun 11, 2020 at 2:51 PM Subbu Subramaniam <[email protected]> > > wrote: > > > > > Hey Ting, > > > > > > I like the URI in metadata as "peer:///segmentName". This way, the URI > > > remains parsable and we can use the scheme to check for a segment > > fetcher, > > > > > > thanks > > > > > > -Subbu > > > > > > On 2020/06/10 01:09:25, TING CHEN <[email protected]> wrote: > > > > As part of the deep store by-passing > > > > < > > > > > https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion > > > > > > > > project, a server is allowed to download segments from peer Pinot > > servers > > > > during Low Level Consumer (LLC) instead of deep segment stores. To > > enable > > > > this feature, we plan to add a new URI format for the field > > > > *segment.realtime.download.url > > > > *in LLCRealtimeSegmentZKMetadata. > > > > > > > > The new URI format serves the purpose of instructing a Pinot server to > > > find > > > > and download the segment from a peer server. Controller writes it to > > > Helix > > > > in case of segment upload failure or no deep store configured at all. > > We > > > > proposed the following format options and want to hear your feedback: > > > > > > > > 1. peer:///segmentName; (my preference) > > > > 2. simply an empty string *''* > > > > > > > > Both are in essence specially markers to indicate that the segment is > > not > > > > found in deep store and servers have to download them from peer > > servers. > > > > (1) has the benefit of better readability than (2) for debugging > > > purposes. > > > > > > > > Please let me know what you think. > > > > > > > > Ting Chen > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
