I thought we agreed on not having a special URL for peer download (peer download is always the backup if URL download failed or URL does not exist). When did we come back to the URL approach? Why do we need this special URL to indicate that the server should try to download from peer? If the regular download fails, the server should always ask the segment from peer right?
On Thu, Jun 11, 2020 at 3:09 PM TING CHEN <[email protected]> wrote: > > You mean multiple URIs in a segment's download url. No for this project. > > On Thu, Jun 11, 2020 at 2:59 PM kishore g <[email protected]> wrote: > > > +1 peer. > > > > unrelated to this - do we support multiple URI's? > > > > On Thu, Jun 11, 2020 at 2:51 PM Subbu Subramaniam <[email protected]> > > wrote: > > > > > Hey Ting, > > > > > > I like the URI in metadata as "peer:///segmentName". This way, the URI > > > remains parsable and we can use the scheme to check for a segment > > fetcher, > > > > > > thanks > > > > > > -Subbu > > > > > > On 2020/06/10 01:09:25, TING CHEN <[email protected]> wrote: > > > > As part of the deep store by-passing > > > > < > > > > > https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion > > > > > > > > project, a server is allowed to download segments from peer Pinot > > servers > > > > during Low Level Consumer (LLC) instead of deep segment stores. To > > enable > > > > this feature, we plan to add a new URI format for the field > > > > *segment.realtime.download.url > > > > *in LLCRealtimeSegmentZKMetadata. > > > > > > > > The new URI format serves the purpose of instructing a Pinot server to > > > find > > > > and download the segment from a peer server. Controller writes it to > > > Helix > > > > in case of segment upload failure or no deep store configured at all. > > We > > > > proposed the following format options and want to hear your feedback: > > > > > > > > 1. peer:///segmentName; (my preference) > > > > 2. simply an empty string *''* > > > > > > > > Both are in essence specially markers to indicate that the segment is > > not > > > > found in deep store and servers have to download them from peer > > servers. > > > > (1) has the benefit of better readability than (2) for debugging > > > purposes. > > > > > > > > Please let me know what you think. > > > > > > > > Ting Chen > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
