chenboat commented on issue #4914: [POC] By-passing deep-store requirement for Realtime segment completion URL: https://github.com/apache/incubator-pinot/pull/4914#issuecomment-610066213 > In phase-2, we will do: > > 1. Intoduce a table config (inside stream config? or maybe in indexing config?) a boolean on whether to abort split commit if upload to deep store fails. This config is not used anywhere as yet. Since this is an architectural change, be sure to email dev mailing list, and also post in slack. Get consensus on where the config should go, and what exactly it means, naming, etc. These things can take time, and are not changeable once introduced. What is the purpose of this new config? It looks like we can achieve the same goal by adding a new parameter to control if to upload segment to controller? This is what I have done in the POC. The new segment committer in 2/3 above can invoke this new endpoint. https://github.com/apache/incubator-pinot/pull/4914/files#diff-f0bd4d0172521b3f6bd52d4ebe12da2b Shouldn't changing the LLC protocol endpoint be easier than adding a new table config? > 2. Change the controller to accept a new URL in the protocol (I suggest we insert the following when segment upload fails: "peer://peer/URI_INTRODUCED_IN_STEP_1". Or, simply the word in all caps "PEER". I like the previous option, because it is in URI format, and the controller can do one check with the URL decoder for the scheme. When the controller detects peer, it should not set the download URL in the metadata (or, insert a null, but I prefer NO url in metadata). Add unit tests to cover this case. Since the server never inserts PEER, we should be good so far. I see. So Pinot servers will not return an empty segment location to controller, right? > 3. Change the server to include logic to fetch from peer (using http or https fetcher as configured) IF the fetch from deep store fails, include retries as discussed in phone conference. (i.e. try deep store, if. it fails, try peer, if that fails, try the whole sequence again, N times). Try not to hook up this logic as yet. Add unit tests > 4. Change realtime segment fixer in the controller to fix segments that are not there in deep store. Add integration tests. > 5. Hook up logic in 6, and change the split committer to use the table config, add an integration test > > You are done at this point
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
