deemoliu commented on issue #9529: URL: https://github.com/apache/pinot/issues/9529#issuecomment-1339740483
After discussion with @Jackie-Jiang @yupeng9 @chenboat We can break down the feature into the following part. - Design doc updates - 1. When committing segment, update replaceSegment to clean up keys - 1.1 clean up keys in primary key indexes - 1.2 generate snapshot locally - 1.3 [Deepstore] upload snapshot to Deepstore - 2. periodic job in pinot controller (upload snapshot if not persisted) - 3. add a download snapshot api on the server side. - 4. when loading segments, get snapshot to avoid re-compute - 4.1 get snapshot from peer server - 4.2 [Deepstore] get snapshot from Deepstore -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
