klsince commented on issue #7320: URL: https://github.com/apache/pinot/issues/7320#issuecomment-901505575
Thanks for the 3rd proposal! I'm doing both option 1 (force download) and 2 (server local cleanup) currently. Option 1 is more like one last resort to bring things back to life, not necessarily just for index cleanup. Option 2 is to clean indices more efficiently and supposed to happen more frequently; and as it's done during segment reloading, it can leverage the existing reloading mechanism to be atomic, thus transparent to queries on the fly. Option 3 - using Minion - I don't see it conflicts with the options above. When minion tasks are combing the segments, they can do cleanup, and also things like merging and rollup as proposed in another project. But as it's happening async and potentially outside server hosts, it might be more costly due to data transfer and non-trivial to swap in the new segment transparently to queries on the fly. Thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
