jihoonson commented on issue #11297: URL: https://github.com/apache/druid/issues/11297#issuecomment-847567539
`IntermediaryDataManger` currently runs on middleManagers and indexers. Do you mean making them to be able to discover and clean up shuffle data in deep storage? I'm not sure how they can work in parallel. Can you elaborate more on your idea? > @jihoonson since you have implemented this feature originally it would be good to hear any ideas you have. The simplest idea I can think of is having an auto cleanup process in the coordinator (or the overlord?). This single process periodically scans deep storage and cleans up all shuffle data that is no longer necessary to keep. This can be expensive, but it may be OK since it's a periodic process that can have enough time between runs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
