pzhdfy opened a new pull request #6988: [Improvement] historical fast restart by lazy load columns metadata URL: https://github.com/apache/incubator-druid/pull/6988 We have large data in druid, historical (12 * 2T SATA HDD) will have over 10k segments and 10TB size. When we want to restart a historical to change configuration or after OOM， it will take 40 minutes , that is too slow. We profile the restart progress, and make a flame graph.  We can see io.druid.segment.IndexIO$V9IndexLoader.deserializeColumn cost most time. So we can make columns metadata lazy load , until it gets first used. After optimize, historical restart will only spend 2 minutes( 20X faster). And the flame graph after optimize is below, we can see load metadata spend little time.  we add a new config druid.segmentCache.lazyLoadOnStart (default is false), whether to do this optimize.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
