clintropolis commented on issue #7919: disable all compression in intermediate segment persists while ingestion URL: https://github.com/apache/incubator-druid/pull/7919#issuecomment-503503838 Ah, I agree with you on all points about merge being a peak of like everything 😅 >When compression is disabled on intermediate segments, then average memory utilization would increase (more page cache used) but overall peak memory usage would decrease due to no decompression buffers allocated at time of merge. This is all I really want to investigate a bit deeper before we go with these defaults, because my gut thinks that it would only be lower peak usage in the cases where there are very large numbers of small segments with lots of columns, because it really takes a lot before the 64KB buffers start to become a significant user of direct memory. I think if this is _not_ the typical merge, then peak memory usage but in the form pressure on os free space, _could_ be a fair bit higher with uncompressed intermediary segments, because merging will need to hit all pages essentially cycling the entire uncompressed segment through free space. If the middle manager has the free space the spare, or sufficiently fast disks there is certainly no issue, but this could be pretty unchill if running co-tenant with a historical, or if free space is tight enough perhaps push into disk becoming the bottleneck due to more faults? I hopefully should have some time to fully review this PR soon, I will try to measure some things to see if I can make myself feel better about this being default :+1:
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
