liupc edited a comment on issue #23602: [SPARK-26674][CORE]Consolidate CompositeByteBuf when reading large frame URL: https://github.com/apache/spark/pull/23602#issuecomment-461706206 After refine, the perf tests show that consolidation can work well for reading 1GB block with little extra memory(use consolidate(cIndex, numComponents) to avoid unnecessary consolidation for already consolidated Components). Here is the newest test results, seems 20MiB is enough for the threshold. Results with my laptop at low battery. ``` Build frame buf with consolidation threshold 1048576 cost 6057 milis Build frame buf with consolidation threshold 5242880 cost 4899 milis Build frame buf with consolidation threshold 10485760 cost 2809 milis Build frame buf with consolidation threshold 20971520 cost 2762 milis Build frame buf with consolidation threshold 31457280 cost 3074 milis Build frame buf with consolidation threshold 52428800 cost 2399 milis Build frame buf with consolidation threshold 83886080 cost 4010 milis Build frame buf with consolidation threshold 104857600 cost 2808 milis Build frame buf with consolidation threshold 314572800 cost 4150 milis Build frame buf with consolidation threshold 524288000 cost 2519 milis Build frame buf with consolidation threshold 9223372036854775807 cost 1664 milis // no consolidation ``` Results with my laptop with normal battery. ``` Build frame buf with consolidation threshold 1048576 cost 3356 milis Build frame buf with consolidation threshold 5242880 cost 2075 milis Build frame buf with consolidation threshold 10485760 cost 1948 milis Build frame buf with consolidation threshold 20971520 cost 1421 milis Build frame buf with consolidation threshold 31457280 cost 1371 milis Build frame buf with consolidation threshold 52428800 cost 1286 milis Build frame buf with consolidation threshold 83886080 cost 1769 milis Build frame buf with consolidation threshold 104857600 cost 2068 milis Build frame buf with consolidation threshold 314572800 cost 2448 milis Build frame buf with consolidation threshold 524288000 cost 2317 milis Build frame buf with consolidation threshold 9223372036854775807 cost 1072 milis ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
