liupc commented on issue #23602: [SPARK-26674][CORE]Consolidate CompositeByteBuf when reading large frame URL: https://github.com/apache/spark/pull/23602#issuecomment-461706206 After refine, the perf tests show that we can save large amount of memory with small time cost. Here is the newest test results, seems 20MiB is enough for making consolidation works well. Results with my laptop at low battery. ``` Build frame buf with consolidation threshold 1048576 cost 6057 milis Build frame buf with consolidation threshold 5242880 cost 4899 milis Build frame buf with consolidation threshold 10485760 cost 2809 milis Build frame buf with consolidation threshold 20971520 cost 2762 milis Build frame buf with consolidation threshold 31457280 cost 3074 milis Build frame buf with consolidation threshold 52428800 cost 2399 milis Build frame buf with consolidation threshold 83886080 cost 4010 milis Build frame buf with consolidation threshold 104857600 cost 2808 milis Build frame buf with consolidation threshold 314572800 cost 4150 milis Build frame buf with consolidation threshold 524288000 cost 2519 milis Build frame buf with consolidation threshold 9223372036854775807 cost 1664 milis ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
