liupc commented on issue #23602: [SPARK-26674][CORE]Consolidate CompositeByteBuf when reading large frame URL: https://github.com/apache/spark/pull/23602#issuecomment-459666639 I have done some benchmark tests on my local machine, seems it can save large memory with small time cost -- ~ 200 milis for 50% saving from a 1GB CompositeByteBuf -------------------------------------------------------------------------------------------------------- Java HotSpot(TM) 64-Bit Server VM 1.8.0_111-b14 on Linux 4.15.0-43-generic Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz [test consolidate 100 buffers each with 10m, 50% used for 1 loop] Allocating 5242880 bytes Time cost with 1 loop for consolidating: 223 millis [test consolidate 100 buffers each with 10m, 100% used for 1 loop] Allocating 10485760 bytes Time cost with 1 loop for consolidating: 451 millis [test consolidate 100 buffers each with 10m, 50% used for 10 loop] Allocating 5242880 bytes Time cost with 10 loop for consolidating: 1924 millis [test consolidate 100 buffers each with 10m, 100% used for 10 loop] Allocating 10485760 bytes Time cost with 10 loop for consolidating: 3787 millis [test consolidate 100 buffers each with 10m, 50% used for 50 loop] Allocating 5242880 bytes Time cost with 50 loop for consolidating: 8870 millis [test consolidate 100 buffers each with 10m, 100% used for 50 loop] Allocating 10485760 bytes Time cost with 50 loop for consolidating: 17472 millis [test consolidate 20 buffers each with 50m, 50% used for 1 loop] Allocating 26214400 bytes Time cost with 1 loop for consolidating: 184 millis [test consolidate 20 buffers each with 50m, 100% used for 1 loop] Allocating 52428800 bytes Time cost with 1 loop for consolidating: 367 millis [test consolidate 20 buffers each with 50m, 50% used for 10 loop] Allocating 26214400 bytes Time cost with 10 loop for consolidating: 1847 millis [test consolidate 20 buffers each with 50m, 100% used for 10 loop] Allocating 52428800 bytes Time cost with 10 loop for consolidating: 3638 millis [test consolidate 20 buffers each with 50m, 50% used for 50 loop] Allocating 26214400 bytes Time cost with 50 loop for consolidating: 9126 millis [test consolidate 20 buffers each with 50m, 100% used for 50 loop] Allocating 52428800 bytes Time cost with 50 loop for consolidating: 19391 millis [test consolidate 10 buffers each with 100m, 50% used for 1 loop] Allocating 52428800 bytes Time cost with 1 loop for consolidating: 211 millis [test consolidate 10 buffers each with 100m, 100% used for 1 loop] Allocating 104857600 bytes Time cost with 1 loop for consolidating: 400 millis [test consolidate 10 buffers each with 100m, 50% used for 10 loop] Allocating 52428800 bytes Time cost with 10 loop for consolidating: 1954 millis [test consolidate 10 buffers each with 100m, 100% used for 10 loop] Allocating 104857600 bytes Time cost with 10 loop for consolidating: 3846 millis [test consolidate 10 buffers each with 100m, 50% used for 50 loop] Allocating 52428800 bytes Time cost with 50 loop for consolidating: 9747 millis [test consolidate 10 buffers each with 100m, 100% used for 50 loop] Allocating 104857600 bytes Time cost with 50 loop for consolidating: 19542 millis
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
