liupc commented on issue #23602: [SPARK-26674][CORE]Consolidate CompositeByteBuf when reading large frame URL: https://github.com/apache/spark/pull/23602#issuecomment-459944614 But, I come up with another idea, we can just check the writerIndex of the compositebytebuf, and if the delta exceeds some threshold then we can do consolidation. **MemoryOverhead for shuffle?** Actually, we can assume sixty percent of memoryOverhead can be used in shuffle, because for most applications, when in shuffle phase, the memoryOverhead are mainly used here. **How to set the threshold?** In the worst case, the writerIndex is equal to the capacity, then we must reserve half of this memory for consolidation, thus we get 0.3 as the threshold. This is a conservative estimation, for most cases, we can make the threshold higher, but we just keep it unchanged for better safety. Then, let's say we fetch a 1GB shuffle block, then we got 300M as the threshold. **How about the performance?** ``` // --- Test Reports for plan 3 --- // [test consolidate 1000 buffers each with 1m, 50% used for 1 loop] // Allocating 524288 bytes // Time cost with 1 loop for consolidating: 116 millis // // [test consolidate 1000 buffers each with 1m, 100% used for 1 loop] // Allocating 1048576 bytes // Time cost with 1 loop for consolidating: 635 millis // // [test consolidate 1000 buffers each with 1m, 50% used for 10 loop] // Allocating 524288 bytes // Time cost with 10 loop for consolidating: 1003 millis // // [test consolidate 1000 buffers each with 1m, 100% used for 10 loop] // Allocating 1048576 bytes // Time cost with 10 loop for consolidating: 5702 millis // // [test consolidate 1000 buffers each with 1m, 50% used for 50 loop] // Allocating 524288 bytes // Time cost with 50 loop for consolidating: 4799 millis // // [test consolidate 1000 buffers each with 1m, 100% used for 50 loop] // Allocating 1048576 bytes // Time cost with 50 loop for consolidating: 28440 millis // // [test consolidate 100 buffers each with 10m, 50% used for 1 loop] // Allocating 5242880 bytes // Time cost with 1 loop for consolidating: 96 millis // // [test consolidate 100 buffers each with 10m, 100% used for 1 loop] // Allocating 10485760 bytes // Time cost with 1 loop for consolidating: 571 millis // // [test consolidate 100 buffers each with 10m, 50% used for 10 loop] // Allocating 5242880 bytes // Time cost with 10 loop for consolidating: 940 millis // // [test consolidate 100 buffers each with 10m, 100% used for 10 loop] // Allocating 10485760 bytes // Time cost with 10 loop for consolidating: 5727 millis // // [test consolidate 100 buffers each with 10m, 50% used for 50 loop] // Allocating 5242880 bytes // Time cost with 50 loop for consolidating: 4739 millis // // [test consolidate 100 buffers each with 10m, 100% used for 50 loop] // Allocating 10485760 bytes // Time cost with 50 loop for consolidating: 28356 millis // // [test consolidate 20 buffers each with 50m, 50% used for 1 loop] // Allocating 26214400 bytes // Time cost with 1 loop for consolidating: 96 millis // // [test consolidate 20 buffers each with 50m, 100% used for 1 loop] // Allocating 52428800 bytes // Time cost with 1 loop for consolidating: 577 millis // // [test consolidate 20 buffers each with 50m, 50% used for 10 loop] // Allocating 26214400 bytes // Time cost with 10 loop for consolidating: 966 millis // // [test consolidate 20 buffers each with 50m, 100% used for 10 loop] // Allocating 52428800 bytes // Time cost with 10 loop for consolidating: 5731 millis // // [test consolidate 20 buffers each with 50m, 50% used for 50 loop] // Allocating 26214400 bytes // Time cost with 50 loop for consolidating: 4826 millis // // [test consolidate 20 buffers each with 50m, 100% used for 50 loop] // Allocating 52428800 bytes // Time cost with 50 loop for consolidating: 28734 millis // // [test consolidate 10 buffers each with 100m, 50% used for 1 loop] // Allocating 52428800 bytes // Time cost with 1 loop for consolidating: 98 millis // // [test consolidate 10 buffers each with 100m, 100% used for 1 loop] // Allocating 104857600 bytes // Time cost with 1 loop for consolidating: 576 millis // // [test consolidate 10 buffers each with 100m, 50% used for 10 loop] // Allocating 52428800 bytes // Time cost with 10 loop for consolidating: 965 millis // // [test consolidate 10 buffers each with 100m, 100% used for 10 loop] // Allocating 104857600 bytes // Time cost with 10 loop for consolidating: 6122 millis // // [test consolidate 10 buffers each with 100m, 50% used for 50 loop] // Allocating 52428800 bytes // Time cost with 50 loop for consolidating: 5228 millis // // [test consolidate 10 buffers each with 100m, 100% used for 50 loop] // Allocating 104857600 bytes // Time cost with 50 loop for consolidating: 30797 millis ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
