liupc commented on a change in pull request #23602: [SPARK-26674][CORE]Consolidate CompositeByteBuf when reading large frame URL: https://github.com/apache/spark/pull/23602#discussion_r255285755
########## File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportFrameDecoder.java ########## @@ -48,14 +49,29 @@ private static final int LENGTH_SIZE = 8; private static final int MAX_FRAME_SIZE = Integer.MAX_VALUE; private static final int UNKNOWN_FRAME_SIZE = -1; + private static final long DEFAULT_CONSOLIDATE_FRAME_BUFS_DELTA_THRESHOLD = 20 * 1024 * 1024; Review comment: What's your suggestion? I'm really torn on this name, what I mean is that if the delta size of input data since last consolidation exceeds the threshold, then the frameBuf should do consolidation. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
