liupc commented on issue #23602: [SPARK-26674][CORE]Consolidate CompositeByteBuf when reading large frame URL: https://github.com/apache/spark/pull/23602#issuecomment-459695341 The only risk now is when doing consolidate, the memory will double before it's done. Or maybe we should just replace ` CompositeByteBuf frame = buffers.getFirst().alloc().compositeBuffer(Integer.MAX_VALUE);` with `CompositeByteBuf frame = buffers.getFirst().alloc().compositeBuffer();` the default maxComponents value is 16 and if components exceeds this threshold, then consolidate would happen. so due to default socket buffer is small(usually <1M, according to net.ipv4.tcp_rmem), it's safe.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
