LuciferYang commented on a change in pull request #35835:
URL: https://github.com/apache/spark/pull/35835#discussion_r825428909
##########
File path:
common/network-common/src/main/java/org/apache/spark/network/util/NettyUtils.java
##########
@@ -160,6 +160,12 @@ public static PooledByteBufAllocator
createPooledByteBufAllocator(
if (numCores == 0) {
numCores = Runtime.getRuntime().availableProcessors();
}
+ // SPARK-38541: After upgrade to Netty 4.1.75, there are 2 behavior
changes of this method:
+ // 1. `PooledByteBufAllocator.defaultMaxOrder()` change from 11 to 9, this
means the default
+ // `PooledByteBufAllocator` chunk size reduce from 16 MiB to 4 MiB, we
need use
+ // `-Dio.netty.allocator.maxOrder=11` to keep the chunk size of
PooledByteBufAllocator to 16m.
+ // 2. `PooledByteBufAllocator.defaultUseCacheForAllThreads()` change from
true to false, we need
+ // to use `-Dio.netty.allocator.useCacheForAllThreads=true` to enable
`useCacheForAllThreads`.
Review comment:
Since the default values of `PooledByteBufAllocator.defaultMaxOrder()`
and `PooledByteBufAllocator.defaultUseCacheForAllThreads()` are changed after
Netty 4.1.75, the `PooledByteBufAllocator` returned by this method change to
disabled `useCacheForAllThreads` as default and the chunk size of
`PooledByteBufAllocator` reduce from 16 MiB to 4 MiB.
I'd like to know what you think of this @srowen @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
