xCASx commented on issue #24732: [SPARK-27868][core] Better default value and documentation for socket server backlog. URL: https://github.com/apache/spark/pull/24732#issuecomment-574883825 First of all, sorry, "the default JVM value" is just a poor wording from my side. I blindly repeated it from the initial post of this thread. What I meant is **OS default**. Here's what it looks like: `org.apache.spark.network.server.TransportServer#init` creates `NioServerSocketChannel`. The channel uses `NioServerSocketChannelConfig`, which is a successor of `io.netty.channel.socket.DefaultServerSocketChannelConfig`. The latter [contains two member fields](https://github.com/netty/netty/blob/0cde4d9cb4d19ddc0ecafc5be7c5f7c781a1f6e9/transport/src/main/java/io/netty/channel/socket/DefaultServerSocketChannelConfig.java#L43): ```java protected final ServerSocket javaSocket; private volatile int backlog = NetUtil.SOMAXCONN; ``` The first one is in fact a `ServerSocket`, which defines default value of `backlog` equal to `50`. But this doesn't matter, because even being defined, this value is not used per se. The real value for backlog comes from the above-mentioned `io.netty.channel.socket.DefaultServerSocketChannelConfig#backlog`, as you can see it [here](https://github.com/netty/netty/blob/6c05d16967df5cadcebcc498223eec3f25bacf89/transport/src/main/java/io/netty/channel/socket/nio/NioServerSocketChannel.java#L132): ```java protected void doBind(SocketAddress localAddress) throws Exception { if (PlatformDependent.javaVersion() >= 7) { javaChannel().bind(localAddress, config.getBacklog()); } else { javaChannel().socket().bind(localAddress, config.getBacklog()); } } ``` And the value of `NetUtil.SOMAXCONN` is [OS dependent](https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/NetUtil.java#L253). For `EpollServerSocketChannel` the logic is similar. So, the default value in fact neither `50` (I assume, it has never been since Spark migration to netty v4.x), nor even hardcoded (except for Windows it is `200`). For Linux the value is taken from `/proc/sys/net/core/somaxconn`. If the file doesn't exist, it will be resolved from kernel attributes `kern.ipc.somaxconn` and `kern.ipc.soacceptqueue`. Only if all mentioned sources aren't defined, the hardcoded value of `128` will be used. Not only the default value is twice (more than three times for Windows) bigger than `64`, but also we forced `/proc/sys/net/core/somaxconn` to be `8192` long time ago...
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
