rickyma opened a new issue, #1651: URL: https://github.com/apache/incubator-uniffle/issues/1651
### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues. ### What would you like to be improved? After https://github.com/apache/incubator-uniffle/pull/1650, we can know that gRPC will not perform very well under high pressure. I think it's time for us to set the default server type to Netty, based on the following reasons. ## Netty brings about a 20% performance improvement compared to gRPC Refer to https://github.com/apache/incubator-uniffle/pull/1650. ## gRPC mode will cause higher CPU load We can see that gRPC mode will cause the machine's load to be much higher than Netty mode: <img width="750" alt="企业微信截图_17132589265203" src="https://github.com/apache/incubator-uniffle/assets/13834479/9402e63d-3d3e-4d9a-800d-12d61c2ef93d";> ## gRPC mode will cause memory usage to double We will find that in gRPC mode, both off-heap and on-heap memory may be heavily occupied. In extreme cases, this may cause memory usage to double:  This is because gRPC enables off-heap memory by default, and this part of the memory will be allocated and used by the gRPC framework. When the request enters the `ShuffleServerGrpcService` method, this part of the memory will be converted into on-heap memory and used in the business code. However, this is completely unnecessary, as we can simply use either off-heap memory or on-heap memory, without the need for conversion. ## gRPC mode will cause inaccurate `usedMemory` leading to OOM gRPC enables off-heap memory by default, and this part of the memory will be allocated and used by the gRPC framework, __which is not calculated in `usedMemory`__. By default, gRPC uses PooledByteBufAllocator to allocate off-heap memory for requests, with a chunkSize of 2MB. Therefore, when doing pre-allocation, it will also require allocating 2MB of off-heap memory. So in high-pressure and high-concurrency scenarios, we may easily encounter the following error exceptions when the shuffle server is receiving plenty of `SendShuffleDataRequest` and `RequireBufferRequest` at the same time: ``` [17:23:23:348] [client-data-transfer-79] ERROR org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.requirePreAllocation:257 - Exception happened when requiring pre-allocated buffer from 30.118.180.77:19977 io.grpc.StatusRuntimeException: UNAVAILABLE: GOAWAY shut down transport. HTTP/2 error code: INTERNAL_ERROR, debug data: failed to allocate 2097152 byte(s) of direct memory (used: 161059176727, max: 161061273600) at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268) ~[rss-client-spark3-shaded-0.9.0-SNAPSHOT.jar:?] at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249) ~[rss-client-spark3-shaded-0.9.0-SNAPSHOT.jar:?] at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167) ~[rss-client-spark3-shaded-0.9.0-SNAPSHOT.jar:?] at org.apache.uniffle.proto.ShuffleServerGrpc$ShuffleServerBlockingStub.requireBuffer(ShuffleServerGrpc.java:842) ~[rss-client-spark3-shaded-0.9.0-SNAPSHOT.jar:?] at org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.requirePreAllocation(ShuffleServerGrpcClient.java:255) ~[rss-client-spark3-shaded-0.9.0-SNAPSHOT.jar:?] at org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.lambda$sendShuffleData$0(ShuffleServerGrpcClient.java:476) ~[rss-client-spark3-shaded-0.9.0-SNAPSHOT.jar:?] at org.apache.uniffle.common.util.RetryUtils.retryWithCondition(RetryUtils.java:81) ~[rss-client-spark3-shaded-0.9.0-SNAPSHOT.jar:?] at org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.sendShuffleData(ShuffleServerGrpcClient.java:473) ~[rss-client-spark3-shaded-0.9.0-SNAPSHOT.jar:?] at org.apache.uniffle.client.impl.ShuffleWriteClientImpl.lambda$sendShuffleDataAsync$1(ShuffleWriteClientImpl.java:189) ~[rss-client-spark3-shaded-0.9.0-SNAPSHOT.jar:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ~[?:1.8.0_352] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_352] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_352] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_352] ``` ## gRPC does not support sending `ByteString` using off-heap memory Refer to https://github.com/grpc/grpc-java/issues/9704. ## More flexible Using Netty is more direct and flexible, and it won't be constrained by the gRPC layer's wrapping. ## Netty's iterative upgrades will be faster Netty's iterative upgrades will be faster, and the gRPC community cannot guarantee timely updates to Netty's version. So I think it's time for us to set the default server type to Netty. ### How should we improve? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
