advancedxy commented on issue #674: URL: https://github.com/apache/incubator-uniffle/issues/674#issuecomment-1455073619
> > This sounds good to me. Do you apply this in your prod env or not? > > No. > > I have to say after applying this optimization, the GC pause is still serious in my point to point writing benchmark. So maybe this is not the main problem. Anyway, this is still an improvement. > > BTW, I want to share the benchmark result and I think uniffle's users will benefit from this. > > Shuffle server environment > > 1. XMX 20G > 2. buffer capacity 10G > 3. read capacity 5G > > Shuffle Writer program > > 1. using spark to write shuffle data with 20 executors. Single executor will total write 1G, and each time write 14M > > Java version ShuffleServer GC Max pause time ThroughOutput > 8 G1 30s 0.3 > 11 G1 2.5s 0.8 > 18 G1 2.5s 0.8 > 18 ZGC 0.2ms 0.99997 > It's amazing for me. I think we should drop java8 version Is there any parameter turning for Java 8 to achieve similar performance gain like Java 11? As we discussed earlier, Hadoop 2 and various Hadoop 3 versions support java 8 only, https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions, I don't think we can drop Java 8 support. I believe that's why Spark still supports Java 8. So I believe we cannot simply change the default JDK version to 11 as you proposed in #683. However, it's always possible to build rss-server with alternative JDK 11 based images. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@uniffle.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
