Re: solr performance with >1 NUMAs
Great updates. Thanks for keeping us all in the loop! On Thu, Oct 22, 2020 at 7:43 PM Wei wrote: > > Hi Shawn, > > I.m circling back with some new findings with our 2 NUMA issue. After a > few iterations, we do see improvement with the useNUMA flag and other JVM > setting changes. Here are the current settings, with Java 11: > > -XX:+UseNUMA > > -XX:+UseG1GC > > -XX:+AlwaysPreTouch > > -XX:+UseTLAB > > -XX:G1MaxNewSizePercent=20 > > -XX:MaxGCPauseMillis=150 > > -XX:+DisableExplicitGC > > -XX:+DoEscapeAnalysis > > -XX:+ParallelRefProcEnabled > > -XX:+UnlockDiagnosticVMOptions > > -XX:+UnlockExperimentalVMOptions > > > Compared to previous Java 8 + CMS on 2 NUMA servers, P99 latency has > improved over 20%. > > > Thanks, > > Wei > > > > > On Mon, Sep 28, 2020 at 4:02 PM Shawn Heisey wrote: > > > On 9/28/2020 12:17 PM, Wei wrote: > > > Thanks Shawn. Looks like Java 11 is the way to go with -XX:+UseNUMA. Do > > you > > > see any backward compatibility issue for Solr 8 with Java 11? Can we run > > > Solr 8 built with JDK 8 in Java 11 JRE, or need to rebuild solr with Java > > > 11 JDK? > > > > I do not know of any problems running the binary release of Solr 8 > > (which is most likely built with the Java 8 JDK) with a newer release > > like Java 11 or higher. > > > > I think Sun was really burned by such problems cropping up in the days > > of Java 5 and 6, and their developers have worked really hard to make > > sure that never happens again. > > > > If you're running Java 11, you will need to pick a different garbage > > collector if you expect the NUMA flag to function. The most recent > > releases of Solr are defaulting to G1GC, which as previously mentioned, > > did not gain NUMA optimizations until Java 14. > > > > It is not clear to me whether the NUMA optimizations will work with any > > collector other than Parallel until Java 14. You would need to check > > Java documentation carefully or ask someone involved with development of > > Java. > > > > If you do see an improvement using the NUMA flag with Java 11, please > > let us know exactly what options Solr was started with. > > > > Thanks, > > Shawn > >
Re: solr performance with >1 NUMAs
Hi Shawn, I.m circling back with some new findings with our 2 NUMA issue. After a few iterations, we do see improvement with the useNUMA flag and other JVM setting changes. Here are the current settings, with Java 11: -XX:+UseNUMA -XX:+UseG1GC -XX:+AlwaysPreTouch -XX:+UseTLAB -XX:G1MaxNewSizePercent=20 -XX:MaxGCPauseMillis=150 -XX:+DisableExplicitGC -XX:+DoEscapeAnalysis -XX:+ParallelRefProcEnabled -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions Compared to previous Java 8 + CMS on 2 NUMA servers, P99 latency has improved over 20%. Thanks, Wei On Mon, Sep 28, 2020 at 4:02 PM Shawn Heisey wrote: > On 9/28/2020 12:17 PM, Wei wrote: > > Thanks Shawn. Looks like Java 11 is the way to go with -XX:+UseNUMA. Do > you > > see any backward compatibility issue for Solr 8 with Java 11? Can we run > > Solr 8 built with JDK 8 in Java 11 JRE, or need to rebuild solr with Java > > 11 JDK? > > I do not know of any problems running the binary release of Solr 8 > (which is most likely built with the Java 8 JDK) with a newer release > like Java 11 or higher. > > I think Sun was really burned by such problems cropping up in the days > of Java 5 and 6, and their developers have worked really hard to make > sure that never happens again. > > If you're running Java 11, you will need to pick a different garbage > collector if you expect the NUMA flag to function. The most recent > releases of Solr are defaulting to G1GC, which as previously mentioned, > did not gain NUMA optimizations until Java 14. > > It is not clear to me whether the NUMA optimizations will work with any > collector other than Parallel until Java 14. You would need to check > Java documentation carefully or ask someone involved with development of > Java. > > If you do see an improvement using the NUMA flag with Java 11, please > let us know exactly what options Solr was started with. > > Thanks, > Shawn >
Re: solr performance with >1 NUMAs
On 9/28/2020 12:17 PM, Wei wrote: Thanks Shawn. Looks like Java 11 is the way to go with -XX:+UseNUMA. Do you see any backward compatibility issue for Solr 8 with Java 11? Can we run Solr 8 built with JDK 8 in Java 11 JRE, or need to rebuild solr with Java 11 JDK? I do not know of any problems running the binary release of Solr 8 (which is most likely built with the Java 8 JDK) with a newer release like Java 11 or higher. I think Sun was really burned by such problems cropping up in the days of Java 5 and 6, and their developers have worked really hard to make sure that never happens again. If you're running Java 11, you will need to pick a different garbage collector if you expect the NUMA flag to function. The most recent releases of Solr are defaulting to G1GC, which as previously mentioned, did not gain NUMA optimizations until Java 14. It is not clear to me whether the NUMA optimizations will work with any collector other than Parallel until Java 14. You would need to check Java documentation carefully or ask someone involved with development of Java. If you do see an improvement using the NUMA flag with Java 11, please let us know exactly what options Solr was started with. Thanks, Shawn
Re: solr performance with >1 NUMAs
Thanks Shawn. Looks like Java 11 is the way to go with -XX:+UseNUMA. Do you see any backward compatibility issue for Solr 8 with Java 11? Can we run Solr 8 built with JDK 8 in Java 11 JRE, or need to rebuild solr with Java 11 JDK? Best, Wei On Sat, Sep 26, 2020 at 6:44 PM Shawn Heisey wrote: > On 9/26/2020 1:39 PM, Wei wrote: > > Thanks Shawn! Currently we are still using the CMS collector for solr > with > > Java 8. When last evaluated with Solr 7, CMS performs better than G1 for > > our case. When using G1, is it better to upgrade from Java 8 to Java 11? > > From > https://lucene.apache.org/solr/guide/8_4/solr-system-requirements.html, > > seems Java 14 is not officially supported for Solr 8. > > It has been a while since I was working with Solr every day, and when I > was, Java 11 did not yet exist. I have no idea whether Java 11 improves > things beyond Java 8. That said ... all software evolves and usually > improves as time goes by. It is likely that the newer version has SOME > benefit. > > Regarding whether or not Java 14 is supported: There are automated > tests where all the important code branches are run with all major > versions of Java, including pre-release versions, and those tests do > include various garbage collectors. Somebody notices when a combination > doesn't work, and big problems with newer Java versions are something > that gets discussed on our mailing lists. > > Java 14 has been out for a while, with no big problems being discussed > so far. So it is likely that it works with Solr. Can I say for sure? > No. I haven't tried it myself. > > I don't have any hardware available where there is more than one NUMA, > or I would look deeper into this myself. It would be interesting to > find out whether the -XX:+UseNUMA option makes a big difference in > performance. > > Thanks, > Shawn >
Re: solr performance with >1 NUMAs
On 9/26/2020 1:39 PM, Wei wrote: Thanks Shawn! Currently we are still using the CMS collector for solr with Java 8. When last evaluated with Solr 7, CMS performs better than G1 for our case. When using G1, is it better to upgrade from Java 8 to Java 11? From https://lucene.apache.org/solr/guide/8_4/solr-system-requirements.html, seems Java 14 is not officially supported for Solr 8. It has been a while since I was working with Solr every day, and when I was, Java 11 did not yet exist. I have no idea whether Java 11 improves things beyond Java 8. That said ... all software evolves and usually improves as time goes by. It is likely that the newer version has SOME benefit. Regarding whether or not Java 14 is supported: There are automated tests where all the important code branches are run with all major versions of Java, including pre-release versions, and those tests do include various garbage collectors. Somebody notices when a combination doesn't work, and big problems with newer Java versions are something that gets discussed on our mailing lists. Java 14 has been out for a while, with no big problems being discussed so far. So it is likely that it works with Solr. Can I say for sure? No. I haven't tried it myself. I don't have any hardware available where there is more than one NUMA, or I would look deeper into this myself. It would be interesting to find out whether the -XX:+UseNUMA option makes a big difference in performance. Thanks, Shawn
Re: solr performance with >1 NUMAs
Thanks Shawn! Currently we are still using the CMS collector for solr with Java 8. When last evaluated with Solr 7, CMS performs better than G1 for our case. When using G1, is it better to upgrade from Java 8 to Java 11? >From https://lucene.apache.org/solr/guide/8_4/solr-system-requirements.html, seems Java 14 is not officially supported for Solr 8. Best, Wei On Fri, Sep 25, 2020 at 5:50 PM Shawn Heisey wrote: > On 9/23/2020 7:42 PM, Wei wrote: > > Recently we deployed solr 8.4.1 on a batch of new servers with 2 NUMAs. I > > noticed that query latency almost doubled compared to deployment on > single > > NUMA machines. Not sure what's causing the huge difference. Is there any > > tuning to boost the performance on multiple NUMA machines? Any pointer is > > appreciated. > > If you're running with standard options, Solr 8.4.1 will start using the > G1 garbage collector. > > As of Java 14, G1 has gained the ability to use the -XX:+UseNUMA option, > which makes better decisions about memory allocations and multiple > NUMAs. If you're running a new enough Java, it would probably be > beneficial to add this to the garbage collector options. Solr itself is > unaware of things like NUMA -- Java must handle that. > > https://openjdk.java.net/jeps/345 > > Thanks, > Shawn >
Re: solr performance with >1 NUMAs
On 9/23/2020 7:42 PM, Wei wrote: Recently we deployed solr 8.4.1 on a batch of new servers with 2 NUMAs. I noticed that query latency almost doubled compared to deployment on single NUMA machines. Not sure what's causing the huge difference. Is there any tuning to boost the performance on multiple NUMA machines? Any pointer is appreciated. If you're running with standard options, Solr 8.4.1 will start using the G1 garbage collector. As of Java 14, G1 has gained the ability to use the -XX:+UseNUMA option, which makes better decisions about memory allocations and multiple NUMAs. If you're running a new enough Java, it would probably be beneficial to add this to the garbage collector options. Solr itself is unaware of things like NUMA -- Java must handle that. https://openjdk.java.net/jeps/345 Thanks, Shawn
Re: solr performance with >1 NUMAs
Thanks Dominique. I'll start with the -XX:+UseNUMA option. Best, Wei On Fri, Sep 25, 2020 at 7:04 AM Dominique Bejean wrote: > Hi, > > This would be a Java VM option, not something Solr itself can know about. > Take a look at this article in comments. May be it will help. > > https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html?showComment=1347033706559#c229885263664926125 > > Regards > > Dominique > > > > Le jeu. 24 sept. 2020 à 03:42, Wei a écrit : > > > Hi, > > > > Recently we deployed solr 8.4.1 on a batch of new servers with 2 NUMAs. I > > noticed that query latency almost doubled compared to deployment on > single > > NUMA machines. Not sure what's causing the huge difference. Is there any > > tuning to boost the performance on multiple NUMA machines? Any pointer is > > appreciated. > > > > Best, > > Wei > > >
Re: solr performance with >1 NUMAs
Hi, This would be a Java VM option, not something Solr itself can know about. Take a look at this article in comments. May be it will help. https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html?showComment=1347033706559#c229885263664926125 Regards Dominique Le jeu. 24 sept. 2020 à 03:42, Wei a écrit : > Hi, > > Recently we deployed solr 8.4.1 on a batch of new servers with 2 NUMAs. I > noticed that query latency almost doubled compared to deployment on single > NUMA machines. Not sure what's causing the huge difference. Is there any > tuning to boost the performance on multiple NUMA machines? Any pointer is > appreciated. > > Best, > Wei >
