Re: high cpu threads (solr 7.5)
On 4/11/2019 1:03 PM, Hari Nakka wrote: I mean the light weight processes (lwp) which were taking high cpu. I pulled the actual threads taking high cpu. full thread dump: *tdump.out* linux lwps: *high-cpu.out* top high cpu lwps mapped to thread nid: *high-cpu-dump.out (included threads taking more than 50% virtual core cpu)* I don't know how to read any of the files you shared and come to any useful conclusions. I did notice that the thread names in the "high-cpu.out" file are truncated to 15 characters, so it is not possible to map them to the thread dump. If I look at the "high-cpu-tdump.out" file and find threads in that dump which are actually running (not waiting), here's what I see: Threads that are doing lookups in one of Solr's caches, probably the documentCache, but I am not sure: qtp1650813924-32709 Not sure what the thread is doing, but it's running Lucene code: qtp1650813924-54372 qtp1650813924-30839 qtp1650813924-44304 qtp1650813924-54344 qtp1650813924-40150 qtp1650813924-46829 qtp1650813924-41093 Threads that are doing a Pivot facet (Solr code): qtp1650813924-43735 qtp1650813924-51360 Threads where Jetty is waiting for a request: qtp1650813924-43734 qtp1650813924-52846 qtp1650813924-46575 qtp1650813924-55623 qtp1650813924-45996 qtp1650813924-48982 If the "jetty waiting" threads are causing the high CPU, then upgrading Java should have taken care of it. If the Lucene or Pivot Facet threads are to blame, then you may be facing a general performance problem, which additional memory MIGHT help with. What requests are you sending to Solr when high CPU happens? How long does the high CPU last? It might be useful to gather the screenshot described at the following web page: https://wiki.apache.org/solr/SolrPerformanceProblems#Asking_for_help_on_a_memory.2Fperformance_issue One piece of information that could be useful and will not show up on the screenshot is how many documents are being handled by that Solr instance. You will need to look at all of the index cores in that instance and add up the "maxDoc" numbers for each. The maxDoc number includes deleted docs, which do affect performance. Thanks, Shawn
Re: high cpu threads (solr 7.5)
I mean the light weight processes (lwp) which were taking high cpu. I pulled the actual threads taking high cpu. full thread dump: *tdump.out* linux lwps: *high-cpu.out* top high cpu lwps mapped to thread nid: *high-cpu-dump.out (included threads taking more than 50% virtual core cpu)* https://drive.google.com/drive/folders/1FsFb8l0u5ZV4qnO-tgqa91JPHqT0lCcR On Thu, Apr 11, 2019 at 1:03 PM Shawn Heisey wrote: > On 4/11/2019 2:21 AM, Hari Nakka wrote: > > Hi Erick, We upgraded JDK to 11. No improvement. Still seeing high cpu > > utilization randomly. > > Attached the full threaddump (tdump.out) and lwp utilization > (high-cpu.out) > > there were more than 30 threads (high-cpu-dump.out)taking high cpu. > > these are different threads. i couldn't find much looking at them. > > Can you take a look and see if there is any known condition that could > > be causing this. > > Maybe we need a prominent note on the "mailing lists" page about > attachments to the mailing list, saying something like the following: > > --- > Attachments rarely make it to the list. They are stripped by the > mailing list software. If you have files to share with the list, you'll > need to find another way to send them. File sharing websites usually > work well. > --- > > We didn't get any attachments from you. > > You said you included something called "lwp utilization" ... the only > lwp I have ever heard of is a perl module for HTTP, which would have > nothing at all to do with high CPU usage in Solr. > > Thread dumps are usually not helpful in diagnosing high CPU usage. They > contain zero information about which threads are consuming the most CPU. > > Thanks, > Shawn >
Re: high cpu threads (solr 7.5)
On 4/11/2019 2:21 AM, Hari Nakka wrote: Hi Erick, We upgraded JDK to 11. No improvement. Still seeing high cpu utilization randomly. Attached the full threaddump (tdump.out) and lwp utilization (high-cpu.out) there were more than 30 threads (high-cpu-dump.out)taking high cpu. these are different threads. i couldn't find much looking at them. Can you take a look and see if there is any known condition that could be causing this. Maybe we need a prominent note on the "mailing lists" page about attachments to the mailing list, saying something like the following: --- Attachments rarely make it to the list. They are stripped by the mailing list software. If you have files to share with the list, you'll need to find another way to send them. File sharing websites usually work well. --- We didn't get any attachments from you. You said you included something called "lwp utilization" ... the only lwp I have ever heard of is a perl module for HTTP, which would have nothing at all to do with high CPU usage in Solr. Thread dumps are usually not helpful in diagnosing high CPU usage. They contain zero information about which threads are consuming the most CPU. Thanks, Shawn
Re: high cpu threads (solr 7.5)
Hi Erick, We upgraded JDK to 11. No improvement. Still seeing high cpu utilization randomly. Attached the full threaddump (tdump.out) and lwp utilization (high-cpu.out) there were more than 30 threads (high-cpu-dump.out)taking high cpu. these are different threads. i couldn't find much looking at them. Can you take a look and see if there is any known condition that could be causing this. On Sat, Apr 6, 2019 at 12:23 PM Erick Erickson wrote: > Here’s what the current state of various Solr x Java versions: > https://wiki.apache.org/solr/SolrJavaVersions > > > On Apr 5, 2019, at 3:16 PM, Hari Nakka wrote: > > > > Thank you. We are planning to upgrade the JDK 11. > > Is solr 7.5 fully compatible with openjdk 11. > > > > > > On Thu, Apr 4, 2019 at 9:58 AM Erick Erickson > > wrote: > > > >> It hasn’t been addressed by any Java 8 releases that I know of. > >> > >> See: https://issues.apache.org/jira/browse/SOLR-13349 > >> > >> The work-around in Solr is trivial, see the patch so it’d be simple to > >> patch/compile on your own. > >> > >> It will be released in a Solr 7.7.2 and Solr 8.1 or later, neither of > >> which have been released yet. > >> > >> Or move to Java 9 or later. > >> > >>> On Apr 3, 2019, at 4:39 PM, Hari Nakka wrote: > >>> > >>> We are noticing high CPU utilization on below threads. Looks like a > >> known > >>> issue with. (https://github.com/netty/netty/issues/327) > >>> > >>> But not sure if this has been addressed in any of the 1.8 releases. > >>> > >>> Can anyone help with this? > >>> > >>> > >>> Version: solr cloud 7.5 > >>> > >>> OS: CentOS 7 > >>> > >>> JDK: Oracle JDK 1.8.0_191 > >>> > >>> > >>> > >>> > >>> > >>> "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x7f4f20018000 > >>> nid=0x4996 runnable [0x7f51fc6d8000] > >>> > >>> java.lang.Thread.State: RUNNABLE > >>> > >>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > >>> > >>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > >>> > >>> at > >> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > >>> > >>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > >>> > >>> - locked <0x00064cded430> (a sun.nio.ch.Util$3) > >>> > >>> - locked <0x00064cded418> (a > >>> java.util.Collections$UnmodifiableSet) > >>> > >>> - locked <0x00064cdf6e38> (a sun.nio.ch.EPollSelectorImpl) > >>> > >>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > >>> > >>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > >>> > >>> at > >>> org.eclipse.jetty.io > >> .ManagedSelector$SelectorProducer.select(ManagedSelector.java:396) > >>> > >>> at > >>> org.eclipse.jetty.io > >> .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) > >>> > >>> at java.lang.Thread.run(Thread.java:748) > >> > >> > >
Re: high cpu threads (solr 7.5)
Here’s what the current state of various Solr x Java versions: https://wiki.apache.org/solr/SolrJavaVersions > On Apr 5, 2019, at 3:16 PM, Hari Nakka wrote: > > Thank you. We are planning to upgrade the JDK 11. > Is solr 7.5 fully compatible with openjdk 11. > > > On Thu, Apr 4, 2019 at 9:58 AM Erick Erickson > wrote: > >> It hasn’t been addressed by any Java 8 releases that I know of. >> >> See: https://issues.apache.org/jira/browse/SOLR-13349 >> >> The work-around in Solr is trivial, see the patch so it’d be simple to >> patch/compile on your own. >> >> It will be released in a Solr 7.7.2 and Solr 8.1 or later, neither of >> which have been released yet. >> >> Or move to Java 9 or later. >> >>> On Apr 3, 2019, at 4:39 PM, Hari Nakka wrote: >>> >>> We are noticing high CPU utilization on below threads. Looks like a >> known >>> issue with. (https://github.com/netty/netty/issues/327) >>> >>> But not sure if this has been addressed in any of the 1.8 releases. >>> >>> Can anyone help with this? >>> >>> >>> Version: solr cloud 7.5 >>> >>> OS: CentOS 7 >>> >>> JDK: Oracle JDK 1.8.0_191 >>> >>> >>> >>> >>> >>> "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x7f4f20018000 >>> nid=0x4996 runnable [0x7f51fc6d8000] >>> >>> java.lang.Thread.State: RUNNABLE >>> >>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>> >>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) >>> >>> at >> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) >>> >>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) >>> >>> - locked <0x00064cded430> (a sun.nio.ch.Util$3) >>> >>> - locked <0x00064cded418> (a >>> java.util.Collections$UnmodifiableSet) >>> >>> - locked <0x00064cdf6e38> (a sun.nio.ch.EPollSelectorImpl) >>> >>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) >>> >>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) >>> >>> at >>> org.eclipse.jetty.io >> .ManagedSelector$SelectorProducer.select(ManagedSelector.java:396) >>> >>> at >>> org.eclipse.jetty.io >> .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333) >>> >>> at >>> >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) >>> >>> at >>> >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) >>> >>> at >>> >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) >>> >>> at >>> >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) >>> >>> at >>> >> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) >>> >>> at >>> >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) >>> >>> at >>> >> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) >>> >>> at java.lang.Thread.run(Thread.java:748) >> >>
Re: high cpu threads (solr 7.5)
Thank you. We are planning to upgrade the JDK 11. Is solr 7.5 fully compatible with openjdk 11. On Thu, Apr 4, 2019 at 9:58 AM Erick Erickson wrote: > It hasn’t been addressed by any Java 8 releases that I know of. > > See: https://issues.apache.org/jira/browse/SOLR-13349 > > The work-around in Solr is trivial, see the patch so it’d be simple to > patch/compile on your own. > > It will be released in a Solr 7.7.2 and Solr 8.1 or later, neither of > which have been released yet. > > Or move to Java 9 or later. > > > On Apr 3, 2019, at 4:39 PM, Hari Nakka wrote: > > > > We are noticing high CPU utilization on below threads. Looks like a > known > > issue with. (https://github.com/netty/netty/issues/327) > > > > But not sure if this has been addressed in any of the 1.8 releases. > > > > Can anyone help with this? > > > > > > Version: solr cloud 7.5 > > > > OS: CentOS 7 > > > > JDK: Oracle JDK 1.8.0_191 > > > > > > > > > > > > "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x7f4f20018000 > > nid=0x4996 runnable [0x7f51fc6d8000] > > > > java.lang.Thread.State: RUNNABLE > > > >at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > >at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > > > >at > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > > > >at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > > > >- locked <0x00064cded430> (a sun.nio.ch.Util$3) > > > >- locked <0x00064cded418> (a > > java.util.Collections$UnmodifiableSet) > > > >- locked <0x00064cdf6e38> (a sun.nio.ch.EPollSelectorImpl) > > > >at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > > > >at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > > > >at > > org.eclipse.jetty.io > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:396) > > > >at > > org.eclipse.jetty.io > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333) > > > >at > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) > > > >at > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) > > > >at > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) > > > >at > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) > > > >at > > > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) > > > >at > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) > > > >at > > > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) > > > >at java.lang.Thread.run(Thread.java:748) > >
Re: high cpu threads (solr 7.5)
It hasn’t been addressed by any Java 8 releases that I know of. See: https://issues.apache.org/jira/browse/SOLR-13349 The work-around in Solr is trivial, see the patch so it’d be simple to patch/compile on your own. It will be released in a Solr 7.7.2 and Solr 8.1 or later, neither of which have been released yet. Or move to Java 9 or later. > On Apr 3, 2019, at 4:39 PM, Hari Nakka wrote: > > We are noticing high CPU utilization on below threads. Looks like a known > issue with. (https://github.com/netty/netty/issues/327) > > But not sure if this has been addressed in any of the 1.8 releases. > > Can anyone help with this? > > > Version: solr cloud 7.5 > > OS: CentOS 7 > > JDK: Oracle JDK 1.8.0_191 > > > > > > "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x7f4f20018000 > nid=0x4996 runnable [0x7f51fc6d8000] > > java.lang.Thread.State: RUNNABLE > >at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > >at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > >at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > >at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > >- locked <0x00064cded430> (a sun.nio.ch.Util$3) > >- locked <0x00064cded418> (a > java.util.Collections$UnmodifiableSet) > >- locked <0x00064cdf6e38> (a sun.nio.ch.EPollSelectorImpl) > >at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > >at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > >at > org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396) > >at > org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333) > >at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) > >at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) > >at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) > >at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) > >at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) > >at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) > >at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) > >at java.lang.Thread.run(Thread.java:748)
high cpu threads (solr 7.5)
We are noticing high CPU utilization on below threads. Looks like a known issue with. (https://github.com/netty/netty/issues/327) But not sure if this has been addressed in any of the 1.8 releases. Can anyone help with this? Version: solr cloud 7.5 OS: CentOS 7 JDK: Oracle JDK 1.8.0_191 "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x7f4f20018000 nid=0x4996 runnable [0x7f51fc6d8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00064cded430> (a sun.nio.ch.Util$3) - locked <0x00064cded418> (a java.util.Collections$UnmodifiableSet) - locked <0x00064cdf6e38> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748)
high cpu threads (solr 7.5) - EPollArrayWrapper.epollWait
Version: solr cloud 7.5 OS: CentOS 7 JDK: Oracle JDK 1.8.0_191 We are noticing high CPU utilization on below threads. Looks like a known issue with. (https://github.com/netty/netty/issues/327) But not sure if this has been addressed in any of the 1.8 releases. Please help. "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x7f4f20018000 nid=0x4996 runnable [0x7f51fc6d8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00064cded430> (a sun.nio.ch.Util$3) - locked <0x00064cded418> (a java.util.Collections$UnmodifiableSet) - locked <0x00064cdf6e38> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748)
