Re: Update to Solr 6 - Amazon EC2 high CPU SYS usage

On 4/27/2017 3:03 AM, Elodie Sannier wrote: > We have migrated from Solr 5.4.1 to Solr 6.4.0 on Amazon EC2 and we have > a high CPU SYS usage and it drastically decreases the Solr performance. > > The JVM version (java-1.8.0-openjdk-1.8.0.131-0.b11.el6_9.x86_64), the > Jetty

Update to Solr 6 - Amazon EC2 high CPU SYS usage

Hello, We have migrated from Solr 5.4.1 to Solr 6.4.0 on Amazon EC2 and we have a high CPU SYS usage and it drastically decreases the Solr performance. The JVM version (java-1.8.0-openjdk-1.8.0.131-0.b11.el6_9.x86_64), the Jetty version (9.3.14) and the OS version (CentOS 6.9) have not changed

Re: High Cpu sys usage

Both of these are anit-patterns. The soft commit interval of 1 second is usually far too aggressive. And committing after every add is also something to avoid. Your original problem statement is high CPU usage. To see if your committing is the culprit, I'd stop committing at all after adding and

Re: High Cpu sys usage

Hi Thanks you Erik. The main collection that stores our trade data is set to the softcomit when we import data using DIH. As you guess that the softcommit intervals is " 1000 " and we have autowarm counts to 0.However there is some collections that store our meta info in which we commit after

Re: High Cpu sys usage

Do not, repeat NOT try to "cure" the "Overlapping onDeckSearchers" by bumping this limit! What that means is that your commits (either hard commit with openSearcher=true or softCommit) are happening far too frequently and your Solr instance is trying to do all sorts of work that is immediately

Re: High Cpu sys usage

Hi Toke The number of collection is just 10.One of collection has 43 shards,each shard has two replicas.We continue importing data from oracle all the time while our systems provide searching service. There are "Overlapping onDeckSearchers" in my solr.logs. What is the meaning about the

Re: High Cpu sys usage

On Tue, 2016-03-29 at 20:12 +0800, YouPeng Yang wrote: > Our system still goes down as times going.We found lots of threads are > WAITING.Here is the threaddump that I copy from the web page.And 4 pictures > for it. > Is there any relationship with my problem? That is a lot of

Re: High Cpu sys usage

Hi Our system still goes down as times going.We found lots of threads are WAITING.Here is the threaddump that I copy from the web page.And 4 pictures for it. Is there any relationship with my problem? https://www.dropbox.com/s/h3wyez091oouwck/threaddump?dl=0

Re: High Cpu sys usage

Hi, On Wed, Mar 16, 2016 at 10:59 AM, Patrick Plaatje wrote: > Hi, > > From the sar output you supplied, it looks like you might have a memory > issue on your hosts. The memory usage just before your crash seems to be > *very* close to 100%. Even the slightest increase (Solr

Re: High Cpu sys usage

Yeah, I did’t pay attention to the cached memory at all, my bad! I remember running into a similar situation a couple of years ago, one of the things to investigate our memory profile was to produce a full heap dump and manually analyse that using a tool like MAT. Cheers, -patrick On

Re: High Cpu sys usage

On 3/16/2016 8:27 PM, YouPeng Yang wrote: > Hi Shawn >Here is my top screenshot: > >https://www.dropbox.com/s/jaw10mkmipz943y/topscreen.jpg?dl=0 > >It is captured when my system is normal.And I have reduced the memory > size down to 48GB originating from 64GB. It looks like you have

Re: High Cpu sys usage

Hi It happened again,and worse thing is that my system went to crash.we can even not connect to it with ssh. I use the sar command to capture the statistics information about it.Here are my details: [1]cpu(by using sar -u),we have to restart our system just as the red font LINUX RESTART in the

Re: High Cpu sys usage

Hi Shawn Here is my top screenshot: https://www.dropbox.com/s/jaw10mkmipz943y/topscreen.jpg?dl=0 It is captured when my system is normal.And I have reduced the memory size down to 48GB originating from 64GB. We have two hardware clusters ,each is comprised of 3 machines,and On one

Re: High Cpu sys usage

On 3/16/2016 8:59 AM, Patrick Plaatje wrote: > From the sar output you supplied, it looks like you might have a memory issue > on your hosts. The memory usage just before your crash seems to be *very* > close to 100%. Even the slightest increase (Solr itself, or possibly by a > system service)

Re: High Cpu sys usage

Hi To Patrick: Never mind .Thank you for your suggestion all the same. To Otis. We do not use SPM. We monintor the JVM just use jstat becasue my system went well before ,so we do not need other tools. But SPM is really awesome . Still looking for help. Best Regards 2016-03-18 6:01

Re: High Cpu sys usage

Hi Shawn Actually,there are three Solr instances(The top three PIDs is the three instances),and the datafile size of the stuff is 851G,592G,49G respectively ,and more and more data will be added as time going.I think it may be rare as the large scope as my solrcloud service .and it is now one

Re: High Cpu sys usage

Hi, >From the sar output you supplied, it looks like you might have a memory issue >on your hosts. The memory usage just before your crash seems to be *very* >close to 100%. Even the slightest increase (Solr itself, or possibly by a >system service) could caused the system crash. What are the

Re: High Cpu sys usage

Hi, I looked at those metrics outputs, but nothing jumps out at me as problematic. How full are your JVM heap memory pools? If you are using SPM to monitor your Solr/Tomcat/Jetty/... look for a chart that looks like this: https://apps.sematext.com/spm-reports/s/zB3JcdZyRn If some of these

Re: High Cpu sys usage

Hello The problem appears several times ,however I could not capture the top output .My script is as follows code. I check the sys cpu usage whether it exceed 30%.the other metric information can be dumpped successfully except the top . Would you like to check my script that I am not able to

Re: High Cpu sys usage

Hi all Thanks for your reply.I do some investigation for much time.and I will post some logs of the 'top' and IO in a few days when the crash come again. 2016-03-08 10:45 GMT+08:00 Shawn Heisey : > On 3/7/2016 2:23 AM, Toke Eskildsen wrote: > > How does this relate to

Re: High Cpu sys usage

On 3/7/2016 2:23 AM, Toke Eskildsen wrote: > How does this relate to YouPeng reporting that the CPU usage increases? > > This is not a snark. YouPeng mentions kernel issues. It might very well > be that IO is the real problem, but that it manifests in a non-intuitive > way. Before memory-mapping

Re: High Cpu sys usage

On Sun, 2016-03-06 at 08:26 -0700, Shawn Heisey wrote: > On 3/5/2016 11:44 PM, YouPeng Yang wrote: > > We are using Solr Cloud 4.6 in our production for searching service > > since 2 years ago.And now it has 700GB in one cluster which is comprised > > of 3 machines with ssd. At beginning

Re: High Cpu sys usage

On 3/5/2016 11:44 PM, YouPeng Yang wrote: > We are using Solr Cloud 4.6 in our production for searching service > since 2 years ago.And now it has 700GB in one cluster which is comprised > of 3 machines with ssd. At beginning ,everything go well,while more and > more business services

High Cpu sys usage

the high cpu sys usage is due to system kernel.But we do not have further more information about the problem other than the solution. Is it perhaps because of the increase of the search concurrence? Do you guys have experiences like us.How do you solve it? Please any suggestion