Re: Solr 8.0.0 - CPU usage 100% when indexed documents
Thanks for your reply.. All 4 CPU core got high by 12 to 15 seconds. we used java 8. I got your point. We will wait solr 8.1 rather upgrade OpenJDK 11. Sent from Outlook<http://aka.ms/weboutlook> From: Shawn Heisey Sent: Wednesday, April 10, 2019 9:07 PM To: solr-user@lucene.apache.org Subject: Re: Solr 8.0.0 - CPU usage 100% when indexed documents On 4/9/2019 10:53 PM, vishal patel wrote: > Still my CPU usage went high and my CPU has 4 core and no other application > running in my machine. I was asking how many CPUs went to 100 percent, not how many CPUs you have. And I also asked how long CPU usage remains at 100 percent after indexing a single document. What Java version are you running? We do have a possible bug that could be affecting you. https://issues.apache.org/jira/browse/SOLR-13349 If this is the problem you're experiencing, the solution would be to either upgrade to Java 11 or wait for Solr 8.1 to be released. Note that Oracle requires payment if you use their Java 11 in production. You're likely going to want to use OpenJDK. Thanks, Shawn
Re: Solr 8.0.0 - CPU usage 100% when indexed documents
On 4/9/2019 10:53 PM, vishal patel wrote: Still my CPU usage went high and my CPU has 4 core and no other application running in my machine. I was asking how many CPUs went to 100 percent, not how many CPUs you have. And I also asked how long CPU usage remains at 100 percent after indexing a single document. What Java version are you running? We do have a possible bug that could be affecting you. https://issues.apache.org/jira/browse/SOLR-13349 If this is the problem you're experiencing, the solution would be to either upgrade to Java 11 or wait for Solr 8.1 to be released. Note that Oracle requires payment if you use their Java 11 in production. You're likely going to want to use OpenJDK. Thanks, Shawn
Re: Solr 8.0.0 - CPU usage 100% when indexed documents
Hi,
After your suggestion i changed code
String SOLR_URL="http://localhost:7991/solr/actionscomments;;
SolrClient solrClient = new HttpSolrClient.Builder(SOLR_URL).build();
SolrInputDocument document = new SolrInputDocument();
document.addField("id","ACTC6401895");
solrClient.add(document);
solrClient.commit();
Still my CPU usage went high and my CPU has 4 core and no other application
running in my machine.
After the lots of try, I found out the below issue.
Before solrconfig.xml (6.1.0)
60
2
false
After the below change in solrconfig.xml
(8.0.0)
${solr.autoCommit.maxTime:15000}
2
false
Actually I am upgrading solr 6.1.0 to 8.0.0. In 6.1.0 it is working fine with
autocommit maxtime 60.
But in 8.0.0, CPU usage goes high.[commitScheduler thread running long time]
Please give me more details why is it happening in solr 8.0.0.
Is any my mistake? In previous mail, I attached solrconfig.xml so please verify
it.
Sent from Outlook<http://aka.ms/weboutlook>
From: Shawn Heisey
Sent: Tuesday, April 9, 2019 1:38 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr 8.0.0 - CPU usage 100% when indexed documents
On 4/8/2019 11:00 PM, vishal patel wrote:
> Sorry my mistake there is no class of that.
>
> I have add the data using below code.
> CloudSolrServer cloudServer = new CloudSolrServer(zkHost);
> cloudServer.setDefaultCollection("actionscomments");
> cloudServer.setParallelUpdates(true);
> List docs = new ArrayList<>();
> SolrInputDocument solrDocument = new SolrInputDocument();
> solrDocument.addField("id", "123");
> docs.add(solrDocument);
> cloudServer.add(docs, 1000);
Side note: This code is not using SolrJ 8.0.0. CloudSolrServer was
deprecated in version 5.0.0 and completely removed in version 6.0.0.
I'm surprised this code even works at all with Solr 8.0.0 -- you need to
upgrade to SolrJ 8 and use CloudSolrClient.
How long does the system remain at 100 percent CPU when you index that
single document that only has one field? If it's longer than a very
small fraction of a second, then my guess is that it's cache warming
queries using the CPU, not the indexing itself.
How many CPU cores are at 100 percent? Is it just one, or multiple? It
would be odd for it to be multiple, unless there is other activity going
on at the same time.
Thanks,
Shawn
Re: Solr 8.0.0 - CPU usage 100% when indexed documents
On 4/8/2019 11:00 PM, vishal patel wrote:
Sorry my mistake there is no class of that.
I have add the data using below code.
CloudSolrServer cloudServer = new CloudSolrServer(zkHost);
cloudServer.setDefaultCollection("actionscomments");
cloudServer.setParallelUpdates(true);
List docs = new ArrayList<>();
SolrInputDocument solrDocument = new SolrInputDocument();
solrDocument.addField("id", "123");
docs.add(solrDocument);
cloudServer.add(docs, 1000);
Side note: This code is not using SolrJ 8.0.0. CloudSolrServer was
deprecated in version 5.0.0 and completely removed in version 6.0.0.
I'm surprised this code even works at all with Solr 8.0.0 -- you need to
upgrade to SolrJ 8 and use CloudSolrClient.
How long does the system remain at 100 percent CPU when you index that
single document that only has one field? If it's longer than a very
small fraction of a second, then my guess is that it's cache warming
queries using the CPU, not the indexing itself.
How many CPU cores are at 100 percent? Is it just one, or multiple? It
would be odd for it to be multiple, unless there is other activity going
on at the same time.
Thanks,
Shawn
Re: Solr 8.0.0 - CPU usage 100% when indexed documents
On 4/8/2019 7:22 AM, vishal patel wrote: I have created two solr shards with 3 zoo keeper. First do upconfig in zoo keeper then start the both solr with different port then create a "actionscomments" collection using API call. When I indexed one document in actionscomments, my CPU utilization go high. You were asked how you are doing the indexing. You still haven't provided that information. You said "AsiteSolrCloudManager" ... but when I google for that, the only thing that comes up is this email thread. I have no idea what AsiteSolrCloudManager is. One thing I *can* say is that it is not part of Solr. I have attached my solrconfig.xml and schema.xml and also thread dump which got from solr admin GUI. The schema and solrconfig came through. The thread dump did not. I'm surprised that ANY attachments made it to the list ... normally they don't. The thread dump also did not come through on your first message. If you need to share files, you'll need to find a mechanism other than attachments to do it. File sharing websites work well. So we don't have the thread dump. But for a problem like this, a thread dump is not going to be helpful. I've never seen anything in a Java thread dump to indicate which threads are using the most CPU. Thanks, Shawn
Re: Solr 8.0.0 - CPU usage 100% when indexed documents
I have created two solr shards with 3 zoo keeper. First do upconfig in zoo
keeper then start the both solr with different port then create a
"actionscomments" collection using API call.
When I indexed one document in actionscomments, my CPU utilization go high.
Note :
upconfig command :: zkcli.bat -zkhost
192.168.100.145:3181,192.168.100.145:3182,192.168.100.145:3183 -cmd upconfig
-confdir E:/SolrCloud-8-0-0/solr1/server/solr/configsets/actionscomments/conf
-confname actionscomments.
[E:\SolrCloud-8-0-0\solr1\server\scripts\cloud-scripts]
Solr start command :: solr start -p 7991 and solr start -p 7992
[E:\SolrCloud-8-0-0\solr1\bin and E:\SolrCloud-8-0-0\solr2\bin]
Create a collection ::
http://192.168.102.150:7991/solr/admin/collections?_=1554285992377=CREATE=false=actionscomments=1=actionscomments=2=1=compositeId=json
Operating system :: windows server 2008 R2 standard
When I indexed document, CPU goes high and in thread dump noticed
commitScheduler-25-thread-2,commitScheduler-48-thread-2
,commitScheduler-21-thread-2. After sometimes it automatically removed and CPU
goes down.
In log file I can not find any error and I indexed document using
AsiteSolrCloudManager.
I have attached my solrconfig.xml and schema.xml and also thread dump which got
from solr admin GUI.
Sent from Outlook<http://aka.ms/weboutlook>
From: Jörn Franke
Sent: Monday, April 8, 2019 4:16 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr 8.0.0 - CPU usage 100% when indexed documents
Can you please describe your scenario in detail ?
How does your load process look like (custom module? How many threads?)?
How many files do you try to index ? What is their format?
How does your solr config look like?
How many cores do you have? What else is installed on the Solr server?
Which Operation System?
What do the log files tell your from Solr and Zookeeper?
What is the Schema looking like?
> Am 08.04.2019 um 12:01 schrieb vishal patel :
>
> Hi
>
> I have configured 2 shards and 3 zoo keeper. When i indexed document in
> collection, my CPU usage becomes a full.
> I have attached thread dump.
> Is there Any changes needed in solrconfig.xml?
>
> Sent from Outlook
id
8.0.0
${solr.data.dir:}
16
1024
${solr.ulog.dir:}
${solr.ulog.numVersionBuckets:65536}
60
2
false
${solr.autoSoftCommit.maxTime:-1}
100
-1
true
true
50
100
*:*
false
none
xml
10
summary
explicit
json
true
text
{!xport}
xsort
false
query
text
text
true
tvComponent
default
org.carrot2.clustering.lingo.LingoClusteringAlgorithm
clustering/carrot2
20
ENGLISH
clustering/carrot2
stc
org.carrot2.clustering.stc.STCClusteringAlgorithm
true
default
true
name
id
features
true
false
edismax
text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4
*:*
10
*,score
clustering
true
false
terms
text/plain; charset=UTF-8
5
*:*
Re: Solr 8.0.0 - CPU usage 100% when indexed documents
Can you please describe your scenario in detail ? How does your load process look like (custom module? How many threads?)? How many files do you try to index ? What is their format? How does your solr config look like? How many cores do you have? What else is installed on the Solr server? Which Operation System? What do the log files tell your from Solr and Zookeeper? What is the Schema looking like? > Am 08.04.2019 um 12:01 schrieb vishal patel : > > Hi > > I have configured 2 shards and 3 zoo keeper. When i indexed document in > collection, my CPU usage becomes a full. > I have attached thread dump. > Is there Any changes needed in solrconfig.xml? > > Sent from Outlook
