5 seconds and 10 seconds is very short for auto commit. 20 Gb is probably too much heap.
Sending the exact same message for every update will create a few very long posting lists. Not sure if that is slow, but it is not realistic. Finally, 26,000 per second is not that slow. That is over 1.5 million/minute. We are indexing bigger documents, but seeing 1 million/minute to a cluster with four shards. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Sep 21, 2017, at 1:18 AM, Emir Arnautović <emir.arnauto...@sematext.com> > wrote: > > Hi, > What are your commit configs? Maybe you are committing too frequently. > > Thanks, > Emir > >> On 21 Sep 2017, at 06:19, saiks <karlapudisam...@gmail.com> wrote: >> >> Hi, >> >> Environment: >> - Solr is running in non-cloud mode on 6.4.2, Sun Java8, Linux >> 4.4.0-31-generic x86_64 >> - Ingesting into a single core >> - SoftCommit = 5 seconds, HardCommit = 10 seconds >> - System has 16 Cpus and 32 Gb of memory (Solr is given 20 Gb of JVM heap) >> - text = StandardTokenizer, id = solr.StrField/docValues, hostname = >> solr.StrField/docValues, app = solr.StrField/docValues, epoch = >> solr.TrieLongField/docValues >> >> I am using jmeter to ingest to Solr core using UpdateRequestHandle >> ("/update/json") and sending in a batch of 1000 messages(same message) in a >> single json array. >> >> Sample message >> [{"text":"May 11 10:18:22 scrooge Web-Requests: May 11 10:18:22 >> @IunAIir1----7k-- EVENT_WR-Y-attack-600 SG_child[823]: [event.error] >> Possible attack - 5 blocked requests within 120 seconds", >> "id":"id1", >> "hostname": "xxxxxxxxxx.com", >> "app": "yyyy", >> "epoch": 1483667347941 >> }, >> ....] >> >> Jmeter is configured to run 10 threads in parallel repeating the request >> 1000 times, which should ingest 10,000,000 messages in total. >> Jmeter post url: >> "/solr/mycore/update/json?overwrite=false&wt=json&commit=false" >> >> Jmeter summary: >> summary = 5000 in 00:03:07 = 26.7/s Avg: 370 Min: 27 Max: 1734 >> Err: 0 (0.00%) >> >> I am only able to ingest 26000 messages per second, looking at system >> resources only one or two cpus are at 25-30% and the rest are sitting idle >> and also Solr heap is flat at 3Gb with no iowait on the devices. >> Increasing parallelism in Jmeter to ingest using 20 threads did not increase >> ingested messages per second, but increased the latency by 2x for each >> request. >> >> I don't understand why Solr is not able to use all the cpus on the host if I >> increase Jmeter parallelism from 10 -> 20 -> 40. What can I do to achieve >> performance gain and make Solr utilize system resources to their maximum. >> >> Please help. >> >> Thank you >> >> >> >> >> >> >> -- >> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >
