[
https://issues.apache.org/jira/browse/SOLR-11200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117144#comment-16117144
]
Amrit Sarkar commented on SOLR-11200:
-------------------------------------
The above patch works just fine, the logging isn't obvious which created
significant confusion ::
{code}
2017-08-07 17:18:18.203 INFO (Lucene Merge Thread #1) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream
[MergeScheduler][Lucene Merge Thread #1]: updateMergeThreads ioThrottle=false
targetMBPerSec=20.0 MB/sec
merge thread Lucene Merge Thread #0 estSize=3.2 MB (written=2.0 MB)
runTime=156.3s (stopped=0.0s, paused=0.0s) rate=unlimited
leave running at Infinity MB/sec
{code}
Even if the targetMBPersec=20, the merges are happening at {{rate=unlimited}},
maximum possible disk write speed.
{code}
merge thread Lucene Merge Thread #0 estSize=29.4 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #1 estSize=77.6 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #2 estSize=86.6 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #1 estSize=77.6 MB (written=76.1 MB)
runTime=10.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #2 estSize=86.6 MB (written=1.0 MB)
runTime=-0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #3 estSize=133.9 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #3 estSize=133.9 MB (written=132.3 MB)
runTime=12.8s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #4 estSize=71.9 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #4 estSize=71.9 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #5 estSize=82.0 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #6 estSize=92.5 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #7 estSize=128.2 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #7 estSize=128.2 MB (written=117.2 MB)
runTime=12.2s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #8 estSize=66.7 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #8 estSize=66.7 MB (written=21.1 MB)
runTime=0.8s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #9 estSize=206.2 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #8 estSize=66.7 MB (written=65.1 MB)
runTime=9.2s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #9 estSize=206.2 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #9 estSize=206.2 MB (written=191.5 MB)
runTime=15.3s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #10 estSize=146.7 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #10 estSize=146.7 MB (written=47.3 MB)
runTime=1.9s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #11 estSize=280.9 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #10 estSize=146.7 MB (written=143.3 MB)
runTime=17.2s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #11 estSize=280.9 MB (written=1.0 MB)
runTime=0.3s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #11 estSize=280.9 MB (written=143.3 MB)
runTime=12.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #12 estSize=100.8 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #11 estSize=280.9 MB (written=208.3 MB)
runTime=24.4s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #11 estSize=280.9 MB (written=235.3 MB)
runTime=28.2s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #13 estSize=193.6 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #13 estSize=193.6 MB (written=79.3 MB)
runTime=4.7s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #13 estSize=193.6 MB (written=155.4 MB)
runTime=16.1s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #14 estSize=35.5 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #13 estSize=193.6 MB (written=190.4 MB)
runTime=20.3s (stopped=0.0s, paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #15 estSize=94.2 MB (written=0.0 MB)
runTime=0.0s (stopped=0.0s, paused=0.0s) rate=unlimited
{code}
I am running some tests, to benchmark and see if we this or altering factors
increase the indexing speed. I will post when I have something substantial.
> provide a config to enable disable ConcurrentMergeSchedule.doAutoIOThrottle
> ---------------------------------------------------------------------------
>
> Key: SOLR-11200
> URL: https://issues.apache.org/jira/browse/SOLR-11200
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Nawab Zada Asad iqbal
> Priority: Minor
> Attachments: SOLR-11200.patch, SOLR-11200.patch
>
>
> This config can be useful while bulk indexing. Lucene introduced it
> https://issues.apache.org/jira/browse/LUCENE-6119 .
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
