Thank you for fixing it! -shinsuke
On Wednesday, October 15, 2014 5:52:15 PM UTC+9, Michael McCandless wrote: > > OK this will be fixed in the next ES release: > https://github.com/elasticsearch/elasticsearch/pull/8087 > > Thank you for reporting this. > > Mike McCandless > > http://blog.mikemccandless.com > > On Wed, Oct 15, 2014 at 2:40 AM, Shinsuke Sugaya <[email protected] > <javascript:>> wrote: > >> Hi, >> >> I encountered a problem in InternalIndexShard#EngineRefresher. >> The problem is, 1 core consumed 100% CPU resource when changing >> refresh_interval to -1. >> >> In ES 1.3, I took a thread dump twice as below: >> >> $ top -n1 -b -H >> ... >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> >> 4074 elastics 20 0 9.9g 5.8g 316m R 99.6 18.5 53:19.94 java >> ... >> $ jstack -F <pid> >> ... >> Thread 4074: (state = IN_JAVA) >> - org.apache.lucene.index.IndexReader.tryIncRef() @bci=23, line=226 >> (Compiled frame; information may be imprecise) >> - >> org.apache.lucene.search.SearcherManager.tryIncRef(org.apache.lucene.search.IndexSearcher) >> >> @bci=4, line=128 (Compiled frame) >> - org.apache.lucene.search.SearcherManager.tryIncRef(java.lang.Object) >> @bci=5, line=58 (Compiled frame) >> - org.apache.lucene.search.ReferenceManager.acquire() @bci=21, line=100 >> (Compiled frame) >> - org.apache.lucene.search.SearcherManager.isSearcherCurrent() @bci=1, >> line=142 (Compiled frame) >> - org.elasticsearch.index.engine.internal.InternalEngine.refreshNeeded() >> @bci=11, line=743 (Compiled frame) >> - >> org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher.run() >> >> @bci=7, line=930 (Compiled frame) >> - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 >> (Compiled frame) >> - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled >> frame) >> - >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) >> >> @bci=1, line=180 (Compiled frame) >> - >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() >> @bci=30, line=293 (Compiled frame) >> - >> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) >> >> @bci=95, line=1142 (Compiled frame) >> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 >> (Interpreted frame) >> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) >> ... >> $ jstack -F <pid> >> ... >> Thread 4074: (state = IN_JAVA) >> - org.elasticsearch.common.unit.TimeValue.millis() @bci=8, line=95 >> (Compiled frame; information may be imprecise) >> - >> org.elasticsearch.threadpool.ThreadPool.schedule(org.elasticsearch.common.unit.TimeValue, >> >> java.lang.String, java.lang.Runnable) @bci=30, line=229 (Compiled frame) >> - >> org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher.run() >> >> @bci=59, line=933 (Compiled frame) >> - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 >> (Compiled frame) >> - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled >> frame) >> - >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) >> >> @bci=1, line=180 (Compiled frame) >> - >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() >> @bci=30, line=293 (Compiled frame) >> - >> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) >> >> @bci=95, line=1142 (Compiled frame) >> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 >> (Interpreted frame) >> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) >> ... >> >> I looked into InternalIndexShard$EngineRefresher#run() and >> InternalIndexShard$ApplyRefreshSettings#onRefreshSettings(). >> If a thread is running in EngineRefresher#run() method when >> changing refresh_interval to -1, invoking >> refreshScheduledFuture.cancel(false) >> in ApplyRefreshSettings#onRefreshSettings() does not cancel >> EngineRefresher thread. Moreover, the refresh_interval is >> changed to -1 and EngineRefresher seems to be invoked >> with no interval(the thread comsumes 100% CPU). >> >> I think that a fix is to check if refreshInterval.millis() > 0. >> For example, the fix is: >> >> class EngineRefresher implements Runnable { >> @Override >> public void run() { >> // we check before if a refresh is needed, if not, we >> reschedule, otherwise, we fork, refresh, and then reschedule >> if (!engine().refreshNeeded()) { >> synchronized (mutex) { >> if (state != IndexShardState.CLOSED && >> refreshInterval.millis() > 0) { // <== HERE >> refreshScheduledFuture = >> threadPool.schedule(refreshInterval, ThreadPool.Names.SAME, this); >> } >> } >> return; >> } >> >> Could you check this problem? >> >> Thanks, >> shinsuke >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/ffd41678-4945-46ce-9405-c165e298d24a%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/ffd41678-4945-46ce-9405-c165e298d24a%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/803ca2e9-9326-401f-9fb1-a64efcff6066%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
