Hi,
I encountered a problem in InternalIndexShard#EngineRefresher.
The problem is, 1 core consumed 100% CPU resource when changing
refresh_interval to -1.
In ES 1.3, I took a thread dump twice as below:
$ top -n1 -b -H
...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4074 elastics 20 0 9.9g 5.8g 316m R 99.6 18.5 53:19.94 java
...
$ jstack -F <pid>
...
Thread 4074: (state = IN_JAVA)
- org.apache.lucene.index.IndexReader.tryIncRef() @bci=23, line=226
(Compiled frame; information may be imprecise)
-
org.apache.lucene.search.SearcherManager.tryIncRef(org.apache.lucene.search.IndexSearcher)
@bci=4, line=128 (Compiled frame)
- org.apache.lucene.search.SearcherManager.tryIncRef(java.lang.Object)
@bci=5, line=58 (Compiled frame)
- org.apache.lucene.search.ReferenceManager.acquire() @bci=21, line=100
(Compiled frame)
- org.apache.lucene.search.SearcherManager.isSearcherCurrent() @bci=1,
line=142 (Compiled frame)
- org.elasticsearch.index.engine.internal.InternalEngine.refreshNeeded()
@bci=11, line=743 (Compiled frame)
-
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher.run()
@bci=7, line=930 (Compiled frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511
(Compiled frame)
- java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame)
-
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
@bci=1, line=180 (Compiled frame)
-
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()
@bci=30, line=293 (Compiled frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
...
$ jstack -F <pid>
...
Thread 4074: (state = IN_JAVA)
- org.elasticsearch.common.unit.TimeValue.millis() @bci=8, line=95
(Compiled frame; information may be imprecise)
-
org.elasticsearch.threadpool.ThreadPool.schedule(org.elasticsearch.common.unit.TimeValue,
java.lang.String, java.lang.Runnable) @bci=30, line=229 (Compiled frame)
-
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher.run()
@bci=59, line=933 (Compiled frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511
(Compiled frame)
- java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame)
-
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
@bci=1, line=180 (Compiled frame)
-
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()
@bci=30, line=293 (Compiled frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
...
I looked into InternalIndexShard$EngineRefresher#run() and
InternalIndexShard$ApplyRefreshSettings#onRefreshSettings().
If a thread is running in EngineRefresher#run() method when
changing refresh_interval to -1, invoking
refreshScheduledFuture.cancel(false)
in ApplyRefreshSettings#onRefreshSettings() does not cancel
EngineRefresher thread. Moreover, the refresh_interval is
changed to -1 and EngineRefresher seems to be invoked
with no interval(the thread comsumes 100% CPU).
I think that a fix is to check if refreshInterval.millis() > 0.
For example, the fix is:
class EngineRefresher implements Runnable {
@Override
public void run() {
// we check before if a refresh is needed, if not, we
reschedule, otherwise, we fork, refresh, and then reschedule
if (!engine().refreshNeeded()) {
synchronized (mutex) {
if (state != IndexShardState.CLOSED &&
refreshInterval.millis() > 0) { // <== HERE
refreshScheduledFuture =
threadPool.schedule(refreshInterval, ThreadPool.Names.SAME, this);
}
}
return;
}
Could you check this problem?
Thanks,
shinsuke
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ffd41678-4945-46ce-9405-c165e298d24a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
