[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515842#comment-15515842
]
Mikhail Khludnev commented on SOLR-9330:
Exception should gone. Sometimes response contains no "search" entry, but only
something like {{Searcher@deadbeaf [collection1] main}}. So, patterns consuming
stats response might, but shouldn't be surprised.
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
>Assignee: Mikhail Khludnev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515812#comment-15515812
]
ASF subversion and git services commented on SOLR-9330:
---
Commit 4278356564d4928dfabfde6d4a2ebf57dcfcf81f in lucene-solr's branch
refs/heads/branch_6x from [~mkhludnev]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4278356 ]
SOLR-9330: Fix AlreadyClosedException on admin/mbeans?stats=true
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515411#comment-15515411
]
ASF subversion and git services commented on SOLR-9330:
---
Commit b50b9106f821915ced14a3efc1e09c265d648fa8 in lucene-solr's branch
refs/heads/master from [~mkhludnev]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b50b910 ]
SOLR-9330: Fix AlreadyClosedException on admin/mbeans?stats=true
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513882#comment-15513882
]
Mike Drob commented on SOLR-9330:
-
Ah, I see where the difference is, yes. In my case, the client process getting
the statistics is an external monitoring application that gets them every 15
seconds and charts them. Since number of replicas can move, grow and shrink to
accommodate usage, solving races like this is a very complicated problem. And
at the end of the day, I don't care if my monitoring system misses one round of
statistics, I'm more concerned about scary exceptions in the log that the ops
team has to deal with.
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513869#comment-15513869
]
Mike Drob commented on SOLR-9330:
-
{code}
lst.add("searcherName", name);
lst.add("caching", cachingEnabled);
lst.add("openedAt", openTime);
if (registerTime != null) lst.add("registeredAt", registerTime);
lst.add("warmupTime", warmupTime);
{code}
Why not put these in the cached list as well? The first three are final and
available before your call to {{snapStatistics}}. The last two are set during
{{register}} which should only be called once, if I understand this correctly.
Then the whole method becomes {{return readerStats;}} -- much simpler and
probably faster too!
{code}
+// core.getInfoRegistry().remove(STATISTICS_KEY, this);
+// decided to comment it, because it might upset users by showing stats,
w/o "searcher" entry
{code}
I don't think there is any reason to keep this in.
Other than those minor points, the patch looks good to me.
I've had a similar issue when calling {{/replication?command=details}}, but am
not able to reproduce it in this test, so I think we're fine to handle that
later.
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513868#comment-15513868
]
Mike Drob commented on SOLR-9330:
-
{code}
lst.add("searcherName", name);
lst.add("caching", cachingEnabled);
lst.add("openedAt", openTime);
if (registerTime != null) lst.add("registeredAt", registerTime);
lst.add("warmupTime", warmupTime);
{code}
Why not put these in the cached list as well? The first three are final and
available before your call to {{snapStatistics}}. The last two are set during
{{register}} which should only be called once, if I understand this correctly.
Then the whole method becomes {{return readerStats;}} -- much simpler and
probably faster too!
{code}
+// core.getInfoRegistry().remove(STATISTICS_KEY, this);
+// decided to comment it, because it might upset users by showing stats,
w/o "searcher" entry
{code}
I don't think there is any reason to keep this in.
Other than those minor points, the patch looks good to me.
I've had a similar issue when calling {{/replication?command=details}}, but am
not able to reproduce it in this test, so I think we're fine to handle that
later.
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513472#comment-15513472
]
Mikhail Khludnev commented on SOLR-9330:
[~mdrob], would you mind to review the last patch?
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509406#comment-15509406
]
Andrey Kudryavtsev commented on SOLR-9330:
--
[~mdrob],
Off course, if you are going to execute in _two separate (clients) threads_
reload() and getStatistics() or deleteCore() and getStatistics() - you might
still have trouble, and solving this races will require additional internal
(probably complicated) synchronization (which by the way will affect all other
cases when you don't really care about races executions)
But in my case I can do this synchronization on client side. I mean just don't
send statistics requests when core is still reloading (I don't care about
reloading core statistics). The only thing I need for this - to know that
reloading is still in progress. Some ability to make reload() fully
synchronized operation. Does anybody know why it is async by default by any
chance?
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508026#comment-15508026
]
Mike Drob commented on SOLR-9330:
-
You're right about there still being a race condition in my solution. For some
reason I wasn't thinking about what happens in that case.
re: jmx exception - yea, it's the same thing. we have a custom jmx handler on
top of the mbean endpoint that has run into this same issue. I should have
cleaned up the commit message there.
[~werder], the reload operation isn't the only place where this can throw
exceptions - it can also happen during core delete or during shutdown. So if we
add synchronization, we will need to look at {{CoreContainer::shutdown}},
{{SolrCores::close}}, etc. I don't have tests written for all of these, but
it's a nearly identical stack trace, given that reload is essentially unload +
load.
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507683#comment-15507683
]
Mikhail Khludnev commented on SOLR-9330:
[~mdrob]
my concern regarding
{code}
if (reader.getRefCount() > 0) {
/// and here core is reloaded in parallel and closes the reader. I think it
can be reproduced with sleep()
lst.add("indexVersion", reader.getVersion());
}
{code}
Can you also share any observation regarding 'jmx exception on shutdown' or
perhaps you have a test? Is it the same case or it's something different?
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507657#comment-15507657
]
Andrey Kudryavtsev commented on SOLR-9330:
--
If we are talking about root cause of the race - it looks like things are a
little bit more easily than I thought. There is just another thread executor
in CoreAdminHandler. If there will be API to avoid it, reload operation will
become more synchronous (patch is attached).
It thought that it will require to do something like SOLR-9390_too_sync.patch,
but not this time probably.
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, SOLR-9390_too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491014#comment-15491014
]
Mike Drob commented on SOLR-9330:
-
I ran into a similar problem recently as well. It can also happen on shutdown
or regular core closing as well.
Is there a reason that {{getVersion}} needs to {{ensureOpen}} first? I noticed
that {{numDocs}} and {{maxDoc}} don't do this (comment claims for performance
reasons) and they are called earlier in the same {{getStatistics}} method.
If we can't remove {{ensureOpen}} from {{StandardDirectoryReader::getVersion}},
then we can also work around this by checking it ourselves. That would look
like:
{code:title=SolrIndexSearcher.java}
-lst.add("indexVersion", reader.getVersion());
+if (reader.getRefCount() > 0) lst.add("indexVersion", reader.getVersion());
{code}
I don't know what the full effects of missing an indexVersion field in the JMX
stats will be, but I am fairly confident that it will be less damaging than the
currently thrown Exception.
[~joel.bernstein] - Do you have any thoughts on which approach to take? Happy
to put up a patch for either one.
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389833#comment-15389833
]
Joel Bernstein commented on SOLR-9330:
--
We are currently looking at exactly this issue at Alfresco. So we'll be happy
to help find the right fix for this.
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
>
> Things happened that we execute this two requests consequentially in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> make it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
