[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[ https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515842#comment-15515842 ] Mikhail Khludnev commented on SOLR-9330: Exception should gone. Sometimes response contains no "search" entry, but only something like {{Searcher@deadbeaf [collection1] main}}. So, patterns consuming stats response might, but shouldn't be surprised. > Race condition between core reload and statistics request > - > > Key: SOLR-9330 > URL: https://issues.apache.org/jira/browse/SOLR-9330 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 5.5 >Reporter: Andrey Kudryavtsev >Assignee: Mikhail Khludnev > Fix For: 6.3, master (7.0) > > Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch, > SOLR-9390.patch, SOLR-9390.patch, too_sync.patch > > > Things happened that we execute this two requests consecutively in Solr 5.5: > * Core reload: /admin/cores?action=RELOAD=_coreName_ > * Check core statistics: /_coreName_/admin/mbeans?stats=true > And sometimes second request ends with this error: > {code} > ERROR org.apache.solr.servlet.HttpSolrCall - > null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is > closed > at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) > at > org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183) > {code} > If my understanding of SolrCore internals is correct, it happens because of > async nature of reload request: > * New searcher is "registered" in separate thread > * Old searcher is closed in same separate thread and only after new one is > registered > * When old searcher is closing, it removes itself from map with MBeans > * If statistic requests happens before old searcher is completely removed > from everywhere - exception can happen. > What do you think if we will introduce new parameter for reload request which > makes it fully synchronized? Basically it will force it to call {code} > SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] > waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[ https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515812#comment-15515812 ] ASF subversion and git services commented on SOLR-9330: --- Commit 4278356564d4928dfabfde6d4a2ebf57dcfcf81f in lucene-solr's branch refs/heads/branch_6x from [~mkhludnev] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4278356 ] SOLR-9330: Fix AlreadyClosedException on admin/mbeans?stats=true > Race condition between core reload and statistics request > - > > Key: SOLR-9330 > URL: https://issues.apache.org/jira/browse/SOLR-9330 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 5.5 >Reporter: Andrey Kudryavtsev > Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch, > SOLR-9390.patch, SOLR-9390.patch, too_sync.patch > > > Things happened that we execute this two requests consecutively in Solr 5.5: > * Core reload: /admin/cores?action=RELOAD=_coreName_ > * Check core statistics: /_coreName_/admin/mbeans?stats=true > And sometimes second request ends with this error: > {code} > ERROR org.apache.solr.servlet.HttpSolrCall - > null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is > closed > at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) > at > org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183) > {code} > If my understanding of SolrCore internals is correct, it happens because of > async nature of reload request: > * New searcher is "registered" in separate thread > * Old searcher is closed in same separate thread and only after new one is > registered > * When old searcher is closing, it removes itself from map with MBeans > * If statistic requests happens before old searcher is completely removed > from everywhere - exception can happen. > What do you think if we will introduce new parameter for reload request which > makes it fully synchronized? Basically it will force it to call {code} > SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] > waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[ https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515411#comment-15515411 ] ASF subversion and git services commented on SOLR-9330: --- Commit b50b9106f821915ced14a3efc1e09c265d648fa8 in lucene-solr's branch refs/heads/master from [~mkhludnev] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b50b910 ] SOLR-9330: Fix AlreadyClosedException on admin/mbeans?stats=true > Race condition between core reload and statistics request > - > > Key: SOLR-9330 > URL: https://issues.apache.org/jira/browse/SOLR-9330 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 5.5 >Reporter: Andrey Kudryavtsev > Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch, > SOLR-9390.patch, SOLR-9390.patch, too_sync.patch > > > Things happened that we execute this two requests consecutively in Solr 5.5: > * Core reload: /admin/cores?action=RELOAD=_coreName_ > * Check core statistics: /_coreName_/admin/mbeans?stats=true > And sometimes second request ends with this error: > {code} > ERROR org.apache.solr.servlet.HttpSolrCall - > null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is > closed > at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) > at > org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183) > {code} > If my understanding of SolrCore internals is correct, it happens because of > async nature of reload request: > * New searcher is "registered" in separate thread > * Old searcher is closed in same separate thread and only after new one is > registered > * When old searcher is closing, it removes itself from map with MBeans > * If statistic requests happens before old searcher is completely removed > from everywhere - exception can happen. > What do you think if we will introduce new parameter for reload request which > makes it fully synchronized? Basically it will force it to call {code} > SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] > waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[ https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513882#comment-15513882 ] Mike Drob commented on SOLR-9330: - Ah, I see where the difference is, yes. In my case, the client process getting the statistics is an external monitoring application that gets them every 15 seconds and charts them. Since number of replicas can move, grow and shrink to accommodate usage, solving races like this is a very complicated problem. And at the end of the day, I don't care if my monitoring system misses one round of statistics, I'm more concerned about scary exceptions in the log that the ops team has to deal with. > Race condition between core reload and statistics request > - > > Key: SOLR-9330 > URL: https://issues.apache.org/jira/browse/SOLR-9330 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 5.5 >Reporter: Andrey Kudryavtsev > Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch, > SOLR-9390.patch, SOLR-9390.patch, too_sync.patch > > > Things happened that we execute this two requests consecutively in Solr 5.5: > * Core reload: /admin/cores?action=RELOAD=_coreName_ > * Check core statistics: /_coreName_/admin/mbeans?stats=true > And sometimes second request ends with this error: > {code} > ERROR org.apache.solr.servlet.HttpSolrCall - > null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is > closed > at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) > at > org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183) > {code} > If my understanding of SolrCore internals is correct, it happens because of > async nature of reload request: > * New searcher is "registered" in separate thread > * Old searcher is closed in same separate thread and only after new one is > registered > * When old searcher is closing, it removes itself from map with MBeans > * If statistic requests happens before old searcher is completely removed > from everywhere - exception can happen. > What do you think if we will introduce new parameter for reload request which > makes it fully synchronized? Basically it will force it to call {code} > SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] > waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[ https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513869#comment-15513869 ] Mike Drob commented on SOLR-9330: - {code} lst.add("searcherName", name); lst.add("caching", cachingEnabled); lst.add("openedAt", openTime); if (registerTime != null) lst.add("registeredAt", registerTime); lst.add("warmupTime", warmupTime); {code} Why not put these in the cached list as well? The first three are final and available before your call to {{snapStatistics}}. The last two are set during {{register}} which should only be called once, if I understand this correctly. Then the whole method becomes {{return readerStats;}} -- much simpler and probably faster too! {code} +// core.getInfoRegistry().remove(STATISTICS_KEY, this); +// decided to comment it, because it might upset users by showing stats, w/o "searcher" entry {code} I don't think there is any reason to keep this in. Other than those minor points, the patch looks good to me. I've had a similar issue when calling {{/replication?command=details}}, but am not able to reproduce it in this test, so I think we're fine to handle that later. > Race condition between core reload and statistics request > - > > Key: SOLR-9330 > URL: https://issues.apache.org/jira/browse/SOLR-9330 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 5.5 >Reporter: Andrey Kudryavtsev > Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch, > SOLR-9390.patch, SOLR-9390.patch, too_sync.patch > > > Things happened that we execute this two requests consecutively in Solr 5.5: > * Core reload: /admin/cores?action=RELOAD=_coreName_ > * Check core statistics: /_coreName_/admin/mbeans?stats=true > And sometimes second request ends with this error: > {code} > ERROR org.apache.solr.servlet.HttpSolrCall - > null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is > closed > at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) > at > org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183) > {code} > If my understanding of SolrCore internals is correct, it happens because of > async nature of reload request: > * New searcher is "registered" in separate thread > * Old searcher is closed in same separate thread and only after new one is > registered > * When old searcher is closing, it removes itself from map with MBeans > * If statistic requests happens before old searcher is completely removed > from everywhere - exception can happen. > What do you think if we will introduce new parameter for reload request which > makes it fully synchronized? Basically it will force it to call {code} > SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] > waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[ https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513868#comment-15513868 ] Mike Drob commented on SOLR-9330: - {code} lst.add("searcherName", name); lst.add("caching", cachingEnabled); lst.add("openedAt", openTime); if (registerTime != null) lst.add("registeredAt", registerTime); lst.add("warmupTime", warmupTime); {code} Why not put these in the cached list as well? The first three are final and available before your call to {{snapStatistics}}. The last two are set during {{register}} which should only be called once, if I understand this correctly. Then the whole method becomes {{return readerStats;}} -- much simpler and probably faster too! {code} +// core.getInfoRegistry().remove(STATISTICS_KEY, this); +// decided to comment it, because it might upset users by showing stats, w/o "searcher" entry {code} I don't think there is any reason to keep this in. Other than those minor points, the patch looks good to me. I've had a similar issue when calling {{/replication?command=details}}, but am not able to reproduce it in this test, so I think we're fine to handle that later. > Race condition between core reload and statistics request > - > > Key: SOLR-9330 > URL: https://issues.apache.org/jira/browse/SOLR-9330 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 5.5 >Reporter: Andrey Kudryavtsev > Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch, > SOLR-9390.patch, SOLR-9390.patch, too_sync.patch > > > Things happened that we execute this two requests consecutively in Solr 5.5: > * Core reload: /admin/cores?action=RELOAD=_coreName_ > * Check core statistics: /_coreName_/admin/mbeans?stats=true > And sometimes second request ends with this error: > {code} > ERROR org.apache.solr.servlet.HttpSolrCall - > null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is > closed > at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) > at > org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183) > {code} > If my understanding of SolrCore internals is correct, it happens because of > async nature of reload request: > * New searcher is "registered" in separate thread > * Old searcher is closed in same separate thread and only after new one is > registered > * When old searcher is closing, it removes itself from map with MBeans > * If statistic requests happens before old searcher is completely removed > from everywhere - exception can happen. > What do you think if we will introduce new parameter for reload request which > makes it fully synchronized? Basically it will force it to call {code} > SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] > waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[ https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513472#comment-15513472 ] Mikhail Khludnev commented on SOLR-9330: [~mdrob], would you mind to review the last patch? > Race condition between core reload and statistics request > - > > Key: SOLR-9330 > URL: https://issues.apache.org/jira/browse/SOLR-9330 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 5.5 >Reporter: Andrey Kudryavtsev > Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch, > SOLR-9390.patch, SOLR-9390.patch, too_sync.patch > > > Things happened that we execute this two requests consecutively in Solr 5.5: > * Core reload: /admin/cores?action=RELOAD=_coreName_ > * Check core statistics: /_coreName_/admin/mbeans?stats=true > And sometimes second request ends with this error: > {code} > ERROR org.apache.solr.servlet.HttpSolrCall - > null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is > closed > at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) > at > org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183) > {code} > If my understanding of SolrCore internals is correct, it happens because of > async nature of reload request: > * New searcher is "registered" in separate thread > * Old searcher is closed in same separate thread and only after new one is > registered > * When old searcher is closing, it removes itself from map with MBeans > * If statistic requests happens before old searcher is completely removed > from everywhere - exception can happen. > What do you think if we will introduce new parameter for reload request which > makes it fully synchronized? Basically it will force it to call {code} > SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] > waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[ https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509406#comment-15509406 ] Andrey Kudryavtsev commented on SOLR-9330: -- [~mdrob], Off course, if you are going to execute in _two separate (clients) threads_ reload() and getStatistics() or deleteCore() and getStatistics() - you might still have trouble, and solving this races will require additional internal (probably complicated) synchronization (which by the way will affect all other cases when you don't really care about races executions) But in my case I can do this synchronization on client side. I mean just don't send statistics requests when core is still reloading (I don't care about reloading core statistics). The only thing I need for this - to know that reloading is still in progress. Some ability to make reload() fully synchronized operation. Does anybody know why it is async by default by any chance? > Race condition between core reload and statistics request > - > > Key: SOLR-9330 > URL: https://issues.apache.org/jira/browse/SOLR-9330 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 5.5 >Reporter: Andrey Kudryavtsev > Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch, > SOLR-9390.patch, too_sync.patch > > > Things happened that we execute this two requests consecutively in Solr 5.5: > * Core reload: /admin/cores?action=RELOAD=_coreName_ > * Check core statistics: /_coreName_/admin/mbeans?stats=true > And sometimes second request ends with this error: > {code} > ERROR org.apache.solr.servlet.HttpSolrCall - > null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is > closed > at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) > at > org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183) > {code} > If my understanding of SolrCore internals is correct, it happens because of > async nature of reload request: > * New searcher is "registered" in separate thread > * Old searcher is closed in same separate thread and only after new one is > registered > * When old searcher is closing, it removes itself from map with MBeans > * If statistic requests happens before old searcher is completely removed > from everywhere - exception can happen. > What do you think if we will introduce new parameter for reload request which > makes it fully synchronized? Basically it will force it to call {code} > SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] > waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[ https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508026#comment-15508026 ] Mike Drob commented on SOLR-9330: - You're right about there still being a race condition in my solution. For some reason I wasn't thinking about what happens in that case. re: jmx exception - yea, it's the same thing. we have a custom jmx handler on top of the mbean endpoint that has run into this same issue. I should have cleaned up the commit message there. [~werder], the reload operation isn't the only place where this can throw exceptions - it can also happen during core delete or during shutdown. So if we add synchronization, we will need to look at {{CoreContainer::shutdown}}, {{SolrCores::close}}, etc. I don't have tests written for all of these, but it's a nearly identical stack trace, given that reload is essentially unload + load. > Race condition between core reload and statistics request > - > > Key: SOLR-9330 > URL: https://issues.apache.org/jira/browse/SOLR-9330 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 5.5 >Reporter: Andrey Kudryavtsev > Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch, > SOLR-9390.patch, too_sync.patch > > > Things happened that we execute this two requests consecutively in Solr 5.5: > * Core reload: /admin/cores?action=RELOAD=_coreName_ > * Check core statistics: /_coreName_/admin/mbeans?stats=true > And sometimes second request ends with this error: > {code} > ERROR org.apache.solr.servlet.HttpSolrCall - > null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is > closed > at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) > at > org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183) > {code} > If my understanding of SolrCore internals is correct, it happens because of > async nature of reload request: > * New searcher is "registered" in separate thread > * Old searcher is closed in same separate thread and only after new one is > registered > * When old searcher is closing, it removes itself from map with MBeans > * If statistic requests happens before old searcher is completely removed > from everywhere - exception can happen. > What do you think if we will introduce new parameter for reload request which > makes it fully synchronized? Basically it will force it to call {code} > SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] > waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[ https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507683#comment-15507683 ] Mikhail Khludnev commented on SOLR-9330: [~mdrob] my concern regarding {code} if (reader.getRefCount() > 0) { /// and here core is reloaded in parallel and closes the reader. I think it can be reproduced with sleep() lst.add("indexVersion", reader.getVersion()); } {code} Can you also share any observation regarding 'jmx exception on shutdown' or perhaps you have a test? Is it the same case or it's something different? > Race condition between core reload and statistics request > - > > Key: SOLR-9330 > URL: https://issues.apache.org/jira/browse/SOLR-9330 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 5.5 >Reporter: Andrey Kudryavtsev > Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch, > SOLR-9390.patch, too_sync.patch > > > Things happened that we execute this two requests consecutively in Solr 5.5: > * Core reload: /admin/cores?action=RELOAD=_coreName_ > * Check core statistics: /_coreName_/admin/mbeans?stats=true > And sometimes second request ends with this error: > {code} > ERROR org.apache.solr.servlet.HttpSolrCall - > null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is > closed > at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) > at > org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183) > {code} > If my understanding of SolrCore internals is correct, it happens because of > async nature of reload request: > * New searcher is "registered" in separate thread > * Old searcher is closed in same separate thread and only after new one is > registered > * When old searcher is closing, it removes itself from map with MBeans > * If statistic requests happens before old searcher is completely removed > from everywhere - exception can happen. > What do you think if we will introduce new parameter for reload request which > makes it fully synchronized? Basically it will force it to call {code} > SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] > waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[ https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507657#comment-15507657 ] Andrey Kudryavtsev commented on SOLR-9330: -- If we are talking about root cause of the race - it looks like things are a little bit more easily than I thought. There is just another thread executor in CoreAdminHandler. If there will be API to avoid it, reload operation will become more synchronous (patch is attached). It thought that it will require to do something like SOLR-9390_too_sync.patch, but not this time probably. > Race condition between core reload and statistics request > - > > Key: SOLR-9330 > URL: https://issues.apache.org/jira/browse/SOLR-9330 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 5.5 >Reporter: Andrey Kudryavtsev > Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch, > SOLR-9390.patch, SOLR-9390_too_sync.patch > > > Things happened that we execute this two requests consecutively in Solr 5.5: > * Core reload: /admin/cores?action=RELOAD=_coreName_ > * Check core statistics: /_coreName_/admin/mbeans?stats=true > And sometimes second request ends with this error: > {code} > ERROR org.apache.solr.servlet.HttpSolrCall - > null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is > closed > at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) > at > org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183) > {code} > If my understanding of SolrCore internals is correct, it happens because of > async nature of reload request: > * New searcher is "registered" in separate thread > * Old searcher is closed in same separate thread and only after new one is > registered > * When old searcher is closing, it removes itself from map with MBeans > * If statistic requests happens before old searcher is completely removed > from everywhere - exception can happen. > What do you think if we will introduce new parameter for reload request which > makes it fully synchronized? Basically it will force it to call {code} > SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] > waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[ https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491014#comment-15491014 ] Mike Drob commented on SOLR-9330: - I ran into a similar problem recently as well. It can also happen on shutdown or regular core closing as well. Is there a reason that {{getVersion}} needs to {{ensureOpen}} first? I noticed that {{numDocs}} and {{maxDoc}} don't do this (comment claims for performance reasons) and they are called earlier in the same {{getStatistics}} method. If we can't remove {{ensureOpen}} from {{StandardDirectoryReader::getVersion}}, then we can also work around this by checking it ourselves. That would look like: {code:title=SolrIndexSearcher.java} -lst.add("indexVersion", reader.getVersion()); +if (reader.getRefCount() > 0) lst.add("indexVersion", reader.getVersion()); {code} I don't know what the full effects of missing an indexVersion field in the JMX stats will be, but I am fairly confident that it will be less damaging than the currently thrown Exception. [~joel.bernstein] - Do you have any thoughts on which approach to take? Happy to put up a patch for either one. > Race condition between core reload and statistics request > - > > Key: SOLR-9330 > URL: https://issues.apache.org/jira/browse/SOLR-9330 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 5.5 >Reporter: Andrey Kudryavtsev > > Things happened that we execute this two requests consecutively in Solr 5.5: > * Core reload: /admin/cores?action=RELOAD=_coreName_ > * Check core statistics: /_coreName_/admin/mbeans?stats=true > And sometimes second request ends with this error: > {code} > ERROR org.apache.solr.servlet.HttpSolrCall - > null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is > closed > at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) > at > org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183) > {code} > If my understanding of SolrCore internals is correct, it happens because of > async nature of reload request: > * New searcher is "registered" in separate thread > * Old searcher is closed in same separate thread and only after new one is > registered > * When old searcher is closing, it removes itself from map with MBeans > * If statistic requests happens before old searcher is completely removed > from everywhere - exception can happen. > What do you think if we will introduce new parameter for reload request which > makes it fully synchronized? Basically it will force it to call {code} > SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] > waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9330) Race condition between core reload and statistics request
[ https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389833#comment-15389833 ] Joel Bernstein commented on SOLR-9330: -- We are currently looking at exactly this issue at Alfresco. So we'll be happy to help find the right fix for this. > Race condition between core reload and statistics request > - > > Key: SOLR-9330 > URL: https://issues.apache.org/jira/browse/SOLR-9330 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 5.5 >Reporter: Andrey Kudryavtsev > > Things happened that we execute this two requests consequentially in Solr 5.5: > * Core reload: /admin/cores?action=RELOAD=_coreName_ > * Check core statistics: /_coreName_/admin/mbeans?stats=true > And sometimes second request ends with this error: > {code} > ERROR org.apache.solr.servlet.HttpSolrCall - > null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is > closed > at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) > at > org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) > at > org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) > at > org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183) > {code} > If my understanding of SolrCore internals is correct, it happens because of > async nature of reload request: > * New searcher is "registered" in separate thread > * Old searcher is closed in same separate thread and only after new one is > registered > * When old searcher is closing, it removes itself from map with MBeans > * If statistic requests happens before old searcher is completely removed > from everywhere - exception can happen. > What do you think if we will introduce new parameter for reload request which > make it fully synchronized? Basically it will force it to call {code} > SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] > waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org