[jira] [Updated] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Khludnev updated SOLR-9330:
---
Fix Version/s: master (7.0)
6.3
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Khludnev updated SOLR-9330:
---
Attachment: SOLR-9390.patch
The best way to win the race is to avoid it. Please check [^SOLR-9390.patch]
and expose your concern, I reduced sweeping fixes, but left one useless note
commented (regarding removing from infoRegistry), let me know if you disagree
to keep it in source. I think it's ready to be committed this week.
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Kudryavtsev updated SOLR-9330:
-
Attachment: (was: SOLR-9390_too_sync.patch)
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Kudryavtsev updated SOLR-9330:
-
Attachment: too_sync.patch
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Kudryavtsev updated SOLR-9330:
-
Attachment: SOLR-9390_too_sync.patch
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch, SOLR-9390_too_sync.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Kudryavtsev updated SOLR-9330:
-
Attachment: SOLR-9390.patch
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch,
> SOLR-9390.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated SOLR-9330:
Attachment: SOLR-9330.patch
[~mkhludnev] - that looks like a good start to a test. I do not think 10
seconds is long enough to relegate something to @Nightly runs.
I'm putting up an alternative fix that seems to also make your test pass. I
think I like my approach better because it may still return partial results
when able, but I'm also not convinced that there won't be other handlers that
suffer from similar problems that are still not addressed. Your approach may be
able to fix everything at once.
Maybe the best solution is to do both things.
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9330.patch, SOLR-9390.patch, SOLR-9390.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Khludnev updated SOLR-9330:
---
Attachment: SOLR-9390.patch
I made a few common sense/best effort changes, but nothing beside try\{}catch()
helps. Perhaps it might be synchronized properly with
{{solrCore._searcherLock}} but it's too much changes I suppose.
I'll try to just cache reader.version or whole reader stats since it shouldn't
be changed during searcher life (will check.)
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9390.patch, SOLR-9390.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Khludnev updated SOLR-9330:
---
Attachment: SOLR-9390.patch
Here is straightforward test reproducing the race, it catches exception at
least at every third run at my laptop. It might be dirty yet, nevertheless.
I wonder if I need to mark it \@Nightly if it run for 10 sec?
> Race condition between core reload and statistics request
> -
>
> Key: SOLR-9330
> URL: https://issues.apache.org/jira/browse/SOLR-9330
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 5.5
>Reporter: Andrey Kudryavtsev
> Attachments: SOLR-9390.patch
>
>
> Things happened that we execute this two requests consecutively in Solr 5.5:
> * Core reload: /admin/cores?action=RELOAD=_coreName_
> * Check core statistics: /_coreName_/admin/mbeans?stats=true
> And sometimes second request ends with this error:
> {code}
> ERROR org.apache.solr.servlet.HttpSolrCall -
> null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
> at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
> at
> org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
> at
> org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
> {code}
> If my understanding of SolrCore internals is correct, it happens because of
> async nature of reload request:
> * New searcher is "registered" in separate thread
> * Old searcher is closed in same separate thread and only after new one is
> registered
> * When old searcher is closing, it removes itself from map with MBeans
> * If statistic requests happens before old searcher is completely removed
> from everywhere - exception can happen.
> What do you think if we will introduce new parameter for reload request which
> makes it fully synchronized? Basically it will force it to call {code}
> SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
> waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Kudryavtsev updated SOLR-9330:
-
Description:
Things happened that we execute this two requests consecutively in Solr 5.5:
* Core reload: /admin/cores?action=RELOAD=_coreName_
* Check core statistics: /_coreName_/admin/mbeans?stats=true
And sometimes second request ends with this error:
{code}
ERROR org.apache.solr.servlet.HttpSolrCall -
null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
at
org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
at
org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
at
org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
at
org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
{code}
If my understanding of SolrCore internals is correct, it happens because of
async nature of reload request:
* New searcher is "registered" in separate thread
* Old searcher is closed in same separate thread and only after new one is
registered
* When old searcher is closing, it removes itself from map with MBeans
* If statistic requests happens before old searcher is completely removed from
everywhere - exception can happen.
What do you think if we will introduce new parameter for reload request which
makes it fully synchronized? Basically it will force it to call {code}
SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
was:
Things happened that we execute this two requests consequentially in Solr 5.5:
* Core reload: /admin/cores?action=RELOAD=_coreName_
* Check core statistics: /_coreName_/admin/mbeans?stats=true
And sometimes second request ends with this error:
{code}
ERROR org.apache.solr.servlet.HttpSolrCall -
null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
at
org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
at
org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
at
org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
at
org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
{code}
If my understanding of SolrCore internals is correct, it happens because of
async nature of reload request:
* New searcher is "registered" in separate thread
* Old searcher is closed in same separate thread and only after new one is
registered
* When old searcher is closing, it removes itself from map with MBeans
* If statistic requests happens before old searcher is completely removed from
everywhere - exception can happen.
What do you think if we will introduce new parameter for reload request which
makes it fully synchronized? Basically it will force it to call {code}
SolrCore#getSearcher(boolean forceNew,
[jira] [Updated] (SOLR-9330) Race condition between core reload and statistics request
[
https://issues.apache.org/jira/browse/SOLR-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Kudryavtsev updated SOLR-9330:
-
Description:
Things happened that we execute this two requests consequentially in Solr 5.5:
* Core reload: /admin/cores?action=RELOAD=_coreName_
* Check core statistics: /_coreName_/admin/mbeans?stats=true
And sometimes second request ends with this error:
{code}
ERROR org.apache.solr.servlet.HttpSolrCall -
null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
at
org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
at
org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
at
org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
at
org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
{code}
If my understanding of SolrCore internals is correct, it happens because of
async nature of reload request:
* New searcher is "registered" in separate thread
* Old searcher is closed in same separate thread and only after new one is
registered
* When old searcher is closing, it removes itself from map with MBeans
* If statistic requests happens before old searcher is completely removed from
everywhere - exception can happen.
What do you think if we will introduce new parameter for reload request which
makes it fully synchronized? Basically it will force it to call {code}
SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[]
waitSearcher, boolean updateHandlerReopens) {code} with waitSearcher!= null
was:
Things happened that we execute this two requests consequentially in Solr 5.5:
* Core reload: /admin/cores?action=RELOAD=_coreName_
* Check core statistics: /_coreName_/admin/mbeans?stats=true
And sometimes second request ends with this error:
{code}
ERROR org.apache.solr.servlet.HttpSolrCall -
null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
at
org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
at
org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
at
org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
at
org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
{code}
If my understanding of SolrCore internals is correct, it happens because of
async nature of reload request:
* New searcher is "registered" in separate thread
* Old searcher is closed in same separate thread and only after new one is
registered
* When old searcher is closing, it removes itself from map with MBeans
* If statistic requests happens before old searcher is completely removed from
everywhere - exception can happen.
What do you think if we will introduce new parameter for reload request which
make it fully synchronized? Basically it will force it to call {code}
SolrCore#getSearcher(boolean
