[
https://issues.apache.org/jira/browse/SOLR-10726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar updated SOLR-10726:
-----------------------------------------
Description:
I was investigating some curious behavior reported by a customer around first
searcher event listeners and multiple searchers being opened when adding a new
replica.
Turns out that if you add a new replica to solrcloud:
1) Searchers are opened at least twice and possibly a third time
2) the first time is because of a new core coming online and opening searcher
on an empty index -- only firstSearcher event listeners are fired here
3) second time is after replication is complete and we have new index files
available -- firstSearcher event listeners are fired again because the old
searcher opened on core load has already been closed and disposed so this is
technically again a first searcher
4) third time happens after documents buffered during recovery are replayed --
if there was no indexing happening on leader then this step is skipped -- a
newSearcher event is fired here because we had already opened a searcher in the
last step
Now if instead of a new replica, a solr node is restarted then there can be
upto four searcher opens -- the additional open is because of log replay on
startup.
So Solr spends a lot of time on unnecessary warming/autowarming on searchers
that are discarded. It is not just warming because sometimes plugins such as
SpellCheckComponent and SuggestComponent can also tie in to these listener
events.
We can probably cut a few of them or at least defer the decision of whether to
fire these listeners to places such as RecoveryStrategy which have a better
idea of whether it is worth it.
was:
I was investigating some curious behavior reported by a customer around first
searcher event listeners and multiple replicas being opened when adding a new
replica.
Turns out that if you add a new replica to solrcloud:
1) Searchers are opened at least twice and possibly a third time
2) the first time is because of a new core coming online and opening searcher
on an empty index -- only firstSearcher event listeners are fired here
3) second time is after replication is complete and we have new index files
available -- firstSearcher event listeners are fired again because the old
searcher opened on core load has already been closed and disposed so this is
technically again a first searcher
4) third time happens after documents buffered during recovery are replayed --
if there was no indexing happening on leader then this step is skipped -- a
newSearcher event is fired here because we had already opened a searcher in the
last step
Now if instead of a new replica, a solr node is restarted then there can be
upto four searcher opens -- the additional open is because of log replay on
startup.
So Solr spends a lot of time on unnecessary warming/autowarming on searchers
that are discarded. It is not just warming because sometimes plugins such as
SpellCheckComponent and SuggestComponent can also tie in to these listener
events.
We can probably cut a few of them or at least defer the decision of whether to
fire these listeners to places such as RecoveryStrategy which have a better
idea of whether it is worth it.
> SolrCloud opens multiple searchers on replica creation/startup
> --------------------------------------------------------------
>
> Key: SOLR-10726
> URL: https://issues.apache.org/jira/browse/SOLR-10726
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search, SolrCloud
> Affects Versions: 6.5.1
> Reporter: Shalin Shekhar Mangar
> Labels: difficulty-medium, impact-high
>
> I was investigating some curious behavior reported by a customer around first
> searcher event listeners and multiple searchers being opened when adding a
> new replica.
> Turns out that if you add a new replica to solrcloud:
> 1) Searchers are opened at least twice and possibly a third time
> 2) the first time is because of a new core coming online and opening searcher
> on an empty index -- only firstSearcher event listeners are fired here
> 3) second time is after replication is complete and we have new index files
> available -- firstSearcher event listeners are fired again because the old
> searcher opened on core load has already been closed and disposed so this is
> technically again a first searcher
> 4) third time happens after documents buffered during recovery are replayed
> -- if there was no indexing happening on leader then this step is skipped --
> a newSearcher event is fired here because we had already opened a searcher in
> the last step
> Now if instead of a new replica, a solr node is restarted then there can be
> upto four searcher opens -- the additional open is because of log replay on
> startup.
> So Solr spends a lot of time on unnecessary warming/autowarming on searchers
> that are discarded. It is not just warming because sometimes plugins such as
> SpellCheckComponent and SuggestComponent can also tie in to these listener
> events.
> We can probably cut a few of them or at least defer the decision of whether
> to fire these listeners to places such as RecoveryStrategy which have a
> better idea of whether it is worth it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
