[
https://issues.apache.org/jira/browse/SOLR-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17770644#comment-17770644
]
Michael Gibney commented on SOLR-11535:
---------------------------------------
This issue is not purely academic -- it can manifest in really problematic
ways. If enough extra StateWatchers get registered, heavy load can trigger a
vicious cycle where zkCallback threadpool balloons, and thread contention
exacerbates the same race condition, and in some cases the node entirely
freezes up. Seen mostly during cluster restarts (lots of CPU, and also lots of
state updates).
I suspect this also explains why thread count sometimes goes quite high upon
restart, but (in cases where the node recovers) then shrinks within a
reasonable amount of time. I had wondered about this behavior because it's
distinct from ballooning of the jetty thread pool, which would shrink extremely
slowly.
> Weird behavior of CollectionStateWatcher
> ----------------------------------------
>
> Key: SOLR-11535
> URL: https://issues.apache.org/jira/browse/SOLR-11535
> Project: Solr
> Issue Type: Bug
> Affects Versions: 7.2, 8.0
> Reporter: Andrzej Bialecki
> Assignee: Michael Gibney
> Priority: Major
> Attachments: test.log
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> While working on SOLR-11320 I noticed a strange behavior in
> {{ActiveReplicaWatcher}}, which is a subclass of {{CollectionStateWatcher}} -
> it appears that its {{onStateChanged}} method can be called from multiple
> threads with exactly the same {{DocCollection}} state, ie. unchanged between
> the calls.
> This seems to run contrary to the javadoc, which implies that this method is
> called only when the state actually changes, and it also doesn't mention
> anything about the need for thread-safety in the method implementation.
> I attached the log, which has a lot of additional debugging - but the most
> pertinent part being where a Watcher-s hashCode is printed together with the
> {{DocCollection}} - notice that these overlapping calls both submit an
> instance of {{DocCollection}} with the same zkVersion.
> [~dragonsinth], [~romseygeek] - could you please take a look at this? If this
> behavior is expected then the javadoc should be updated to state clearly that
> multiple calls can be made concurrently, with exactly the same state (which
> is kind of a weak guarantee for a method called {{onStateChanged}} ;) ).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
