[
https://issues.apache.org/jira/browse/SOLR-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864604#comment-16864604
]
Lucene/Solr QA commented on SOLR-13490:
---------------------------------------
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 2m 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 1m 53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 1m 53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 49m
0s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
44s{color} | {color:green} solrj in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} test-framework in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-13490 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12971853/SOLR-13490.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon
Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / abb5ea0 |
| ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 |
| Default Java | LTS |
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/436/testReport/ |
| modules | C: solr/core solr/solrj solr/test-framework U: solr |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/436/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> waitForState/registerCollectionStateWatcher can see stale liveNodes data due
> to (Zk) Watcher race condition
> -----------------------------------------------------------------------------------------------------------
>
> Key: SOLR-13490
> URL: https://issues.apache.org/jira/browse/SOLR-13490
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
> Assignee: Hoss Man
> Priority: Major
> Attachments: SOLR-13490.patch, SOLR-13490.patch, SOLR-13490.patch,
> SOLR-13490.patch
>
>
> I was investigating some failures in
> {{TestCloudSearcherWarming.testRepFactor1LeaderStartup}} which lead me to the
> hunch that {{waitForState}} wasn't ensuring that the predicates registered
> would always be called if/when a node was shutdown.
> Digging into it a bit more, I found that the root cause seems to be the way
> the {{CollectionStateWatcher}} / {{CollectionStatePredicate}} APIs pass in
> *both* the {{DocCollection}}, and the "current" {{liveNodes}} - but are only
> _triggered_ by the {{StateWatcher}} on the {{state.json}} (which is used to
> rebuild the {{DocCollection}}) - when the {{CollectionStateWatcher}} /
> {{CollectionStatePredicate}} are called, they get the "fresh"
> {{DocCollection}} but they get the _cached_ {{ZkStateReader.liveNodes}}
> Meanwhile, the {{LiveNodeWatcher}} only calls {{refreshLiveNodes()}} only
> updates {{ZkStateReader.liveNodes}} and triggers any {{LiveNodesListener}} -
> it does *NOT* invoke any {{CollectionStateWatcher}} that may have replicas
> hosted on any of changed nodes.
> Since there is no garunteed order that Watchers will be triggered, this means
> there is a race condition where the following can happen...
> * client1 has a ZkStateReader with cached {{liveNodes=[N1, N2, N3]}}
> * client1 registers a {{CollectionStateWatcher}} "watcherZ" that cares if
> "replicaX" of collectionA is on a "down" node
> * client2 causes shutdown of node N1 which is hosting replicaX
> * client1's zkStateReader gets a WatchedEvent for state.json of collectionA
> ** DocCollection for collectionA is rebuilt
> ** watcherZ is fired w/cached {{liveNodes=[N1, N2, N3]}} and the new
> DocCollection
> *** watcherZ sees that replicaX is on N1, but thinks N1 is live
> *** watcherZ says "everything ok, not the event i was waiting for" and
> doesn't take any action
> * client1's zkStateReader gets a WatchedEvent for LIVE_NODES_ZKNODE
> ** zkStateReader.liveNodes is rebuilt
> ...at no point in this sequence (or after this) will watcherZ be notified
> fired with the updated liveNodes (unless/until another {{state.json}} change
> is made for collectionA.
> ----
> While this is definitely be problematic in _tests_ that deal with node
> lifecyle and use things like {{SolrCloudTestCase.waitForState(...,
> SolrCloudTestCase.clusterShape(...))}} to check for the expected
> shards/replicas, a cursory search of how/where
> {{ZkStateReader.waitForState(...)}} and
> {{ZkStateReader.registerCollectionStateWatcher(...)}} are used in solr-core
> suggests that could also lead to bad behavior in situations like reacting to
> shard leader loss, waiting for all leaders of SYSTEM_COLL to come online for
> upgrade, running PrepRecoveryOp, etc... (anywhere that liveNodes is used by
> the watcher/predicate)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
