[ https://issues.apache.org/jira/browse/SOLR-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863964#comment-16863964 ]
Lucene/Solr QA commented on SOLR-13490: --------------------------------------- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 34s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Release audit (RAT) {color} | {color:green} 3m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Check forbidden APIs {color} | {color:green} 3m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Validate source patterns {color} | {color:green} 3m 10s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 9s{color} | {color:red} core in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 37s{color} | {color:green} solrj in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s{color} | {color:green} test-framework in the patch passed. {color} | | {color:black}{color} | {color:black} {color} | {color:black}122m 44s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | solr.cloud.AliasIntegrationTest | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | SOLR-13490 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12971740/SOLR-13490.patch | | Optional Tests | compile javac unit ratsources checkforbiddenapis validatesourcepatterns | | uname | Linux lucene2-us-west.apache.org 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | ant | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh | | git revision | master / 81e8b38 | | ant | version: Apache Ant(TM) version 1.9.6 compiled on July 20 2018 | | Default Java | LTS | | unit | https://builds.apache.org/job/PreCommit-SOLR-Build/435/artifact/out/patch-unit-solr_core.txt | | Test Results | https://builds.apache.org/job/PreCommit-SOLR-Build/435/testReport/ | | modules | C: solr/core solr/solrj solr/test-framework U: solr | | Console output | https://builds.apache.org/job/PreCommit-SOLR-Build/435/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > waitForState/registerCollectionStateWatcher can see stale liveNodes data due > to (Zk) Watcher race condition > ----------------------------------------------------------------------------------------------------------- > > Key: SOLR-13490 > URL: https://issues.apache.org/jira/browse/SOLR-13490 > Project: Solr > Issue Type: Bug > Reporter: Hoss Man > Assignee: Hoss Man > Priority: Major > Attachments: SOLR-13490.patch, SOLR-13490.patch, SOLR-13490.patch > > > I was investigating some failures in > {{TestCloudSearcherWarming.testRepFactor1LeaderStartup}} which lead me to the > hunch that {{waitForState}} wasn't ensuring that the predicates registered > would always be called if/when a node was shutdown. > Digging into it a bit more, I found that the root cause seems to be the way > the {{CollectionStateWatcher}} / {{CollectionStatePredicate}} APIs pass in > *both* the {{DocCollection}}, and the "current" {{liveNodes}} - but are only > _triggered_ by the {{StateWatcher}} on the {{state.json}} (which is used to > rebuild the {{DocCollection}}) - when the {{CollectionStateWatcher}} / > {{CollectionStatePredicate}} are called, they get the "fresh" > {{DocCollection}} but they get the _cached_ {{ZkStateReader.liveNodes}} > Meanwhile, the {{LiveNodeWatcher}} only calls {{refreshLiveNodes()}} only > updates {{ZkStateReader.liveNodes}} and triggers any {{LiveNodesListener}} - > it does *NOT* invoke any {{CollectionStateWatcher}} that may have replicas > hosted on any of changed nodes. > Since there is no garunteed order that Watchers will be triggered, this means > there is a race condition where the following can happen... > * client1 has a ZkStateReader with cached {{liveNodes=[N1, N2, N3]}} > * client1 registers a {{CollectionStateWatcher}} "watcherZ" that cares if > "replicaX" of collectionA is on a "down" node > * client2 causes shutdown of node N1 which is hosting replicaX > * client1's zkStateReader gets a WatchedEvent for state.json of collectionA > ** DocCollection for collectionA is rebuilt > ** watcherZ is fired w/cached {{liveNodes=[N1, N2, N3]}} and the new > DocCollection > *** watcherZ sees that replicaX is on N1, but thinks N1 is live > *** watcherZ says "everything ok, not the event i was waiting for" and > doesn't take any action > * client1's zkStateReader gets a WatchedEvent for LIVE_NODES_ZKNODE > ** zkStateReader.liveNodes is rebuilt > ...at no point in this sequence (or after this) will watcherZ be notified > fired with the updated liveNodes (unless/until another {{state.json}} change > is made for collectionA. > ---- > While this is definitely be problematic in _tests_ that deal with node > lifecyle and use things like {{SolrCloudTestCase.waitForState(..., > SolrCloudTestCase.clusterShape(...))}} to check for the expected > shards/replicas, a cursory search of how/where > {{ZkStateReader.waitForState(...)}} and > {{ZkStateReader.registerCollectionStateWatcher(...)}} are used in solr-core > suggests that could also lead to bad behavior in situations like reacting to > shard leader loss, waiting for all leaders of SYSTEM_COLL to come online for > upgrade, running PrepRecoveryOp, etc... (anywhere that liveNodes is used by > the watcher/predicate) -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org