Hi Ariel, test_simple_bootstrap_mixed_versions issue is related to CASSANDRA-13004 <https://issues.apache.org/jira/browse/CASSANDRA-13004>, which introduced "cassandra.force_3_0_protocol_version" for schema migrations during upgrades from 3.0.14 upwards. This flag is missing in `test_simple_bootstrap_mixed_versions` upgrade test while we are adding/bootstrapping 3.11.4 node to an existing 3.5 version of C* node. This resulted in `ks` keyspace schema/data not being bootstrapped to the new node.
I debugged and confirmed that MigrationManager::is30Compatible <https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/service/MigrationManager.java#L181-L185> is returning false which is forcing MigrationManager::shouldPullSchemaFrom <https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/service/MigrationManager.java#L168-L177> to return false as well. *From debug logs:* DEBUG [GossipStage:1] 2019-02-06 23:20:47,392 MigrationManager.java:115 - Not pulling schema because versions match or shouldPullSchemaFrom returned false Here is the updated dtest branch: https://github.com/vinaykumarchella/cassandra-dtest/tree/fix_failing_upgradetest dtests on CircleCI: https://circleci.com/gh/vinaykumarchella/cassandra/345 P.S: While MigrationManager <https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/service/MigrationManager.java#L181-L185> confirms that schema migrations from pre 3.11 are not allowed without `cassandra.force_3_0_protocol_version` option, release notes for 3.11 are confusing - docs <https://github.com/apache/cassandra/blob/cassandra-3.11/NEWS.txt#L173-L174> Let me know if this looks good to you, I will send a patch to cassandra-dtest Thanks, Vinay Chella On Wed, Feb 6, 2019 at 8:07 PM Vinay Chella <vinaykumar...@gmail.com> wrote: > Hi Ariel, > > Sure, I am volunteering to debug this. Will update the progress here. > > Thanks, > Vinay > > > On Wed, Feb 6, 2019 at 1:41 PM Ariel Weisberg <ar...@weisberg.ws> wrote: > >> Hi, >> >> It fails consistently. I don't know why the data is not evenly >> distributed. Can someone volunteer to debug this failing test to make sure >> there isn't an issue with bootstrap in 3.11? >> >> https://circleci.com/gh/aweisberg/cassandra/2593 >> >> Thanks, >> Ariel >> On Wed, Feb 6, 2019, at 3:11 PM, Ariel Weisberg wrote: >> > Hi, >> > >> > -0 >> > >> > bootstrap_upgrade_test.py test_simple_bootstrap_mixed_versions fails >> > because it doesn't see the expected on disk size within 30% of the >> > expected value. It's bootstrapping a new version node and runs cleanup >> > on the existing node. If the data were evenly distributed the on disk >> > size should be similar. >> > >> > https://circleci.com/gh/aweisberg/cassandra/2591#tests/containers/40 >> > >> > I don't have time to see if this reproduces manually. I kicked off the >> > tests again to see if reproduces. >> > https://circleci.com/gh/aweisberg/cassandra/2593 >> > >> > Ariel >> > >> > On Wed, Feb 6, 2019, at 5:02 AM, Marcus Eriksson wrote: >> > > +1 >> > > >> > > Den ons 6 feb. 2019 kl 10:52 skrev Benedict Elliott Smith < >> > > bened...@apache.org>: >> > > >> > > > +1 >> > > > >> > > > > On 6 Feb 2019, at 08:01, Tommy Stendahl < >> tommy.stend...@ericsson.com> >> > > > wrote: >> > > > > >> > > > > +1 (non-binding) >> > > > > >> > > > > /Tommy >> > > > > >> > > > > On lör, 2019-02-02 at 18:31 -0600, Michael Shuler wrote: >> > > > > >> > > > > I propose the following artifacts for release as 3.11.4. >> > > > > >> > > > > sha1: fd47391aae13bcf4ee995abcde1b0e180372d193 >> > > > > Git: >> > > > > >> > > > >> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.11.4-tentative >> > > > > Artifacts: >> > > > > >> > > > >> https://repository.apache.org/content/repositories/orgapachecassandra-1170/org/apache/cassandra/apache-cassandra/3.11.4/ >> > > > > Staging repository: >> > > > > >> > > > >> https://repository.apache.org/content/repositories/orgapachecassandra-1170/ >> > > > > >> > > > > The Debian and RPM packages are available here: >> > > > > http://people.apache.org/~mshuler >> > > > > >> > > > > The vote will be open for 72 hours (longer if needed). >> > > > > >> > > > > [1]: CHANGES.txt: >> > > > > >> > > > >> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.4-tentative >> > > > > [2]: NEWS.txt: >> > > > > >> > > > >> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.4-tentative >> > > > > >> > > > > >> > > > >> > > > >> > > > >> --------------------------------------------------------------------- >> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org >> > > > >> > > > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> > For additional commands, e-mail: dev-h...@cassandra.apache.org >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: dev-h...@cassandra.apache.org >> >>
