Russ Hatch created CASSANDRA-12171:
--------------------------------------
Summary: counter mismatch during rolling upgrade from 2.2 to 3.0
Key: CASSANDRA-12171
URL: https://issues.apache.org/jira/browse/CASSANDRA-12171
Project: Cassandra
Issue Type: Bug
Reporter: Russ Hatch
Assignee: Aleksey Yeschenko
This may occur on other versions, but 3.0 is where I observed it recently.
N=RF=3, counter writes at quorum, reads at quorum.
This is being seen on some upgrade tests I'm currently repairing here:
https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix (this
branch is to resolve an issue where counters were not being properly tested
during rolling upgrade tests).
The test runs a continuous counter incrementing process, as well as a
continuous counter checking process. Once a counter value has been verified,
the test code makes it eligible to be incremented again.
The test is encountering the problem when trying to check an expected counter
value and not matching expectations, for example:
{noformat}
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File
"/home/rhatch/git/cstar/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
line 210, in counter_checker
tester.assertEqual(expected_count, actual_count)
File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
raise self.failureException(msg)
AssertionError: 1 != 2
ERROR
{noformat}
To check if something else could be going on, I did an experiment where I
changed the test to not upgrade nodes (just drain, stop, start) and the
mismatch didn't occur in several attempts. So it appears something about
upgrading is possibly the culprit.
To run the test and repro locally:
{noformat}
grab my dtest branch at
https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix
export UPGRADE_TEST_RUN=true
nosetests -v
upgrade_tests/upgrade_through_versions_test.py:TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_test
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
