[
https://issues.apache.org/jira/browse/CASSANDRA-12171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433120#comment-15433120
]
Russ Hatch edited comment on CASSANDRA-12171 at 8/23/16 4:16 PM:
-----------------------------------------------------------------
Though it's a different test, I wonder if the dtest code fix used on
CASSANDRA-12092 might help out here (same symptom of counters not being at
expected value).
was (Author: rhatch):
Though it's a different test, I wonder if the test code fix used on
CASSANDRA-12092 might help out here (same symptom of counters not being at
expected value).
> counter mismatch during rolling upgrade from 2.2 to 3.0
> -------------------------------------------------------
>
> Key: CASSANDRA-12171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12171
> Project: Cassandra
> Issue Type: Bug
> Reporter: Russ Hatch
> Assignee: Aleksey Yeschenko
>
> This may occur on other versions, but 3.0 is where I observed it recently.
> N=RF=3, counter writes at quorum, reads at quorum.
> This is being seen on some upgrade tests I'm currently repairing here:
> https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix (this
> branch is to resolve an issue where counters were not being properly tested
> during rolling upgrade tests).
> The test runs a continuous counter incrementing process, as well as a
> continuous counter checking process. Once a counter value has been verified,
> the test code makes it eligible to be incremented again.
> The test is encountering the problem when trying to check an expected counter
> value and not matching expectations, for example:
> {noformat}
> Traceback (most recent call last):
> File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
> _bootstrap
> self.run()
> File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
> File
> "/home/rhatch/git/cstar/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 210, in counter_checker
> tester.assertEqual(expected_count, actual_count)
> File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
> assertion_func(first, second, msg=msg)
> File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
> raise self.failureException(msg)
> AssertionError: 1 != 2
> ERROR
> {noformat}
> To check if something else could be going on, I did an experiment where I
> changed the test to not upgrade nodes (just drain, stop, start) and the
> mismatch didn't occur in several attempts. So it appears something about
> upgrading is possibly the culprit.
> To run the test and repro locally:
> {noformat}
> grab my dtest branch at
> https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix
> export UPGRADE_TEST_RUN=true
> nosetests -v
> upgrade_tests/upgrade_through_versions_test.py:TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_test
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
