[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435656#comment-15435656
]
Yuki Morishita commented on CASSANDRA-12192:
+1
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
>Assignee: Tyler Hobbs
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435645#comment-15435645
]
Tyler Hobbs commented on CASSANDRA-12192:
-
The dtest run was good. I've merged in trunk and restarted the testall runs.
Assuming those look good as well, are you +1 on this committing [~yukim]?
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
>Assignee: Tyler Hobbs
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424962#comment-15424962
]
Tyler Hobbs commented on CASSANDRA-12192:
-
bq. One nit: can you modify the comment here to fit with your change?
I've pushed a second commit to update this, thanks.
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
>Assignee: Tyler Hobbs
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424933#comment-15424933
]
Yuki Morishita commented on CASSANDRA-12192:
I don't know the reason behind CASSANDRA-5393 not retrying for all (maybe
because the scope of ticket was just for repair), but it looks like we can turn
on retry to all messages as Tyler suggests. On trunk, we try to reconnect when
writing to socket fails, and if we cannot reconnect we clear backlog so
retrying message won't accumulate and be problematic.
One nit: can you modify the comment
[here|https://github.com/thobbs/cassandra/blob/6c46045ec5e22437b5493ee66851623b0bda1209/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L314-L315]
to fit with your change?
Let's see how tests go.
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
>Assignee: Tyler Hobbs
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419578#comment-15419578
]
Tyler Hobbs commented on CASSANDRA-12192:
-
Here's basically what's happening:
The 3.x node tries to add an index while the cluster is half-upgraded. This
basically triggers the symptoms from CASSANDRA-12236 (a deserialization error
on the 3.0.x node). The 3.0.x node closes its end of the connection, but of
course the 3.x node won't notice this until it tries to write on that
connection again. The truncate is the next thing that happens. All nodes
truncate successfully, but when the 3.x node tries to write its response, it
gets a broken pipe error. The truncate then times out on the coordinator.
I've opened a [dtest PR|https://github.com/riptano/cassandra-dtest/pull/1229]
to remove the unnecessary schema change from the test. However, there is one
improvement we could make in C* to make these sorts of errors (which especially
tend to happen during upgrades) less problematic. We already have a mechanism
for retrying messages once after reconnecting. This is currently reserved for
non-droppable verbs (like schema change messages, repair messages, etc). It
seems like retrying _all_ verbs once after reconnecting would be reasonable.
This approach would also fix the test. I'm not sure why non-droppable messages
were omitted from this behavior in CASSANDRA-5393 (where the retry mech was
created), so there might be some potential issues I'm unaware of.
Can anybody offer a reason why we shouldn't retry _all_ verbs after
reconnecting?
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
>Assignee: Tyler Hobbs
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418039#comment-15418039
]
Tyler Hobbs commented on CASSANDRA-12192:
-
It looks like the truncate _does_ happen successfully on all nodes, but the
response from the upgraded node to the old node doesn't get through. This
appears to be caused by this schema change:
{noformat}
if is_upgraded:
if self.get_node_version(is_upgraded) >= '3.0':
cursor.execute("CREATE INDEX ON test(m)")
else:
assert_invalid(cursor, "CREATE INDEX on test(m)")
{noformat}
When that schema change is removed, all of the truncates complete successfully.
I'm guessing what happens is that the schema change in a mixed-version cluster
puts the old node in a state where it doesn't correctly handle the truncate
message from the upgraded node. I'll look into this more, but it does seem
like this is a case where we should adjust the test to not perform an
unsupported operation (schema change in a mixed cluster) and expect everything
to work correctly afterwards.
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
>Assignee: Tyler Hobbs
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418023#comment-15418023
]
Tyler Hobbs commented on CASSANDRA-12192:
-
I'm looking into this since Sam's on vacation for another week and a half.
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
>Assignee: Tyler Hobbs
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384507#comment-15384507
]
Philip Thompson commented on CASSANDRA-12192:
-
This could just be CASSANDRA-12236
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
>Assignee: Sam Tunnicliffe
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382687#comment-15382687
]
Philip Thompson commented on CASSANDRA-12192:
-
I see node1 and node2 log "blocking truncate operation", but not node3. I do
see node3 "discard sstable data for truncate" though, so I'm not really sure
what is supposed to be happening on trunk here, and I think I need a dev to
take a look.
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382675#comment-15382675
]
Philip Thompson commented on CASSANDRA-12192:
-
Okay, so then disregard my copy from the logs. This error is already ignored,
meaning if it isn't causing the truncate to fail, there is a different problem.
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382581#comment-15382581
]
Russ Hatch commented on CASSANDRA-12192:
That is to say we should set the class up with an ignore pattern for this.
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382580#comment-15382580
]
Russ Hatch commented on CASSANDRA-12192:
>From what I understand this one is safe to ignore. See CASSANDRA-12026.
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382549#comment-15382549
]
Philip Thompson commented on CASSANDRA-12192:
-
So, I have the following error in the logs of nodes2 and 3:
{code}
ERROR [MessagingService-Incoming-/127.0.0.1] 2016-07-13 05:23:15,952
CassandraDaemon.java:201 - Exception in thread
Thread[MessagingService-Incoming-/127.0.0.1,5,main]
java.lang.RuntimeException: Unknown column cdc during deserialization
at
org.apache.cassandra.db.Columns$Serializer.deserialize(Columns.java:432)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.db.SerializationHeader$Serializer.deserializeForMessaging(SerializationHeader.java:427)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.deserializeHeader(UnfilteredRowIteratorSerializer.java:190)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:657)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:645)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:344)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:353)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:609)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:592)
~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
~[apache-cassandra-3.0.8.jar:3.0.8]
DEBUG [GossipStage:1] 2016-07-13 05:23:16,855 MigrationManager.java:101 -
Submitting migration task for /127.0.0.1
ERROR [MessagingService-Incoming-/127.0.0.1] 2016-07-13 05:23:17,383
CassandraDaemon.java:201 - Exception in thread
Thread[MessagingService-Incoming-/127.0.0.1,5,main]
java.lang.RuntimeException: Unknown column cdc during deserialization
at
org.apache.cassandra.db.Columns$Serializer.deserialize(Columns.java:432)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.db.SerializationHeader$Serializer.deserializeForMessaging(SerializationHeader.java:427)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.deserializeHeader(UnfilteredRowIteratorSerializer.java:190)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:657)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:645)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:344)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:353)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:609)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:592)
~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
~[apache-cassandra-3.0.8.jar:3.0.8]
DEBUG [GossipStage:1] 2016-07-13 05:23:17,858 StorageService.java:1899 - Node
/127.0.0.1 state NORMAL, token [-1378122049260494817, -1491944349266034000,
-1622944791425543487, -2011387665415943034, -2604861794090361113,
-2623806159584154975, -2661418387004022901, -3056348922832743393,
-3634522926981587558,
