[jira] [Commented] (CASSANDRA-12196) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412276#comment-15412276 ] Philip Thompson commented on CASSANDRA-12196: - I bumped up that timeout I mentioned, and am running here: http://cassci.datastax.com/view/trunk/job/trunk_large_dtest/21/ > dtest failure in > upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test > -- > > Key: CASSANDRA-12196 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12196 > Project: Cassandra > Issue Type: Test >Reporter: Sean McCarthy >Assignee: DS Test Eng > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log, > node4.log, node4_debug.log, node4_gc.log > > > example failure: > http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_1_x_To_indev_3_x/bootstrap_test > Failed on CassCI build upgrade_tests-all #59 > {code} > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 707, in bootstrap_test > self.upgrade_scenario(after_upgrade_call=(self._bootstrap_new_node,)) > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 383, in upgrade_scenario > call() > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 688, in _bootstrap_new_node > nnode.start(use_jna=True, wait_other_notice=True, > wait_for_binary_proto=True) > File "/home/automaton/ccm/ccmlib/node.py", line 634, in start > node.watch_log_for_alive(self, from_mark=mark) > File "/home/automaton/ccm/ccmlib/node.py", line 481, in watch_log_for_alive > self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout, > filename=filename) > File "/home/automaton/ccm/ccmlib/node.py", line 449, in watch_log_for > raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + " > [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" + > reads[:50] + ".\nSee {} for remainder".format(filename)) > "13 Jul 2016 02:23:05 [node2] Missing: ['127.0.0.4.* now UP']:\nINFO > [HANDSHAKE-/127.0.0.4] 2016-07-13 02:21:00,2.\nSee system.log for > remainder > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12196) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412087#comment-15412087 ] Philip Thompson commented on CASSANDRA-12196: - So, this failure is still occurring over in large dtest > dtest failure in > upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test > -- > > Key: CASSANDRA-12196 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12196 > Project: Cassandra > Issue Type: Test >Reporter: Sean McCarthy >Assignee: DS Test Eng > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log, > node4.log, node4_debug.log, node4_gc.log > > > example failure: > http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_1_x_To_indev_3_x/bootstrap_test > Failed on CassCI build upgrade_tests-all #59 > {code} > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 707, in bootstrap_test > self.upgrade_scenario(after_upgrade_call=(self._bootstrap_new_node,)) > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 383, in upgrade_scenario > call() > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 688, in _bootstrap_new_node > nnode.start(use_jna=True, wait_other_notice=True, > wait_for_binary_proto=True) > File "/home/automaton/ccm/ccmlib/node.py", line 634, in start > node.watch_log_for_alive(self, from_mark=mark) > File "/home/automaton/ccm/ccmlib/node.py", line 481, in watch_log_for_alive > self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout, > filename=filename) > File "/home/automaton/ccm/ccmlib/node.py", line 449, in watch_log_for > raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + " > [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" + > reads[:50] + ".\nSee {} for remainder".format(filename)) > "13 Jul 2016 02:23:05 [node2] Missing: ['127.0.0.4.* now UP']:\nINFO > [HANDSHAKE-/127.0.0.4] 2016-07-13 02:21:00,2.\nSee system.log for > remainder > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12196) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402800#comment-15402800 ] Russ Hatch commented on CASSANDRA-12196: We should re-check this failure against the dtest large job, now that the upgrade tests have moved over there. Job here: http://cassci.datastax.com/view/trunk/job/trunk_large_dtest/ > dtest failure in > upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test > -- > > Key: CASSANDRA-12196 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12196 > Project: Cassandra > Issue Type: Test >Reporter: Sean McCarthy >Assignee: DS Test Eng > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log, > node4.log, node4_debug.log, node4_gc.log > > > example failure: > http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_1_x_To_indev_3_x/bootstrap_test > Failed on CassCI build upgrade_tests-all #59 > {code} > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 707, in bootstrap_test > self.upgrade_scenario(after_upgrade_call=(self._bootstrap_new_node,)) > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 383, in upgrade_scenario > call() > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 688, in _bootstrap_new_node > nnode.start(use_jna=True, wait_other_notice=True, > wait_for_binary_proto=True) > File "/home/automaton/ccm/ccmlib/node.py", line 634, in start > node.watch_log_for_alive(self, from_mark=mark) > File "/home/automaton/ccm/ccmlib/node.py", line 481, in watch_log_for_alive > self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout, > filename=filename) > File "/home/automaton/ccm/ccmlib/node.py", line 449, in watch_log_for > raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + " > [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" + > reads[:50] + ".\nSee {} for remainder".format(filename)) > "13 Jul 2016 02:23:05 [node2] Missing: ['127.0.0.4.* now UP']:\nINFO > [HANDSHAKE-/127.0.0.4] 2016-07-13 02:21:00,2.\nSee system.log for > remainder > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12196) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382545#comment-15382545 ] Philip Thompson commented on CASSANDRA-12196: - The node comes up. Do we just need to wait longer for the rest of the cluster to recognize it? [~rhatch], what do you think of increasing the wait_other_notice timeout on the upgrade tests? > dtest failure in > upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test > -- > > Key: CASSANDRA-12196 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12196 > Project: Cassandra > Issue Type: Test >Reporter: Sean McCarthy >Assignee: DS Test Eng > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log, > node4.log, node4_debug.log, node4_gc.log > > > example failure: > http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_1_x_To_indev_3_x/bootstrap_test > Failed on CassCI build upgrade_tests-all #59 > {code} > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 707, in bootstrap_test > self.upgrade_scenario(after_upgrade_call=(self._bootstrap_new_node,)) > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 383, in upgrade_scenario > call() > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 688, in _bootstrap_new_node > nnode.start(use_jna=True, wait_other_notice=True, > wait_for_binary_proto=True) > File "/home/automaton/ccm/ccmlib/node.py", line 634, in start > node.watch_log_for_alive(self, from_mark=mark) > File "/home/automaton/ccm/ccmlib/node.py", line 481, in watch_log_for_alive > self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout, > filename=filename) > File "/home/automaton/ccm/ccmlib/node.py", line 449, in watch_log_for > raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + " > [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" + > reads[:50] + ".\nSee {} for remainder".format(filename)) > "13 Jul 2016 02:23:05 [node2] Missing: ['127.0.0.4.* now UP']:\nINFO > [HANDSHAKE-/127.0.0.4] 2016-07-13 02:21:00,2.\nSee system.log for > remainder > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)