[jira] [Commented] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
[ https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036582#comment-16036582 ] Kurt Greaves commented on CASSANDRA-13209: -- Yeah not sure why the counts are flaky on ASF infra. I rarely ever got the client request timeouts running on a standalone machine, even with a smaller heap. Anyway, probably best those tests are fixed in their own tickets/in dtests where applicable so may as well finish up with this ticket. Thanks for the review Stefania, and yes I need you to commit. > test failure in > cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections > -- > > Key: CASSANDRA-13209 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13209 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Shuler >Assignee: Kurt Greaves > Labels: dtest, test-failure > Attachments: 13209.patch, node1.log, node2.log, node3.log, node4.log, > node5.log > > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections > {noformat} > Error Message > errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4 > >> begin captured logging << > dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j > dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory > dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > cassandra.policies: INFO: Using datacenter 'datacenter1' for > DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify > a local_dc to the constructor, or limit contact points to local cluster nodes > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > dtest: DEBUG: Running stress with user profile > /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml > - >> end captured logging << - > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped > f(obj) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2571, in test_bulk_round_trip_blogposts_with_max_connections > copy_from_options={'NUMPROCESSES': 2}) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2500, in _test_bulk_round_trip > num_records = create_records() > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2473, in create_records > ret = rows_to_list(self.session.execute(count_statement))[0][0] > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 1998, in execute > return self.execute_async(query, parameters, trace, custom_payload, > timeout, execution_profile, paging_state).result() > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 3784, in result > raise self._final_exception > "errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4\n > >> begin captured logging << \ndtest: DEBUG: cluster ccm > directory: /tmp/dtest-792s6j\ndtest: DEBUG: Done setting configuration > options:\n{ 'initial_token': None,\n'num_tokens': '32',\n > 'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n > 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n > 'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms': > 1}\ndtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j\ndtest: > DEBUG: clearing
[jira] [Commented] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
[ https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036465#comment-16036465 ] Stefania commented on CASSANDRA-13209: -- CI for 2.2 looks good as far as the proposed patch is concerned: only the known failures in cqlshlib and a client request timeout in {{test_bulk_round_trip_blogposts}} when invoking {{SELECT COUNT}} at line 2475, unrelated to this patch. I imagine we would need to reduce the number of records in the bulk tests to fix this sort of problems if they happen on ASF infra, or remove the {{SELECT COUNT}} altogether, if at all possible. So \+1 for the patch in 2.2\+, even though if may not stabilize the bulk tests fully. Do you need me to commit? > test failure in > cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections > -- > > Key: CASSANDRA-13209 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13209 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Shuler >Assignee: Kurt Greaves > Labels: dtest, test-failure > Attachments: 13209.patch, node1.log, node2.log, node3.log, node4.log, > node5.log > > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections > {noformat} > Error Message > errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4 > >> begin captured logging << > dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j > dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory > dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > cassandra.policies: INFO: Using datacenter 'datacenter1' for > DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify > a local_dc to the constructor, or limit contact points to local cluster nodes > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > dtest: DEBUG: Running stress with user profile > /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml > - >> end captured logging << - > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped > f(obj) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2571, in test_bulk_round_trip_blogposts_with_max_connections > copy_from_options={'NUMPROCESSES': 2}) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2500, in _test_bulk_round_trip > num_records = create_records() > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2473, in create_records > ret = rows_to_list(self.session.execute(count_statement))[0][0] > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 1998, in execute > return self.execute_async(query, parameters, trace, custom_payload, > timeout, execution_profile, paging_state).result() > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 3784, in result > raise self._final_exception > "errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4\n > >> begin captured logging << \ndtest: DEBUG: cluster ccm > directory: /tmp/dtest-792s6j\ndtest: DEBUG: Done setting configuration > options:\n{ 'initial_token': None,\n'num_tokens': '32',\n > 'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n > 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms':
[jira] [Commented] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
[ https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034404#comment-16034404 ] Stefania commented on CASSANDRA-13209: -- I've run the entire cqlsh tests internally for 3.0, 3.11 and trunk, no problems found. OK for committing in 2.2+. > test failure in > cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections > -- > > Key: CASSANDRA-13209 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13209 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Shuler >Assignee: Kurt Greaves > Labels: dtest, test-failure > Attachments: 13209.patch, node1.log, node2.log, node3.log, node4.log, > node5.log > > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections > {noformat} > Error Message > errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4 > >> begin captured logging << > dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j > dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory > dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > cassandra.policies: INFO: Using datacenter 'datacenter1' for > DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify > a local_dc to the constructor, or limit contact points to local cluster nodes > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > dtest: DEBUG: Running stress with user profile > /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml > - >> end captured logging << - > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped > f(obj) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2571, in test_bulk_round_trip_blogposts_with_max_connections > copy_from_options={'NUMPROCESSES': 2}) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2500, in _test_bulk_round_trip > num_records = create_records() > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2473, in create_records > ret = rows_to_list(self.session.execute(count_statement))[0][0] > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 1998, in execute > return self.execute_async(query, parameters, trace, custom_payload, > timeout, execution_profile, paging_state).result() > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 3784, in result > raise self._final_exception > "errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4\n > >> begin captured logging << \ndtest: DEBUG: cluster ccm > directory: /tmp/dtest-792s6j\ndtest: DEBUG: Done setting configuration > options:\n{ 'initial_token': None,\n'num_tokens': '32',\n > 'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n > 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n > 'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms': > 1}\ndtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j\ndtest: > DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory\ndtest: DEBUG: > cluster ccm directory: /tmp/dtest-uNMsuW\ndtest: DEBUG: Done setting > configuration options:\n{ 'initial_token': None,\n'num_tokens': '32',\n >'phi_convict_threshold': 5,\n
[jira] [Commented] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
[ https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034261#comment-16034261 ] Kurt Greaves commented on CASSANDRA-13209: -- The whole critical fixes only has always annoyed me as an operator of many clusters of many versions so I'm biased in this regard. I don't see any harm in applying the fix to 2.2 (or 2.1 for that matter). The way I see it is 2.2 will be around for a long time as people won't move to 3, so at the very least we should apply it there. Plus it is a very small patch after all. I'll have a look at extending on {{test_reading_max_insert_errors}}, thanks for the pointer. > test failure in > cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections > -- > > Key: CASSANDRA-13209 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13209 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Shuler >Assignee: Kurt Greaves > Labels: dtest, test-failure > Attachments: 13209.patch, node1.log, node2.log, node3.log, node4.log, > node5.log > > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections > {noformat} > Error Message > errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4 > >> begin captured logging << > dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j > dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory > dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > cassandra.policies: INFO: Using datacenter 'datacenter1' for > DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify > a local_dc to the constructor, or limit contact points to local cluster nodes > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > dtest: DEBUG: Running stress with user profile > /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml > - >> end captured logging << - > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped > f(obj) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2571, in test_bulk_round_trip_blogposts_with_max_connections > copy_from_options={'NUMPROCESSES': 2}) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2500, in _test_bulk_round_trip > num_records = create_records() > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2473, in create_records > ret = rows_to_list(self.session.execute(count_statement))[0][0] > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 1998, in execute > return self.execute_async(query, parameters, trace, custom_payload, > timeout, execution_profile, paging_state).result() > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 3784, in result > raise self._final_exception > "errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4\n > >> begin captured logging << \ndtest: DEBUG: cluster ccm > directory: /tmp/dtest-792s6j\ndtest: DEBUG: Done setting configuration > options:\n{ 'initial_token': None,\n'num_tokens': '32',\n > 'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n > 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n > 'truncate_request_timeout_in_ms': 1,\n
[jira] [Commented] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
[ https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034026#comment-16034026 ] Stefania commented on CASSANDRA-13209: -- bq. Patch simply makes it so we count the total insert errors based on failing all attempts, rather than just one attempt. +1, it is clearly a mistake to count attempts as insert errors. I've applied your patch to [this branch|https://github.com/stef1927/cassandra/tree/13209-3.0] and I'm currently running the full cqlsh tests on our CI system. I will post the results later. Where do you think we should commit this, technically 2.1 is critical fixes only, and this isn't critical. I'm not sure about 2.2 either. I would suggest 3.0+, WDYT? We can set {{MAXINSERTERRORS}} to -1 to stabilize the test on the branches without this fix. bq. It might be nice to have a dtest that does the same blogpost COPY test but with MAXATTEMPTS=1(or more?) and forced failure to test that it is calculating correctly We have [{{test_reading_max_insert_errors}}|https://github.com/stef1927/cassandra-dtest/blob/master/cqlsh_tests/cqlsh_copy_tests.py#L1176] for this. It's currently setting {{MAXATTEMPTS}} to 1, you could extend it to use {{MAXATTEMPTS}} > 1 if you wanted to test that attempts are not counted as insert errors. > test failure in > cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections > -- > > Key: CASSANDRA-13209 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13209 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Shuler >Assignee: Kurt Greaves > Labels: dtest, test-failure > Attachments: 13209.patch, node1.log, node2.log, node3.log, node4.log, > node5.log > > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections > {noformat} > Error Message > errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4 > >> begin captured logging << > dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j > dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory > dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > cassandra.policies: INFO: Using datacenter 'datacenter1' for > DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify > a local_dc to the constructor, or limit contact points to local cluster nodes > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > dtest: DEBUG: Running stress with user profile > /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml > - >> end captured logging << - > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped > f(obj) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2571, in test_bulk_round_trip_blogposts_with_max_connections > copy_from_options={'NUMPROCESSES': 2}) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2500, in _test_bulk_round_trip > num_records = create_records() > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2473, in create_records > ret = rows_to_list(self.session.execute(count_statement))[0][0] > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 1998, in execute > return self.execute_async(query, parameters, trace, custom_payload, > timeout, execution_profile, paging_state).result() > File
[jira] [Commented] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
[ https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024019#comment-16024019 ] Stefania commented on CASSANDRA-13209: -- bq. That's awfully misleading... I'm not sure what you are referring to but, if it is the print statement on the number of attempts you could consider adding a statement when a batch completes with number of attempts > 0, I would keep it at debug level though. bq. It seems to me that this means that if there enough first attempt failures that trigger the log message to add up to over 1000 rows the COPY FROM will stop. Is that correct? Yes. bq. If so it could account for some of the failures but not all, Also correct, if it completed with fewer rows than expected then there is something else we don't understand yet. > test failure in > cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections > -- > > Key: CASSANDRA-13209 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13209 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Shuler >Assignee: Kurt Greaves > Labels: dtest, test-failure > Attachments: node1.log, node2.log, node3.log, node4.log, node5.log > > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections > {noformat} > Error Message > errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4 > >> begin captured logging << > dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j > dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory > dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > cassandra.policies: INFO: Using datacenter 'datacenter1' for > DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify > a local_dc to the constructor, or limit contact points to local cluster nodes > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > dtest: DEBUG: Running stress with user profile > /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml > - >> end captured logging << - > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped > f(obj) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2571, in test_bulk_round_trip_blogposts_with_max_connections > copy_from_options={'NUMPROCESSES': 2}) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2500, in _test_bulk_round_trip > num_records = create_records() > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2473, in create_records > ret = rows_to_list(self.session.execute(count_statement))[0][0] > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 1998, in execute > return self.execute_async(query, parameters, trace, custom_payload, > timeout, execution_profile, paging_state).result() > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 3784, in result > raise self._final_exception > "errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4\n > >> begin captured logging << \ndtest: DEBUG: cluster ccm > directory: /tmp/dtest-792s6j\ndtest: DEBUG: Done setting configuration > options:\n{ 'initial_token': None,\n'num_tokens': '32',\n > 'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n >
[jira] [Commented] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
[ https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022594#comment-16022594 ] Kurt Greaves commented on CASSANDRA-13209: -- Thanks [~Stefania]. That's awfully misleading... It seems to me that this means that if there enough first attempt failures that trigger the log message to add up to over 1000 rows the COPY FROM will stop. Is that correct? If so it could account for _some_ of the failures but not all, as some of the failed jobs don't have the {{Exceeded maximum number of insert errors 1000}} error. > test failure in > cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections > -- > > Key: CASSANDRA-13209 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13209 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Shuler >Assignee: Kurt Greaves > Labels: dtest, test-failure > Attachments: node1.log, node2.log, node3.log, node4.log, node5.log > > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections > {noformat} > Error Message > errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4 > >> begin captured logging << > dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j > dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory > dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > cassandra.policies: INFO: Using datacenter 'datacenter1' for > DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify > a local_dc to the constructor, or limit contact points to local cluster nodes > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > dtest: DEBUG: Running stress with user profile > /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml > - >> end captured logging << - > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped > f(obj) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2571, in test_bulk_round_trip_blogposts_with_max_connections > copy_from_options={'NUMPROCESSES': 2}) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2500, in _test_bulk_round_trip > num_records = create_records() > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2473, in create_records > ret = rows_to_list(self.session.execute(count_statement))[0][0] > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 1998, in execute > return self.execute_async(query, parameters, trace, custom_payload, > timeout, execution_profile, paging_state).result() > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 3784, in result > raise self._final_exception > "errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4\n > >> begin captured logging << \ndtest: DEBUG: cluster ccm > directory: /tmp/dtest-792s6j\ndtest: DEBUG: Done setting configuration > options:\n{ 'initial_token': None,\n'num_tokens': '32',\n > 'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n > 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n > 'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms': > 1}\ndtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j\ndtest: >
[jira] [Commented] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
[ https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019158#comment-16019158 ] Stefania commented on CASSANDRA-13209: -- bq. however it will never go past 1. If it succeeds importing rows in a following attempt, it would not log attempt no. 2. bq. Going to fix the error handling in the COPY FROM command to actually do retries (like COPY TO does), COPY FROM does retry on timeouts, otherwise {{test_bulk_round_trip_with_timeouts}} would always fail. The code that retries is in the [error callback|https://github.com/apache/cassandra/blob/trunk/pylib/cqlshlib/copyutil.py#L2583]. These [two lines|https://github.com/apache/cassandra/blob/trunk/pylib/cqlshlib/copyutil.py#L2579] just above are because of [PYTHON-652|https://datastax-oss.atlassian.net/browse/PYTHON-652], which doesn't seem fixed yet. You may want to try and see if by any chance these lines cause some time outs not to get retried but I doubt it. The reason why COPY TO and COPY FROM have difference retry mechanisms is performance (CASSANDRA-11053). bq. Also once the COPY FROM code in cqlsh gets over 1000 failed rows it exits, This is configurable, see [here|https://github.com/apache/cassandra/blob/trunk/pylib/cqlshlib/copyutil.py#L362]. > test failure in > cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections > -- > > Key: CASSANDRA-13209 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13209 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Shuler >Assignee: Kurt Greaves > Labels: dtest, test-failure > Attachments: node1.log, node2.log, node3.log, node4.log, node5.log > > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections > {noformat} > Error Message > errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4 > >> begin captured logging << > dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j > dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory > dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > cassandra.policies: INFO: Using datacenter 'datacenter1' for > DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify > a local_dc to the constructor, or limit contact points to local cluster nodes > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > dtest: DEBUG: Running stress with user profile > /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml > - >> end captured logging << - > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped > f(obj) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2571, in test_bulk_round_trip_blogposts_with_max_connections > copy_from_options={'NUMPROCESSES': 2}) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2500, in _test_bulk_round_trip > num_records = create_records() > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2473, in create_records > ret = rows_to_list(self.session.execute(count_statement))[0][0] > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 1998, in execute > return self.execute_async(query, parameters, trace, custom_payload, > timeout, execution_profile, paging_state).result() > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 3784, in result >
[jira] [Commented] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
[ https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016784#comment-16016784 ] Kurt Greaves commented on CASSANDRA-13209: -- actually it's always less, and it appears the majority of the issues come from the COPY FROM. Not every test failure seems to be caused by the same thing, but the majority appear to be because when timeouts occur in the COPY FROM they don't actually get retried, so some rows don't get written. Also once the COPY FROM code in cqlsh gets over 1000 failed rows it exits, which kind of explains why a lot of the failures are because of a difference of a 1000 rows. Corresponding error is the following, note it mentions the # of attempts, however it will never go past 1. {code}:2:Failed to import 9 rows: OperationTimedOut - errors={'127.0.0.3': 'Client request timeout. See Session.execute[_async](timeout)'}, last_host=127.0.0.3, will retry later, attempt 1 of 5{code} Going to fix the error handling in the COPY FROM command to actually do retries (like COPY TO does), that should make the tests less flaky, as COPY TO also suffers from the timeouts but effectively retries and thus rarely has issues. There is still the underlying problem of why COPY is timing out in the first place, but to be honest I'd put it down to the command and nodes simply using too much resources on the servers. If it's still very flaky after fixing the retries we can look into performance issues more. > test failure in > cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections > -- > > Key: CASSANDRA-13209 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13209 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Shuler >Assignee: Kurt Greaves > Labels: dtest, test-failure > Attachments: node1.log, node2.log, node3.log, node4.log, node5.log > > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections > {noformat} > Error Message > errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4 > >> begin captured logging << > dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j > dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory > dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > cassandra.policies: INFO: Using datacenter 'datacenter1' for > DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify > a local_dc to the constructor, or limit contact points to local cluster nodes > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > dtest: DEBUG: Running stress with user profile > /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml > - >> end captured logging << - > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped > f(obj) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2571, in test_bulk_round_trip_blogposts_with_max_connections > copy_from_options={'NUMPROCESSES': 2}) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2500, in _test_bulk_round_trip > num_records = create_records() > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2473, in create_records > ret = rows_to_list(self.session.execute(count_statement))[0][0] > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 1998, in execute > return
[jira] [Commented] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
[ https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011969#comment-16011969 ] Kurt Greaves commented on CASSANDRA-13209: -- although timeouts do appear in the failures, most of them seem to be occurring because the CSV's don't have the expected number of lines. sometimes it's more, sometimes less. A better example is https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-dtest/134/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections/ seems that the blogposts tests have previously been an issue. https://issues.apache.org/jira/browse/CASSANDRA-10938 > test failure in > cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections > -- > > Key: CASSANDRA-13209 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13209 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Shuler >Assignee: Kurt Greaves > Labels: dtest, test-failure > Attachments: node1.log, node2.log, node3.log, node4.log, node5.log > > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections > {noformat} > Error Message > errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4 > >> begin captured logging << > dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j > dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory > dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW > dtest: DEBUG: Done setting configuration options: > { 'initial_token': None, > 'num_tokens': '32', > 'phi_convict_threshold': 5, > 'range_request_timeout_in_ms': 1, > 'read_request_timeout_in_ms': 1, > 'request_timeout_in_ms': 1, > 'truncate_request_timeout_in_ms': 1, > 'write_request_timeout_in_ms': 1} > cassandra.policies: INFO: Using datacenter 'datacenter1' for > DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify > a local_dc to the constructor, or limit contact points to local cluster nodes > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > cassandra.cluster: INFO: New Cassandra host > discovered > dtest: DEBUG: Running stress with user profile > /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml > - >> end captured logging << - > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped > f(obj) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2571, in test_bulk_round_trip_blogposts_with_max_connections > copy_from_options={'NUMPROCESSES': 2}) > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2500, in _test_bulk_round_trip > num_records = create_records() > File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", > line 2473, in create_records > ret = rows_to_list(self.session.execute(count_statement))[0][0] > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 1998, in execute > return self.execute_async(query, parameters, trace, custom_payload, > timeout, execution_profile, paging_state).result() > File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line > 3784, in result > raise self._final_exception > "errors={'127.0.0.4': 'Client request timeout. See > Session.execute[_async](timeout)'}, last_host=127.0.0.4\n > >> begin captured logging << \ndtest: DEBUG: cluster ccm > directory: /tmp/dtest-792s6j\ndtest: DEBUG: Done setting configuration > options:\n{ 'initial_token': None,\n'num_tokens': '32',\n > 'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n > 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n > 'truncate_request_timeout_in_ms': 1,\n