[jira] [Updated] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections

2017-06-05 Thread Stefania (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-13209:
-
   Resolution: Fixed
Fix Version/s: 3.11.0
   3.0.14
   2.2.10
   Status: Resolved  (was: Ready to Commit)

Committed to 2.2 as 
[5807ec|https://github.com/apache/cassandra/commit/5807eca8bbb30091b632e031441939c763b3e054]
 and merged upwards.

Thank you for the patch!

> test failure in 
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
> --
>
> Key: CASSANDRA-13209
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13209
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Michael Shuler
>Assignee: Kurt Greaves
>  Labels: dtest, test-failure
> Fix For: 2.2.10, 3.0.14, 3.11.0
>
> Attachments: 13209.patch, node1.log, node2.log, node3.log, node4.log, 
> node5.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections
> {noformat}
> Error Message
> errors={'127.0.0.4': 'Client request timeout. See 
> Session.execute[_async](timeout)'}, last_host=127.0.0.4
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j
> dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> cassandra.policies: INFO: Using datacenter 'datacenter1' for 
> DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify 
> a local_dc to the constructor, or limit contact points to local cluster nodes
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> dtest: DEBUG: Running stress with user profile 
> /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml
> - >> end captured logging << -
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped
> f(obj)
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2571, in test_bulk_round_trip_blogposts_with_max_connections
> copy_from_options={'NUMPROCESSES': 2})
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2500, in _test_bulk_round_trip
> num_records = create_records()
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2473, in create_records
> ret = rows_to_list(self.session.execute(count_statement))[0][0]
>   File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 
> 1998, in execute
> return self.execute_async(query, parameters, trace, custom_payload, 
> timeout, execution_profile, paging_state).result()
>   File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 
> 3784, in result
> raise self._final_exception
> "errors={'127.0.0.4': 'Client request timeout. See 
> Session.execute[_async](timeout)'}, last_host=127.0.0.4\n 
> >> begin captured logging << \ndtest: DEBUG: cluster ccm 
> directory: /tmp/dtest-792s6j\ndtest: DEBUG: Done setting configuration 
> options:\n{   'initial_token': None,\n'num_tokens': '32',\n
> 'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n
> 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n   
>  'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms': 
> 1}\ndtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j\ndtest: 
> DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] 

[jira] [Updated] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections

2017-06-04 Thread Stefania (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-13209:
-
Status: Ready to Commit  (was: Patch Available)

> test failure in 
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
> --
>
> Key: CASSANDRA-13209
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13209
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Michael Shuler
>Assignee: Kurt Greaves
>  Labels: dtest, test-failure
> Attachments: 13209.patch, node1.log, node2.log, node3.log, node4.log, 
> node5.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections
> {noformat}
> Error Message
> errors={'127.0.0.4': 'Client request timeout. See 
> Session.execute[_async](timeout)'}, last_host=127.0.0.4
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j
> dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> cassandra.policies: INFO: Using datacenter 'datacenter1' for 
> DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify 
> a local_dc to the constructor, or limit contact points to local cluster nodes
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> dtest: DEBUG: Running stress with user profile 
> /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml
> - >> end captured logging << -
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped
> f(obj)
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2571, in test_bulk_round_trip_blogposts_with_max_connections
> copy_from_options={'NUMPROCESSES': 2})
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2500, in _test_bulk_round_trip
> num_records = create_records()
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2473, in create_records
> ret = rows_to_list(self.session.execute(count_statement))[0][0]
>   File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 
> 1998, in execute
> return self.execute_async(query, parameters, trace, custom_payload, 
> timeout, execution_profile, paging_state).result()
>   File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 
> 3784, in result
> raise self._final_exception
> "errors={'127.0.0.4': 'Client request timeout. See 
> Session.execute[_async](timeout)'}, last_host=127.0.0.4\n 
> >> begin captured logging << \ndtest: DEBUG: cluster ccm 
> directory: /tmp/dtest-792s6j\ndtest: DEBUG: Done setting configuration 
> options:\n{   'initial_token': None,\n'num_tokens': '32',\n
> 'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n
> 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n   
>  'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms': 
> 1}\ndtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j\ndtest: 
> DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory\ndtest: DEBUG: 
> cluster ccm directory: /tmp/dtest-uNMsuW\ndtest: DEBUG: Done setting 
> configuration options:\n{   'initial_token': None,\n'num_tokens': '32',\n 
>'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n   
>  'read_request_timeout_in_ms': 1,\n

[jira] [Updated] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections

2017-06-04 Thread Stefania (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-13209:
-
Reviewer: Stefania

> test failure in 
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
> --
>
> Key: CASSANDRA-13209
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13209
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Michael Shuler
>Assignee: Kurt Greaves
>  Labels: dtest, test-failure
> Attachments: 13209.patch, node1.log, node2.log, node3.log, node4.log, 
> node5.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections
> {noformat}
> Error Message
> errors={'127.0.0.4': 'Client request timeout. See 
> Session.execute[_async](timeout)'}, last_host=127.0.0.4
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j
> dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> cassandra.policies: INFO: Using datacenter 'datacenter1' for 
> DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify 
> a local_dc to the constructor, or limit contact points to local cluster nodes
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> dtest: DEBUG: Running stress with user profile 
> /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml
> - >> end captured logging << -
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped
> f(obj)
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2571, in test_bulk_round_trip_blogposts_with_max_connections
> copy_from_options={'NUMPROCESSES': 2})
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2500, in _test_bulk_round_trip
> num_records = create_records()
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2473, in create_records
> ret = rows_to_list(self.session.execute(count_statement))[0][0]
>   File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 
> 1998, in execute
> return self.execute_async(query, parameters, trace, custom_payload, 
> timeout, execution_profile, paging_state).result()
>   File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 
> 3784, in result
> raise self._final_exception
> "errors={'127.0.0.4': 'Client request timeout. See 
> Session.execute[_async](timeout)'}, last_host=127.0.0.4\n 
> >> begin captured logging << \ndtest: DEBUG: cluster ccm 
> directory: /tmp/dtest-792s6j\ndtest: DEBUG: Done setting configuration 
> options:\n{   'initial_token': None,\n'num_tokens': '32',\n
> 'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n
> 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n   
>  'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms': 
> 1}\ndtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j\ndtest: 
> DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory\ndtest: DEBUG: 
> cluster ccm directory: /tmp/dtest-uNMsuW\ndtest: DEBUG: Done setting 
> configuration options:\n{   'initial_token': None,\n'num_tokens': '32',\n 
>'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n   
>  'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n  
>   

[jira] [Updated] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections

2017-06-01 Thread Kurt Greaves (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Greaves updated CASSANDRA-13209:
-
Attachment: 13209.patch

> test failure in 
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
> --
>
> Key: CASSANDRA-13209
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13209
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Michael Shuler
>Assignee: Kurt Greaves
>  Labels: dtest, test-failure
> Attachments: 13209.patch, node1.log, node2.log, node3.log, node4.log, 
> node5.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections
> {noformat}
> Error Message
> errors={'127.0.0.4': 'Client request timeout. See 
> Session.execute[_async](timeout)'}, last_host=127.0.0.4
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j
> dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> cassandra.policies: INFO: Using datacenter 'datacenter1' for 
> DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify 
> a local_dc to the constructor, or limit contact points to local cluster nodes
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> dtest: DEBUG: Running stress with user profile 
> /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml
> - >> end captured logging << -
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped
> f(obj)
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2571, in test_bulk_round_trip_blogposts_with_max_connections
> copy_from_options={'NUMPROCESSES': 2})
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2500, in _test_bulk_round_trip
> num_records = create_records()
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2473, in create_records
> ret = rows_to_list(self.session.execute(count_statement))[0][0]
>   File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 
> 1998, in execute
> return self.execute_async(query, parameters, trace, custom_payload, 
> timeout, execution_profile, paging_state).result()
>   File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 
> 3784, in result
> raise self._final_exception
> "errors={'127.0.0.4': 'Client request timeout. See 
> Session.execute[_async](timeout)'}, last_host=127.0.0.4\n 
> >> begin captured logging << \ndtest: DEBUG: cluster ccm 
> directory: /tmp/dtest-792s6j\ndtest: DEBUG: Done setting configuration 
> options:\n{   'initial_token': None,\n'num_tokens': '32',\n
> 'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n
> 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n   
>  'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms': 
> 1}\ndtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j\ndtest: 
> DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory\ndtest: DEBUG: 
> cluster ccm directory: /tmp/dtest-uNMsuW\ndtest: DEBUG: Done setting 
> configuration options:\n{   'initial_token': None,\n'num_tokens': '32',\n 
>'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 1,\n   
>  'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n 

[jira] [Updated] (CASSANDRA-13209) test failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections

2017-06-01 Thread Kurt Greaves (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Greaves updated CASSANDRA-13209:
-
Status: Patch Available  (was: Open)

{quote}I'm not sure what you are referring to but, if it is the print statement 
on the number of attempts you could consider adding a statement when a batch 
completes with number of attempts > 0, I would keep it at debug level though.
{quote}
One could say that the bug made it misleading. Now that I'm more familiar with 
the error handling between the tasks and the processes it makes sense, however 
the fact that insert errors were counted for every attempt made it misleading.

So I think the attached 1 line patch is the way to go, and from my testing it 
seems to make the test a lot more stable. Occasionally still get client read 
timeout errors early on which break the test, but this appears to be a resource 
issue, and doesn't happen when I run the test on a machine more powerful than 
my laptop.

Patch simply makes it so we count the total insert errors based on failing all 
attempts, rather than just one attempt. It might be nice to have a dtest that 
does the same blogpost COPY test but with MAXATTEMPTS=1(or more?) and forced 
failure to test that it is calculating correctly, but I'll only go down that 
path if people agree this makes sense.

The alternative is to increase the dtest's MAXINSERTERRORS but personally I 
don't think it makes sense to count failed attempts as errors.

> test failure in 
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_bulk_round_trip_blogposts_with_max_connections
> --
>
> Key: CASSANDRA-13209
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13209
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Michael Shuler
>Assignee: Kurt Greaves
>  Labels: dtest, test-failure
> Attachments: 13209.patch, node1.log, node2.log, node3.log, node4.log, 
> node5.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/528/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts_with_max_connections
> {noformat}
> Error Message
> errors={'127.0.0.4': 'Client request timeout. See 
> Session.execute[_async](timeout)'}, last_host=127.0.0.4
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-792s6j
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-792s6j
> dtest: DEBUG: clearing ssl stores from [/tmp/dtest-792s6j] directory
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-uNMsuW
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> cassandra.policies: INFO: Using datacenter 'datacenter1' for 
> DCAwareRoundRobinPolicy (via host '127.0.0.1'); if incorrect, please specify 
> a local_dc to the constructor, or limit contact points to local cluster nodes
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> cassandra.cluster: INFO: New Cassandra host  
> discovered
> dtest: DEBUG: Running stress with user profile 
> /home/automaton/cassandra-dtest/cqlsh_tests/blogposts.yaml
> - >> end captured logging << -
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/dtest.py", line 1090, in wrapped
> f(obj)
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2571, in test_bulk_round_trip_blogposts_with_max_connections
> copy_from_options={'NUMPROCESSES': 2})
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2500, in _test_bulk_round_trip
> num_records = create_records()
>   File "/home/automaton/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py", 
> line 2473, in create_records
> ret = rows_to_list(self.session.execute(count_statement))[0][0]
>   File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 
> 1998, in execute
>