[jira] [Comment Edited] (CASSANDRA-11687) dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384487#comment-15384487
]
Jim Witschey edited comment on CASSANDRA-11687 at 7/19/16 4:55 PM:
---
That PR also improved error handling, and here's the error I get running
locally:
{code}
==
ERROR: rebuild_test.py:TestRebuild.simple_rebuild_test
--
Traceback (most recent call last):
File "/home/mambocab/cstar_src/cassandra-dtest/rebuild_test.py", line 96, in
simple_rebuild_test
rebuild()
File "/home/mambocab/cstar_src/cassandra-dtest/rebuild_test.py", line 88, in
rebuild
raise e
NodetoolError: Nodetool command
'/home/mambocab/.ccm/repository/gitCOLONtrunk/bin/nodetool -h localhost -p 7200
rebuild dc1' failed; exit status: 1; stdout: nodetool: Unable to find
sufficient sources for streaming range
(-3287869951390391138,-1624006824486474209] in keyspace system_auth with
RF=1.If you want to ignore this, consider using system property
-Dcassandra.consistent.rangemovement=false.
See 'nodetool help' or 'nodetool help '.
>> begin captured logging <<
ccm: INFO: Fetching Cassandra updates...
dtest: DEBUG: cluster ccm directory: /tmp/dtest-QDgw0_
ccm: INFO: Fetching Cassandra updates...
dtest: DEBUG: Done setting configuration options:
{ 'num_tokens': None, 'phi_convict_threshold': 5, 'start_rpc': 'true'}
cassandra.cluster: INFO: New Cassandra host discovered
cassandra.cluster: WARNING: Host 127.0.0.1 has been marked down
cassandra.pool: INFO: Successful reconnection to 127.0.0.1, marking node up if
it isn't already
cassandra.cluster: INFO: Host 127.0.0.1 may be up; will prepare queries and
open connection pool
cassandra.cluster: INFO: New Cassandra host discovered
- >> end captured logging << -
--
{code}
-So, looks like we're calling {{rebuild}} incorrectly.- This shouldn't fail
this way. The thing to do here, as the error says, may be to set
{{cassandra.consistent.rangemovement}} to {{false}}.
EDIT: for the record, this was the PR:
https://github.com/riptano/cassandra-dtest/pull/1106
was (Author: mambocab):
That PR also improved error handling, and here's the error I get running
locally:
{code}
==
ERROR: rebuild_test.py:TestRebuild.simple_rebuild_test
--
Traceback (most recent call last):
File "/home/mambocab/cstar_src/cassandra-dtest/rebuild_test.py", line 96, in
simple_rebuild_test
rebuild()
File "/home/mambocab/cstar_src/cassandra-dtest/rebuild_test.py", line 88, in
rebuild
raise e
NodetoolError: Nodetool command
'/home/mambocab/.ccm/repository/gitCOLONtrunk/bin/nodetool -h localhost -p 7200
rebuild dc1' failed; exit status: 1; stdout: nodetool: Unable to find
sufficient sources for streaming range
(-3287869951390391138,-1624006824486474209] in keyspace system_auth with
RF=1.If you want to ignore this, consider using system property
-Dcassandra.consistent.rangemovement=false.
See 'nodetool help' or 'nodetool help '.
>> begin captured logging <<
ccm: INFO: Fetching Cassandra updates...
dtest: DEBUG: cluster ccm directory: /tmp/dtest-QDgw0_
ccm: INFO: Fetching Cassandra updates...
dtest: DEBUG: Done setting configuration options:
{ 'num_tokens': None, 'phi_convict_threshold': 5, 'start_rpc': 'true'}
cassandra.cluster: INFO: New Cassandra host discovered
cassandra.cluster: WARNING: Host 127.0.0.1 has been marked down
cassandra.pool: INFO: Successful reconnection to 127.0.0.1, marking node up if
it isn't already
cassandra.cluster: INFO: Host 127.0.0.1 may be up; will prepare queries and
open connection pool
cassandra.cluster: INFO: New Cassandra host discovered
- >> end captured logging << -
--
{code}
So, looks like we're calling {{rebuild}} incorrectly.
EDIT: for the record, this was the PR:
https://github.com/riptano/cassandra-dtest/pull/1106
> dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
> -
>
> Key: CASSANDRA-11687
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11687
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
> Labels: dtest
>
> single failure on most recent run (3.0 no-vnode)
> {noformat}
> concurrent rebuild should not be allowed, but one rebuild command should have
>
[jira] [Comment Edited] (CASSANDRA-11687) dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384487#comment-15384487
]
Jim Witschey edited comment on CASSANDRA-11687 at 7/19/16 4:50 PM:
---
That PR also improved error handling, and here's the error I get running
locally:
{code}
==
ERROR: rebuild_test.py:TestRebuild.simple_rebuild_test
--
Traceback (most recent call last):
File "/home/mambocab/cstar_src/cassandra-dtest/rebuild_test.py", line 96, in
simple_rebuild_test
rebuild()
File "/home/mambocab/cstar_src/cassandra-dtest/rebuild_test.py", line 88, in
rebuild
raise e
NodetoolError: Nodetool command
'/home/mambocab/.ccm/repository/gitCOLONtrunk/bin/nodetool -h localhost -p 7200
rebuild dc1' failed; exit status: 1; stdout: nodetool: Unable to find
sufficient sources for streaming range
(-3287869951390391138,-1624006824486474209] in keyspace system_auth with
RF=1.If you want to ignore this, consider using system property
-Dcassandra.consistent.rangemovement=false.
See 'nodetool help' or 'nodetool help '.
>> begin captured logging <<
ccm: INFO: Fetching Cassandra updates...
dtest: DEBUG: cluster ccm directory: /tmp/dtest-QDgw0_
ccm: INFO: Fetching Cassandra updates...
dtest: DEBUG: Done setting configuration options:
{ 'num_tokens': None, 'phi_convict_threshold': 5, 'start_rpc': 'true'}
cassandra.cluster: INFO: New Cassandra host discovered
cassandra.cluster: WARNING: Host 127.0.0.1 has been marked down
cassandra.pool: INFO: Successful reconnection to 127.0.0.1, marking node up if
it isn't already
cassandra.cluster: INFO: Host 127.0.0.1 may be up; will prepare queries and
open connection pool
cassandra.cluster: INFO: New Cassandra host discovered
- >> end captured logging << -
--
{code}
So, looks like we're calling {{rebuild}} incorrectly.
EDIT: for the record, this was the PR:
https://github.com/riptano/cassandra-dtest/pull/1106
was (Author: mambocab):
That PR also improved error handling, and here's the error I get running
locally:
{code}
==
ERROR: rebuild_test.py:TestRebuild.simple_rebuild_test
--
Traceback (most recent call last):
File "/home/mambocab/cstar_src/cassandra-dtest/rebuild_test.py", line 96, in
simple_rebuild_test
rebuild()
File "/home/mambocab/cstar_src/cassandra-dtest/rebuild_test.py", line 88, in
rebuild
raise e
NodetoolError: Nodetool command
'/home/mambocab/.ccm/repository/gitCOLONtrunk/bin/nodetool -h localhost -p 7200
rebuild dc1' failed; exit status: 1; stdout: nodetool: Unable to find
sufficient sources for streaming range
(-3287869951390391138,-1624006824486474209] in keyspace system_auth with
RF=1.If you want to ignore this, consider using system property
-Dcassandra.consistent.rangemovement=false.
See 'nodetool help' or 'nodetool help '.
>> begin captured logging <<
ccm: INFO: Fetching Cassandra updates...
dtest: DEBUG: cluster ccm directory: /tmp/dtest-QDgw0_
ccm: INFO: Fetching Cassandra updates...
dtest: DEBUG: Done setting configuration options:
{ 'num_tokens': None, 'phi_convict_threshold': 5, 'start_rpc': 'true'}
cassandra.cluster: INFO: New Cassandra host discovered
cassandra.cluster: WARNING: Host 127.0.0.1 has been marked down
cassandra.pool: INFO: Successful reconnection to 127.0.0.1, marking node up if
it isn't already
cassandra.cluster: INFO: Host 127.0.0.1 may be up; will prepare queries and
open connection pool
cassandra.cluster: INFO: New Cassandra host discovered
- >> end captured logging << -
--
{code}
So, looks like we're calling {{rebuild}} incorrectly.
> dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
> -
>
> Key: CASSANDRA-11687
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11687
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
> Labels: dtest
>
> single failure on most recent run (3.0 no-vnode)
> {noformat}
> concurrent rebuild should not be allowed, but one rebuild command should have
> succeeded.
> {noformat}
> http://cassci.datastax.com/job/cassandra-3.0_novnode_dtest/217/testReport/rebuild_test/TestRebuild/simple_rebuild_test
> Failed on CassCI build cassandra-3.0_novnode_dtest #217
--
This message was sent by
[jira] [Comment Edited] (CASSANDRA-11687) dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368434#comment-15368434
]
Jim Witschey edited comment on CASSANDRA-11687 at 7/8/16 8:36 PM:
--
Saw 8 failures, all of which looked like this:
{code}
Error Message
concurrent rebuild should not be allowed, but one rebuild command should have
succeeded.
>> begin captured logging <<
dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-w6QEHl
dtest: DEBUG: Done setting configuration options:
{ 'initial_token': None,
'num_tokens': '32',
'phi_convict_threshold': 5,
'start_rpc': 'true'}
cassandra.cluster: INFO: New Cassandra host discovered
cassandra.cluster: WARNING: Host 127.0.0.1 has been marked down
cassandra.pool: INFO: Successful reconnection to 127.0.0.1, marking node up if
it isn't already
cassandra.cluster: INFO: Host 127.0.0.1 may be up; will prepare queries and
open connection pool
cassandra.cluster: INFO: New Cassandra host discovered
- >> end captured logging << -
Stacktrace
File "/usr/lib/python2.7/unittest/case.py", line 329, in run
testMethod()
File "/home/automaton/cassandra-dtest/rebuild_test.py", line 106, in
simple_rebuild_test
msg='concurrent rebuild should not be allowed, but one rebuild command
should have succeeded.')
File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
raise self.failureException(msg)
"concurrent rebuild should not be allowed, but one rebuild command should have
succeeded.\n >> begin captured logging <<
\ndtest: DEBUG: cluster ccm directory:
/mnt/tmp/dtest-w6QEHl\ndtest: DEBUG: Done setting configuration options:\n{
'initial_token': None,\n'num_tokens': '32',\n'phi_convict_threshold':
5,\n'start_rpc': 'true'}\ncassandra.cluster: INFO: New Cassandra host
discovered\ncassandra.cluster: WARNING: Host 127.0.0.1
has been marked down\ncassandra.pool: INFO: Successful reconnection to
127.0.0.1, marking node up if it isn't already\ncassandra.cluster: INFO: Host
127.0.0.1 may be up; will prepare queries and open connection
pool\ncassandra.cluster: INFO: New Cassandra host
discovered\n- >> end captured logging <<
-"
{code}
EDIT: Ah, interesting -- this is _with_ vnodes, so it looks like it can fail
with or without.
was (Author: mambocab):
Saw 8 failures, all of which looked like this:
{code}
Error Message
concurrent rebuild should not be allowed, but one rebuild command should have
succeeded.
>> begin captured logging <<
dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-w6QEHl
dtest: DEBUG: Done setting configuration options:
{ 'initial_token': None,
'num_tokens': '32',
'phi_convict_threshold': 5,
'start_rpc': 'true'}
cassandra.cluster: INFO: New Cassandra host discovered
cassandra.cluster: WARNING: Host 127.0.0.1 has been marked down
cassandra.pool: INFO: Successful reconnection to 127.0.0.1, marking node up if
it isn't already
cassandra.cluster: INFO: Host 127.0.0.1 may be up; will prepare queries and
open connection pool
cassandra.cluster: INFO: New Cassandra host discovered
- >> end captured logging << -
Stacktrace
File "/usr/lib/python2.7/unittest/case.py", line 329, in run
testMethod()
File "/home/automaton/cassandra-dtest/rebuild_test.py", line 106, in
simple_rebuild_test
msg='concurrent rebuild should not be allowed, but one rebuild command
should have succeeded.')
File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
raise self.failureException(msg)
"concurrent rebuild should not be allowed, but one rebuild command should have
succeeded.\n >> begin captured logging <<
\ndtest: DEBUG: cluster ccm directory:
/mnt/tmp/dtest-w6QEHl\ndtest: DEBUG: Done setting configuration options:\n{
'initial_token': None,\n'num_tokens': '32',\n'phi_convict_threshold':
5,\n'start_rpc': 'true'}\ncassandra.cluster: INFO: New Cassandra host
discovered\ncassandra.cluster: WARNING: Host 127.0.0.1
has been marked down\ncassandra.pool: INFO: Successful reconnection to
127.0.0.1, marking node up if it isn't already\ncassandra.cluster: INFO: Host
127.0.0.1 may be up; will prepare queries and open connection
pool\ncassandra.cluster: INFO: New Cassandra host
discovered\n- >> end captured logging <<
-"
{code}
> dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
>
