subject:"Re\: Rebuild failing when adding new datacenter \(3.0.8\)"

Re: Rebuild failing when adding new datacenter (3.0.8)

2016-09-22 Thread Timo Ahokas

Hi Yabin/Alain,

I changed the replication strategies for system_distributed, system_auth
and system_traces to use NetworkTopologyStrategies and repaired the
affected keyspaces. Now the rebuild process starts up ok without errors.

Thanks a lot for your help!

Best regards,
Timo

On 22 September 2016 at 21:16, Yabin Meng  wrote:

> It is a Cassandra bug. The workaround is to change system_distributed
> keyspce replication strategy to something as below:
>
>   alter keyspace  system_distributed with replication = {'class':
> 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3', 'DC3': '3'};
>
> You may see similar problem for other system keyspaces. Do the same thing.
>
> Cheers,
>
> Yabin
>
> On Thu, Sep 22, 2016 at 1:44 PM, Timo Ahokas 
> wrote:
>
>> Hi Alain,
>>
>> Our normal user keyspaces have RF3 in all DCs, e.g:
>>
>> create keyspace reporting with replication = {'class':
>> 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3', 'DC3': '3'};
>>
>> Any idea would it be safe to change the system_distributed keyspace to
>> match this?
>>
>> -Timo
>>
>> On 22 September 2016 at 19:23, Timo Ahokas  wrote:
>>
>>> Hi Alain,
>>>
>>> Thanks a lot for a helping out!
>>>
>>> Some of the basic keyspace / cluster info you requested:
>>>
>>> # echo "DESCRIBE KEYSPACE system_distributed;" | cqlsh
>>>
>>> CREATE KEYSPACE system_distributed WITH replication = {'class':
>>> 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;
>>>
>>> CREATE TABLE system_distributed.repair_history (
>>>
>>>keyspace_name text,
>>>
>>>columnfamily_name text,
>>>
>>>id timeuuid,
>>>
>>>coordinator inet,
>>>
>>>exception_message text,
>>>
>>>exception_stacktrace text,
>>>
>>>finished_at timestamp,
>>>
>>>parent_id timeuuid,
>>>
>>>participants set,
>>>
>>>range_begin text,
>>>
>>>range_end text,
>>>
>>>started_at timestamp,
>>>
>>>status text,
>>>
>>>PRIMARY KEY ((keyspace_name, columnfamily_name), id)
>>>
>>> ) WITH CLUSTERING ORDER BY (id ASC)
>>>
>>>AND bloom_filter_fp_chance = 0.01
>>>
>>>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>>
>>>AND comment = 'Repair history'
>>>
>>>AND compaction = {'class': 'org.apache.cassandra.db.compa
>>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>>> 'min_threshold': '4'}
>>>
>>>AND compression = {'chunk_length_in_kb': '64', 'class': '
>>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>>
>>>AND crc_check_chance = 1.0
>>>
>>>AND dclocal_read_repair_chance = 0.0
>>>
>>>AND default_time_to_live = 0
>>>
>>>AND gc_grace_seconds = 0
>>>
>>>AND max_index_interval = 2048
>>>
>>>AND memtable_flush_period_in_ms = 360
>>>
>>>AND min_index_interval = 128
>>>
>>>AND read_repair_chance = 0.0
>>>
>>>AND speculative_retry = '99PERCENTILE';
>>>
>>> CREATE TABLE system_distributed.parent_repair_history (
>>>
>>>parent_id timeuuid PRIMARY KEY,
>>>
>>>columnfamily_names set,
>>>
>>>exception_message text,
>>>
>>>exception_stacktrace text,
>>>
>>>finished_at timestamp,
>>>
>>>keyspace_name text,
>>>
>>>requested_ranges set,
>>>
>>>started_at timestamp,
>>>
>>>successful_ranges set
>>>
>>> ) WITH bloom_filter_fp_chance = 0.01
>>>
>>>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>>
>>>AND comment = 'Repair history'
>>>
>>>AND compaction = {'class': 'org.apache.cassandra.db.compa
>>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>>> 'min_threshold': '4'}
>>>
>>>AND compression = {'chunk_length_in_kb': '64', 'class': '
>>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>>
>>>AND crc_check_chance = 1.0
>>>
>>>AND dclocal_read_repair_chance = 0.0
>>>
>>>AND default_time_to_live = 0
>>>
>>>AND gc_grace_seconds = 0
>>>
>>>AND max_index_interval = 2048
>>>
>>>AND memtable_flush_period_in_ms = 360
>>>
>>>AND min_index_interval = 128
>>>
>>>AND read_repair_chance = 0.0
>>>
>>>AND speculative_retry = '99PERCENTILE';
>>>
>>>
>>> CREATE TABLE system_distributed.repair_history (
>>>
>>>keyspace_name text,
>>>
>>>columnfamily_name text,
>>>
>>>id timeuuid,
>>>
>>>coordinator inet,
>>>
>>>exception_message text,
>>>
>>>exception_stacktrace text,
>>>
>>>finished_at timestamp,
>>>
>>>parent_id timeuuid,
>>>
>>>participants set,
>>>
>>>range_begin text,
>>>
>>>range_end text,
>>>
>>>started_at timestamp,
>>>
>>>status text,
>>>
>>>PRIMARY KEY ((keyspace_name, columnfamily_name), id)
>>>
>>> ) WITH CLUSTERING ORDER BY (id ASC)
>>>
>>>AND bloom_filter_fp_chance = 0.01
>>>
>>>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>>
>>>AND comment = 'Repair history'
>>>
>>>AND compaction = {'class': 'org.apache.cassandra.db.compa
>>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>>> 'min_threshold':

Re: Rebuild failing when adding new datacenter (3.0.8)

2016-09-22 Thread Yabin Meng

It is a Cassandra bug. The workaround is to change system_distributed
keyspce replication strategy to something as below:

  alter keyspace  system_distributed with replication = {'class':
'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3', 'DC3': '3'};

You may see similar problem for other system keyspaces. Do the same thing.

Cheers,

Yabin

On Thu, Sep 22, 2016 at 1:44 PM, Timo Ahokas  wrote:

> Hi Alain,
>
> Our normal user keyspaces have RF3 in all DCs, e.g:
>
> create keyspace reporting with replication = {'class':
> 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3', 'DC3': '3'};
>
> Any idea would it be safe to change the system_distributed keyspace to
> match this?
>
> -Timo
>
> On 22 September 2016 at 19:23, Timo Ahokas  wrote:
>
>> Hi Alain,
>>
>> Thanks a lot for a helping out!
>>
>> Some of the basic keyspace / cluster info you requested:
>>
>> # echo "DESCRIBE KEYSPACE system_distributed;" | cqlsh
>>
>> CREATE KEYSPACE system_distributed WITH replication = {'class':
>> 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;
>>
>> CREATE TABLE system_distributed.repair_history (
>>
>>keyspace_name text,
>>
>>columnfamily_name text,
>>
>>id timeuuid,
>>
>>coordinator inet,
>>
>>exception_message text,
>>
>>exception_stacktrace text,
>>
>>finished_at timestamp,
>>
>>parent_id timeuuid,
>>
>>participants set,
>>
>>range_begin text,
>>
>>range_end text,
>>
>>started_at timestamp,
>>
>>status text,
>>
>>PRIMARY KEY ((keyspace_name, columnfamily_name), id)
>>
>> ) WITH CLUSTERING ORDER BY (id ASC)
>>
>>AND bloom_filter_fp_chance = 0.01
>>
>>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>
>>AND comment = 'Repair history'
>>
>>AND compaction = {'class': 'org.apache.cassandra.db.compa
>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>> 'min_threshold': '4'}
>>
>>AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>
>>AND crc_check_chance = 1.0
>>
>>AND dclocal_read_repair_chance = 0.0
>>
>>AND default_time_to_live = 0
>>
>>AND gc_grace_seconds = 0
>>
>>AND max_index_interval = 2048
>>
>>AND memtable_flush_period_in_ms = 360
>>
>>AND min_index_interval = 128
>>
>>AND read_repair_chance = 0.0
>>
>>AND speculative_retry = '99PERCENTILE';
>>
>> CREATE TABLE system_distributed.parent_repair_history (
>>
>>parent_id timeuuid PRIMARY KEY,
>>
>>columnfamily_names set,
>>
>>exception_message text,
>>
>>exception_stacktrace text,
>>
>>finished_at timestamp,
>>
>>keyspace_name text,
>>
>>requested_ranges set,
>>
>>started_at timestamp,
>>
>>successful_ranges set
>>
>> ) WITH bloom_filter_fp_chance = 0.01
>>
>>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>
>>AND comment = 'Repair history'
>>
>>AND compaction = {'class': 'org.apache.cassandra.db.compa
>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>> 'min_threshold': '4'}
>>
>>AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>
>>AND crc_check_chance = 1.0
>>
>>AND dclocal_read_repair_chance = 0.0
>>
>>AND default_time_to_live = 0
>>
>>AND gc_grace_seconds = 0
>>
>>AND max_index_interval = 2048
>>
>>AND memtable_flush_period_in_ms = 360
>>
>>AND min_index_interval = 128
>>
>>AND read_repair_chance = 0.0
>>
>>AND speculative_retry = '99PERCENTILE';
>>
>>
>> CREATE TABLE system_distributed.repair_history (
>>
>>keyspace_name text,
>>
>>columnfamily_name text,
>>
>>id timeuuid,
>>
>>coordinator inet,
>>
>>exception_message text,
>>
>>exception_stacktrace text,
>>
>>finished_at timestamp,
>>
>>parent_id timeuuid,
>>
>>participants set,
>>
>>range_begin text,
>>
>>range_end text,
>>
>>started_at timestamp,
>>
>>status text,
>>
>>PRIMARY KEY ((keyspace_name, columnfamily_name), id)
>>
>> ) WITH CLUSTERING ORDER BY (id ASC)
>>
>>AND bloom_filter_fp_chance = 0.01
>>
>>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>
>>AND comment = 'Repair history'
>>
>>AND compaction = {'class': 'org.apache.cassandra.db.compa
>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>> 'min_threshold': '4'}
>>
>>AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>
>>AND crc_check_chance = 1.0
>>
>>AND dclocal_read_repair_chance = 0.0
>>
>>AND default_time_to_live = 0
>>
>>AND gc_grace_seconds = 0
>>
>>AND max_index_interval = 2048
>>
>>AND memtable_flush_period_in_ms = 360
>>
>>AND min_index_interval = 128
>>
>>AND read_repair_chance = 0.0
>>
>>AND speculative_retry = '99PERCENTILE';
>>
>> CREATE TABLE system_distributed.parent_repair_history (
>>
>>

Re: Rebuild failing when adding new datacenter (3.0.8)

2016-09-22 Thread Timo Ahokas

Hi Alain,

Our normal user keyspaces have RF3 in all DCs, e.g:

create keyspace reporting with replication = {'class':
'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3', 'DC3': '3'};

Any idea would it be safe to change the system_distributed keyspace to
match this?

-Timo

On 22 September 2016 at 19:23, Timo Ahokas  wrote:

> Hi Alain,
>
> Thanks a lot for a helping out!
>
> Some of the basic keyspace / cluster info you requested:
>
> # echo "DESCRIBE KEYSPACE system_distributed;" | cqlsh
>
> CREATE KEYSPACE system_distributed WITH replication = {'class':
> 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;
>
> CREATE TABLE system_distributed.repair_history (
>
>keyspace_name text,
>
>columnfamily_name text,
>
>id timeuuid,
>
>coordinator inet,
>
>exception_message text,
>
>exception_stacktrace text,
>
>finished_at timestamp,
>
>parent_id timeuuid,
>
>participants set,
>
>range_begin text,
>
>range_end text,
>
>started_at timestamp,
>
>status text,
>
>PRIMARY KEY ((keyspace_name, columnfamily_name), id)
>
> ) WITH CLUSTERING ORDER BY (id ASC)
>
>AND bloom_filter_fp_chance = 0.01
>
>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>
>AND comment = 'Repair history'
>
>AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
>
>AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
>
>AND crc_check_chance = 1.0
>
>AND dclocal_read_repair_chance = 0.0
>
>AND default_time_to_live = 0
>
>AND gc_grace_seconds = 0
>
>AND max_index_interval = 2048
>
>AND memtable_flush_period_in_ms = 360
>
>AND min_index_interval = 128
>
>AND read_repair_chance = 0.0
>
>AND speculative_retry = '99PERCENTILE';
>
> CREATE TABLE system_distributed.parent_repair_history (
>
>parent_id timeuuid PRIMARY KEY,
>
>columnfamily_names set,
>
>exception_message text,
>
>exception_stacktrace text,
>
>finished_at timestamp,
>
>keyspace_name text,
>
>requested_ranges set,
>
>started_at timestamp,
>
>successful_ranges set
>
> ) WITH bloom_filter_fp_chance = 0.01
>
>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>
>AND comment = 'Repair history'
>
>AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
>
>AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
>
>AND crc_check_chance = 1.0
>
>AND dclocal_read_repair_chance = 0.0
>
>AND default_time_to_live = 0
>
>AND gc_grace_seconds = 0
>
>AND max_index_interval = 2048
>
>AND memtable_flush_period_in_ms = 360
>
>AND min_index_interval = 128
>
>AND read_repair_chance = 0.0
>
>AND speculative_retry = '99PERCENTILE';
>
>
> CREATE TABLE system_distributed.repair_history (
>
>keyspace_name text,
>
>columnfamily_name text,
>
>id timeuuid,
>
>coordinator inet,
>
>exception_message text,
>
>exception_stacktrace text,
>
>finished_at timestamp,
>
>parent_id timeuuid,
>
>participants set,
>
>range_begin text,
>
>range_end text,
>
>started_at timestamp,
>
>status text,
>
>PRIMARY KEY ((keyspace_name, columnfamily_name), id)
>
> ) WITH CLUSTERING ORDER BY (id ASC)
>
>AND bloom_filter_fp_chance = 0.01
>
>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>
>AND comment = 'Repair history'
>
>AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
>
>AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
>
>AND crc_check_chance = 1.0
>
>AND dclocal_read_repair_chance = 0.0
>
>AND default_time_to_live = 0
>
>AND gc_grace_seconds = 0
>
>AND max_index_interval = 2048
>
>AND memtable_flush_period_in_ms = 360
>
>AND min_index_interval = 128
>
>AND read_repair_chance = 0.0
>
>AND speculative_retry = '99PERCENTILE';
>
> CREATE TABLE system_distributed.parent_repair_history (
>
>parent_id timeuuid PRIMARY KEY,
>
>columnfamily_names set,
>
>exception_message text,
>
>exception_stacktrace text,
>
>finished_at timestamp,
>
>keyspace_name text,
>
>requested_ranges set,
>
>started_at timestamp,
>
>successful_ranges set
>
> ) WITH bloom_filter_fp_chance = 0.01
>
>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>
>AND comment = 'Repair history'
>
>AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
>
>AND compression = {'chunk_length_in_kb': '64', 'class': '
>

Re: Rebuild failing when adding new datacenter (3.0.8)

2016-09-22 Thread Timo Ahokas

Hi Alain,

Thanks a lot for a helping out!

Some of the basic keyspace / cluster info you requested:

# echo "DESCRIBE KEYSPACE system_distributed;" | cqlsh

CREATE KEYSPACE system_distributed WITH replication = {'class':
'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

CREATE TABLE system_distributed.repair_history (

   keyspace_name text,

   columnfamily_name text,

   id timeuuid,

   coordinator inet,

   exception_message text,

   exception_stacktrace text,

   finished_at timestamp,

   parent_id timeuuid,

   participants set,

   range_begin text,

   range_end text,

   started_at timestamp,

   status text,

   PRIMARY KEY ((keyspace_name, columnfamily_name), id)

) WITH CLUSTERING ORDER BY (id ASC)

   AND bloom_filter_fp_chance = 0.01

   AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

   AND comment = 'Repair history'

   AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}

   AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}

   AND crc_check_chance = 1.0

   AND dclocal_read_repair_chance = 0.0

   AND default_time_to_live = 0

   AND gc_grace_seconds = 0

   AND max_index_interval = 2048

   AND memtable_flush_period_in_ms = 360

   AND min_index_interval = 128

   AND read_repair_chance = 0.0

   AND speculative_retry = '99PERCENTILE';

CREATE TABLE system_distributed.parent_repair_history (

   parent_id timeuuid PRIMARY KEY,

   columnfamily_names set,

   exception_message text,

   exception_stacktrace text,

   finished_at timestamp,

   keyspace_name text,

   requested_ranges set,

   started_at timestamp,

   successful_ranges set

) WITH bloom_filter_fp_chance = 0.01

   AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

   AND comment = 'Repair history'

   AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}

   AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}

   AND crc_check_chance = 1.0

   AND dclocal_read_repair_chance = 0.0

   AND default_time_to_live = 0

   AND gc_grace_seconds = 0

   AND max_index_interval = 2048

   AND memtable_flush_period_in_ms = 360

   AND min_index_interval = 128

   AND read_repair_chance = 0.0

   AND speculative_retry = '99PERCENTILE';


CREATE TABLE system_distributed.repair_history (

   keyspace_name text,

   columnfamily_name text,

   id timeuuid,

   coordinator inet,

   exception_message text,

   exception_stacktrace text,

   finished_at timestamp,

   parent_id timeuuid,

   participants set,

   range_begin text,

   range_end text,

   started_at timestamp,

   status text,

   PRIMARY KEY ((keyspace_name, columnfamily_name), id)

) WITH CLUSTERING ORDER BY (id ASC)

   AND bloom_filter_fp_chance = 0.01

   AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

   AND comment = 'Repair history'

   AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}

   AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}

   AND crc_check_chance = 1.0

   AND dclocal_read_repair_chance = 0.0

   AND default_time_to_live = 0

   AND gc_grace_seconds = 0

   AND max_index_interval = 2048

   AND memtable_flush_period_in_ms = 360

   AND min_index_interval = 128

   AND read_repair_chance = 0.0

   AND speculative_retry = '99PERCENTILE';

CREATE TABLE system_distributed.parent_repair_history (

   parent_id timeuuid PRIMARY KEY,

   columnfamily_names set,

   exception_message text,

   exception_stacktrace text,

   finished_at timestamp,

   keyspace_name text,

   requested_ranges set,

   started_at timestamp,

   successful_ranges set

) WITH bloom_filter_fp_chance = 0.01

   AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

   AND comment = 'Repair history'

   AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}

   AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}

   AND crc_check_chance = 1.0

   AND dclocal_read_repair_chance = 0.0

   AND default_time_to_live = 0

   AND gc_grace_seconds = 0

   AND max_index_interval = 2048

   AND memtable_flush_period_in_ms = 360

   AND min_index_interval = 128

   AND read_repair_chance = 0.0

   AND speculative_retry = '99PERCENTILE';



# nodetool status

Datacenter: DC1

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address Load   Tokens   OwnsHost ID
  Rack

UN  xxx.xxx.145.5693,63 GB  256  ?
  6f1a0fdd-e3f9-474d-9a49-7bfeeadb3f56  RAC1

UN  xxx.xxx.145.225

Re: Rebuild failing when adding new datacenter (3.0.8)

2016-09-22 Thread Alain RODRIGUEZ

It could be a bug.

Yet I am not very aware of this system_distributed keyspace, but from what
I see, it is using a simple strategy:

root@tlp-cassandra-2:~# echo "DESCRIBE KEYSPACE system_distributed;" |
cqlsh $(hostname -I | awk '{print $1}')

CREATE KEYSPACE system_distributed WITH replication = {'class':
'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

Let's first check some stuff. Could you share the output of:


   - echo "DESCRIBE KEYSPACE system_distributed;" | cqlsh
   [ip_address_of_the_server]
   - nodetool status
   - nodetool status system_distributed
   - Let us know about the snitch you are using and the corresponding
   configuration.


I am trying to make sure the command you used is expected to work, given
your setup.

My guess is this you might need to alter this keyspace accordingly to your
cluster setup.

Just guessing, hope that helps.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-09-22 15:47 GMT+02:00 Timo Ahokas :

> Hi,
>
> We have a Cassandra 3.0.8 cluster (recently upgraded from 2.1.15)
> currently running in two data centers (13 and 19 nodes, RF3 in both). We
> are adding a third data center before decommissioning one of the earlier
> ones. Installing Cassandra (3.0.8) goes fine and all the nodes join the
> cluster (not set to bootstrap, as documented in
> https://docs.datastax.com/en/cassandra/3.0/cassandra/
> operations/opsAddDCToCluster.html).
>
> When trying to rebuild nodes in the new DC from a previous DC (nodetool
> rebuild -- DC1), we get the following error:
>
> Unable to find sufficient sources for streaming range 
> (597769692463489739,597931451954862346]
> in keyspace system_distributed
>
> The same error occurs which ever of the 2 existing DCs we try to rebuild
> from.
>
> We run pr repairs (nodetool repair -pr) on all nodes twice a week via cron.
>
> Any advice on how to get the rebuild started?
>
> Best regards,
> Timo
>
>
>
>
>

Re: Rebuild failing when adding new datacenter (3.0.8)

Re: Rebuild failing when adding new datacenter (3.0.8)

Re: Rebuild failing when adding new datacenter (3.0.8)

Re: Rebuild failing when adding new datacenter (3.0.8)

Re: Rebuild failing when adding new datacenter (3.0.8)

5 matches

Site Navigation

Mail list logo

Footer information