Re:Re: Re: stream failed when bootstrap

[dayu]

Hi kurt, a rolling restart means run disablebinary, disablethrift, 
disablegossip, drain, stop cassandra and start cassandra command one by one, 
right?
Only one node is executed at a time


Dayu




At 2018-06-28 11:37:43, "kurt greaves"  wrote:

Best off trying a rolling restart.


On 28 June 2018 at 03:18, dayu  wrote:

the output of nodetool describecluster
Cluster Information:
Name: online-xxx
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
c3f00d61-1ad7-3702-8703-af2a29e401c1: [10.136.71.43]


0568e8c1-48ba-3fb0-bb3c-462438978d7b: [10.136.71.33, ]


after I run nodetool resetlocalschema, a error log outcome


ERROR [InternalResponseStage:209417] 2018-06-28 11:14:12,904 
MigrationTask.java:96 - Configuration
exception merging remote schema
org.apache.cassandra.exceptions.ConfigurationException: Column family ID 
mismatch (found 5552bba0-2
dc6-11e8-9b5c-254242d97235; expected 53f6d520-2dc6-11e8-948d-ab7caa3c8c36)
at 
org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:790)
 ~[apac
he-cassandra-3.0.10.jar:3.0.10]
at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:750) 
~[apache-cassandra-3.0
.10.jar:3.0.10]
at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) 
~[apache-cassandra-3.0.1
0.jar:3.0.10]
at 
org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1348)
 ~[ap
ache-cassandra-3.0.10.jar:3.0.10]






At 2018-06-28 10:01:52, "Jeff Jirsa"  wrote:
You can sometimes bounce your way through it (or use nodetool resetlocalschema 
if it’s a single node that’s wrong), but there are some edge cases from which 
it’s very hard to recover


What’s the output of nodetool describecluster?


If you select from the schema tables, do you see that CFID on any real tables? 


-- 
Jeff Jirsa



On Jun 27, 2018, at 7:58 PM, dayu  wrote:


That sound reasonable, I have seen schema mismatch error before.
So any advise to deal with schema mismatches?
Dayu


At 2018-06-28 09:50:37, "Jeff Jirsa"  wrote:
>That log message says you did:
>
> CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming
>
>If you’re absolutely sure you didn’t, you should look for schema mismatches in 
>your cluster
>
>
>-- 
>Jeff Jirsa
>
>
>> On Jun 27, 2018, at 7:49 PM, dayu  wrote:
>> 
>> CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming
>
>-
>To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>For additional commands, e-mail: user-h...@cassandra.apache.org




 





 

Re: Re: stream failed when bootstrap

[kurt greaves]

Best off trying a rolling restart.

On 28 June 2018 at 03:18, dayu  wrote:

> the output of nodetool describecluster
> Cluster Information:
> Name: online-xxx
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> c3f00d61-1ad7-3702-8703-af2a29e401c1: [10.136.71.43]
>
> 0568e8c1-48ba-3fb0-bb3c-462438978d7b: [10.136.71.33, ]
>
> after I run nodetool resetlocalschema, a error log outcome
>
> ERROR [InternalResponseStage:209417] 2018-06-28 11:14:12,904
> MigrationTask.java:96 - Configuration
> exception merging remote schema
> org.apache.cassandra.exceptions.ConfigurationException: Column family ID
> mismatch (found 5552bba0-2
> dc6-11e8-9b5c-254242d97235; expected 53f6d520-2dc6-11e8-948d-ab7caa3c8c36)
> at 
> org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:790)
> ~[apac
> he-cassandra-3.0.10.jar:3.0.10]
> at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:750)
> ~[apache-cassandra-3.0
> .10.jar:3.0.10]
> at org.apache.cassandra.config.Schema.updateTable(Schema.java:661)
> ~[apache-cassandra-3.0.1
> 0.jar:3.0.10]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1348)
> ~[ap
> ache-cassandra-3.0.10.jar:3.0.10]
>
>
>
>
>
> At 2018-06-28 10:01:52, "Jeff Jirsa"  wrote:
>
> You can sometimes bounce your way through it (or use nodetool
> resetlocalschema if it’s a single node that’s wrong), but there are some
> edge cases from which it’s very hard to recover
>
> What’s the output of nodetool describecluster?
>
> If you select from the schema tables, do you see that CFID on any real
> tables?
>
> --
> Jeff Jirsa
>
>
> On Jun 27, 2018, at 7:58 PM, dayu  wrote:
>
> That sound reasonable, I have seen schema mismatch error before.
> So any advise to deal with schema mismatches?
>
> Dayu
>
> At 2018-06-28 09:50:37, "Jeff Jirsa"  wrote:
> >That log message says you did:
> >
> > CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming
> >
> >If you’re absolutely sure you didn’t, you should look for schema mismatches 
> >in your cluster
> >
> >
> >--
> >Jeff Jirsa
> >
> >
> >> On Jun 27, 2018, at 7:49 PM, dayu  wrote:
> >>
> >> CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming
> >
> >-
> >To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >For additional commands, e-mail: user-h...@cassandra.apache.org
>
>
>
>
>
>
>
>
>

Re:Re: stream failed when bootstrap

[dayu]

the output of nodetool describecluster
Cluster Information:
Name: online-xxx
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
c3f00d61-1ad7-3702-8703-af2a29e401c1: [10.136.71.43]


0568e8c1-48ba-3fb0-bb3c-462438978d7b: [10.136.71.33, ]


after I run nodetool resetlocalschema, a error log outcome


ERROR [InternalResponseStage:209417] 2018-06-28 11:14:12,904 
MigrationTask.java:96 - Configuration
exception merging remote schema
org.apache.cassandra.exceptions.ConfigurationException: Column family ID 
mismatch (found 5552bba0-2
dc6-11e8-9b5c-254242d97235; expected 53f6d520-2dc6-11e8-948d-ab7caa3c8c36)
at 
org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:790)
 ~[apac
he-cassandra-3.0.10.jar:3.0.10]
at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:750) 
~[apache-cassandra-3.0
.10.jar:3.0.10]
at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) 
~[apache-cassandra-3.0.1
0.jar:3.0.10]
at 
org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1348)
 ~[ap
ache-cassandra-3.0.10.jar:3.0.10]






At 2018-06-28 10:01:52, "Jeff Jirsa"  wrote:
You can sometimes bounce your way through it (or use nodetool resetlocalschema 
if it’s a single node that’s wrong), but there are some edge cases from which 
it’s very hard to recover


What’s the output of nodetool describecluster?


If you select from the schema tables, do you see that CFID on any real tables? 


-- 
Jeff Jirsa



On Jun 27, 2018, at 7:58 PM, dayu  wrote:


That sound reasonable, I have seen schema mismatch error before.
So any advise to deal with schema mismatches?
Dayu


At 2018-06-28 09:50:37, "Jeff Jirsa"  wrote:
>That log message says you did:
>
> CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming
>
>If you’re absolutely sure you didn’t, you should look for schema mismatches in 
>your cluster
>
>
>-- 
>Jeff Jirsa
>
>
>> On Jun 27, 2018, at 7:49 PM, dayu  wrote:
>> 
>> CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming
>
>-
>To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>For additional commands, e-mail: user-h...@cassandra.apache.org




 

Re: stream failed when bootstrap

[Jeff Jirsa]

You can sometimes bounce your way through it (or use nodetool resetlocalschema 
if it’s a single node that’s wrong), but there are some edge cases from which 
it’s very hard to recover

What’s the output of nodetool describecluster?

If you select from the schema tables, do you see that CFID on any real tables? 

-- 
Jeff Jirsa


> On Jun 27, 2018, at 7:58 PM, dayu  wrote:
> 
> That sound reasonable, I have seen schema mismatch error before.
> So any advise to deal with schema mismatches?
> Dayu
> 
> At 2018-06-28 09:50:37, "Jeff Jirsa"  wrote:
> >That log message says you did:
> >
> > CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming
> >
> >If you’re absolutely sure you didn’t, you should look for schema mismatches 
> >in your cluster
> >
> >
> >-- 
> >Jeff Jirsa
> >
> >
> >> On Jun 27, 2018, at 7:49 PM, dayu  wrote:
> >> 
> >> CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming
> >
> >-
> >To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >For additional commands, e-mail: user-h...@cassandra.apache.org
> 
> 
>  

Re:Re: stream failed when bootstrap

[dayu]

That sound reasonable, I have seen schema mismatch error before.
So any advise to deal with schema mismatches?
Dayu


At 2018-06-28 09:50:37, "Jeff Jirsa"  wrote:
>That log message says you did:
>
> CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming
>
>If you’re absolutely sure you didn’t, you should look for schema mismatches in 
>your cluster
>
>
>-- 
>Jeff Jirsa
>
>
>> On Jun 27, 2018, at 7:49 PM, dayu  wrote:
>> 
>> CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming
>
>-
>To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>For additional commands, e-mail: user-h...@cassandra.apache.org

Re: stream failed when bootstrap

[Jeff Jirsa]

That log message says you did:

 CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming

If you’re absolutely sure you didn’t, you should look for schema mismatches in 
your cluster


-- 
Jeff Jirsa


> On Jun 27, 2018, at 7:49 PM, dayu  wrote:
> 
> CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re:Re: stream failed when bootstrap

[dayu]

Hi Jeff
Thanks for your help. But I didn't remove any tables during bootstrap. I 
have tried 3 times, still got this error
BTW the cassandra version is 3.0.10.


dayu


 ERROR [STREAM-IN-/10.136.71.33] 2018-06-28 08:51:33,330 StreamSession.java:534 
- [Stream #712021f0-7a25-11e8-b622-e7402ff35894] Streaming error occurred
java.io.IOException: CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during 
streaming
at 
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77)
 ~[apache-cassandra-3.0.10.jar:3.0.10]
at 
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:54)
 ~[apache-cassandra-3.0.10.jar:3.0.10]
at 
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:43)
 ~[apache-cassandra-3.0.10.jar:3.0.10]
at 
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
 ~[apache-cassandra-3.0.10.jar:3.0.10]
at 
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:290)
 ~[apache-cassandra-3.0.10.jar:3.0.10]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]






At 2018-06-28 09:42:29, "Jeff Jirsa"  wrote:
Try again and this time don’t remove tables during bootstrap (the streaming 
code doesn’t handle removing tables very well).


-- 
Jeff Jirsa



On Jun 27, 2018, at 7:30 PM, dayu  wrote:


Hi everyone
I am joining a new node to a cluster. but it failed even if I use nodetool 
bootstrap resume.
I found error in log. what can I do?



Thanks !
Dayu




 

Re: stream failed when bootstrap

[Jeff Jirsa]

Try again and this time don’t remove tables during bootstrap (the streaming 
code doesn’t handle removing tables very well).

-- 
Jeff Jirsa


> On Jun 27, 2018, at 7:30 PM, dayu  wrote:
> 
> Hi everyone
> I am joining a new node to a cluster. but it failed even if I use 
> nodetool bootstrap resume.
> I found error in log. what can I do?
> 
> 
> Thanks !
> Dayu
> 
> 
>  

stream failed when bootstrap

[dayu]

Hi everyone
I am joining a new node to a cluster. but it failed even if I use nodetool 
bootstrap resume.
I found error in log. what can I do?


Thanks !
Dayu

Re: Maximum SSTable size

[Jeff Jirsa]

That’s a target and not a maximum, and it can go over that even on LCS with 
wide partitions

In general, the practical limit is hundreds of gigabytes (I’ve seen 500-600 a 
lot, but not more than a terabyte very often). You’ll typically hit other 
issues and need to tune some params before you hit hard caps (unless you know 
what you’re doing).

-- 
Jeff Jirsa


> On Jun 27, 2018, at 3:11 PM, ZAIDI, ASAD A  wrote:
> 
> With  Leveled compaction strategy  , you can set target size with 
> sstable_size_in_mb attribute however actual size can still be larger than 
> target given large partition size.
>  
> Thanks/Asad
>  
> From: Lucas Benevides [mailto:lu...@maurobenevides.com.br] 
> Sent: Wednesday, June 27, 2018 7:02 AM
> To: user@cassandra.apache.org
> Subject: Maximum SSTable size
>  
> Hello Community,
>  
> Is there a maximum SSTable Size?  
> If there is not, does it go up to the maximum Operational System values? 
>  
> Thanks in advance,
> Lucas Benevides

Re: [External] Maximum SSTable size

[Tom van der Woerdt]

I’ve had SSTables as big as 11TB. It works, read performance is fine. But,
compaction is hell, because you’ll need twice that in disk space and it
will take many hours 

Avoid large SSTables unless you really know what you’re doing. LCS is a
great default for almost every workload, especially if your cluster has a
single large table. STCS is the actual Cassandra default but it often
causes more trouble than it solves, because of large SSTables 

Hope that helps!

Tom


On Wed, 27 Jun 2018 at 08:02, Lucas Benevides 
wrote:

> Hello Community,
>
> Is there a maximum SSTable Size?
> If there is not, does it go up to the maximum Operational System values?
>
> Thanks in advance,
> Lucas Benevides
>
-- 
Tom van der Woerdt
Site Reliability Engineer

Booking.com B.V.
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
Direct +31207153426
[image: Booking.com] 
The world's #1 accommodation site
43 languages, 198+ offices worldwide, 120,000+ global destinations,
1,550,000+ room nights booked every day
No booking fees, best price always guaranteed
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)

RE: Maximum SSTable size

[ZAIDI, ASAD A]

With  Leveled compaction 
strategy  
, you can set target size with sstable_size_in_mb attribute however actual size 
can still be larger than target given large partition size.

Thanks/Asad

From: Lucas Benevides [mailto:lu...@maurobenevides.com.br]
Sent: Wednesday, June 27, 2018 7:02 AM
To: user@cassandra.apache.org
Subject: Maximum SSTable size

Hello Community,

Is there a maximum SSTable Size?
If there is not, does it go up to the maximum Operational System values?

Thanks in advance,
Lucas Benevides

Re: Is it ok to add more than one node to a exist cluster

[kurt greaves]

To clarify, this is after the node reaches NORMAL state. don't bootstrap
other nodes while one is in JOINING unless you really know what you are
doing and aren't using vnodes.

On Wed., 27 Jun. 2018, 21:02 Abdul Patel,  wrote:

> Theres always an 2 minute rule ..after adding one node wait for 2 minutes
> before addding second node.
>
> On Wednesday, June 27, 2018, dayu  wrote:
>
>> Thanks for your reply, kurt.
>>
>> another question, Can I bootstrap a new node when some node is in Joining
>> state ? Or I should wait until Joining node becoming Normal ?
>>
>> Dayu
>>
>>
>>
>> At 2018-06-27 17:50:34, "kurt greaves"  wrote:
>>
>> Don't bootstrap nodes simultaneously unless you really know what you're
>> doing, and you're using single tokens. It's not straightforward and will
>> likely lead to data loss/inconsistencies. This applies for all current
>> versions.
>>
>> On 27 June 2018 at 10:21, dayu  wrote:
>>
>>> Hi,
>>> I have read a warning of not Simultaneously bootstrapping more than
>>> one new node from the same rack in version 2.1  link
>>> 
>>> My cassandra cluster version is 3.0.10.
>>> So I wonder to know Is it ok to add more than one node to a exist
>>> cluster in 3.0.10.
>>>
>>> Thanks!
>>> Dayu
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>

Maximum SSTable size

[Lucas Benevides]

Hello Community,

Is there a maximum SSTable Size?
If there is not, does it go up to the maximum Operational System values?

Thanks in advance,
Lucas Benevides

Re: Is it ok to add more than one node to a exist cluster

[Abdul Patel]

Theres always an 2 minute rule ..after adding one node wait for 2 minutes
before addding second node.

On Wednesday, June 27, 2018, dayu  wrote:

> Thanks for your reply, kurt.
>
> another question, Can I bootstrap a new node when some node is in Joining
> state ? Or I should wait until Joining node becoming Normal ?
>
> Dayu
>
>
>
> At 2018-06-27 17:50:34, "kurt greaves"  wrote:
>
> Don't bootstrap nodes simultaneously unless you really know what you're
> doing, and you're using single tokens. It's not straightforward and will
> likely lead to data loss/inconsistencies. This applies for all current
> versions.
>
> On 27 June 2018 at 10:21, dayu  wrote:
>
>> Hi,
>> I have read a warning of not Simultaneously bootstrapping more than
>> one new node from the same rack in version 2.1  link
>> 
>> My cassandra cluster version is 3.0.10.
>> So I wonder to know Is it ok to add more than one node to a exist
>> cluster in 3.0.10.
>>
>> Thanks!
>> Dayu
>>
>>
>>
>>
>
>
>
>
>

Re:Re: Is it ok to add more than one node to a exist cluster

[dayu]

Thanks Abdul !



dayu




At 2018-06-27 18:02:08, "Abdul Patel"  wrote:
Theres always an 2 minute rule ..after adding one node wait for 2 minutes 
before addding second node.

On Wednesday, June 27, 2018, dayu  wrote:

Thanks for your reply, kurt.


another question, Can I bootstrap a new node when some node is in Joining state 
? Or I should wait until Joining node becoming Normal ?

Dayu




At 2018-06-27 17:50:34, "kurt greaves"  wrote:

Don't bootstrap nodes simultaneously unless you really know what you're doing, 
and you're using single tokens. It's not straightforward and will likely lead 
to data loss/inconsistencies. This applies for all current versions.


On 27 June 2018 at 10:21, dayu  wrote:

Hi,
I have read a warning of not Simultaneously bootstrapping more than one new 
node from the same rack in version 2.1  link
My cassandra cluster version is 3.0.10.
So I wonder to know Is it ok to add more than one node to a exist cluster 
in 3.0.10. 


Thanks!
Dayu




 







 

Re:Re: Is it ok to add more than one node to a exist cluster

[dayu]

Thanks for your reply, kurt.


another question, Can I bootstrap a new node when some node is in Joining state 
? Or I should wait until Joining node becoming Normal ?

Dayu




At 2018-06-27 17:50:34, "kurt greaves"  wrote:

Don't bootstrap nodes simultaneously unless you really know what you're doing, 
and you're using single tokens. It's not straightforward and will likely lead 
to data loss/inconsistencies. This applies for all current versions.


On 27 June 2018 at 10:21, dayu  wrote:

Hi,
I have read a warning of not Simultaneously bootstrapping more than one new 
node from the same rack in version 2.1  link
My cassandra cluster version is 3.0.10.
So I wonder to know Is it ok to add more than one node to a exist cluster 
in 3.0.10. 


Thanks!
Dayu




 

Re: Is it ok to add more than one node to a exist cluster

[kurt greaves]

Don't bootstrap nodes simultaneously unless you really know what you're
doing, and you're using single tokens. It's not straightforward and will
likely lead to data loss/inconsistencies. This applies for all current
versions.

On 27 June 2018 at 10:21, dayu  wrote:

> Hi,
> I have read a warning of not Simultaneously bootstrapping more than
> one new node from the same rack in version 2.1  link
> 
> My cassandra cluster version is 3.0.10.
> So I wonder to know Is it ok to add more than one node to a exist
> cluster in 3.0.10.
>
> Thanks!
> Dayu
>
>
>
>

Is it ok to add more than one node to a exist cluster

[dayu]

Hi,
I have read a warning of not Simultaneously bootstrapping more than one new 
node from the same rack in version 2.1  link
My cassandra cluster version is 3.0.10.
So I wonder to know Is it ok to add more than one node to a exist cluster 
in 3.0.10. 


Thanks!
Dayu