回复： Data Loss irreparabley so

[Peng Xiao]

Due to the tombstone,we have set GC_GRACE_SECONDS to 6 hours.And for a huge 
table with 4T size,repair is a hard thing for us.




-- 原始邮件 --
发件人: "kurt";;
发送时间: 2017年8月3日(星期四) 中午12:08
收件人: "User"; 

主题: Re: Data Loss irreparabley so



You should run repairs every GC_GRACE_SECONDS. If a node is overloaded/goes 
down, you should run repairs. LOCAL_QUORUM will somewhat maintain consistency 
within a DC, but certainly doesn't mean you can get away without running 
repairs. You need to run repairs even if you are using QUORUM or ONE.​

Re: Data Loss irreparabley so

[kurt greaves]

You should run repairs every GC_GRACE_SECONDS. If a node is overloaded/goes
down, you should run repairs. LOCAL_QUORUM will somewhat maintain
consistency within a DC, but certainly doesn't mean you can get away
without running repairs. You need to run repairs even if you are using
QUORUM or ONE.​

回复： Data Loss irreparabley so

[Peng Xiao]

Hi,
We are also experiencing the same issue.we have 3 DCs(DC1 RF=3,DC2 
RF=3,DC3,RF=1),if we use local_quorum,we are not meant to loss any data,right?
if we use local_one, maybe loss data? then we need to run repair regularly?
Could anyone advise?


Thanks






-- 原始邮件 --
发件人: "Jon Haddad";;
发送时间: 2017年7月28日(星期五) 凌晨1:37
收件人: "user"; 

主题: Re: Data Loss irreparabley so



We (The Last Pickle) maintain an open source tool to help manage repairs across 
your clusters called Reaper.  It’s a lot easier to set up and manage than 
trying to manage it through cron.


http://thelastpickle.com/reaper.html

On Jul 27, 2017, at 12:38 AM, Daniel Hölbling-Inzko 
 wrote:

In that vein, Cassandra support Auto compaction and incremental repair. 
Does this mean I have to set up cron jobs on each node to do a nodetool repair 
or is this taken care of by Cassandra anyways?
How often should I run nodetool repair

Greetings Daniel
Jeff Jirsa  schrieb am Do. 27. Juli 2017 um 07:48:


 
 On 2017-07-25 15:49 (-0700), Roger Warner  wrote:
 > This is a quick informational question. I know that Cassandra can detect 
 > failures of nodes and repair them given replication and multiple DC.
 >
 > My question is can Cassandra tell if data was lost after a failure and 
 > node(s) “fixed” and resumed operation?
 >
 
 Sorta concerned by the way you're asking this - Cassandra doesn't "fix" failed 
nodes. It can route requests around a down node, but the "fixing" is entirely 
manual.
 
 If you have a node go down temporarily, and it comes back up (with it's disk 
intact), you can see it "repair" data with a combination of active 
(anti-entropy) repair via nodetool repair, or by watching 'nodetool netstats' 
and see the read repair counters increase over time (which will happen 
naturally as data is requested and mismatches are detected in the data, based 
on your consistency level).
 
 
 
 -
 To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra Data migration from 2.2.3 to 3.7

[Jeff Jirsa]

Anytime you copy sstables around (sstableloader is excluded here), make sure 
you don't copy sstables from 2 nodes to the same directories without checking 
that the names don't collide or you'll lose one of the sstables


-- 
Jeff Jirsa


> On Aug 2, 2017, at 10:38 AM, Harika Vangapelli -T (hvangape - AKRAYA INC at 
> Cisco)  wrote:
> 
> Jeff,  what is the meaning of this line you mentioned in below email..
> ‘making sure you dont overwrite any sstables with the same name’
>  
> 
>  
> Harika Vangapelli
> Engineer - IT
> hvang...@cisco.com
> Tel:
> Cisco Systems, Inc.
> 
> 
> 
> United States
> cisco.com
> Think before you print.
> This email may contain confidential and privileged material for the sole use 
> of the intended recipient. Any review, use, distribution or disclosure by 
> others is strictly prohibited. If you are not the intended recipient (or 
> authorized to receive for the recipient), please contact the sender by reply 
> email and delete all copies of this message.
> Please click here for Company Registration Information.
>  
> From: Jeff Jirsa [mailto:jji...@gmail.com] 
> Sent: Tuesday, August 1, 2017 5:13 PM
> To: cassandra 
> Subject: Re: Cassandra Data migration from 2.2.3 to 3.7
>  
> Roughly that approach should work, modulo copying data in/out to the right 
> paths, and making sure you dont overwrite any sstables with the same name.
> 
> (You dont need to refresh after the upgradesstables unless you've got 2 3.7 
> clusters, and you may need to refresh before the upgradesstables if the 3.7 
> cluster in step 3 is already running)
>  
>  
> On Tue, Aug 1, 2017 at 3:44 PM, Harika Vangapelli -T (hvangape - AKRAYA INC 
> at Cisco)  wrote:
> Jeff, I tried the below steps for just 3 rows of data, It looks to be 
> working. But Confirm me this is right approach.
> 1.   Taking snapshot (using nodetool snapshot) in the existing cluster 
> (2.2 cluster)
> 
> 2.   Bring the snapshot to new cluster and copy in keyspce/table 
> (2.2->3.7)
> 
> 3.   Nodetool upgrade sstbales on new cluster (3.7)
> 
> 4.   Use noedtool refresh --   (3.7)
> 
> Does this approach work?
>  
> 
>  
> Harika Vangapelli
> Engineer - IT
> hvang...@cisco.com
> Tel:
> Cisco Systems, Inc.
> 
> 
> 
> United States
> cisco.com
>  
> Think before you print.
> This email may contain confidential and privileged material for the sole use 
> of the intended recipient. Any review, use, distribution or disclosure by 
> others is strictly prohibited. If you are not the intended recipient (or 
> authorized to receive for the recipient), please contact the sender by reply 
> email and delete all copies of this message.
> Please click here for Company Registration Information.
>  
> From: Jeff Jirsa [mailto:jji...@gmail.com] 
> Sent: Tuesday, August 1, 2017 3:36 PM
> 
> To: user@cassandra.apache.org
> Subject: Re: Cassandra Data migration from 2.2.3 to 3.7
>  
> You can't stream across versions, so you either need to use the native 
> protocol (do writes with the normal INSERT operations) or you have to get to 
> the right sstable version
>  
> That means one of the three of these: 
>  
> - export to CSV (copy to/copy from, or Brian Hess' Cassandra-loader on github)
> - export to CSV and use CQLSSTableWriter to generate sstables you can bulk 
> load
> - create a minimal cluster, load the sstables in, and run upgradesstables to 
> rev the version and then stop that cluster and stream them into the new 
> cluster
> 
> 
> -- 
> Jeff Jirsa
>  
> 
> On Aug 1, 2017, at 3:30 PM, Harika Vangapelli -T (hvangape - AKRAYA INC at 
> Cisco)  wrote:
> 
> Jeff, Thanks for initial pointer. I want to add some more information to it.
>  
> This is to migrate from one cluster (2.2 version) to 3.7 . The client who is 
> in 2.2 cluster they don’t need down time.
>  
> So How the data is migrated to new cluster without down time with different 
> versions?
>  
> Thanks,
> Harika
>  
> 
>  
> Harika Vangapelli
> Engineer - IT
> hvang...@cisco.com
> Tel:
> Cisco Systems, Inc.
> 
> 
> 
> United States
> cisco.com
>  
> Think before you print.
> This email may contain confidential and privileged material for the sole use 
> of the intended recipient. Any review, use, distribution or disclosure by 
> others is strictly prohibited. If you are not the intended recipient (or 
> authorized to receive for the recipient), please contact the sender by reply 
> email and delete all copies of this message.
> Please click here for Company Registration Information.
>  
> From: Jeff Jirsa [mailto:jji...@gmail.com] 
> Sent: Tuesday, August 1, 2017 2:48 PM
> To: user@cassandra.apache.org
> Subject: Re: Cassandra Data migration from 2.2.3 to 3.7
>  
> If you upgrade in place:
>  
> Stop each node one at a time
> Swap binaries
> Start each node
> Run upgradesstables when all nodes are bounced
>  
> If you want to take data from 2.2 and put it into a 3.x cluster, you'll need 

Re: Cassandra isn't compacting old files

[Sotirios Delimanolis]

Turns out there are already logs for this in Tracker.java. I enabled those and 
clearly saw the old files are being tracked.
What else can I look at for hints about whether these files are later 
invalidated/filtered out somehow?

On Tuesday, August 1, 2017, 3:29:38 PM PDT, Sotirios Delimanolis 
 wrote:

There aren't any ERROR logs for failure to load these files and they do get 
compacted away. I'll try to plug some DEBUG logs in a custom Cassandra 
version.On Tuesday, August 1, 2017, 12:13:09 PM PDT, Jeff Jirsa 
 wrote:

I don't have time to dive deep into the code of your version, but it may be ( 
https://issues.apache.org/jira/browse/CASSANDRA-13620 ) , or it may be 
something else.
I wouldn't expect compaction to touch them if they're invalid. The handle may 
be a leftover from trying to load them. 


On Tue, Aug 1, 2017 at 10:01 AM, Sotirios Delimanolis 
 wrote:

@Jeff, why does compaction clear them and why does Cassandra keep a handle to 
them? Shouldn't they be ignored entirely? Is there an error log I can enable to 
detect them?
@kurt, there are no such logs for any of these tables. We have a custom log in 
our build of Cassandra that does shows that compactions are happening for that 
table but only ever include the files from July.
On Tuesday, August 1, 2017, 12:55:53 AM PDT, kurt greaves 
 wrote:

Seeing as there aren't even 100 SSTables in L2, LCS should be gradually trying 
to compact L3 with L2. You could search the logs for "Adding high-level (L3)" 
to check if this is happening. ​

RE: Cassandra Data migration from 2.2.3 to 3.7

[Harika Vangapelli -T (hvangape - AKRAYA INC at Cisco)]

Jeff,  what is the meaning of this line you mentioned in below email..
‘making sure you dont overwrite any sstables with the same name’

[http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signaturetool/images/logo/logo_gradient.png]



Harika Vangapelli
Engineer - IT
hvang...@cisco.com
Tel:

Cisco Systems, Inc.



United States
cisco.com


[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif]Think before you 
print.

This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.
Please click 
here for 
Company Registration Information.


From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: Tuesday, August 1, 2017 5:13 PM
To: cassandra 
Subject: Re: Cassandra Data migration from 2.2.3 to 3.7

Roughly that approach should work, modulo copying data in/out to the right 
paths, and making sure you dont overwrite any sstables with the same name.

(You dont need to refresh after the upgradesstables unless you've got 2 3.7 
clusters, and you may need to refresh before the upgradesstables if the 3.7 
cluster in step 3 is already running)


On Tue, Aug 1, 2017 at 3:44 PM, Harika Vangapelli -T (hvangape - AKRAYA INC at 
Cisco) > wrote:
Jeff, I tried the below steps for just 3 rows of data, It looks to be working. 
But Confirm me this is right approach.

1.   Taking snapshot (using nodetool snapshot) in the existing cluster (2.2 
cluster)

2.   Bring the snapshot to new cluster and copy in keyspce/table (2.2->3.7)

3.   Nodetool upgrade sstbales on new cluster (3.7)

4.   Use noedtool refresh --   (3.7)
Does this approach work?

[http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signaturetool/images/logo/logo_gradient.png]



Harika Vangapelli
Engineer - IT
hvang...@cisco.com
Tel:

Cisco Systems, Inc.



United States
cisco.com


[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif]Think before you 
print.

This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.
Please click 
here for 
Company Registration Information.


From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: Tuesday, August 1, 2017 3:36 PM

To: user@cassandra.apache.org
Subject: Re: Cassandra Data migration from 2.2.3 to 3.7

You can't stream across versions, so you either need to use the native protocol 
(do writes with the normal INSERT operations) or you have to get to the right 
sstable version

That means one of the three of these:

- export to CSV (copy to/copy from, or Brian Hess' Cassandra-loader on github)
- export to CSV and use CQLSSTableWriter to generate sstables you can bulk load
- create a minimal cluster, load the sstables in, and run upgradesstables to 
rev the version and then stop that cluster and stream them into the new cluster


--
Jeff Jirsa


On Aug 1, 2017, at 3:30 PM, Harika Vangapelli -T (hvangape - AKRAYA INC at 
Cisco) > wrote:
Jeff, Thanks for initial pointer. I want to add some more information to it.

This is to migrate from one cluster (2.2 version) to 3.7 . The client who is in 
2.2 cluster they don’t need down time.

So How the data is migrated to new cluster without down time with different 
versions?

Thanks,
Harika





Harika Vangapelli
Engineer - IT
hvang...@cisco.com
Tel:

Cisco Systems, Inc.



United States
cisco.com


Think before you print.

This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.
Please click 
here for 
Company Registration Information.


From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: Tuesday, August 1, 2017 2:48 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra Data migration from 2.2.3 to 3.7

If you upgrade in place:

Stop each node one at a 

Re: Bootstrapping a new Node with Consistency=ONE

[Jeff Jirsa]

By the time bootstrap is complete it should be as consistent as the source node 
- you can change start_native_transport to false to avoid serving clients 
directly (tcp/9042), but it'll still serve reads via the storage service 
(tcp/7000), but the guarantee is that data should be consistent by the time 
bootstrap finishes




-- 
Jeff Jirsa


> On Aug 2, 2017, at 1:53 AM, Daniel Hölbling-Inzko 
>  wrote:
> 
> Hi,
> It's probably a strange question but I have a heavily read-optimized payload 
> where data integrity is not a big deal. So to keep latencies low I am reading 
> with Consistency ONE from my Multi-DC Cluster.
> 
> Now the issue I saw is that I needed to add another Cassandra node (for 
> redundancy reasons). 
> Since I want this for renduncancy I booted the node and then changed the 
> Replication of my Keyspace to include the new node (all nodes have 100% of 
> the data).
> 
> The issue I was seeing is that clients that connected to the new Node 
> afterwards were seeing incomplete data - so the Key would already be present, 
> but the columns would all be null values.
> I expect this to die down once the node is fully replicated, but in the 
> meantime a lot of my connected clients were in trouble. (The application can 
> handle seeing old data - incomplete is another matter all together)
> 
> The total data in question is a negligible 500kb (so nothing that should 
> really take any amount of time in my opinion but it took a few minutes for 
> the data to replicate over and I am still not sure everything is replicated 
> correctly).
> 
> Increasing the RF to something higher won't really help as the setup is dc1: 
> 3; dc2: 2 (I added the second node in dc2). So a LOCAL_QUORUM in dc2 would 
> still be 2 nodes which means I just can't loose either of them. Adding a 
> third node is not really cost effective for the current workloads these nodes 
> need to handle.
> 
> Any advice on how to avoid this in the future? Is there a way to start up a 
> node that does not serve client requests but does replicate data?
> 
> greetings Daniel

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra data loss in come DC

[Jeff Jirsa]

Cassandra doesn't guarantee writes make it to all replicas unless you use a 
sufficiently high consistency level or run nodetool repair

What consistency level did you use for writes?
Have you run repair?


-- 
Jeff Jirsa


> On Aug 2, 2017, at 6:27 AM, Peng Xiao <2535...@qq.com> wrote:
> 
> Hi there,
> 
> We have a three DCs Cluster (two DCs with RF=3,one remote DC with RF=1),we 
> currently find that in DC1/DC2 select count(*) from t=1250,while in DC3  
> select count(*) from t=750.
> looks some data is missing in DC3(remote DC).there are no node down or 
> anything exceptional.
> we only upgrade this DC from 2.1.13 to 2.1.18,but this seems won't cause data 
> loss.
> 
> Could anyone please advise?
> 
> Thanks,
> Peng

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Cassandra data loss in come DC

[Peng Xiao]

Hi there,


We have a three DCs Cluster (two DCs with RF=3,one remote DC with RF=1),we 
currently find that in DC1/DC2 select count(*) from t=1250,while in DC3  select 
count(*) from t=750.
looks some data is missing in DC3(remote DC).there are no node down or anything 
exceptional.
we only upgrade this DC from 2.1.13 to 2.1.18,but this seems won't cause data 
loss.


Could anyone please advise?


Thanks,
Peng

Re: Bootstrapping a new Node with Consistency=ONE

[kurt greaves]

only in this one case might that work (RF==N)

Re: Bootstrapping a new Node with Consistency=ONE

[Oleksandr Shulgin]

On Wed, Aug 2, 2017 at 10:53 AM, Daniel Hölbling-Inzko <
daniel.hoelbling-in...@bitmovin.com> wrote:

>
> Any advice on how to avoid this in the future? Is there a way to start up
> a node that does not serve client requests but does replicate data?
>

Would it not work if you first increase the RF and then add the new node?

--
Alex

Re: UndeclaredThrowableException, C* 3.11

[Micha]

ok, thanks, so I'll just start it again...



On 02.08.2017 11:51, kurt greaves wrote:
> If the repair command failed, repair also failed. Regarding % repaired,
> no it's unlikely you will see 100% repaired after a single repair. Maybe
> after a few consecutive repairs with no data load you might get it to 100%.​

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Bootstrapping a new Node with Consistency=ONE

[kurt greaves]

You can't just add a new DC and then tell their clients to connect to the
new one (after migrating all the data to it obv.)? If you can't achieve
that you should probably use GossipingPropertyFileSnitch.​ Your best plan
is to have the desired RF/redundancy from the start. Changing RF in
production is not fun and can be costly.

Re: Bootstrapping a new Node with Consistency=ONE

[Daniel Hölbling-Inzko]

Thanks for the pointers Kurt!

I did increase the RF to N so that would not have been the issue.
DC migration is also a problem since I am using the Google Cloud Snitch. So
I'd have to take down the whole DC and restart anew (which would mess with
my clients as they only connect to their local DC).

As I said this was a small issue here - we only  were seeing the issue for
5 minutes. But considering how miniscule the amount of data to replicate
was (400 rows with a total of 500kb) I am a bit worried on how to do this
once loads increases.

greetings Daniel

On Wed, 2 Aug 2017 at 11:50 kurt greaves  wrote:

> If you want to change RF on a live system your best bet is through DC
> migration (add another DC with the desired # of nodes and RF), and migrate
> your clients to use that DC. There is a way to boot a node and not join the
> ring, however I don't think it will work for new nodes (have not
> confirmed), also increasing RF in this way would only not be completely
> catastrophic if you were increasing RF to N (num nodes).​
>

Re: UndeclaredThrowableException, C* 3.11

[kurt greaves]

If the repair command failed, repair also failed. Regarding % repaired, no
it's unlikely you will see 100% repaired after a single repair. Maybe after
a few consecutive repairs with no data load you might get it to 100%.​

Re: Bootstrapping a new Node with Consistency=ONE

[kurt greaves]

If you want to change RF on a live system your best bet is through DC
migration (add another DC with the desired # of nodes and RF), and migrate
your clients to use that DC. There is a way to boot a node and not join the
ring, however I don't think it will work for new nodes (have not
confirmed), also increasing RF in this way would only not be completely
catastrophic if you were increasing RF to N (num nodes).​

Bootstrapping a new Node with Consistency=ONE

[Daniel Hölbling-Inzko]

Hi,
It's probably a strange question but I have a heavily read-optimized
payload where data integrity is not a big deal. So to keep latencies low I
am reading with Consistency ONE from my Multi-DC Cluster.

Now the issue I saw is that I needed to add another Cassandra node (for
redundancy reasons).
Since I want this for renduncancy I booted the node and then changed the
Replication of my Keyspace to include the new node (all nodes have 100% of
the data).

The issue I was seeing is that clients that connected to the new Node
afterwards were seeing incomplete data - so the Key would already be
present, but the columns would all be null values.
I expect this to die down once the node is fully replicated, but in the
meantime a lot of my connected clients were in trouble. (The application
can handle seeing old data - incomplete is another matter all together)

The total data in question is a negligible 500kb (so nothing that should
really take any amount of time in my opinion but it took a few minutes for
the data to replicate over and I am still not sure everything is replicated
correctly).

Increasing the RF to something higher won't really help as the setup is
dc1: 3; dc2: 2 (I added the second node in dc2). So a LOCAL_QUORUM in dc2
would still be 2 nodes which means I just can't loose either of them.
Adding a third node is not really cost effective for the current workloads
these nodes need to handle.

Any advice on how to avoid this in the future? Is there a way to start up a
node that does not serve client requests but does replicate data?

greetings Daniel

UndeclaredThrowableException, C* 3.11

[Micha]

Hi,

has someone experienced this?


I added a fourth node to my cluster, after the boostrap I changed RP
from 2 to 3 and ran nodetool repair on the new node.

A few hours later the repair command exited with the
UndeclaredThrowableException and the node was down.

In the logs I don't see a reason for the exception or shutdown.
How can I know that the repair was successful?

There are messages with the repair id and "Session with... is complete"
and "All sessions completed" and "Sync completed using session..."


Is this an indicate for a completed repair?

The output of nodetool info shows "percent Repaired 26.18%"   Should
this be 100% after a completed repair?


Thanks,
 Michael


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org