Re: Cassandra Files Taking up Much More Space than CF

[Reynald Bourtembourg]

Hi Nate,

Are you using incremental backups?

Extract from the documentation ( 
http://www.datastax.com/documentation/cassandra/2.1/cassandra/operations/ops_backup_incremental_t.html 
):


/When incremental backups are enabled (disabled by default), Cassandra 
hard-links each flushed SSTable to a backups directory under the 
keyspace data directory. This allows storing backups offsite without 
transferring entire snapshots. Also, incremental backups combine with 
snapshots to provide a dependable, up-to-date backup mechanism./

//

/As with snapshots, Cassandra does not automatically clear incremental 
backup files. *DataStax recommends setting up a process to clear 
incremental backup hard-links each time a new snapshot is created.*/


These backups are stored in directories named backups at the same 
level as the snapshots' directories.


Reynald

On 09/12/2014 18:13, Nate Yoder wrote:
Thanks for the advice.  Totally makes sense.  Once I figure out how to 
make my data stop taking up more than 2x more space without being 
useful I'll definitely make the change :)


Nate



--
*Nathanael Yoder*
Principal Engineer  Data Scientist, Whistle
415-944-7344 // n...@whistle.com mailto:n...@whistle.com

On Tue, Dec 9, 2014 at 9:02 AM, Jonathan Haddad j...@jonhaddad.com 
mailto:j...@jonhaddad.com wrote:


Well, I personally don't like RF=2.  It means if you're using
CL=QUORUM and a node goes down, you're going to have a bad time.
(downtime) If you're using CL=ONE then you'd be ok. However, I am
not wild about losing a node and having only 1 copy of my data
available in prod.


On Tue Dec 09 2014 at 8:40:37 AM Nate Yoder n...@whistle.com
mailto:n...@whistle.com wrote:

Thanks Jonathan.  So there is nothing too idiotic about my
current set-up with 6 boxes each with 256 vnodes each and a RF
of 2?

I appreciate the help,
Nate



--
*Nathanael Yoder*
Principal Engineer  Data Scientist, Whistle
415-944-7344 // n...@whistle.com mailto:n...@whistle.com

On Tue, Dec 9, 2014 at 8:31 AM, Jonathan Haddad
j...@jonhaddad.com mailto:j...@jonhaddad.com wrote:

You don't need a prime number of nodes in your ring, but
it's not a bad idea to it be a multiple of your RF when
your cluster is small.


On Tue Dec 09 2014 at 8:29:35 AM Nate Yoder
n...@whistle.com mailto:n...@whistle.com wrote:

Hi Ian,

Thanks for the suggestion but I had actually already
done that prior to the scenario I described (to get
myself some free space) and when I ran nodetool
cfstats it listed 0 snapshots as expected, so
unfortunately I don't think that is where my space went.

One additional piece of information I forgot to point
out is that when I ran nodetool status on the node it
included all 6 nodes.

I have also heard it mentioned that I may want to have
a prime number of nodes which may help protect against
split-brain.  Is this true?  If so does it still apply
when I am using vnodes?

Thanks again,
Nate

--
*Nathanael Yoder*
Principal Engineer  Data Scientist, Whistle
415-944-7344 // n...@whistle.com mailto:n...@whistle.com

On Tue, Dec 9, 2014 at 7:42 AM, Ian Rose
ianr...@fullstory.com mailto:ianr...@fullstory.com
wrote:

Try `nodetool clearsnapshot` which will delete any
snapshots you have.  I have never taken a snapshot
with nodetool yet I found several snapshots on my
disk recently (which can take a lot of space).  So
perhaps they are automatically generated by some
operation? No idea.  Regardless, nuking those
freed up a ton of space for me.

- Ian


On Mon, Dec 8, 2014 at 8:12 PM, Nate Yoder
n...@whistle.com mailto:n...@whistle.com wrote:

Hi All,

I am new to Cassandra so I apologise in
advance if I have missed anything obvious but
this one currently has me stumped.

I am currently running a 6 node Cassandra
2.1.1 cluster on EC2 using C3.2XLarge nodes
which overall is working very well for us.
However, after letting it run for a while I
seem to get into a situation where the amount
of disk space used far exceeds the total
amount of data on each node and I haven't been

Re: Repairing OpsCenter rollups60 Results in Snapshot Errors

[Reynald Bourtembourg]

Hi Paul,

There is a JIRA ticket about this issue:
https://issues.apache.org/jira/browse/CASSANDRA-8696

I have seen these errors too the last time I ran nodetool repair.
I would also be interested to know the answer to the questions you were 
asking:


Are these errors problematic? Should I just let the repair process 
continue for however long it takes? 

I am wondering whether this is making the repair ineffectual.

Best regards

Reynald


On 29/01/2015 17:03, Paul Nickerson wrote:
I am running a 6 node cluster using Apache Cassandra 2.1.2 with 
DataStax OpsCenter 5.0.2 from the AWS EC2 AMI DataStax 
Auto-Clustering AMI 2.5.1-hvm (DataStax Community AMI). When I try to 
run a repair on the rollups60 column family in the OpsCenter keyspace, 
I get errors about failed snapshot creation in the Cassandra system 
log. The repair seems to continue, and then finishes with errors.


I am wondering whether this is making the repair ineffectual.

I am running the command

nodetool repair OpsCenter rollups60

on one of the nodes (10.63.74.70). From the command, I get this output:

[2015-01-23 19:36:06,261] Starting repair command #9, repairing 
511 ranges for keyspace OpsCenter (seq=true, full=true)
[2015-01-23 21:08:16,242] Repair session 
67772db0-a337-11e4-9e78-37e5027a626b for range 
(5848435723460298978,5868916338423419522] failed with error 
java.io.IOException: Failed during snapshot creation.


The error is repeated many times, and they all appear right at the 
end. Here is an example of what I see in the log on that same system 
(the one that I'm running the command from, and the one that's trying 
to snapshot):


INFO  [AntiEntropyStage:1] 2015-01-23 19:38:28,235 
RepairSession.java:171 - [repair 
#138b42e0-a337-11e4-9e78-37e5027a626b] Received merkle tree for 
rollups60 from /10.63.74.70 http://10.63.74.70
INFO  [AntiEntropySessions:9] 2015-01-23 19:38:28,236 
RepairSession.java:260 - [repair 
#67772db0-a337-11e4-9e78-37e5027a626b] new session: will sync 
/10.63.74.70 http://10.63.74.70, /10.51.180.16 http://10.51.180.16 
on range (5848435723460298978,5868916338423419522] for 
OpsCenter.[rollups60]
INFO  [RepairJobTask:3] 2015-01-23 19:38:28,237 
Differencer.java:74 - [repair #138b42e0-a337-11e4-9e78-37e5027a626b] 
Endpoints /10.13.157.190 http://10.13.157.190 and /10.63.74.70 
http://10.63.74.70 have 1 range(s) out of sync for rollups60
INFO  [AntiEntropyStage:1] 2015-01-23 19:38:28,237 
ColumnFamilyStore.java:840 - Enqueuing flush of rollups60: 465365 (0%) 
on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:25] 2015-01-23 19:38:28,238 
Memtable.java:325 - Writing Memtable-rollups60@204861223(51960 
serialized bytes, 1395 ops, 0%/0% of on/off-heap limit)
INFO  [RepairJobTask:3] 2015-01-23 19:38:28,239 
StreamingRepairTask.java:68 - [streaming task 
#138b42e0-a337-11e4-9e78-37e5027a626b] Performing streaming repair of 
1 ranges with /10.13.157.190 http://10.13.157.190
INFO  [MemtableFlushWriter:25] 2015-01-23 19:38:28,262 
Memtable.java:364 - Completed flushing 
/raid0/cassandra/data/OpsCenter/rollups60-445613507ca411e4bd3f1927a2a71193/OpsCenter-rollups60-ka-331933-Data.db 
(29998 bytes) for commitlog position 
ReplayPosition(segmentId=1422038939094, position=31047766)
ERROR [RepairJobTask:2] 2015-01-23 19:38:39,067 RepairJob.java:127 
- Error occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at 
/10.63.74.70 http://10.63.74.70
at 
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77) 
~[apache-cassandra-2.1.2.jar:2.1.2]
at 
org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347) 
~[apache-cassandra-2.1.2.jar:2.1.2]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_51]
at 
java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_51]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_51]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_51]

at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
INFO  [AntiEntropySessions:10] 2015-01-23 19:38:39,068 
RepairSession.java:260 - [repair 
#6dec29c0-a337-11e4-9e78-37e5027a626b] new session: will sync 
/10.63.74.70 http://10.63.74.70, /10.51.180.16 http://10.51.180.16 
on range (-6918744323658665195,-6916171087863528821] for 
OpsCenter.[rollups60]
ERROR [AntiEntropySessions:9] 2015-01-23 19:38:39,068 
RepairSession.java:303 - [repair 
#67772db0-a337-11e4-9e78-37e5027a626b] session completed with the 
following error

java.io.IOException: Failed during snapshot creation.
at 
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) 
~[apache-cassandra-2.1.2.jar:2.1.2]
at 
org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128) 

Re: [Marketing Mail] Migrating to incremental repairs

[Reynald Bourtembourg]

Well,

By re-reading my e-mail, I understood the rationale behind doing a full 
sequential repair for each node.
I was confused by the fact that in our case, we have 3 nodes with RF = 
3, so all the nodes are storing all replicas.

So we are in a special case.
As soon as you have more than 3 nodes, this is no longer the case.

In any case, in our special case (3 nodes and RF=3), could we apply the 
following migration procedure?:

- disableautocompaction on all nodes at the same time
 - run the full sequential repair
 - For each node:
- stop the node
- Use the tool sstablerepairedset to mark all the SSTables that 
were created before you disabled compaction.

- Restart Cassandra

I'd be glad if someone could answer to my other questions in any case ;-).

Thanks in advance for your help

Reynald


On 18/11/2015 16:45, Reynald Bourtembourg wrote:

Hi,

We currently have a 3 nodes Cassandra cluster with RF = 3.
We are using Cassandra 2.1.7.

We would like to start using incremental repairs.
We have some tables using LCS compaction strategy and some others 
using STCS.


Here is the procedure written in the documentation:

To migrate to incremental repair, one node at a time:

 1. Disable compaction on the node using nodetool disableautocompaction.
 2. Run the default full, sequential repair.
 3. Stop the node.
 4. Use the tool sstablerepairedset to mark all the SSTables that were
created before you disabled compaction.
 5. Restart cassandra


In our case, a full sequential repair takes about 5 days.
If I follow the procedure described above and if my understanding is 
correct, it's gonna take at least 15 days (3 repairs of 5 days) before 
to be able to use the incremental repairs, right(?), since we need to 
do it one node at a time (one full sequential repair per node?).


If my understanding is correct, what is the rationale behind the fact 
that we need to run a full sequential repair once for each node?
I understood a full sequential repair would repair all the sstables on 
all the nodes. So doing it only once should be enough, right?


Is it possible to do the following instead of what is written in the 
documentation?:

 - disableautocompaction on all nodes at the same time
 - run the full sequential repair
 - For each node:
- stop one node
- Use the tool sstablerepairedset to mark all the SSTables 
that were created before you disabled compaction.

- Restart Cassandra
Without having to run the full sequential repair 3 times?

The documentation states that if we don't execute this migration 
procedure, the first time we will run incremental repair, Cassandra 
will perform size-tiering on all SSTables because the 
repair/unrepaired status is unknown and this operation can take a long 
time.

Do you think this operation could take more than 15 days in our case?

I understood that we only need to use sstablerepairedset on the 
SSTables related to the tables using LCS compaction strategy and which 
were created before the auto compaction was disabled.

Is my understanding correct?

The documentation is not very explicit but I suppose the following 
sentence:
"4. Use the tool sstablerepairedset to mark all the SSTables that were 
created before you disabled compaction."
means we need to invoke "sstablerepairedset --is-repaired -f 
list_of_sstable_names.txt" on the LCS SSTables that were created 
before the compaction was disabled.


Is this correct?

Do we need to enableautocompaction again after the Cassandra restart 
or is it done automatically?


Would you recommend us to upgrade our Cassandra version before 
starting the incremental repair migration?


Thank you for your help and sorry for the long e-mail.

Reynald

Migrating to incremental repairs

[Reynald Bourtembourg]

Hi,

We currently have a 3 nodes Cassandra cluster with RF = 3.
We are using Cassandra 2.1.7.

We would like to start using incremental repairs.
We have some tables using LCS compaction strategy and some others using 
STCS.


Here is the procedure written in the documentation:

To migrate to incremental repair, one node at a time:

1. Disable compaction on the node using nodetool disableautocompaction.
2. Run the default full, sequential repair.
3. Stop the node.
4. Use the tool sstablerepairedset to mark all the SSTables that were
   created before you disabled compaction.
5. Restart cassandra


In our case, a full sequential repair takes about 5 days.
If I follow the procedure described above and if my understanding is 
correct, it's gonna take at least 15 days (3 repairs of 5 days) before 
to be able to use the incremental repairs, right(?), since we need to do 
it one node at a time (one full sequential repair per node?).


If my understanding is correct, what is the rationale behind the fact 
that we need to run a full sequential repair once for each node?
I understood a full sequential repair would repair all the sstables on 
all the nodes. So doing it only once should be enough, right?


Is it possible to do the following instead of what is written in the 
documentation?:

 - disableautocompaction on all nodes at the same time
 - run the full sequential repair
 - For each node:
- stop one node
- Use the tool sstablerepairedset to mark all the SSTables that 
were created before you disabled compaction.

- Restart Cassandra
Without having to run the full sequential repair 3 times?

The documentation states that if we don't execute this migration 
procedure, the first time we will run incremental repair, Cassandra will 
perform size-tiering on all SSTables because the repair/unrepaired 
status is unknown and this operation can take a long time.

Do you think this operation could take more than 15 days in our case?

I understood that we only need to use sstablerepairedset on the SSTables 
related to the tables using LCS compaction strategy and which were 
created before the auto compaction was disabled.

Is my understanding correct?

The documentation is not very explicit but I suppose the following sentence:
"4. Use the tool sstablerepairedset to mark all the SSTables that were 
created before you disabled compaction."
means we need to invoke "sstablerepairedset --is-repaired -f 
list_of_sstable_names.txt" on the LCS SSTables that were created before 
the compaction was disabled.


Is this correct?

Do we need to enableautocompaction again after the Cassandra restart or 
is it done automatically?


Would you recommend us to upgrade our Cassandra version before starting 
the incremental repair migration?


Thank you for your help and sorry for the long e-mail.

Reynald

Re: [Marketing Mail] Re: [Marketing Mail] can't make any permissions change in 2.2.4

[Reynald Bourtembourg]

Done:
https://issues.apache.org/jira/browse/CASSANDRA-10904


On 18/12/2015 10:51, Sylvain Lebresne wrote:
On Fri, Dec 18, 2015 at 8:55 AM, Reynald Bourtembourg 
<reynald.bourtembo...@esrf.fr <mailto:reynald.bourtembo...@esrf.fr>> 
wrote:


This does not seem to be explained in the Cassandra 2.2 Upgrading
section of the NEWS.txt file:


https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-2.2.4


This is indeed an oversight. Would you mind opening a JIRA ticket so 
we don't forget to add it now?


--
Sylvain

Re: [Marketing Mail] can't make any permissions change in 2.2.4

[Reynald Bourtembourg]

Hi,

Maybe your problem comes from the new role based access control in 
Cassandra introduced in Cassandra 2.2.


http://www.datastax.com/dev/blog/role-based-access-control-in-cassandra

The /Upgrading /section of this blog post is specifying the following:

"/For systems already using the internal auth implementations, the 
process for converting existing data during a rolling upgrade is 
straightforward. As each node is restarted, it will attempt to convert 
any data in the legacy tables into the new schema. Until enough nodes to 
satisfy the replication strategy for the //|system_auth|//keyspace are 
upgraded and so have the new schema, this conversion will fail with the 
failure being reported in the system log. During the upgrade, 
Cassandra’s internal auth classes will continue to use the legacy 
tables, so clients experience no disruption. Issuing //DCL 
//statements during 
an upgrade is not supported. Once all nodes are upgraded, an operator 
with superuser privileges should drop the legacy tables, which will 
prompt Cassandra to switch over to the new tables without requiring any 
further intervention./


//

/As I mentioned, this is just a part of a wider reworking of the auth 
subsystem in Cassandra planned for inclusion in the 2.2 release. You can 
check out more detail and follow progress in //CASSANDRA-8394 
//./"


This does not seem to be explained in the Cassandra 2.2 Upgrading 
section of the NEWS.txt file:


https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-2.2.4

Hoping this helps,
Reynald

On 17/12/2015 18:10, Kai Wang wrote:
I used to able to add/drop users and modify permissions in 2.1.1. 
After upgrading to 2.2.4, I can't modify any of those. "List all 
permissions" returns me all the permissions I setup before the 
upgrade. But I can't add new permission or add new users in cqlsh. 
"create user" and "grant" didn't report any errors.

Re: [Marketing Mail] Migrating to incremental repairs

[Reynald Bourtembourg]

Dear Lorina,

Thank you very much for your answer and to your support team.
I understand why you won't document the special case RF = N = 3. Thank 
you for clarifying this special case.


I've found this answer from Marcus Eriksson stating that autocompaction 
should be reenabled after the migration:

https://www.mail-archive.com/user@cassandra.apache.org/msg40305.html

It would be nice to add this step in the documentation if it's not 
automatic after the cassandra restart.


What about what Stefano was asking? (thanks by the way, Stefano!)
Can anyone confirm the recent versions of Cassandra (from which version) 
do not require anything specific but the -inc switch when migrating to 
incremental repair?
I can understand the migration procedure is not mandatory, especially 
when there is not much data to migrate, but the documentation seems to 
imply that if the migration steps are not done, the first incremental 
repair could take a very long time.

Can anyone clarify this point please?
Did anyone try incremental repairs without the migration procedure with 
a sensible amount of data to migrate?

How much longer did it take?

Thank you very much for your help

Kind regards

Reynald

On 19/11/2015 22:43, Lorina Poland wrote:

Hi Reynald,

I asked Support about your email. They said that since your RF = N (3 
nodes, 3 replicates), one full repair on any node will suffice. You 
are an edge case, so we won't be documenting this solution.
As for your other questions, I'm guessing you are not a DataStax 
customer. I would recommend the Apache Cassandra user forum 
(http://www.mail-archive.com/user@cassandra.apache.org/), where you 
can ask your questions and receive feedback. Stackoverflow is another 
good forum for Cassandra questions 
(http://stackoverflow.com/questions/tagged/cassandra).


Sincerely,
Lorina Poland

datastax_logo.png <http://www.datastax.com/>

LORINA POLAND

Senior Technical Writer | lor...@datastax.com <mailto:lor...@datastax.com>

linkedin.png <https://www.linkedin.com/in/lorinapoland>twitter.png 
<https://twitter.com/datastax>


<http://cassandrasummit-datastax.com/?utm_campaign=summit15_medium=summiticon_source=emailsignature>

On Thu, Nov 19, 2015 at 3:13 AM, Stefano Ortolani <ostef...@gmail.com 
<mailto:ostef...@gmail.com>> wrote:


As far as I know, docs is quite inconsistent on the matter.
Based on some research here and on IRC, recent versions of
Cassandra do no require anything specific when migrating to
incremental repairs but the the -inc switch even on LCS.
Any confirmation on the matter is more than welcome.
Regards,
Stefano
    On Wed, Nov 18, 2015 at 3:59 PM, Reynald Bourtembourg
<reynald.bourtembo...@esrf.fr
<mailto:reynald.bourtembo...@esrf.fr>> wrote:

Well, By re-reading my e-mail, I understood the rationale
behind doing a full sequential repair for each node. I was
confused by the fact that in our case, we have 3 nodes with RF
= 3, so all the nodes are storing all replicas. So we are in a
special case. As soon as you have more than 3 nodes, this is
no longer the case. In any case, in our special case (3 nodes
and RF=3), could we apply the following migration procedure?:-
disableautocompaction on all nodes at the same time  - run the
full sequential repair  - For each node: - stop the
node- Use the tool sstablerepairedset to mark all the
SSTables that were created before you disabled compaction.
- Restart Cassandra I'd be glad if someone could answer to

my other questions in any case ;-). Thanks in advance for your
    helpReynald
On 18/11/2015 16:45, Reynald Bourtembourg wrote:

Hi, We currently have a 3 nodes Cassandra cluster with RF =
3. We are using Cassandra 2.1.7. We would like to start using
incremental repairs. We have some tables using LCS compaction
strategy and some others using STCS. Here is the procedure
written in the documentation:

To migrate to incremental repair, one node at a time:

 1. Disable compaction on the node using nodetool
disableautocompaction.
 2. Run the default full, sequential repair.
 3. Stop the node.
 4. Use the tool sstablerepairedset to mark all the SSTables
that were created before you disabled compaction.
 5. Restart cassandra

In our case, a full sequential repair takes about 5 days. If
I follow the procedure described above and if my
understanding is correct, it's gonna take at least 15 days (3
repairs of 5 days) before to be able to use the incremental
repairs, right(?), since we need to do it one node at a time
(one full sequential repair per node?). If my understanding
is correct, what is the rationale behind the fact that we

Re: [Marketing Mail] Cassandra 2.1: Snapshot data changing while transferring

[Reynald Bourtembourg]

Hi Paul,

I guess this might come from the incremental repairs...
The repair time is stored in the sstable (RepairedAt timestamp metadata).

Cheers,
Reynald

On 31/05/2016 11:03, Paul Dunkler wrote:

Hi there,

i am sometimes running in very strange errors while backing up 
snapshots from a cassandra cluster.


Cassandra version:
2.1.11

What i basically do:
1. nodetool snapshot
2. tar all snapshot folders into one file
3. transfer them to another server

What happens is that tar just sometimes give the error message "file 
changed as we read it" while its adding a .db-file from the folder of 
the previously created snapshot.
If i understand everything correct, this SHOULD never happen. 
Snapshots should be totally immutable, right?


Am i maybe hitting a bug or is there some rare case with running 
repair operations or what-so-ever which can change snapshotted data?
I already searched through cassandra jira but couldn't find a bug 
which looks related to this behaviour.


Would love to get some help on this.

—
Paul Dunkler

Re: [Marketing Mail] Re: [Marketing Mail] Cassandra 2.1: Snapshot data changing while transferring

[Reynald Bourtembourg]

Hi Paul,

If I understand correctly, you are making a tar file with all the 
folders named "snapshots" (i.e. the folder under which all the snapshots 
are created. So you have one /snapshots /folder per table).
If this is the case, when you are executing "nodetool repair", Cassandra 
will create a snapshot at the beginning of the repair, creating a new 
directory under each /snapshots/ directories. If this happens while you 
are creating your tar file, you will get the error you saw.


If you are not yet doing it, I advise you to use the -t option of the 
"nodetool snapshot" command to specify a specific name to your snapshot.
Then you should copy only the directories named 
snapshots/ in your tar file.


Can you confirm that you are creating your tar file from the snapshots 
directories directly (resulting in taking all the currently generated 
snapshots)?


Kind regards

Reynald

On 01/06/2016 13:27, Paul Dunkler wrote:

I guess this might come from the incremental repairs...
The repair time is stored in the sstable (RepairedAt timestamp metadata).


By the way: We are not using incremental repairs at all. So can't be 
the case here.


It really seems like there is somewhat that can still change data in 
snapshot directories. I feel like it's something to do with flushing / 
compaction. But no clue, what... :(




Cheers,
Reynald

On 31/05/2016 11:03, Paul Dunkler wrote:

Hi there,

i am sometimes running in very strange errors while backing up 
snapshots from a cassandra cluster.


Cassandra version:
2.1.11

What i basically do:
1. nodetool snapshot
2. tar all snapshot folders into one file
3. transfer them to another server

What happens is that tar just sometimes give the error message "file 
changed as we read it" while its adding a .db-file from the folder 
of the previously created snapshot.
If i understand everything correct, this SHOULD never happen. 
Snapshots should be totally immutable, right?


Am i maybe hitting a bug or is there some rare case with running 
repair operations or what-so-ever which can change snapshotted data?
I already searched through cassandra jira but couldn't find a bug 
which looks related to this behaviour.


Would love to get some help on this.

—
Paul Dunkler




—
Paul Dunkler

Re: [Marketing Mail] Re: Memory leak and lockup on our 2.2.7 Cassandra cluster.

[Reynald Bourtembourg]

Hi,

Maybe Ben was referring to this issue which has been mentioned recently 
on this mailing list:

https://issues.apache.org/jira/browse/CASSANDRA-11887

Cheers,
Reynald

On 03/08/2016 18:09, Romain Hardouin wrote:

>Curious why the 2.2 to 3.x upgrade path is risky at best.
I guess that upgrade from 2.2 is less tested by DataStax QA because 
DSE4 used C* 2.1, not 2.2.

I would say the safest upgrade is 2.1 to 3.0.x

Best,

Romain

Re: 回复： data loss in different DC

[Reynald Bourtembourg]

Do you mean how often we should run repairs in this situation (write at 
CL=EACH_QUORUM and CL=LOCAL_QUORUM)?


I don't consider myself as an expert in this domain, so I will let the 
real experts answer to this question...
You can also refer to this mailing list history (many threads on this 
snbject) and to the recommendations in the documentation:

http://cassandra.apache.org/doc/latest/operating/repair.html#usage-and-best-practices

Kind regards
Reynald


On 28/09/2017 14:49, Jacob Shadix wrote:

How often are you running repairs?

-- Jacob Shadix

On Thu, Sep 28, 2017 at 7:53 AM, Reynald Bourtembourg 
<reynald.bourtembo...@esrf.fr <mailto:reynald.bourtembo...@esrf.fr>> 
wrote:


Hi,

You can write with CL=EACH_QUORUM and read with CL=LOCAL_QUORUM to
get strong consistency.

Kind regards,
Reynald


On 28/09/2017 13:46, Peng Xiao wrote:

even with CL=QUORUM,there is no guarantee to be sure to read the
same data in DC2,right?
then multi DCs looks make no sense?


-- 原始邮件 --
*发件人:* "DuyHai Doan";<doanduy...@gmail.com>
<mailto:doanduy...@gmail.com>;
*发送时间:* 2017年9月28日(星期四) 下午5:45
*收件人:* "user"<user@cassandra.apache.org>
<mailto:user@cassandra.apache.org>;
*主题:* Re: data loss in different DC

If you're writing into DC1 with CL = LOCAL_xxx, there is no
guarantee to be sure to read the same data in DC2. Only repair
will help you

On Thu, Sep 28, 2017 at 11:41 AM, Peng Xiao <2535...@qq.com
<mailto:2535...@qq.com>> wrote:

Dear All,

We have a cluster with one DC1:RF=3,another DC DC2:RF=1 only
for ETL,but we found that sometimes we can query records in
DC1,while not able not find the same record in DC2 with
local_quorum.How it happens?
Could anyone please advise?
looks we can only run repair to fix it.

Thanks,
Peng Xiao

Re: 回复： data loss in different DC

[Reynald Bourtembourg]

Hi,

You can write with CL=EACH_QUORUM and read with CL=LOCAL_QUORUM to get 
strong consistency.


Kind regards,
Reynald

On 28/09/2017 13:46, Peng Xiao wrote:
even with CL=QUORUM,there is no guarantee to be sure to read the same 
data in DC2,right?

then multi DCs looks make no sense?


--?0?2?0?2--
*??:*?0?2"DuyHai Doan";;
*:*?0?22017??9??28??(??) 5:45
*??:*?0?2"user";
*:*?0?2Re: data loss in different DC

If you're writing into DC1 with CL = LOCAL_xxx, there is no guarantee 
to be sure to read the same data in DC2. Only repair will help you


On Thu, Sep 28, 2017 at 11:41 AM, Peng Xiao <2535...@qq.com 
> wrote:


Dear All,

We have a cluster with one DC1:RF=3,another DC DC2:RF=1 only for
ETL,but we found that sometimes we can query records in DC1,while
not able not find the same record in DC2 with local_quorum.How it
happens?
Could anyone please advise?
looks we can only run repair to fix it.

Thanks,
Peng Xiao