Hi Paul,
If I understand correctly, you are making a tar file with all the
folders named "snapshots" (i.e. the folder under which all the snapshots
are created. So you have one /snapshots /folder per table).
If this is the case, when you are executing "nodetool repair", Cassandra
will create a snapshot at the beginning of the repair, creating a new
directory under each /snapshots/ directories. If this happens while you
are creating your tar file, you will get the error you saw.
If you are not yet doing it, I advise you to use the -t option of the
"nodetool snapshot" command to specify a specific name to your snapshot.
Then you should copy only the directories named
snapshots/<your_specific_snapshot_name> in your tar file.
Can you confirm that you are creating your tar file from the snapshots
directories directly (resulting in taking all the currently generated
snapshots)?
Kind regards
Reynald
On 01/06/2016 13:27, Paul Dunkler wrote:
I guess this might come from the incremental repairs...
The repair time is stored in the sstable (RepairedAt timestamp metadata).
By the way: We are not using incremental repairs at all. So can't be
the case here.
It really seems like there is somewhat that can still change data in
snapshot directories. I feel like it's something to do with flushing /
compaction. But no clue, what... :(
Cheers,
Reynald
On 31/05/2016 11:03, Paul Dunkler wrote:
Hi there,
i am sometimes running in very strange errors while backing up
snapshots from a cassandra cluster.
Cassandra version:
2.1.11
What i basically do:
1. nodetool snapshot
2. tar all snapshot folders into one file
3. transfer them to another server
What happens is that tar just sometimes give the error message "file
changed as we read it" while its adding a .db-file from the folder
of the previously created snapshot.
If i understand everything correct, this SHOULD never happen.
Snapshots should be totally immutable, right?
Am i maybe hitting a bug or is there some rare case with running
repair operations or what-so-ever which can change snapshotted data?
I already searched through cassandra jira but couldn't find a bug
which looks related to this behaviour.
Would love to get some help on this.
—
Paul Dunkler
—
Paul Dunkler
