Another thing to keep in mind is that if you are hitting the issue I
described, waiting 60 seconds will not absolutely solve your problem, it
will only make it less likely to occur. If a memtable has been partially
flushed at the 60 second mark you will end up with the same corrupt sstable.
On F
+1 for tablesnap
On Fri, Mar 28, 2014 at 4:28 PM, Jonathan Haddad wrote:
> I will +1 the recommendation on using tablesnap over EBS. S3 is at least
> predictable.
>
> Additionally, from a practical standpoint, you may want to back up your
> sstables somewhere. If you use S3, it's easy to pull
As I tried to say, EBS snapshots require much care or you get corruption
such as you have encountered.
Does Cassandra quiesce the file system after a snapshot using fsfreeze or
xfs_freeze? Somehow I doubt it...
On Fri, Mar 28, 2014 at 4:17 PM, Jonathan Haddad wrote:
> I have a nagging memory o
I will +1 the recommendation on using tablesnap over EBS. S3 is at least
predictable.
Additionally, from a practical standpoint, you may want to back up your
sstables somewhere. If you use S3, it's easy to pull just the new tables
out via aws-cli tools (s3 sync), to your remote, non-aws server,
I have a nagging memory of reading about issues with virtualization and not
actually having durable versions of your data even after an fsync (within
the VM). Googling around lead me to this post:
http://petercai.com/virtualization-is-bad-for-database-integrity/
It's possible you're hitting this
Robert,
That is what I thought as well. But apparently something is happening. The
only way I can get away with doing this is adding a sleep 60 right after the
nodetool snapshot is executed. I can reproduce this 100% of the time by not
issuing a sleep after nodetool snapshot.
This is the er
On Fri, Mar 28, 2014 at 12:21 PM, Russ Lavoie wrote:
> Thank you for your quick response.
>
> Is there a way to tell when a snapshot is completely done?
>
IIRC, the JMX call blocks until the snapshot completes. It should be done
when nodetool returns.
=Rob
In your step 4, be sure you create a consistent EBS snapshot. You may have
pieces of your sstables that have not actually been flushed all the way to
EBS.
See https://github.com/alestic/ec2-consistent-snapshot
ml
On Fri, Mar 28, 2014 at 3:21 PM, Russ Lavoie wrote:
> Thank you for your quick r
Thank you for your quick response.
Is there a way to tell when a snapshot is completely done?
On Friday, March 28, 2014 1:30 PM, Robert Coli wrote:
On Fri, Mar 28, 2014 at 11:15 AM, Russ Lavoie wrote:
We are using cassandra 1.2.10 (With JNA installed) on ubuntu 12.04.3 and are
running our
On Fri, Mar 28, 2014 at 11:15 AM, Russ Lavoie wrote:
> We are using cassandra 1.2.10 (With JNA installed) on ubuntu 12.04.3 and
> are running our instances in Amazon Web Services.
>
> Our cassandra systems data is on an EBS volume
>
Best practice for Cassandra on AWS is to run on ephemeral str
10 matches
Mail list logo