> Alex, just want to make sure that I understand your point correctly. Are you
> suggesting this sequence of operations with TCM?
>
> * Make config changes
> * Do the initial data copy
> * Make destination part of write placements (same as source)
> * Start destination instance
> * Decommission
On Wed, May 1, 2024 at 3:34 AM Alex Petrov wrote:
>
> We can implement CEP-40 using a similar approach: we can leave the source
> node as both a read and write target, and allow the new node to be a target
> for (pending) writes. Unfortunately, this does not help with availability
> (in fact, it
Replies from my side for the other points of the discussion:
*Managing C* life cycle with Sidecar*
>lifecycle / orchestration portion is the more challenging aspect. It would
be nice to address that as well so we don’t end up with something like
repair where the building blocks are there but the
Thank you for input!
> Would it be possible to create a new type of write target node? The new
> write target node is notified of writes (like any other write node) but does
> not participate in the write availability calculation.
We could make a some kind of optional write, but
Alex,
you write:
> We can implement CEP-40 using a similar approach: we can leave the source
> node as both a read and write target, and allow the new node to be a target
> for (pending) writes. Unfortunately, this does not help with availability
> (in fact, it decreases write availability,
Thank you for submitting this CEP!
Wanted to discuss this point from the description:
> How to bring up/down Cassandra/Sidecar instances or making/applying config
> changes are outside the scope of this document.
One advantage of doing migration via sidecar is the fact that we can stream
Just realised that for this live migration case, I think it is not just
about validating whether the SSTables present in local are valid or not,
but also about validating whether the destination has identical file as
source. Then in that case binary level content verification needs to be
done
>In the next iteration, I am going to address the below point.
>> I would like to see more abstraction of how the files get moved / put in
place with the proposed solution being the default implementation. That
would allow others to plug in alternatives means of data movement like
pulling down
On Tue, Apr 23, 2024 at 11:37 AM Venkata Hari Krishna Nukala <
n.v.harikrishna.apa...@gmail.com> wrote:
> reason why I called out binary level verification out of initial scope is
> because of these two reasons: 1) Calculating digest for each file may
> increase CPU utilisation and 2) Disk would
our implementation of this concept. It has the
> >>> added benefit that the backup / restore path gets exercised much more
> >>> regularly than it would in normal operations, finding edge case bugs at a
> >>> time when you still have other ways of rec
> might get roasted for scope creep
This community *would never*.
What you've outlined seems like a very reasonable stretch goal or v2 to keep in
mind so we architect something in v1 that's also supportive of a v2 keyspace
only migration.
On Thu, Apr 25, 2024, at 1:57 PM, Venkata Hari Krishna
than it would in normal operations, finding edge case bugs at a
>>> time when you still have other ways of recovering rather than in a full
>>> disaster scenario.
>>>
>>>
>>>
>>> Cheers
>>>
>>> Ben
>>>
>>>
>
ill have other ways of recovering rather than in a full
>> disaster scenario.
>>
>>
>>
>> Cheers
>>
>> Ben
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From: *Jordan West
>> *Date: *Sunday, 21 Ap
> *Date: *Sunday, 21 April 2024 at 05:38
> *To: *dev@cassandra.apache.org
> *Subject: *Re: [DISCUSS] CEP-40: Data Transfer Using Cassandra Sidecar
> for Live Migrating Instances
>
> *EXTERNAL EMAIL - USE CAUTION when clicking links or attachments *
>
>
>
> I do really like th
disaster scenario.
Cheers
Ben
From: Jordan West
Date: Sunday, 21 April 2024 at 05:38
To: dev@cassandra.apache.org
Subject: Re: [DISCUSS] CEP-40: Data Transfer Using Cassandra Sidecar for Live
Migrating Instances
EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
I do really like
I do really like the framing of replacing a node is restoring a node and
then kicking off a replace. That is effectively what we do internally.
I also agree we should be able to do data movement well both internal to
Cassandra and externally for a variety of reasons.
We’ve seen great performance
Jeff, this is probably the best explanation and justification of the idea
that I've heard so far.
I like it because
1) we really should have something official for backups
2) backups / object store would be great for analytics
3) it solves a much bigger problem than the single goal of moving
I think Jordan and German had an interesting insight, or at least their comment made me think about this slightly differently, and I’m going to repeat it so it’s not lost in the discussion about zerocopy / sendfile.The CEP treats this as “move a live instance from one machine to another”. I know
On Thu, Apr 18, 2024 at 12:46 PM Ariel Weisberg wrote:
>
> If there is a faster/better way to replace a node why not have Cassandra
> support that natively without the sidecar so people who aren’t running the
> sidecar can benefit?
>
I am not the author of the CEP so take whatever I say with a
On Fri, Apr 19, 2024 at 3:12 PM Jon Haddad wrote:
> I haven't looked at streaming over TLS, so I might be way off base here,
> but our own docs (
> https://cassandra.apache.org/doc/latest/cassandra/architecture/streaming.html)
> say ZCS is not available when using encryption, and if we have to
I haven't looked at streaming over TLS, so I might be way off base here,
but our own docs (
https://cassandra.apache.org/doc/latest/cassandra/architecture/streaming.html)
say ZCS is not available when using encryption, and if we have to bring the
data into the JVM then I'm not sure how it would
These are the salient points here for me, yes:> My understanding from the proposal is that Sidecar would be able to migrate from a Cassandra instance that is already dead and cannot recover.> That’s one thing I like about having it an external process — not that it’s bullet proof but it’s one less
If we are considering the main process then we have to do some additional
work to ensure that it doesn’t put pressure on the JVM and introduce
latency. That’s one thing I like about having it an external process — not
that it’s bullet proof but it’s one less thing to worry about.
Jordan
On Thu,
My understanding from the proposal is that Sidecar would be able to migrate
from a Cassandra instance that is already dead and cannot recover. This is a
scenario that is possible where Sidecar should still be able to migrate to a new
instance.
Alternatively, Cassandra itself could have some flag
Hmm... I guess if you're using encryption you can't use ZCS so there's that.
It probably makes sense to implement kernel TLS:
https://www.kernel.org/doc/html/v5.7/networking/tls.html
Then we can get ZCS all the time, for bootstrap & replacements.
Jon
On Thu, Apr 18, 2024 at 12:50 PM Jon
Ariel, having it in C* process makes sense to me.
Please correct me if I'm wrong here, but shouldn't using ZCS to transfer
have no distinguishable difference in overhead from doing it using the
sidecar? Since the underlying call is sendfile, never hitting userspace, I
can't see why we'd opt for
Hi,
If there is a faster/better way to replace a node why not have Cassandra
support that natively without the sidecar so people who aren’t running the
sidecar can benefit?
Copying files over a network shouldn’t be slow in C* and it would also already
have all the connectivity issues solved.
14, 2024 12:27 PM
> *To:* dev@cassandra.apache.org
> *Subject:* [EXTERNAL] Re: [DISCUSS] CEP-40: Data Transfer Using Cassandra
> Sidecar for Live Migrating Instances
>
> Thanks for proposing this CEP! We have something like this internally so I
> have some familiarity with the appr
, April 14, 2024 12:27 PM
To: dev@cassandra.apache.org
Subject: [EXTERNAL] Re: [DISCUSS] CEP-40: Data Transfer Using Cassandra Sidecar
for Live Migrating Instances
Thanks for proposing this CEP! We have something like this internally so I have
some familiarity with the approach
I am sympathetic to having a native fast instance migration solution as
this is a common operational recipe.
> It's valuable for Cassandra to have an ecosystem-native mechanism of
migrating data between physical/virtual instances outside the standard
streaming path. As Hari mentions, the current
Thanks for proposing this CEP! We have something like this internally so I
have some familiarity with the approach and the challenges. After reading
the CEP a couple things come to mind:
1. I would like to see more abstraction of how the files get moved / put in
place with the proposed solution
First off, let me apologize for my initial reply, it came off harsher than
I had intended.
I know I didn't say it initially, but I like the idea of making it easier
to replace a node. I think it's probably not obvious to folks that you can
use rsync (with stunnel, or alternatively rclone), and
On Mon, Apr 8, 2024 at 10:23 AM Jon Haddad wrote:
> This seems like a lot of work to create an rsync alternative. I can't
> really say I see the point. I noticed your "rejected alternatives"
> mentions it with this note:
>
I want to point out a few things before dismissing it as an 'rsync
Thanks Jon & Scott for taking time to go through this CEP and providing
inputs.
I am completely with what Scott had mentioned earlier (I would have added
more details into the CEP). Adding a few more points to the same.
Having a solution with Sidecar can make the migration easy without
depending
Oh, one note on this item:> The operator can ensure that files in the destination matches with the source. In the first iteration of this feature, an API is introduced to calculate digest for the list of file names and their lengths to identify any mismatches. It does not validate the file
Hi Jon, Thanks for taking the time to read and reply to this proposal. Would encourage you to approach it from
an attitude of seeking understanding on the part of the first-time CEP author, as this reply casts it off
pretty quickly as NIH. The proposal isn't mine, but I'll offer a few notes on
This seems like a lot of work to create an rsync alternative. I can't
really say I see the point. I noticed your "rejected alternatives"
mentions it with this note:
- However, it might not be permitted by the administrator or available
in various environments such as Kubernetes or
Hi all,
I have filed CEP-40 [1] for live migrating Cassandra instances using the
Cassandra Sidecar.
When someone needs to move all or a portion of the Cassandra nodes
belonging to a cluster to different hosts, the traditional approach of
Cassandra node replacement can be time-consuming due to
38 matches
Mail list logo
