Re: Traffic inconsistent across nodes

[Anishek Agarwal]

Looks like some problem with our monitoring framework. Thanks for you help !

On Mon, Apr 18, 2016 at 2:46 PM, Anishek Agarwal  wrote:

> OS used : Cent OS 6 on all nodes except *10*.125.138.59 ( which runs Cent
> OS 7)
> All of them are running Cassandra 2.0.17
>
> output of the test :
>
> host ip: 10.124.114.113
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.108
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.110
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.118
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.125.138.59
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.97
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.105
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.98
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
>
> On Fri, Apr 15, 2016 at 6:47 PM, Eric Stevens  wrote:
>
>> Thanks for that, that helps a lot.  The next thing to check might be
>> whether or not your application actually has access to the other nodes.
>> With that topology, and assuming all the nodes you included in your
>> original graph are in the 'WDC' data center, I'd be inclined to look for a
>> network issue of some kind.
>>
>> Also, it probably doesn't matter, but what OS / Distribution are you
>> running the servers and clients on?
>>
>> Check with netcat or something that you can reach all the configured
>> ports from your application server, but also the driver itself offers some
>> introspection into its view of individual connection health.  This is a
>> little bit ugly, but this is how we include information about connection
>> status in an API for health monitoring from a Scala application using the
>> Java driver; hopefully you can use it to see how to access information
>> about the driver's view of host health from the application's perspective.
>> Most importantly I'd suggest looking for host.isUp status and
>> LoadBalancingPolicy.distance(host) to see that it considers all the hosts
>> in your target datacenter to be LOCAL.
>>
>> "hosts" -> {
>>   val hosts: Map[String, Map[String, mutable.Set[Host]]] =
>> connection.getMetadata
>>   .getAllHosts.asScala
>>   .groupBy(_.getDatacenter)
>>   .mapValues(_.groupBy(_.getRack))
>>   val lbp: LoadBalancingPolicy = 
>> connection.getConfiguration.getPolicies.getLoadBalancingPolicy
>>   JsObject(hosts.map { case (dc: String, rackAndHosts) =>
>> dc -> JsObject(rackAndHosts.map { case (rack: String, hosts: 
>> mutable.Set[Host]) =>
>>   rack -> JsArray(hosts.map { host =>
>> Json.obj(
>>   "address"  -> host.getAddress.toString,
>>   "socketAddress"-> host.getSocketAddress.toString,
>>   "cassandraVersion" -> host.getCassandraVersion.toString,
>>   "isUp" -> host.isUp,
>>   "hostDistance" -> lbp.distance(host).toString
>> )
>>   }.toSeq)
>> }.toSeq)
>>   }.toSeq)
>> },
>>
>>
>> On Wed, Apr 13, 2016 at 10:50 PM Anishek Agarwal 
>> wrote:
>>
>>> here is the output:  every node in a single DC is in the same rack.
>>>
>>> Datacenter: WDC5
>>>
>>> 
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> --  Address Load   Tokens  Owns (effective)  Host ID
>>>   Rack
>>>
>>> UN  10.125.138.33   299.22 GB  256 64.2%
>>> 8aaa6015-d444-4551-a3c5-3257536df476  RAC1
>>>
>>> UN  10.125.138.125  329.38 GB  256 70.3%
>>> 70be44a2-de17-41f1-9d3a-6a0be600eedf  RAC1
>>>
>>> UN  10.125.138.129  305.11 GB  256 65.5%
>>> 0fbc7f44-7062-4996-9eba-2a05ae1a7032  RAC1
>>>
>>> Datacenter: WDC
>>>
>>> ===
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> --  Address Load   Tokens  Owns (effective)  Host ID
>>>   Rack
>>>
>>> UN  10.124.114.105  151.09 GB  256 38.0%
>>> c432357d-bf81-4eef-98e1-664c178a3c23  RAC1
>>>
>>> UN  10.124.114.110  150.15 GB  256 36.9%
>>> 6f92d32e-1c64-4145-83d7-265c331ea408  RAC1
>>>
>>> UN  10.124.114.108  170.1 GB   256 41.3%
>>> 040ae7e5-3f1e-4874-8738-45edbf576e12  RAC1
>>>
>>> UN  10.124.114.98   165.34 GB  256 37.6%
>>> cdc69c7d-b9d6-4abd-9388-1cdcd35d946c  RAC1
>>>
>>> UN  10.124.114.113  145.22 GB  256 35.7%
>>> 1557af04-e658-4751-b984-8e0cdc41376e  RAC1
>>>
>>> UN  10.125.138.59   162.65 GB  256 38.6%
>>> 9ba1b7b6-5655-456e-b1a1-6f429750fc96  RAC1
>>>
>>> UN  

Re: Unable to perform COUNT(*) on CASSANDRA table

[Steve Robenalt]

Hi Lokesh,

Jack Krupansky sent an email to this list just this morning with all the
detail you should need to understand why this happens.

Steve

On Mon, Apr 18, 2016 at 4:38 PM, Lokesh Ceeba - Vendor <
lokesh.ce...@walmart.com> wrote:

> Hello Team
>
>The Kafka-Storm is running and the logs says,
> it processed 500,000 records.
>
>
>
> But the table count(*) fails – timeout  -- Do you know why ??
>
>
>
>
>
> cqlsh:sandbox> select count(*) from events;
>
> OperationTimedOut: errors={}, last_host=10.226.68.248
>
>
>
>
>
> DDL –setting parameters
>
>
>
>
>
> AND bloom_filter_fp_chance = 0.01
>
> AND comment = ''
>
> AND dclocal_read_repair_chance = 0.1
>
> AND default_time_to_live = 0
>
> AND gc_grace_seconds = 864000
>
> AND max_index_interval = 2048
>
> AND memtable_flush_period_in_ms = 0
>
> AND min_index_interval = 128
>
> AND read_repair_chance = 0.0
>
> AND speculative_retry = '99.0PERCENTILE'
>
> AND caching = {
>
> 'keys' : 'ALL',
>
> 'rows_per_partition' : 'NONE'
>
> }
>
> AND compression = {
>
> 'chunk_length_kb' : 64,
>
> 'crc_check_chance' : 1.0,
>
> 'sstable_compression' : 'LZ4Compressor'
>
> }
>
> AND compaction = {
>
> 'bucket_high' : 1.5,
>
> 'bucket_low' : 0.5,
>
> 'class' : 'LeveledCompactionStrategy',
>
> 'cold_reads_to_omit' : 0.05,
>
> 'enabled' : true,
>
> 'max_threshold' : 32,
>
> 'min_sstable_size' : 50,
>
> 'min_threshold' : 4,
>
> 'tombstone_compaction_interval' : 86400,
>
> 'tombstone_threshold' : 0.2,
>
> 'unchecked_tombstone_compaction' : false
>
> };
>
>
>
>
>
>
>
>
>
>
>
> Thanks
>
> Lokesh
> This email and any files transmitted with it are confidential and intended
> solely for the individual or entity to whom they are addressed. If you have
> received this email in error destroy it immediately. *** Walmart
> Confidential ***
>



-- 
Steve Robenalt
Software Architect
sroben...@highwire.org 
(office/cell): 916-505-1785

HighWire Press, Inc.
425 Broadway St, Redwood City, CA 94063
www.highwire.org

Technology for Scholarly Communication

Unable to perform COUNT(*) on CASSANDRA table

[Lokesh Ceeba - Vendor]

Hello Team
   The Kafka-Storm is running and the logs says, it 
processed 500,000 records.

But the table count(*) fails – timeout  -- Do you know why ??


cqlsh:sandbox> select count(*) from events;
OperationTimedOut: errors={}, last_host=10.226.68.248


DDL –setting parameters


AND bloom_filter_fp_chance = 0.01
AND comment = ''
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE'
AND caching = {
'keys' : 'ALL',
'rows_per_partition' : 'NONE'
}
AND compression = {
'chunk_length_kb' : 64,
'crc_check_chance' : 1.0,
'sstable_compression' : 'LZ4Compressor'
}
AND compaction = {
'bucket_high' : 1.5,
'bucket_low' : 0.5,
'class' : 'LeveledCompactionStrategy',
'cold_reads_to_omit' : 0.05,
'enabled' : true,
'max_threshold' : 32,
'min_sstable_size' : 50,
'min_threshold' : 4,
'tombstone_compaction_interval' : 86400,
'tombstone_threshold' : 0.2,
'unchecked_tombstone_compaction' : false
};





Thanks
Lokesh

This email and any files transmitted with it are confidential and intended 
solely for the individual or entity to whom they are addressed. If you have 
received this email in error destroy it immediately. *** Walmart Confidential 
***

Re: Experience with Kubernetes

[David Aronchick]

We're working on that very documentation, but there's nothing about the
product that is inherently single datacenter. That is to say, if you
ALREADY wire together multiple data centers (via VPN, etc), then there's
zero about Kubernetes that wouldn't work great today (as folks are already
doing). What's your multi-DC solution today with VMs?

As far as using local, persistent storage, it's definitely not the core of
the containers design philosophy - the whole 'pets v. cattle' debate and
what not. However, if you'd like to use local storage, that's also trivial
to do in Kubernetes - It's called "hostPath" and it lets you mount a local
directory into your container.
http://kubernetes.io/docs/user-guide/volumes/#hostpath

However, you'd want to either limit your nodes to one container per node,
or you'll have IO contention. You can use that in Kubernetes using an
object called "DaemonSet" - http://kubernetes.io/docs/admin/daemons/


On Fri, Apr 15, 2016 at 3:31 PM Jack Krupansky 
wrote:

> It sounds as if Kubernetes is oriented towards a single data center, DC in
> Cassandra parlance, and maybe Ubernetes attempts to address that. If there
> is indeed some simple, obvious, way to trick Kubernetes into
> Cassandra-style multi-DC operation, it sure would be helpful if it was
> documented more obviously.
>
> Also, it seems like a lot of these cluster tools are oriented towards
> applications that in turn use networked services such as Cassandra for
> storage of the vast bulk of their data. IOW, they persist their real data
> in an external database such as Cassandra, and that any other persisted
> data or state is relatively inconsequential. How that model applies to a
> database server that works best with fairly large amounts of ultra-fast
> local data storage is not so obvious. Maybe that simply wasn't a design
> goal?
>
> -- Jack Krupansky
>
> On Fri, Apr 15, 2016 at 3:48 PM, David Aronchick 
> wrote:
>
>> Hi--
>>
>> It's trivial to do this in Kubernetes, even without Ubernetes. Please
>> feel free to send me a note and I'll walk you through it.
>>
>> Disclosure: I work on Google on Kubernetes.
>>
>> On Thu, Apr 14, 2016 at 9:10 AM Joe Stein  wrote:
>>
>>> You can do that with the Mesos scheduler
>>> https://github.com/elodina/datastax-enterprise-mesos and layout
>>> clusters and racks for datacenters based on attributes
>>> http://mesos.apache.org/documentation/latest/attributes-resources/
>>>
>>> ~ Joestein
>>> On Apr 14, 2016 12:05 PM, "Nate McCall"  wrote:
>>>

 Does anybody here have any experience, positive or negative, with
> deploying Cassandra (or DSE) clusters using Kubernetes? I don't have any
> immediate need (or experience), but I am curious about the pros and cons.
>
>

 The last time I played around with kubernetes+cassandra, you could not
 specify node allocations across failure boundaries (AZs, Regions, etc).

 To me, that makes it not interesting outside of development or trivial
 setups.

 It does look like they are getting farther along on "ubernetes" which
 should fix this:

 https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federation.md



 --
 -
 Nate McCall
 Austin, TX
 @zznate

 Co-Founder & Sr. Technical Consultant
 Apache Cassandra Consulting
 http://www.thelastpickle.com

>>>
>

Bootstrapping Question for Cassandra 2.0

[Mir Tanvir Hossain]

Hi, could anyone please tell me whether is it possible to remove an
existing node from cluster while another node is joining the same cluster?

Thanks,
Mir

Proper use of COUNT

[Jack Krupansky]

Based on a recent inquiry and a recent thread of my own, and the coming
support for wide rows, I'll focus in on this question that I feel needs
better documentation of recommended best practice:

When can the COUNT(*) aggregate row-counting function be used?

Accept for relatively small or narrow queries, it seems to have a
propensity for timing out.

>From what I gather (and without specific testing myself), SELECT COUNT(*)
seems to be warranted for use cases where:

1. The table is relatively small, or
2. The number of rows selected by the WHERE clause is relatively small, due
to a combination of a relatively small number of partitions and a
relatively narrow slice of rows in a partition.

One can always approximate a larger SELECT by performing multiple selects,
each specifying a limited range of tokens in the WHERE clause.

But, even if the number of partitions selected is relatively small, it is
possible that the partitions might be wide so that the number of rows is
not so small.

And with CASSANDRA-11206
, very large
partitions (2 GB) could have a non-trivial number of rows.

So, an open question is how small is reasonable for COUNT. Thousands? Tens
of thousands? Only Hundreds?

Obviously it does depend on your particular data and hardware, but just on
a general basis.

In any case, does anyone have any specific experiences to share as to what
recommendations they would have for the use of COUNT?

A companion question is whether COUNT(column_name) has the same limitations
and recommendations. It does have to actually fetch the column values as
opposed to simply determining the existence of the row, but how
consequential that additional processing is, I couldn't say.

-- Jack Krupansky

Re: Most stable version?

[Carlos Rolo]

Sorry to resurrect this now, but I don't consider anything after 3.0.x
stable.

I wrote a blog post about this to be clear:
https://www.pythian.com/blog/cassandra-version-production/

Use it and pick a version based on your needs.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
*
Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649
www.pythian.com

On Fri, Apr 15, 2016 at 12:44 PM, Jean Tremblay <
jean.tremb...@zen-innovations.com> wrote:

> Thank you Jack.
> Jean
>
> On 14 Apr 2016, at 22:00 , Jack Krupansky 
> wrote:
>
> Normally, since 3.5 just came out, it would be wise to see if people
> report any problems over the next few weeks.
>
> But... the new tick-tock release process is designed to assure that these
> odd-numbered releases are only incremental bug fixes from the last
> even-numbered feature release, which was 3.4. So, 3.5 should be reasonably
> stable.
>
> That said, a bug-fix release of 3.0 is probably going to be more stable
> than a bug fix release of a more recent feature release (3.4).
>
> Usually it comes down to whether you need any of the new features or
> improvements in 3.x, or whether you might want to keep your chosen release
> in production for longer than the older 3.0 releases will be in production.
>
> Ultimately, this is a personality test: Are you adventuresome or
> conservative?
>
> To be clear, with the new tick-tock release scheme, 3.5 is designed to be
> a stable release.
>
> -- Jack Krupansky
>
> On Thu, Apr 14, 2016 at 3:23 PM, Jean Tremblay <
> jean.tremb...@zen-innovations.com> wrote:
>
>> Hi,
>> Could someone give his opinion on this?
>> What should be considered more stable, Cassandra 3.0.5 or Cassandra 3.5?
>>
>> Thank you
>> Jean
>>
>> > On 12 Apr,2016, at 07:00, Jean Tremblay <
>> jean.tremb...@zen-innovations.com> wrote:
>> >
>> > Hi,
>> > Which version of Cassandra should considered most stable in the version
>> 3?
>> > I see two main branch: the branch with the version 3.0.* and the
>> tick-tock one 3.*.*.
>> > So basically my question is: which one is most stable, version 3.0.5 or
>> version 3.3?
>> > I know odd versions in tick-took are bug fix.
>> > Thanks
>> > Jean
>>
>
>
>

-- 


--

Multiple clustering column and ordering issue.

[Michel]

Hello,

We have just started exploring cassandra for our project and learning
basics for designing tables to meet our queries.

We will have a table with following fields :
user_id
list_id
fname
lname
email
AND so on.

We have each user with multiple lists (csv data) so have created table as
below

CREATE TABLE IF NOT EXISTS mykeyspace.table3 (
user_id int,
list_id bigint,
id timeuuid,
fname text,
lname text,
PRIMARY KEY (user_id,list_id,id)
);

With above table we can execute below queries
1) fetch all data of one user_id with "WHERE user_id = ?".
2) fetch all data of one specific list_id of one user with "WHERE user_id =
? AND list_id = ?".

Also using SASI index on fname, lname, email and other fields we can
perform search
3) search from all records of one user_id (regardless of list_id)
4) search from all records of one list_id for one user_id

Now the only issue with this table is ordering of data. With this table we
can not order data by id field as below
WITH CLUSTERING ORDER BY (id DESC);

Records are ordered by list_id by default which is not useful for default
ordering.

Can anyone help here or suggest if we are missing something?

Thanks!

Re: Traffic inconsistent across nodes

[Anishek Agarwal]

OS used : Cent OS 6 on all nodes except *10*.125.138.59 ( which runs Cent
OS 7)
All of them are running Cassandra 2.0.17

output of the test :

host ip: 10.124.114.113

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.124.114.108

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.124.114.110

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.124.114.118

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.125.138.59

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.124.114.97

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.124.114.105

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.124.114.98

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17


On Fri, Apr 15, 2016 at 6:47 PM, Eric Stevens  wrote:

> Thanks for that, that helps a lot.  The next thing to check might be
> whether or not your application actually has access to the other nodes.
> With that topology, and assuming all the nodes you included in your
> original graph are in the 'WDC' data center, I'd be inclined to look for a
> network issue of some kind.
>
> Also, it probably doesn't matter, but what OS / Distribution are you
> running the servers and clients on?
>
> Check with netcat or something that you can reach all the configured ports
> from your application server, but also the driver itself offers some
> introspection into its view of individual connection health.  This is a
> little bit ugly, but this is how we include information about connection
> status in an API for health monitoring from a Scala application using the
> Java driver; hopefully you can use it to see how to access information
> about the driver's view of host health from the application's perspective.
> Most importantly I'd suggest looking for host.isUp status and
> LoadBalancingPolicy.distance(host) to see that it considers all the hosts
> in your target datacenter to be LOCAL.
>
> "hosts" -> {
>   val hosts: Map[String, Map[String, mutable.Set[Host]]] =
> connection.getMetadata
>   .getAllHosts.asScala
>   .groupBy(_.getDatacenter)
>   .mapValues(_.groupBy(_.getRack))
>   val lbp: LoadBalancingPolicy = 
> connection.getConfiguration.getPolicies.getLoadBalancingPolicy
>   JsObject(hosts.map { case (dc: String, rackAndHosts) =>
> dc -> JsObject(rackAndHosts.map { case (rack: String, hosts: 
> mutable.Set[Host]) =>
>   rack -> JsArray(hosts.map { host =>
> Json.obj(
>   "address"  -> host.getAddress.toString,
>   "socketAddress"-> host.getSocketAddress.toString,
>   "cassandraVersion" -> host.getCassandraVersion.toString,
>   "isUp" -> host.isUp,
>   "hostDistance" -> lbp.distance(host).toString
> )
>   }.toSeq)
> }.toSeq)
>   }.toSeq)
> },
>
>
> On Wed, Apr 13, 2016 at 10:50 PM Anishek Agarwal 
> wrote:
>
>> here is the output:  every node in a single DC is in the same rack.
>>
>> Datacenter: WDC5
>>
>> 
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> --  Address Load   Tokens  Owns (effective)  Host ID
>>   Rack
>>
>> UN  10.125.138.33   299.22 GB  256 64.2%
>> 8aaa6015-d444-4551-a3c5-3257536df476  RAC1
>>
>> UN  10.125.138.125  329.38 GB  256 70.3%
>> 70be44a2-de17-41f1-9d3a-6a0be600eedf  RAC1
>>
>> UN  10.125.138.129  305.11 GB  256 65.5%
>> 0fbc7f44-7062-4996-9eba-2a05ae1a7032  RAC1
>>
>> Datacenter: WDC
>>
>> ===
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> --  Address Load   Tokens  Owns (effective)  Host ID
>>   Rack
>>
>> UN  10.124.114.105  151.09 GB  256 38.0%
>> c432357d-bf81-4eef-98e1-664c178a3c23  RAC1
>>
>> UN  10.124.114.110  150.15 GB  256 36.9%
>> 6f92d32e-1c64-4145-83d7-265c331ea408  RAC1
>>
>> UN  10.124.114.108  170.1 GB   256 41.3%
>> 040ae7e5-3f1e-4874-8738-45edbf576e12  RAC1
>>
>> UN  10.124.114.98   165.34 GB  256 37.6%
>> cdc69c7d-b9d6-4abd-9388-1cdcd35d946c  RAC1
>>
>> UN  10.124.114.113  145.22 GB  256 35.7%
>> 1557af04-e658-4751-b984-8e0cdc41376e  RAC1
>>
>> UN  10.125.138.59   162.65 GB  256 38.6%
>> 9ba1b7b6-5655-456e-b1a1-6f429750fc96  RAC1
>>
>> UN  10.124.114.97   164.03 GB  256 36.9%
>> c918e497-498e-44c3-ab01-ab5cb4d48b09  RAC1
>>
>> UN  10.124.114.118  139.62 GB  256 35.1%
>> 2bb0c265-a5d4-4cd4-8f50-13b5a9a891c9  RAC1
>>
>> On Thu, Apr 14, 2016 at 4:48 AM, Eric Stevens  wrote:
>>
>>> The output of nodetool status would really help answer some questions.
>>> I take it the 8 hosts in your graph are in the same 

Alternative approach to setting up new DC

[Jens Rantil]

Hi,

I am provisioning a new datacenter for an existing cluster. A rather shaky
VPN connection is hindering me from making a "nodetool rebuild" bootstrap
on the new DC. Interestingly, I have a full fresh database snapshot/backup
at the same location as the new DC (transferred outside of the VPN). I am
now considering the following approach:

   1. Make sure my clients are using the old DC.
   2. Provision the new nodes in new DC.
   3. ALTER the keyspace to enable replicas on the new DC. This will start
   replicating all writes from old DC to new DC.
   4. Before gc_grace_seconds after operation 3) above, use sstableloader
   to stream my backup to the new nodes.
   5. For safety precaution, do a full repair.

Could you see any issues with doing this?

Cheers,
Jens
-- 

Jens Rantil
Backend Developer @ Tink

Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.