Re: Traffic inconsistent across nodes

[Anishek Agarwal]

here is the output:  every node in a single DC is in the same rack.

Datacenter: WDC5



Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address Load   Tokens  Owns (effective)  Host ID
Rack

UN  10.125.138.33   299.22 GB  256 64.2%
8aaa6015-d444-4551-a3c5-3257536df476  RAC1

UN  10.125.138.125  329.38 GB  256 70.3%
70be44a2-de17-41f1-9d3a-6a0be600eedf  RAC1

UN  10.125.138.129  305.11 GB  256 65.5%
0fbc7f44-7062-4996-9eba-2a05ae1a7032  RAC1

Datacenter: WDC

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address Load   Tokens  Owns (effective)  Host ID
Rack

UN  10.124.114.105  151.09 GB  256 38.0%
c432357d-bf81-4eef-98e1-664c178a3c23  RAC1

UN  10.124.114.110  150.15 GB  256 36.9%
6f92d32e-1c64-4145-83d7-265c331ea408  RAC1

UN  10.124.114.108  170.1 GB   256 41.3%
040ae7e5-3f1e-4874-8738-45edbf576e12  RAC1

UN  10.124.114.98   165.34 GB  256 37.6%
cdc69c7d-b9d6-4abd-9388-1cdcd35d946c  RAC1

UN  10.124.114.113  145.22 GB  256 35.7%
1557af04-e658-4751-b984-8e0cdc41376e  RAC1

UN  10.125.138.59   162.65 GB  256 38.6%
9ba1b7b6-5655-456e-b1a1-6f429750fc96  RAC1

UN  10.124.114.97   164.03 GB  256 36.9%
c918e497-498e-44c3-ab01-ab5cb4d48b09  RAC1

UN  10.124.114.118  139.62 GB  256 35.1%
2bb0c265-a5d4-4cd4-8f50-13b5a9a891c9  RAC1

On Thu, Apr 14, 2016 at 4:48 AM, Eric Stevens  wrote:

> The output of nodetool status would really help answer some questions.  I
> take it the 8 hosts in your graph are in the same DC.  Are the four serving
> writes in the same logical or physical rack (as Cassandra sees it), while
> the others are not?
>
> On Tue, Apr 12, 2016 at 10:48 PM Anishek Agarwal 
> wrote:
>
>> We have two DC one with the above 8 nodes and other with 3 nodes.
>>
>>
>>
>> On Tue, Apr 12, 2016 at 8:06 PM, Eric Stevens  wrote:
>>
>>> Maybe include nodetool status here?  Are the four nodes serving reads in
>>> one DC (local to your driver's config) while the others are in another?
>>>
>>> On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal  wrote:
>>>
 hello,

 we have 8 nodes in one cluster and attached is the traffic patterns
 across the nodes.

 its very surprising that only 4 nodes show transmitting (purple)
 packets.

 our driver configuration on clients has the following load balancing
 configuration  :

 new TokenAwarePolicy(
 new 
 DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME,
  "WDC")),
 true)


 any idea what is that we are missing which is leading to this skewed
 data read patterns

 cassandra drivers as below:

 
 com.datastax.cassandra
 cassandra-driver-core
 2.1.6
 
 
 com.datastax.cassandra
 cassandra-driver-mapping
 2.1.6
 

 cassandra version is 2.0.17

 Thanks in advance for the help.

 Anishek


>>

Re: Balancing tokens over 2 datacenter

[Jeff Jirsa]

100% ownership on all nodes isn’t wrong with 3 nodes in each of 2 Dcs with RF=3 
in both of those Dcs. That’s exactly what you’d expect it to be, and a 
perfectly viable production config for many workloads.



From:  Anuj Wadehra
Reply-To:  "user@cassandra.apache.org"
Date:  Wednesday, April 13, 2016 at 6:02 PM
To:  "user@cassandra.apache.org"
Subject:  Re: Balancing tokens over 2 datacenter

Hi Stephen Walsh, 

As per the nodetool output, every node owns 100% of the range. This indicates 
wrong configuration. It would be good, if you verify and share following 
properties of yaml on all nodes:

Num tokens,seeds, cluster name,listen address, initial token.

Also, which snitch are you using? If you use propertyfilesnitch, please share 
cassandra-topology.properties too.



Thanks
Anuj

Sent from Yahoo Mail on Android

On Wed, 13 Apr, 2016 at 9:46 PM, Walsh, Stephen
 wrote:
Right again Alain
We use the DCAwareRoundRobinPolicy in our java datastax driver in each DC 
application to point to that Cassandra DC’s.



From: Alain RODRIGUEZ 
Reply-To: "user@cassandra.apache.org" 
Date: Wednesday, 13 April 2016 at 15:52
To: "user@cassandra.apache.org" 
Subject: Re: Balancing tokens over 2 datacenter

Steve,

This cluster looks just great.

Now, due to a miss configuration in our application, we saw that our 
application in both DC’s where pointing to DC1.

This is the only thing to solve, and it happens in the client side 
configuration.

What client do you use?

Are you using something like 'new DCAwareRoundRobinPolicy("DC1"));' as pointed 
in Bhuvan's link 
http://stackoverflow.com/questions/22813045/ability-to-write-to-a-particular-cassandra-node
 ? You can use some other 

Then make sure to deploy this on clients on that need to use 'DC1' and 'new 
DCAwareRoundRobinPolicy("DC2")' on client that should be using 'DC2'.

Make sure ports are open.

This should be it,

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com



2016-04-13 16:28 GMT+02:00 Walsh, Stephen :
Thanks for your helps guys,

As you guessed our schema is 
{'class': 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3'}  AND 
durable_writes = false;



Our reads and writes on LOCAL_ONE with each application (now) using its own DC 
as its preferred DC

Here is the nodetool status for one of our tables (all tabes are created the 
same way)

Datacenter: DC1

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address Load   Tokens  Owns (effective)  Host ID
   Rack

UN  X.0.0.149  14.6 MB256 100.0%
0f497235-a0bb-4e47-9434-dd0e126aa432  RAC3

UN  X.0.0.251  12.33 MB   256 100.0%
a1307717-4b61-4d57-8658-50460d6d54a1  RAC1

UN  X.0.0.79   21.54 MB   256 100.0%
f353c8f3-6b7c-483b-ad9a-3d66d469079e  RAC2

Datacenter: DC2

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address Load   Tokens  Owns (effective)  Host ID
   Rack

UN  X.0.2.32   18.08 MB   256 100.0%
103a1cb3-6580-44bd-bf97-28ae160e1119  RAC6

UN  X.0.2.211  12.46 MB   256 100.0%
8c8dd5ba-806d-43eb-9ee5-af463e443f46  RAC5

UN  X.0.2.186  12.58 MB   256 100.0%
aef904ba-aaab-47f1-9bdc-cc1e0c676f61  RAC4



We ran the nodetool repair and cleanup in case the nodes where balanced but 
needed cleaning up – this was not the case :(


Steve


From: Alain RODRIGUEZ 
Reply-To: "user@cassandra.apache.org" 
Date: Wednesday, 13 April 2016 at 14:48
To: "user@cassandra.apache.org" 
Subject: Re: Balancing tokens over 2 datacenter

Hi Steve,
 
As such, all keyspaces and tables where created on DC1.
The effect of this is that all reads are now going to DC1 and ignoring DC2

I think this is not exactly true. When tables are created, they are created on 
a specific keyspace, no matter where you send the alter schema command, schema 
will propagate to all the datacenters the keyspace is replicated to.

So the question is: Is your keyspace using 'DC1: 3, DC2: 3' as replication 
factors? Could you show us the schema and a nodetool status as well?

WE’ve tried doing , nodetool repair / cleanup – but the reads always go to DC1

Trying to do random things is often not a good idea. For example, as each node 
holds 100% of the data, cleanup is an expensive no-op :-).

Anyone know how to rebalance the tokens over DC’s?

Yes, I can help on that, but I need to know your current status.

Basically, your(s) keyspace(s) must be using RF of 3 on the 2 DCs as mentioned, 
your client to be configured to stick to the DC in their zone (use a DCAware 
policy with the DC name + LOCAL_ONE/QUORUM, see Bhuvan's links) 

RE: Leak Detected while bootstrap

[Anubhav Kale]

Thanks, Updated with logs.

From: Tyler Hobbs [mailto:ty...@datastax.com]
Sent: Wednesday, April 13, 2016 3:36 PM
To: user@cassandra.apache.org
Subject: Re: Leak Detected while bootstrap

This looks like it might be 
https://issues.apache.org/jira/browse/CASSANDRA-11374.
  Can you comment on that ticket and share your logs leading up to the error?

On Wed, Apr 13, 2016 at 3:37 PM, Anubhav Kale 
> wrote:
Hello,

Since we upgraded to Cassandra 2.1.12, we are noticing that below happens when 
we are trying to bootstrap nodes, and the process just gets stuck. Restarting 
the process / VM does not help. Our nodes are around ~300 GB and run on local 
SSDs and we haven’t seen this problem on older versions (specifically 2.1.9).

Is this a known issue / any workarounds ?

ERROR [Reference-Reaper:1] 2016-04-13 20:33:53,394  Ref.java:179 - LEAK 
DETECTED: a reference 
(org.apache.cassandra.utils.concurrent.Ref$State@15e611a3)
 to class 
org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$1@203187780:[[OffHeapBitSet]]
 was not released before the reference was garbage collected

Thanks !



--
Tyler Hobbs
DataStax

Re: Leak Detected while bootstrap

[Tyler Hobbs]

This looks like it might be
https://issues.apache.org/jira/browse/CASSANDRA-11374.  Can you comment on
that ticket and share your logs leading up to the error?

On Wed, Apr 13, 2016 at 3:37 PM, Anubhav Kale 
wrote:

> Hello,
>
>
>
> Since we upgraded to Cassandra 2.1.12, we are noticing that * below*
> happens when we are trying to bootstrap nodes, and the process just gets
> stuck. Restarting the process / VM does not help. Our nodes are around ~300
> GB and run on local SSDs and we haven’t seen this problem on older versions
> (specifically 2.1.9).
>
>
>
> Is this a known issue / any workarounds ?
>
>
>
> *ERROR [Reference-Reaper:1] 2016-04-13 20:33:53,394  Ref.java:179 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@15e611a3) to class
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$1@203187780:[[OffHeapBitSet]]
> was not released before the reference was garbage collected*
>
>
>
> Thanks !
>



-- 
Tyler Hobbs
DataStax 

Re: Compaction Error When upgrading from 2.1.9 to 3.0.2

[Tyler Hobbs]

Can you open a ticket here with your schema and the stacktrace?
https://issues.apache.org/jira/browse/CASSANDRA

I'm also curious why you're not upgrading to 3.0.5 instead of 3.0.2.

On Wed, Apr 13, 2016 at 4:37 PM, Anthony Verslues <
anthony.versl...@mezocliq.com> wrote:

> I got this compaction error when running ‘nodetool upgradesstable –a’
> while upgrading from 2.1.9 to 3.0.2. According to documentation this
> upgrade should work.
>
>
>
> Would upgrading to another intermediate version help?
>
>
>
>
>
> This is the line number:
> https://github.com/apache/cassandra/blob/cassandra-3.0.2/src/java/org/apache/cassandra/db/LegacyLayout.java#L1124
>
>
>
>
>
> error: null
>
> -- StackTrace --
>
> java.lang.AssertionError
>
> at
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1124)
>
> at
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1099)
>
> at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:444)
>
> at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
>
> at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:289)
>
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:134)
>
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
>
> at
> org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:329)
>
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
>
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65)
>
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
>
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
>
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:206)
>
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:159)
>
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
>
> at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
>
> at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
>
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
>
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
>
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
>
> at
> org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:572)
>
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>



-- 
Tyler Hobbs
DataStax 

Compaction Error When upgrading from 2.1.9 to 3.0.2

[Anthony Verslues]

I got this compaction error when running 'nodetool upgradesstable -a' while 
upgrading from 2.1.9 to 3.0.2. According to documentation this upgrade should 
work.

Would upgrading to another intermediate version help?


This is the line number: 
https://github.com/apache/cassandra/blob/cassandra-3.0.2/src/java/org/apache/cassandra/db/LegacyLayout.java#L1124


error: null
-- StackTrace --
java.lang.AssertionError
at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1124)
at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1099)
at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:444)
at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:289)
at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:134)
at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
at 
org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:329)
at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65)
at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
at 
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:206)
at 
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:159)
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
at 
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
at 
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at 
org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:572)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

RE: Set up authentication on a live production cluster

[SEAN_R_DURITY]

Do the clients already send the credentials? That is the first thing to address.

Setting up a cluster for authentication (and authorization) requires a restart 
with the properties turned on in cassandra.yaml. However, the actual keyspace 
(system_auth) and tables are not created until the last node is restarted with 
the parameters changed. So, as you are changing each node, what you get is 
individual nodes that are requiring a password, but have no system_auth 
keyspace to authenticate against. Thus, clients cannot connect to these nodes.

With open source Cassandra you cannot implement authentication without at least 
a brief degradation of service (as nodes can’t authenticate) and an outage 
(while the keyspace and tables are created, users are created, and permissions 
are granted). The outage can be relatively brief, depending on cluster size, 
CL, speed to restart, etc.

With DataStax Enterprise, there is a TransitionalAuthenticator (and Authorizer) 
that lets you implement security without a full outage. You basically switch to 
the Transitional classes so that system_auth gets created. You create all your 
security objects. Then you switch to PasswordAuthenticator and 
CassandraAuthorizer. It takes two rolling bounces to get it done, but no outage.

I have done both of the above. The DataStax stuff is very helpful, when 
downtime is a concern. Perhaps you could write your own implementation of the 
various interfaces to do something like TransitionalAuthenticator, but we have 
seen that the security interfaces change, so you will probably break/rewrite in 
later versions. (For one-time use, maybe it is worth a shot?)

For anyone setting up new clusters, just start with security turned on so that 
you don’t end up in the It’s-Production-Can’t-Stop quandary above.


Sean Durity

From: Vigneshwaran [mailto:vigneshwaran2...@gmail.com]
Sent: Wednesday, April 13, 2016 3:36 AM
To: user@cassandra.apache.org
Subject: Set up authentication on a live production cluster

Hi,

I have setup a 16 node cluster (8 per DC; C* 2.2.4) up and running in our 
production setup. We use Datastax Java driver 2.1.8.

I would like to set up Authentication and Authorization in the cluster without 
breaking the live clients.

From the references I found by googling, I can setup credentials for a new 
cluster. But it is not clear to me what steps I should take for setting up 
credentials in an already running cluster without breaking existing clients.

Can someone clarify me or link me to a reference I may have missed? I'd really 
appreciate it.

Thank you,
Vigneshwaran



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

[RELEASE] Apache Cassandra 3.5 released

[Jake Luciani]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 3.5.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 3.5 series. As always, please
pay
attention to the release notes[2] and Let us know[3] if you were to
encounter
any problem.

Enjoy!

[1]: http://goo.gl/FchTrl (CHANGES.txt)
[2]: http://goo.gl/0zpkJU (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Leak Detected while bootstrap

[Anubhav Kale]

Hello,

Since we upgraded to Cassandra 2.1.12, we are noticing that below happens when 
we are trying to bootstrap nodes, and the process just gets stuck. Restarting 
the process / VM does not help. Our nodes are around ~300 GB and run on local 
SSDs and we haven't seen this problem on older versions (specifically 2.1.9).

Is this a known issue / any workarounds ?

ERROR [Reference-Reaper:1] 2016-04-13 20:33:53,394  Ref.java:179 - LEAK 
DETECTED: a reference 
(org.apache.cassandra.utils.concurrent.Ref$State@15e611a3) to class 
org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$1@203187780:[[OffHeapBitSet]]
 was not released before the reference was garbage collected

Thanks !

Re: Cassandra Golang Driver and Support

[Bryan Cheng]

Hi Yawei,

While you're right that there's no first-party driver, we've had good luck
using gocql (https://github.com/gocql/gocql) in production at moderate
scale. What features in particular are you looking for that are missing?

--Bryan

On Tue, Apr 12, 2016 at 10:06 PM, Yawei Li  wrote:

> Hi,
>
> It looks like to me that DataStax doesn't provide official golang driver
> yet and the goland client libs are overall lagging behind the Java driver
> in terms of feature set, supported version and possibly production
> stability?
>
> We are going to support a large number of services  in both Java and Go.
> if the above impression is largely true, we are considering the option of
> focusing on Java client and having GoLang program talk to the Java service
> via RPC for data access. Anyone has tried similar approach?
>
> Thanks
>

Re: Balancing tokens over 2 datacenter

[Walsh, Stephen]

Right again Alain
We use the DCAwareRoundRobinPolicy in our java datastax driver in each DC 
application to point to that Cassandra DC’s.



From: Alain RODRIGUEZ >
Reply-To: "user@cassandra.apache.org" 
>
Date: Wednesday, 13 April 2016 at 15:52
To: "user@cassandra.apache.org" 
>
Subject: Re: Balancing tokens over 2 datacenter

Steve,

This cluster looks just great.

Now, due to a miss configuration in our application, we saw that our 
application in both DC’s where pointing to DC1.

This is the only thing to solve, and it happens in the client side 
configuration.

What client do you use?

Are you using something like 'new DCAwareRoundRobinPolicy("DC1"));' as pointed 
in Bhuvan's link 
http://stackoverflow.com/questions/22813045/ability-to-write-to-a-particular-cassandra-node
 ? You can use some other

Then make sure to deploy this on clients on that need to use 'DC1' and 'new 
DCAwareRoundRobinPolicy("DC2")' on client that should be using 'DC2'.

Make sure ports are open.

This should be it,

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com



2016-04-13 16:28 GMT+02:00 Walsh, Stephen 
>:
Thanks for your helps guys,

As you guessed our schema is

{'class': 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3'}  AND 
durable_writes = false;


Our reads and writes on LOCAL_ONE with each application (now) using its own DC 
as its preferred DC

Here is the nodetool status for one of our tables (all tabes are created the 
same way)


Datacenter: DC1

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address Load   Tokens  Owns (effective)  Host ID
   Rack

UN  X.0.0.149  14.6 MB256 100.0%
0f497235-a0bb-4e47-9434-dd0e126aa432  RAC3

UN  X.0.0.251  12.33 MB   256 100.0%
a1307717-4b61-4d57-8658-50460d6d54a1  RAC1

UN  X.0.0.79   21.54 MB   256 100.0%
f353c8f3-6b7c-483b-ad9a-3d66d469079e  RAC2

Datacenter: DC2

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address Load   Tokens  Owns (effective)  Host ID
   Rack

UN  X.0.2.32   18.08 MB   256 100.0%
103a1cb3-6580-44bd-bf97-28ae160e1119  RAC6

UN  X.0.2.211  12.46 MB   256 100.0%
8c8dd5ba-806d-43eb-9ee5-af463e443f46  RAC5

UN  X.0.2.186  12.58 MB   256 100.0%
aef904ba-aaab-47f1-9bdc-cc1e0c676f61  RAC4


We ran the nodetool repair and cleanup in case the nodes where balanced but 
needed cleaning up – this was not the case :(


Steve


From: Alain RODRIGUEZ >
Reply-To: "user@cassandra.apache.org" 
>
Date: Wednesday, 13 April 2016 at 14:48
To: "user@cassandra.apache.org" 
>
Subject: Re: Balancing tokens over 2 datacenter

Hi Steve,

As such, all keyspaces and tables where created on DC1.
The effect of this is that all reads are now going to DC1 and ignoring DC2

I think this is not exactly true. When tables are created, they are created on 
a specific keyspace, no matter where you send the alter schema command, schema 
will propagate to all the datacenters the keyspace is replicated to.

So the question is: Is your keyspace using 'DC1: 3, DC2: 3' as replication 
factors? Could you show us the schema and a nodetool status as well?

WE’ve tried doing , nodetool repair / cleanup – but the reads always go to DC1

Trying to do random things is often not a good idea. For example, as each node 
holds 100% of the data, cleanup is an expensive no-op :-).

Anyone know how to rebalance the tokens over DC’s?

Yes, I can help on that, but I need to know your current status.

Basically, your(s) keyspace(s) must be using RF of 3 on the 2 DCs as mentioned, 
your client to be configured to stick to the DC in their zone (use a DCAware 
policy with the DC name + LOCAL_ONE/QUORUM, see Bhuvan's links) and things 
should be better.

If you need more detailed help, let us know what is unclear to you and provide 
us with 'nodetool status' output and with your schema (at least keyspaces 
config).

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com







2016-04-13 15:32 GMT+02:00 Bhuvan Rawal 
>:
This could be because of the way 

Re: Balancing tokens over 2 datacenter

[Alain RODRIGUEZ]

Steve,

This cluster looks just great.

Now, due to a miss configuration in our application, we saw that our
> application in both DC’s where pointing to DC1.


This is the only thing to solve, and it happens in the client side
configuration.

What client do you use?

Are you using something like 'new DCAwareRoundRobinPolicy("DC1"));' as
pointed in Bhuvan's link
http://stackoverflow.com/questions/22813045/ability-to-write-to-a-particular-cassandra-node
? You can use some other

Then make sure to deploy this on clients on that need to use 'DC1' and 'new
DCAwareRoundRobinPolicy("DC2")' on client that should be using 'DC2'.

Make sure ports are open.

This should be it,

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com



2016-04-13 16:28 GMT+02:00 Walsh, Stephen :

> Thanks for your helps guys,
>
> As you guessed our schema is
>
> {'class': 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3'}  AND
> durable_writes = false;
>
>
> Our reads and writes on LOCAL_ONE with each application (now) using its
> own DC as its preferred DC
>
> Here is the nodetool status for one of our tables (all tabes are created
> the same way)
>
> Datacenter: DC1
>
> ===
>
> Status=Up/Down
>
> |/ State=Normal/Leaving/Joining/Moving
>
> --  Address Load   Tokens  Owns (effective)  Host ID
> Rack
>
> UN  X.0.0.149  14.6 MB256 100.0%
> 0f497235-a0bb-4e47-9434-dd0e126aa432  RAC3
>
> UN  X.0.0.251  12.33 MB   256 100.0%
> a1307717-4b61-4d57-8658-50460d6d54a1  RAC1
>
> UN  X.0.0.79   21.54 MB   256 100.0%
> f353c8f3-6b7c-483b-ad9a-3d66d469079e  RAC2
>
> Datacenter: DC2
>
> ===
>
> Status=Up/Down
>
> |/ State=Normal/Leaving/Joining/Moving
>
> --  Address Load   Tokens  Owns (effective)  Host ID
> Rack
>
> UN  X.0.2.32   18.08 MB   256 100.0%
> 103a1cb3-6580-44bd-bf97-28ae160e1119  RAC6
>
> UN  X.0.2.211  12.46 MB   256 100.0%
> 8c8dd5ba-806d-43eb-9ee5-af463e443f46  RAC5
>
> UN  X.0.2.186  12.58 MB   256 100.0%
> aef904ba-aaab-47f1-9bdc-cc1e0c676f61  RAC4
>
>
> We ran the nodetool repair and cleanup in case the nodes where balanced
> but needed cleaning up – this was not the case :(
>
>
> Steve
>
>
> From: Alain RODRIGUEZ 
> Reply-To: "user@cassandra.apache.org" 
> Date: Wednesday, 13 April 2016 at 14:48
> To: "user@cassandra.apache.org" 
> Subject: Re: Balancing tokens over 2 datacenter
>
> Hi Steve,
>
>
>> As such, all keyspaces and tables where created on DC1.
>> The effect of this is that all reads are now going to DC1 and ignoring DC2
>>
>
> I think this is not exactly true. When tables are created, they are
> created on a specific keyspace, no matter where you send the alter schema
> command, schema will propagate to all the datacenters the keyspace is
> replicated to.
>
> So the question is: Is your keyspace using 'DC1: 3, DC2: 3' as replication
> factors? Could you show us the schema and a nodetool status as well?
>
> WE’ve tried doing , nodetool repair / cleanup – but the reads always go to
>> DC1
>
>
> Trying to do random things is often not a good idea. For example, as each
> node holds 100% of the data, cleanup is an expensive no-op :-).
>
> Anyone know how to rebalance the tokens over DC’s?
>
>
> Yes, I can help on that, but I need to know your current status.
>
> Basically, your(s) keyspace(s) must be using RF of 3 on the 2 DCs as
> mentioned, your client to be configured to stick to the DC in their zone
> (use a DCAware policy with the DC name + LOCAL_ONE/QUORUM, see Bhuvan's
> links) and things should be better.
>
> If you need more detailed help, let us know what is unclear to you and
> provide us with 'nodetool status' output and with your schema (at least
> keyspaces config).
>
> C*heers,
> ---
> Alain Rodriguez - al...@thelastpickle.com
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
>
>
>
>
>
> 2016-04-13 15:32 GMT+02:00 Bhuvan Rawal :
>
>> This could be because of the way you have configured the policy, have a
>> look at the below links for configuring the policy
>>
>> https://datastax.github.io/python-driver/api/cassandra/policies.html
>>
>>
>> http://stackoverflow.com/questions/22813045/ability-to-write-to-a-particular-cassandra-node
>>
>> Regards,
>> Bhuvan
>>
>> On Wed, Apr 13, 2016 at 6:54 PM, Walsh, Stephen > > wrote:
>>
>>> Hi there,
>>>
>>> So we have 2 datacenter with 3 nodes each.
>>> Replication factor is 3 per DC (so each node has all data)
>>>
>>> We have an application in each DC that writes that Cassandra DC.
>>>
>>> Now, due to a miss configuration in our application, we saw that our
>>> application in both DC’s where pointing to DC1.
>>>
>>> As such, all keyspaces and tables where created 

Re: Balancing tokens over 2 datacenter

[Walsh, Stephen]

Thanks for your helps guys,

As you guessed our schema is

{'class': 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3'}  AND 
durable_writes = false;


Our reads and writes on LOCAL_ONE with each application (now) using its own DC 
as its preferred DC

Here is the nodetool status for one of our tables (all tabes are created the 
same way)


Datacenter: DC1

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address Load   Tokens  Owns (effective)  Host ID
   Rack

UN  X.0.0.149  14.6 MB256 100.0%
0f497235-a0bb-4e47-9434-dd0e126aa432  RAC3

UN  X.0.0.251  12.33 MB   256 100.0%
a1307717-4b61-4d57-8658-50460d6d54a1  RAC1

UN  X.0.0.79   21.54 MB   256 100.0%
f353c8f3-6b7c-483b-ad9a-3d66d469079e  RAC2

Datacenter: DC2

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address Load   Tokens  Owns (effective)  Host ID
   Rack

UN  X.0.2.32   18.08 MB   256 100.0%
103a1cb3-6580-44bd-bf97-28ae160e1119  RAC6

UN  X.0.2.211  12.46 MB   256 100.0%
8c8dd5ba-806d-43eb-9ee5-af463e443f46  RAC5

UN  X.0.2.186  12.58 MB   256 100.0%
aef904ba-aaab-47f1-9bdc-cc1e0c676f61  RAC4


We ran the nodetool repair and cleanup in case the nodes where balanced but 
needed cleaning up – this was not the case :(


Steve


From: Alain RODRIGUEZ >
Reply-To: "user@cassandra.apache.org" 
>
Date: Wednesday, 13 April 2016 at 14:48
To: "user@cassandra.apache.org" 
>
Subject: Re: Balancing tokens over 2 datacenter

Hi Steve,

As such, all keyspaces and tables where created on DC1.
The effect of this is that all reads are now going to DC1 and ignoring DC2

I think this is not exactly true. When tables are created, they are created on 
a specific keyspace, no matter where you send the alter schema command, schema 
will propagate to all the datacenters the keyspace is replicated to.

So the question is: Is your keyspace using 'DC1: 3, DC2: 3' as replication 
factors? Could you show us the schema and a nodetool status as well?

WE’ve tried doing , nodetool repair / cleanup – but the reads always go to DC1

Trying to do random things is often not a good idea. For example, as each node 
holds 100% of the data, cleanup is an expensive no-op :-).

Anyone know how to rebalance the tokens over DC’s?

Yes, I can help on that, but I need to know your current status.

Basically, your(s) keyspace(s) must be using RF of 3 on the 2 DCs as mentioned, 
your client to be configured to stick to the DC in their zone (use a DCAware 
policy with the DC name + LOCAL_ONE/QUORUM, see Bhuvan's links) and things 
should be better.

If you need more detailed help, let us know what is unclear to you and provide 
us with 'nodetool status' output and with your schema (at least keyspaces 
config).

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com







2016-04-13 15:32 GMT+02:00 Bhuvan Rawal 
>:
This could be because of the way you have configured the policy, have a look at 
the below links for configuring the policy

https://datastax.github.io/python-driver/api/cassandra/policies.html

http://stackoverflow.com/questions/22813045/ability-to-write-to-a-particular-cassandra-node

Regards,
Bhuvan

On Wed, Apr 13, 2016 at 6:54 PM, Walsh, Stephen 
> wrote:
Hi there,

So we have 2 datacenter with 3 nodes each.
Replication factor is 3 per DC (so each node has all data)

We have an application in each DC that writes that Cassandra DC.

Now, due to a miss configuration in our application, we saw that our 
application in both DC’s where pointing to DC1.

As such, all keyspaces and tables where created on DC1.
The effect of this is that all reads are now going to DC1 and ignoring DC2

WE’ve tried doing , nodetool repair / cleanup – but the reads always go to DC1?

Anyone know how to rebalance the tokens over DC’s?


Regards
Steve


P.S I know about this article
http://www.datastax.com/dev/blog/balancing-your-cassandra-cluster
But its doesn’t answer my question regarding 2 DC’s token balancing

This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.


This email 

Re: Balancing tokens over 2 datacenter

[Alain RODRIGUEZ]

Hi Steve,


> As such, all keyspaces and tables where created on DC1.
> The effect of this is that all reads are now going to DC1 and ignoring DC2
>

I think this is not exactly true. When tables are created, they are created
on a specific keyspace, no matter where you send the alter schema command,
schema will propagate to all the datacenters the keyspace is replicated to.

So the question is: Is your keyspace using 'DC1: 3, DC2: 3' as replication
factors? Could you show us the schema and a nodetool status as well?

WE’ve tried doing , nodetool repair / cleanup – but the reads always go to
> DC1


Trying to do random things is often not a good idea. For example, as each
node holds 100% of the data, cleanup is an expensive no-op :-).

Anyone know how to rebalance the tokens over DC’s?


Yes, I can help on that, but I need to know your current status.

Basically, your(s) keyspace(s) must be using RF of 3 on the 2 DCs as
mentioned, your client to be configured to stick to the DC in their zone
(use a DCAware policy with the DC name + LOCAL_ONE/QUORUM, see Bhuvan's
links) and things should be better.

If you need more detailed help, let us know what is unclear to you and
provide us with 'nodetool status' output and with your schema (at least
keyspaces config).

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com







2016-04-13 15:32 GMT+02:00 Bhuvan Rawal :

> This could be because of the way you have configured the policy, have a
> look at the below links for configuring the policy
>
> https://datastax.github.io/python-driver/api/cassandra/policies.html
>
>
> http://stackoverflow.com/questions/22813045/ability-to-write-to-a-particular-cassandra-node
>
> Regards,
> Bhuvan
>
> On Wed, Apr 13, 2016 at 6:54 PM, Walsh, Stephen 
> wrote:
>
>> Hi there,
>>
>> So we have 2 datacenter with 3 nodes each.
>> Replication factor is 3 per DC (so each node has all data)
>>
>> We have an application in each DC that writes that Cassandra DC.
>>
>> Now, due to a miss configuration in our application, we saw that our
>> application in both DC’s where pointing to DC1.
>>
>> As such, all keyspaces and tables where created on DC1.
>> The effect of this is that all reads are now going to DC1 and ignoring DC2
>>
>> WE’ve tried doing , nodetool repair / cleanup – but the reads always go
>> to DC1?
>>
>> Anyone know how to rebalance the tokens over DC’s?
>>
>>
>> Regards
>> Steve
>>
>>
>> P.S I know about this article
>> http://www.datastax.com/dev/blog/balancing-your-cassandra-cluster
>> But its doesn’t answer my question regarding 2 DC’s token balancing
>>
>> This email (including any attachments) is proprietary to Aspect Software,
>> Inc. and may contain information that is confidential. If you have received
>> this message in error, please do not read, copy or forward this message.
>> Please notify the sender immediately, delete it from your system and
>> destroy any copies. You may not further disclose or distribute this email
>> or its attachments.
>>
>
>

Re: Creation of Async Datacenter for Monitoring purposes and Engineering Services purpose

[Alain RODRIGUEZ]

>
> Live Data delay of 1-2 Hours is acceptable. It is essential that
> replication to this DC to not impact the other 2 data centers.
>

Well, you can have immediate replication with no impact on other DC.
Basically set your clients to use LOCAL_ONE/QUORUM and specify a DCAWARE
policy on the client side too. On server side, just add the RF 'DC3: 2' (or
3 or whatever).

This way, all the writes to the chosen keyspaces will go to DC3 also, but
client won't care about how successful this write to DC3 is. So those
requirements are respected.

Is there a way to copy data in async fashion to new DC3 so that it doesnt
> impact existing DC's


What would be bad with this 'standard' approach?

Cheers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-04-13 11:52 GMT+02:00 Bhuvan Rawal :

> Hi All,
>
> We have 2 Running Datacenters in physically seperate DC's with 3 Nodes
> each. There is a requirement of an Audit DC for issuing queries which will
> not be concerned with live application traffic. Live Data delay of 1-2
> Hours is acceptable. It is essential that replication to this DC to not
> impact the other 2 data centers.
>
> Is there a way to copy data in async fashion to new DC3 so that it doesnt
> impact existing DC's, possibly without querying (using sstable/commitlog).
>
> Regards,
> Bhuvan
>
>
>

Re: Balancing tokens over 2 datacenter

[Bhuvan Rawal]

This could be because of the way you have configured the policy, have a
look at the below links for configuring the policy

https://datastax.github.io/python-driver/api/cassandra/policies.html

http://stackoverflow.com/questions/22813045/ability-to-write-to-a-particular-cassandra-node

Regards,
Bhuvan

On Wed, Apr 13, 2016 at 6:54 PM, Walsh, Stephen 
wrote:

> Hi there,
>
> So we have 2 datacenter with 3 nodes each.
> Replication factor is 3 per DC (so each node has all data)
>
> We have an application in each DC that writes that Cassandra DC.
>
> Now, due to a miss configuration in our application, we saw that our
> application in both DC’s where pointing to DC1.
>
> As such, all keyspaces and tables where created on DC1.
> The effect of this is that all reads are now going to DC1 and ignoring DC2
>
> WE’ve tried doing , nodetool repair / cleanup – but the reads always go to
> DC1?
>
> Anyone know how to rebalance the tokens over DC’s?
>
>
> Regards
> Steve
>
>
> P.S I know about this article
> http://www.datastax.com/dev/blog/balancing-your-cassandra-cluster
> But its doesn’t answer my question regarding 2 DC’s token balancing
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>

Re: C* 1.2.x vs Gossip marking DOWN/UP

[Alain RODRIGUEZ]

Hi Michael,

I had critical issues using 1.2 (.11, I believe) around gossip (but it was
like 2 years ago...).

Are you using the last C* 1.2.19 minor version? If not, you probably should
go there asap.

A lot of issues like this one
https://issues.apache.org/jira/browse/CASSANDRA-6297 have been fixed since
then on C* 1.2, 2.0, 2.1, 2.2, 3.0.X, 3.X. You got to go through steps to
upgrade. It should be safe and enough to go to the last 1.2 minor to solve
this issue.

For your information, even C* 2.0 is no longer supported. The minimum
version you should use now is 2.1.last.

This technical debt might end up costing you more in terms of time, money
and Quality of Service that taking care of upgrades. The most probable
thing is that your bug is fixed already on newer versions. Plus it is not
very interesting for us to help you as we would have to go through old
code, to find issues that are most likely already fixed. If you want some
support (from community or commercial one) you really should upgrade this
cluster. Make sure your clients are compatible too.

I did not know that some people were still using C* < 2.0 :-).

Cheers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-04-13 10:58 GMT+02:00 Michael Fong :

> Hi, all
>
>
>
>
>
> We have been a Cassandra 4-node cluster (C* 1.2.x) where a node marked all
> the other 3 nodes DOWN, and came back UP a few seconds later. There was a
> compaction that kicked in a minute before, roughly 10~MB in size, followed
> by marking all the other nodes DOWN later. In the other words, in the
> system.log we see
>
> 00:00:00 Compacting ….
>
> 00:00:03 Compacted 8 sstables … 10~ megabytes
>
> 00:01:06 InetAddress /x.x.x.4 is now DOWN
>
> 00:01:06 InetAddress /x.x.x.3 is now DOWN
>
> 00:01:06 InetAddress /x.x.x.1 is now DOWN
>
>
>
> There was no significant GC activities in gc.log. We have heard that busy
> compaction activities would cause this behavior, but we cannot reason why
> this could happen logically. How come a compaction operation would stop the
> Gossip thread to perform heartbeat check? Has anyone experienced this kind
> of behavior before?
>
>
>
> Thanks in advanced!
>
>
>
> Sincerely,
>
>
>
> Michael Fong
>

Balancing tokens over 2 datacenter

[Walsh, Stephen]

Hi there,

So we have 2 datacenter with 3 nodes each.
Replication factor is 3 per DC (so each node has all data)

We have an application in each DC that writes that Cassandra DC.

Now, due to a miss configuration in our application, we saw that our 
application in both DC’s where pointing to DC1.

As such, all keyspaces and tables where created on DC1.
The effect of this is that all reads are now going to DC1 and ignoring DC2

WE’ve tried doing , nodetool repair / cleanup – but the reads always go to DC1?

Anyone know how to rebalance the tokens over DC’s?


Regards
Steve


P.S I know about this article
http://www.datastax.com/dev/blog/balancing-your-cassandra-cluster
But its doesn’t answer my question regarding 2 DC’s token balancing

This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

Creation of Async Datacenter for Monitoring purposes and Engineering Services purpose

[Bhuvan Rawal]

Hi All,

We have 2 Running Datacenters in physically seperate DC's with 3 Nodes
each. There is a requirement of an Audit DC for issuing queries which will
not be concerned with live application traffic. Live Data delay of 1-2
Hours is acceptable. It is essential that replication to this DC to not
impact the other 2 data centers.

Is there a way to copy data in async fashion to new DC3 so that it doesnt
impact existing DC's, possibly without querying (using sstable/commitlog).

Regards,
Bhuvan

C* 1.2.x vs Gossip marking DOWN/UP

[Michael Fong]

Hi, all


We have been a Cassandra 4-node cluster (C* 1.2.x) where a node marked all the 
other 3 nodes DOWN, and came back UP a few seconds later. There was a 
compaction that kicked in a minute before, roughly 10~MB in size, followed by 
marking all the other nodes DOWN later. In the other words, in the system.log 
we see
00:00:00 Compacting 
00:00:03 Compacted 8 sstables ... 10~ megabytes
00:01:06 InetAddress /x.x.x.4 is now DOWN
00:01:06 InetAddress /x.x.x.3 is now DOWN
00:01:06 InetAddress /x.x.x.1 is now DOWN

There was no significant GC activities in gc.log. We have heard that busy 
compaction activities would cause this behavior, but we cannot reason why this 
could happen logically. How come a compaction operation would stop the Gossip 
thread to perform heartbeat check? Has anyone experienced this kind of behavior 
before?

Thanks in advanced!

Sincerely,

Michael Fong

Set up authentication on a live production cluster

[Vigneshwaran]

Hi,

I have setup a 16 node cluster (8 per DC; C* 2.2.4) up and running in our
production setup. We use Datastax Java driver 2.1.8.

I would like to set up Authentication and Authorization in the cluster
without breaking the live clients.

>From the references I found by googling, I can setup credentials for a new
cluster. But it is not clear to me what steps I should take for setting up
credentials in an already running cluster without breaking existing clients.

Can someone clarify me or link me to a reference I may have missed? I'd
really appreciate it.

Thank you,
Vigneshwaran