Re: Lightweight transaction in Multi DC

[Charulata Sharma (charshar)]

Thanks for your reply. I understand that LOCAL_SERIAL is for within a DC , will 
setting up SERIAL not slow down the operation?
And should I set SERIAL for both read and write phase or just Read phase? This 
is becoming a big problem for us. It happens for a small percentage of
data set but it happens daily because of the highly concurrent and random DC 
routing of the received events.

We had a counter implementation prior to CAS, which used to fail most of the 
times. CAS provided a big relief but it is still not totally without errors.
Would really appreciate the community’s feedback.

Thanks,
Charu

From: vasu gunja 
Reply-To: "user@cassandra.apache.org" 
Date: Friday, September 8, 2017 at 1:56 PM
To: "user@cassandra.apache.org" 
Subject: Re: Lightweight transaction in Multi DC

LOCAL_SERIAL is dc level, SERIAL checks for complete cluster level.

On Fri, Sep 8, 2017 at 2:33 PM, Charulata Sharma (charshar) 
> wrote:
Yes …it is with LOCAL_SERIAL. Should I be using SERIAL ?

Thanks,
Charu

From: DuyHai Doan >
Reply-To: "user@cassandra.apache.org" 
>
Date: Friday, September 8, 2017 at 12:30 PM
To: "user@cassandra.apache.org" 
>
Subject: Re: Lightweight transaction in Multi DC

Are you using CAS with SERIAL consistency level for your multi-DC setup ?

On Fri, Sep 8, 2017 at 9:27 PM, Charulata Sharma (charshar) 
> wrote:
Hi,
  We are facing a serious issue with CAS in a multi DC setup and I wanted 
to get some input on it from the forum.

We have a Column family which stores counts for the number of events our 
application receives. When the counts reach a certain threshold,
there is another process which kicks in. The issue that is happening is, 
sometimes when concurrent events come in 2 Data Centers, both the events read 
the same count
and increment it to the same value which causes the next process to not kick in.

When concurrent events come in the same DC, this does not happen, because in 
this case, the CAS explicitly fails for one of them.
However, in case of Multiple DCs this is not happening. Has anyone faced a 
similar issue and is there any resolution to this?? I was unable to find any 
open Jira on this.

Thanks,
Charu

Re: Lightweight transaction in Multi DC

[vasu gunja]

LOCAL_SERIAL is dc level, SERIAL checks for complete cluster level.

On Fri, Sep 8, 2017 at 2:33 PM, Charulata Sharma (charshar) <
chars...@cisco.com> wrote:

> Yes …it is with LOCAL_SERIAL. Should I be using SERIAL ?
>
>
>
> Thanks,
>
> Charu
>
>
>
> *From: *DuyHai Doan 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Friday, September 8, 2017 at 12:30 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Lightweight transaction in Multi DC
>
>
>
> Are you using CAS with SERIAL consistency level for your multi-DC setup ?
>
>
>
> On Fri, Sep 8, 2017 at 9:27 PM, Charulata Sharma (charshar) <
> chars...@cisco.com> wrote:
>
> Hi,
>
>   We are facing a serious issue with CAS in a multi DC setup and I
> wanted to get some input on it from the forum.
>
>
>
> We have a Column family which stores counts for the number of events our
> application receives. When the counts reach a certain threshold,
>
> there is another process which kicks in. The issue that is happening is,
> sometimes when concurrent events come in 2 Data Centers, both the events
> read the same count
>
> and increment it to the same value which causes the next process to not
> kick in.
>
>
>
> When concurrent events come in the same DC, this does not happen, because
> in this case, the CAS explicitly fails for one of them.
>
> However, in case of Multiple DCs this is not happening. Has anyone faced a
> similar issue and is there any resolution to this?? I was unable to find
> any open Jira on this.
>
>
>
> Thanks,
>
> Charu
>
>
>
>
>
>
>

Re: How to do cassandra routine maintenance

[Romain Hardouin]

 Hi,
You should read about repair maintenance: 
http://cassandra.apache.org/doc/latest/operating/repair.htmlConsider installing 
and running C* reaper to do so: http://cassandra-reaper.io/STCS doesn't work 
well with TTL. I saw you have done some tuning, hard to say if it's OK without 
knowing the workload.LCS is better for TTL (but requires fast disks) and if 
you're working with time series consider TWCS.If CPU are not overloaded you can 
also consider Snappy compression (btw check compression ratio).Again depending 
on your data model and your queries, chunk_length_in_kb might be increased to 
have a more effective compression (generally speaking we tend to lower it to 
improve read latency).
Best,
Romain
Le samedi 2 septembre 2017 à 04:17:22 UTC+2, qf zhou  
a écrit :  
 
 I am using the cluster with 3 cassandra  nodes, the cluster version is 3.0.9. 
Each day about 200~300 million records are inserted into the cluster.
As time goes by,  more and more data occupied more and more disk space. 
Currently,    the data distribution  on each node is  as  the following:

UN  172.20.5.4  2.5 TiB    256          66.3%            
c5271e74-19a1-4cee-98d7-dc169cf87e95  rack1
UN  172.20.5.2  1.73 TiB  256          67.0%            
c623bbc0-9839-4d2d-8ff3-db7115719d59  rack1
UN  172.20.5.3  1.86 TiB  256          66.7%            
c555e44c-9590-4f45-aea4-f5eca68180b2  rack1 

There is only one datacenter.  

The compaciton strategy is here:
    compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '12', 'tombstone_threshold': '0.1', 
'unchecked_tombstone_compaction': 'true'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 864
    AND gc_grace_seconds = 432000

I really want to know  about how to do cassandra routine maintenance ?

I found the data seems to grow faster  and  the disk is in heavy load. 
Sometimes I found the data inconsistency: two different results appear with the 
same query.

So what I shoud I do to keep the cluster healthy,  how to maintain the cluster?

I hope  some help  very much!  Thanks a lot ! 



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
  

Re: Cassandra 3.11 is compacting forever

[Romain Hardouin]

Hi,
It might be useful to enable compaction logging with log_all subproperties.
Best,
Romain 
Le vendredi 8 septembre 2017 à 00:15:19 UTC+2, kurt greaves 
 a écrit :  
 
 Might be worth turning on debug logging for that node and when the compaction 
kicks off and CPU skyrockets send through the logs.​

Re: Lightweight transaction in Multi DC

[Charulata Sharma (charshar)]

Yes …it is with LOCAL_SERIAL. Should I be using SERIAL ?

Thanks,
Charu

From: DuyHai Doan 
Reply-To: "user@cassandra.apache.org" 
Date: Friday, September 8, 2017 at 12:30 PM
To: "user@cassandra.apache.org" 
Subject: Re: Lightweight transaction in Multi DC

Are you using CAS with SERIAL consistency level for your multi-DC setup ?

On Fri, Sep 8, 2017 at 9:27 PM, Charulata Sharma (charshar) 
> wrote:
Hi,
  We are facing a serious issue with CAS in a multi DC setup and I wanted 
to get some input on it from the forum.

We have a Column family which stores counts for the number of events our 
application receives. When the counts reach a certain threshold,
there is another process which kicks in. The issue that is happening is, 
sometimes when concurrent events come in 2 Data Centers, both the events read 
the same count
and increment it to the same value which causes the next process to not kick in.

When concurrent events come in the same DC, this does not happen, because in 
this case, the CAS explicitly fails for one of them.
However, in case of Multiple DCs this is not happening. Has anyone faced a 
similar issue and is there any resolution to this?? I was unable to find any 
open Jira on this.

Thanks,
Charu

Re: Lightweight transaction in Multi DC

[DuyHai Doan]

Are you using CAS with SERIAL consistency level for your multi-DC setup ?

On Fri, Sep 8, 2017 at 9:27 PM, Charulata Sharma (charshar) <
chars...@cisco.com> wrote:

> Hi,
>
>   We are facing a serious issue with CAS in a multi DC setup and I
> wanted to get some input on it from the forum.
>
>
>
> We have a Column family which stores counts for the number of events our
> application receives. When the counts reach a certain threshold,
>
> there is another process which kicks in. The issue that is happening is,
> sometimes when concurrent events come in 2 Data Centers, both the events
> read the same count
>
> and increment it to the same value which causes the next process to not
> kick in.
>
>
>
> When concurrent events come in the same DC, this does not happen, because
> in this case, the CAS explicitly fails for one of them.
>
> However, in case of Multiple DCs this is not happening. Has anyone faced a
> similar issue and is there any resolution to this?? I was unable to find
> any open Jira on this.
>
>
>
> Thanks,
>
> Charu
>
>
>
>
>

Lightweight transaction in Multi DC

[Charulata Sharma (charshar)]

Hi,
  We are facing a serious issue with CAS in a multi DC setup and I wanted 
to get some input on it from the forum.

We have a Column family which stores counts for the number of events our 
application receives. When the counts reach a certain threshold,
there is another process which kicks in. The issue that is happening is, 
sometimes when concurrent events come in 2 Data Centers, both the events read 
the same count
and increment it to the same value which causes the next process to not kick in.

When concurrent events come in the same DC, this does not happen, because in 
this case, the CAS explicitly fails for one of them.
However, in case of Multiple DCs this is not happening. Has anyone faced a 
similar issue and is there any resolution to this?? I was unable to find any 
open Jira on this.

Thanks,
Charu

Re: Limit on number of connections to Cassandra

[Andy Tolbert]

Hello,

If I'm understanding the question correctly, as of C* 2.0.15 / 2.1.5 via
CASSANDRA-8086  you
can limit the maximum number of connections allowed to a C* node via
native_transport_max_concurrent_connections

in
cassandra.yaml.

As far as the java driver goes, newer versions (i.e. 2.1.10+, 3.0.1+)
behave in such a way that as long as the driver can maintain at least one
connection to a node it considers it up.  If it can no longer maintain
connection it will not send requests to it and will try reconnecting per
the configured retry policy.  By default with C* 2.1+ (protocol version
V3+), the driver will only maintain one connection per host in local data
center, although you can tweak this using PoolingOptions

.

Thanks,
Andy


On Fri, Sep 8, 2017 at 1:27 PM techpyaasa .  wrote:

> Hi
>
> Is there any limit on number of client connections to Cassandra just like
> MySQL etc., ?
>
> If YES, what is that & how can we set that?
>
> If NO , how will get to know that node has reached it's capacity serving
> client requests/over loaded?
>
> Using C*-2.1.17 , datastax java driver
>
>
> Thanks
> Techpyaasa
>

Limit on number of connections to Cassandra

[techpyaasa .]

Hi

Is there any limit on number of client connections to Cassandra just like
MySQL etc., ?

If YES, what is that & how can we set that?

If NO , how will get to know that node has reached it's capacity serving
client requests/over loaded?

Using C*-2.1.17 , datastax java driver


Thanks
Techpyaasa

Re: New Column addition and Locking

[Jeff Jirsa]

CASSANDRA-13004

-- 
Jeff Jirsa


> On Sep 8, 2017, at 7:33 AM, Deepak Panda  wrote:
> 
> Many Thanks Jeff for the answer and the relevant details. Can you
> please share the bug#?
> 
> Are there any opensource tools available for Cassandra which manage
> schema changes for Cassandra?
> 
>> On Fri, Sep 8, 2017 at 7:57 PM, Jeff Jirsa  wrote:
>> There is no lock and it doesn't block existing reads or writes
>> 
>> You're already on 3.11, but as a reminder for others there's a bug in 3.0 
>> versions between (3.0.0 and 3.0.14) and (3.0 and 3.11.0) that can cause 
>> corruption when you add columns. It's fixed in both 3.11.0 and 3.0.14, so if 
>> you need to alter your table and you're on a 3.something version that isn't 
>> one of those, upgrade first.
>> 
>> --
>> Jeff Jirsa
>> 
>> 
>>> On Sep 8, 2017, at 7:13 AM, Deepak Panda  wrote:
>>> 
>>> Hi,
>>> 
>>> For Cassandra 3.11, when we add a new column is added to an existing
>>> table having huge number of rows and huge data manipulation going on
>>> will there be any issues?
>>> 
>>> Will the operations blocked till the column is added?
>>> 
>>> Is the table locked during column addition? If yes, what type of lock?
>>> 
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>> 
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: New Column addition and Locking

[Deepak Panda]

Many Thanks Jeff for the answer and the relevant details. Can you
please share the bug#?

Are there any opensource tools available for Cassandra which manage
schema changes for Cassandra?

On Fri, Sep 8, 2017 at 7:57 PM, Jeff Jirsa  wrote:
> There is no lock and it doesn't block existing reads or writes
>
> You're already on 3.11, but as a reminder for others there's a bug in 3.0 
> versions between (3.0.0 and 3.0.14) and (3.0 and 3.11.0) that can cause 
> corruption when you add columns. It's fixed in both 3.11.0 and 3.0.14, so if 
> you need to alter your table and you're on a 3.something version that isn't 
> one of those, upgrade first.
>
> --
> Jeff Jirsa
>
>
>> On Sep 8, 2017, at 7:13 AM, Deepak Panda  wrote:
>>
>> Hi,
>>
>> For Cassandra 3.11, when we add a new column is added to an existing
>> table having huge number of rows and huge data manipulation going on
>> will there be any issues?
>>
>> Will the operations blocked till the column is added?
>>
>> Is the table locked during column addition? If yes, what type of lock?
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: New Column addition and Locking

[Jeff Jirsa]

There is no lock and it doesn't block existing reads or writes

You're already on 3.11, but as a reminder for others there's a bug in 3.0 
versions between (3.0.0 and 3.0.14) and (3.0 and 3.11.0) that can cause 
corruption when you add columns. It's fixed in both 3.11.0 and 3.0.14, so if 
you need to alter your table and you're on a 3.something version that isn't one 
of those, upgrade first. 

-- 
Jeff Jirsa


> On Sep 8, 2017, at 7:13 AM, Deepak Panda  wrote:
> 
> Hi,
> 
> For Cassandra 3.11, when we add a new column is added to an existing
> table having huge number of rows and huge data manipulation going on
> will there be any issues?
> 
> Will the operations blocked till the column is added?
> 
> Is the table locked during column addition? If yes, what type of lock?
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

New Column addition and Locking

[Deepak Panda]

Hi,

For Cassandra 3.11, when we add a new column is added to an existing
table having huge number of rows and huge data manipulation going on
will there be any issues?

Will the operations blocked till the column is added?

Is the table locked during column addition? If yes, what type of lock?

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Stop writes to a cassandra node for maintenance purpose & enable them after some time

[Indraja and Arvydas Jonusonis]

Keep in mind that hints will accumulate for this node across the cluster
for *max_hint_window_in_ms* (default: 3hrs)

On Fri, Sep 8, 2017 at 8:20 AM, Jeff Jirsa  wrote:

> nodetool disablebinary (will make the node down for native cql clients,
> stopping client apps from writing directly over tcp/9042)
> nodetool disablethrift (will make the node down for thrift clients,
> stopping client apps from writing directly over tcp/9160)
> nodetool disablegossip (will make the node appear down to other nodes in
> the ring, stopping writes from the internode messaging protocols)
>
>
>
> On Thu, Sep 7, 2017 at 10:32 PM, Akshit Jain 
> wrote:
>
>> Hi,
>> I want to stop writes to a cassandra node for some time and then start
>> them.Is there any way to do that?
>> Regards
>>
>>
>

Re: Stop writes to a cassandra node for maintenance purpose & enable them after some time

[Jeff Jirsa]

nodetool disablebinary (will make the node down for native cql clients,
stopping client apps from writing directly over tcp/9042)
nodetool disablethrift (will make the node down for thrift clients,
stopping client apps from writing directly over tcp/9160)
nodetool disablegossip (will make the node appear down to other nodes in
the ring, stopping writes from the internode messaging protocols)



On Thu, Sep 7, 2017 at 10:32 PM, Akshit Jain 
wrote:

> Hi,
> I want to stop writes to a cassandra node for some time and then start
> them.Is there any way to do that?
> Regards
>
>