Re: default_time_to_live vs TTL on insert statement

[kurt greaves]

The Datastax documentation is wrong. It won't error, and it shouldn't. If
you want to fix that documentation I suggest contacting Datastax.

On 11 July 2018 at 19:56, Nitan Kainth  wrote:

> Hi DuyHai,
>
> Could you please explain in what case C* will error based on documented
> statement:
>
> You can set a default TTL for an entire table by setting the table's
> default_time_to_live
> 
>  property. If you try to set a TTL for a specific column that is longer
> than the time defined by the table TTL, Cassandra returns an error.
>
>
>
> On Wed, Jul 11, 2018 at 2:34 PM, DuyHai Doan  wrote:
>
>> default_time_to_live
>> 
>>  property applies if you don't specify any TTL on your CQL statement
>>
>> However you can always override the default_time_to_live
>> 
>>  property by specifying a custom value for each CQL statement
>>
>> The behavior is correct, nothing wrong here
>>
>> On Wed, Jul 11, 2018 at 7:31 PM, Nitan Kainth 
>> wrote:
>>
>>> Hi,
>>>
>>> As per document: https://docs.datastax.com/en/cql/3.3/cql/cql_using
>>> /useExpireExample.html
>>>
>>>
>>>-
>>>
>>>You can set a default TTL for an entire table by setting the table's
>>>default_time_to_live
>>>
>>> 
>>> property. If you try to set a TTL for a specific column that is
>>>longer than the time defined by the table TTL, Cassandra returns an 
>>> error.
>>>
>>>
>>> When I tried to test this statement, i found, we can insert data with
>>> TTL greater than default_time_to_live. Is the document needs correction, or
>>> am I mis-understanding it?
>>>
>>> CREATE TABLE test (
>>>
>>> name text PRIMARY KEY,
>>>
>>> description text
>>>
>>> ) WITH bloom_filter_fp_chance = 0.01
>>>
>>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>>
>>> AND comment = ''
>>>
>>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>>> 'min_threshold': '4'}
>>>
>>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>>
>>> AND crc_check_chance = 1.0
>>>
>>> AND dclocal_read_repair_chance = 0.1
>>>
>>> AND default_time_to_live = 240
>>>
>>> AND gc_grace_seconds = 864000
>>>
>>> AND max_index_interval = 2048
>>>
>>> AND memtable_flush_period_in_ms = 0
>>>
>>> AND min_index_interval = 128
>>>
>>> AND read_repair_chance = 0.0
>>>
>>> AND speculative_retry = '99PERCENTILE';
>>>
>>> insert into test (name, description) values ('name5', 'name
>>> description5') using ttl 360;
>>>
>>> select * from test ;
>>>
>>>
>>>  name  | description
>>>
>>> ---+---
>>>
>>>  name5 | name description5
>>>
>>>
>>> SELECT TTL (description) from test;
>>>
>>>
>>>  ttl(description)
>>>
>>> --
>>>
>>>  351
>>>
>>> Can someone please clear this for me?
>>>
>>>
>>>
>>>
>>>
>>>
>>
>

Re: Inconsistent Quorum Read after Quorum Write

[Mick Semb Wever]

Li,

I did not reset repairedAt and ran repair with -pr directly. That’s
> probably why the inconsistency occurred.
>

Yes, this will be a likely cause. There's enough docs out there to help you
with this. Shout out if not.



> As our tables are pretty big, full repair takes many days to finish. Given
> the 10 days gc period, it means repair almost will run all the time.
> Consistency is more important to us and the cluster takes the same amount
> of write and read requests. Temporary outage is allowed but if a dead node
> can’t come back in time, we will go back to Quorum mode.
>

Yes, repairs can be a real headache.
Install Reaper. Seriously. http://cassandra-reaper.io/



> What’s new in 3.11.3? We’ve been running on C* for almost 2 years. The
> biggest pain point is about repair. Especially with 3.11.1, incremental
> repair doesn’t work well compared to our experience with 3.10. Maybe it’s
> just because our data size wasn’t that big before upgrade...
>


3.11.2 and 3.11.3 are just patch releases on top of 3.11.1.
It's definitely recommended to always *test* and upgrade to the latest
patch release. And it's kinda a prerequisite if you want help from the open
source community, none of us really enjoy debugging old code  :-)

regards,
Mick

-- 
Mick Semb Wever
Australia

The Last Pickle
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Check Cluster Health

[Thouraya TH]

Hi,
Ok.
Thanks a lot for answers.
Kind regards.


2018-07-11 16:13 GMT+01:00 Furkan Cifci :

> No, you dont need to install Prometheus on each node. Install Prometheus
> on one machine and configure it. For basic configuration use this
> documentation: https://www.robustperception.io/monitoring-cassandra-with-
> prometheus/
> You need to use exporter on each node for collecting metrics.
>
> Node exporter will give you system metrics like cpu,ram,disk I/O etc.
> https://github.com/prometheus/node_exporter
> Jmx/Cassandra Exporter will give you  detailed Cassandra metrics like
> Client r-w latencies, node status, jvm metrics and such.
> https://github.com/prometheus/jmx_exporter
>
>
>
>
> 2018-07-06 18:01 GMT+03:00 Thouraya TH :
>
>>
>>
>> 2018-07-06 13:04 GMT+01:00 Simon Fontana Oscarsson <
>> simon.fontana.oscars...@ericsson.com>:
>>
>>> Running nodetool status is okay if you want the simplest solution.
>>> But it generates a lot of output and creates a new JMX connection for
>>> every execution.
>>> Cassandra uses JMX to expose metrics via mbeans.
>>> Read this to get a first understanding: https://docs.da
>>> tastax.com/en/cassandra/2.1/cassandra/operations/ops_monitoring_c.html
>>> Use Jconsole to explore the different metrics. Use documentation as
>>> reference: https://cassandra.apache.org/doc/latest/operating
>>> /metrics.html
>>>
>>> As for your solution I recommend one of the following:
>>> * Create a simple JMX client and add your beans. You can do some simple
>>> logging with Logback or log4j. You can get some help by googling.
>>> * Use a monitoring system such as Prometheus. This is the best solution
>>> but most time consuming.
>>>
>>
>> If i use  Prometheus , i have to install it on each node on my data
>> center ? It will give me details about all nodes connections as does
>> nodetool status?
>>
>>>
>>> --
>>> SIMON FONTANA OSCARSSON
>>> Software Developer
>>>
>>> Ericsson
>>> Ölandsgatan 1
>>> 
>>> 37133 Karlskrona, Sweden
>>> simon.fontana.oscars...@ericsson.com
>>> www.ericsson.com
>>>
>>> On fre, 2018-07-06 at 11:18 +0100, Thouraya TH wrote:
>>> > Hi,
>>> > Thank you so much for answers.
>>> >
>>> > Please, can you explain more what's metric libraries ? and give me
>>> some examples ?
>>> >
>>> > Using nodetool status, to generate the history of my data center, i
>>> intend to proceed as follows:
>>> >
>>> > From a node A:
>>> >
>>> > For i  1 ..24 hours  (every 2 minutes do)
>>> >
>>> > ./nodetool status >> file.txt
>>> >
>>> > End For
>>> >
>>> >
>>> > is it a good idea?
>>> >
>>> > Thanks a lot.
>>> > Kind regards.
>>> >
>>> > 2018-07-05 1:30 GMT+01:00 Anthony Grasso :
>>> > > Hi,
>>> > >
>>> > > Yes, you can use nodetool status to inspect the health/status of the
>>> cluster. Using nodetool status  will show the cluster
>>> health/status as well as the amount of data that each node has
>>> > > for the specified .  Using nodetool status without the
>>>  argument will only show the cluster health/status.
>>> > >
>>> > > Unless there is a special reason for using nodetool to capture
>>> history, you may want to consider using metric libraries to capture and
>>> push information about each node to a metric server. It is
>>> > > much easier to view the data captured on the metric server as there
>>> are tools already made for this. Using metrics libraries will save you time
>>> creating and maintaining a parser for the nodetool
>>> > > output. It also makes monitoring the health of cluster very easy.
>>> > >
>>> > > Regards,
>>> > > Anthony
>>> > >
>>> > > On Sun, 1 Jul 2018 at 20:19, Thouraya TH 
>>> wrote:
>>> > > > Hi,
>>> > > > Thank you so much for answer.
>>> > > > Please, is it possible to use this command ?
>>> > > > nodetool status mykeyspace
>>> > > >
>>> > > > Datacenter: datacenter1
>>> > > > ===
>>> > > > Status=Up/Down
>>> > > > |/ State=Normal/Leaving/Joining/Moving
>>> > > > --  AddressLoad   Tokens  OwnsHost
>>> ID   Rack
>>> > > > UN  127.0.0.1  47.66 KB   1   33.3%   aaa1b7c1-
>>> 6049-4a08-ad3e-3697a0e30e10  rack1
>>> > > > UN  127.0.0.2  47.67 KB   1   33.3%   1848c369-
>>> 4306-4874-afdf-5c1e95b8732e  rack1
>>> > > > UN
>>> > > > Thank you so much.
>>> > > > Kind regards.
>>> > > >
>>> > > > 2018-06-29 1:40 GMT+01:00 Rahul Singh <
>>> rahul.xavier.si...@gmail.com>:
>>> > > > >
>>> > > > >
>>> > > > > When you run TPstats or Tablestats subcommands in nodetool you
>>> are actually accessing data inside Cassandra via JMX.
>>> > > > >
>>> > > > > You can start there at first.
>>> > > > >
>>> > > > > Rahul
>>> > > > > On Jun 28, 2018, 10:55 AM -0500, Thouraya TH <
>>> thouray...@gmail.com>, wrote:
>>> > > > > > Hi,
>>> > > > > >
>>> > > > > > Please, how can check the health of my cluster / data center
>>> using cassandra ?
>>> > > > > > In fact i'd like to generate a hitory of the state of each
>>> node. an history 

Re: default_time_to_live vs TTL on insert statement

[Nitan Kainth]

Hi DuyHai,

Could you please explain in what case C* will error based on documented
statement:

You can set a default TTL for an entire table by setting the table's
default_time_to_live

 property. If you try to set a TTL for a specific column that is longer
than the time defined by the table TTL, Cassandra returns an error.



On Wed, Jul 11, 2018 at 2:34 PM, DuyHai Doan  wrote:

> default_time_to_live
> 
>  property applies if you don't specify any TTL on your CQL statement
>
> However you can always override the default_time_to_live
> 
>  property by specifying a custom value for each CQL statement
>
> The behavior is correct, nothing wrong here
>
> On Wed, Jul 11, 2018 at 7:31 PM, Nitan Kainth 
> wrote:
>
>> Hi,
>>
>> As per document: https://docs.datastax.com/en/cql/3.3/cql/cql_
>> using/useExpireExample.html
>>
>>
>>-
>>
>>You can set a default TTL for an entire table by setting the table's
>>default_time_to_live
>>
>> 
>> property. If you try to set a TTL for a specific column that is
>>longer than the time defined by the table TTL, Cassandra returns an error.
>>
>>
>> When I tried to test this statement, i found, we can insert data with TTL
>> greater than default_time_to_live. Is the document needs correction, or am
>> I mis-understanding it?
>>
>> CREATE TABLE test (
>>
>> name text PRIMARY KEY,
>>
>> description text
>>
>> ) WITH bloom_filter_fp_chance = 0.01
>>
>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>
>> AND comment = ''
>>
>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>> 'min_threshold': '4'}
>>
>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>
>> AND crc_check_chance = 1.0
>>
>> AND dclocal_read_repair_chance = 0.1
>>
>> AND default_time_to_live = 240
>>
>> AND gc_grace_seconds = 864000
>>
>> AND max_index_interval = 2048
>>
>> AND memtable_flush_period_in_ms = 0
>>
>> AND min_index_interval = 128
>>
>> AND read_repair_chance = 0.0
>>
>> AND speculative_retry = '99PERCENTILE';
>>
>> insert into test (name, description) values ('name5', 'name
>> description5') using ttl 360;
>>
>> select * from test ;
>>
>>
>>  name  | description
>>
>> ---+---
>>
>>  name5 | name description5
>>
>>
>> SELECT TTL (description) from test;
>>
>>
>>  ttl(description)
>>
>> --
>>
>>  351
>>
>> Can someone please clear this for me?
>>
>>
>>
>>
>>
>>
>

Re: default_time_to_live vs TTL on insert statement

[DuyHai Doan]

default_time_to_live

 property applies if you don't specify any TTL on your CQL statement

However you can always override the default_time_to_live

 property by specifying a custom value for each CQL statement

The behavior is correct, nothing wrong here

On Wed, Jul 11, 2018 at 7:31 PM, Nitan Kainth  wrote:

> Hi,
>
> As per document: https://docs.datastax.com/en/cql/3.3/cql/
> cql_using/useExpireExample.html
>
>
>-
>
>You can set a default TTL for an entire table by setting the table's
>default_time_to_live
>
> 
> property. If you try to set a TTL for a specific column that is
>longer than the time defined by the table TTL, Cassandra returns an error.
>
>
> When I tried to test this statement, i found, we can insert data with TTL
> greater than default_time_to_live. Is the document needs correction, or am
> I mis-understanding it?
>
> CREATE TABLE test (
>
> name text PRIMARY KEY,
>
> description text
>
> ) WITH bloom_filter_fp_chance = 0.01
>
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>
> AND comment = ''
>
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
>
> AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
>
> AND crc_check_chance = 1.0
>
> AND dclocal_read_repair_chance = 0.1
>
> AND default_time_to_live = 240
>
> AND gc_grace_seconds = 864000
>
> AND max_index_interval = 2048
>
> AND memtable_flush_period_in_ms = 0
>
> AND min_index_interval = 128
>
> AND read_repair_chance = 0.0
>
> AND speculative_retry = '99PERCENTILE';
>
> insert into test (name, description) values ('name5', 'name description5')
> using ttl 360;
>
> select * from test ;
>
>
>  name  | description
>
> ---+---
>
>  name5 | name description5
>
>
> SELECT TTL (description) from test;
>
>
>  ttl(description)
>
> --
>
>  351
>
> Can someone please clear this for me?
>
>
>
>
>
>

default_time_to_live vs TTL on insert statement

[Nitan Kainth]

Hi,

As per document:
https://docs.datastax.com/en/cql/3.3/cql/cql_using/useExpireExample.html


   -

   You can set a default TTL for an entire table by setting the table's
   default_time_to_live
   

property. If you try to set a TTL for a specific column that is longer
   than the time defined by the table TTL, Cassandra returns an error.


When I tried to test this statement, i found, we can insert data with TTL
greater than default_time_to_live. Is the document needs correction, or am
I mis-understanding it?

CREATE TABLE test (

name text PRIMARY KEY,

description text

) WITH bloom_filter_fp_chance = 0.01

AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

AND comment = ''

AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}

AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}

AND crc_check_chance = 1.0

AND dclocal_read_repair_chance = 0.1

AND default_time_to_live = 240

AND gc_grace_seconds = 864000

AND max_index_interval = 2048

AND memtable_flush_period_in_ms = 0

AND min_index_interval = 128

AND read_repair_chance = 0.0

AND speculative_retry = '99PERCENTILE';

insert into test (name, description) values ('name5', 'name description5')
using ttl 360;

select * from test ;


 name  | description

---+---

 name5 | name description5


SELECT TTL (description) from test;


 ttl(description)

--

 351

Can someone please clear this for me?

Re: Inconsistent Quorum Read after Quorum Write

[Visa]

Hi Mick,

Thanks for replying!

I did not reset repairedAt and ran repair with -pr directly. That’s probably 
why the inconsistency occurred.

As our tables are pretty big, full repair takes many days to finish. Given the 
10 days gc period, it means repair almost will run all the time. Consistency is 
more important to us and the cluster takes the same amount of write and read 
requests. Temporary outage is allowed but if a dead node can’t come back in 
time, we will go back to Quorum mode.

What’s new in 3.11.3? We’ve been running on C* for almost 2 years. The biggest 
pain point is about repair. Especially with 3.11.1, incremental repair doesn’t 
work well compared to our experience with 3.10. Maybe it’s just because our 
data size wasn’t that big before upgrade...

Thanks,
Li

> On Jul 11, 2018, at 05:32, Mick Semb Wever  wrote:
> 
> 
> Li,
> 
> 
>> I’ve confirmed that the inconsistency issues disappeared after repair 
>> finished.
>> 
>> Anything changed with repair in 3.11.1? One difference I noticed is that the 
>> validation step during repair could turn down the node upon large tables, 
>> which never happen in 3.10. I had to throttle validation requests to let it 
>> pass. Also I switched back to -pr instead of incremental repair which is a 
>> resource killer and often hangs for the first node to be repaired.
> 
> 
> When you switched back to non-incremental did you set `repairedAt` on all 
> sstables (on all nodes) back to zero (or unrepaired state)?
> This should have been done with `sstablerepairedset --is-unrepaired … ` while 
> the node is stopped.
>  
>  
>> To address the inconsistency issue, I could do Write All and Read One by 
>> giving up availability and stop running repair. Any comments on that?
> 
> 
> You loose availability doing this, and at the number of reads you're doing I 
> would not recommend it.
> You could think about using a fallback strategy that initially tries CL.ALL 
> and falls back to CL.QUORUM. But this is a hack, could overload your cluster, 
> and if there's any correlation to dropped messages or flapping nodes won't 
> help.
> 
> I'd also be prepared to upgrade to 3.11.3, when it does get released.
> 
> regards,
> Mick
> 
> -- 
> Mick Semb Wever
> Australia
> 
> The Last Pickle
> Apache Cassandra Consulting
> http://www.thelastpickle.com

Re: Check Cluster Health

[Furkan Cifci]

No, you dont need to install Prometheus on each node. Install Prometheus on
one machine and configure it. For basic configuration use this
documentation:
https://www.robustperception.io/monitoring-cassandra-with-prometheus/
You need to use exporter on each node for collecting metrics.

Node exporter will give you system metrics like cpu,ram,disk I/O etc.
https://github.com/prometheus/node_exporter
Jmx/Cassandra Exporter will give you  detailed Cassandra metrics like
Client r-w latencies, node status, jvm metrics and such.
https://github.com/prometheus/jmx_exporter




2018-07-06 18:01 GMT+03:00 Thouraya TH :

>
>
> 2018-07-06 13:04 GMT+01:00 Simon Fontana Oscarsson <
> simon.fontana.oscars...@ericsson.com>:
>
>> Running nodetool status is okay if you want the simplest solution.
>> But it generates a lot of output and creates a new JMX connection for
>> every execution.
>> Cassandra uses JMX to expose metrics via mbeans.
>> Read this to get a first understanding: https://docs.da
>> tastax.com/en/cassandra/2.1/cassandra/operations/ops_monitoring_c.html
>> Use Jconsole to explore the different metrics. Use documentation as
>> reference: https://cassandra.apache.org/doc/latest/operating/metrics.html
>>
>> As for your solution I recommend one of the following:
>> * Create a simple JMX client and add your beans. You can do some simple
>> logging with Logback or log4j. You can get some help by googling.
>> * Use a monitoring system such as Prometheus. This is the best solution
>> but most time consuming.
>>
>
> If i use  Prometheus , i have to install it on each node on my data center
> ? It will give me details about all nodes connections as does nodetool
> status?
>
>>
>> --
>> SIMON FONTANA OSCARSSON
>> Software Developer
>>
>> Ericsson
>> Ölandsgatan 1
>> 
>> 37133 Karlskrona, Sweden
>> simon.fontana.oscars...@ericsson.com
>> www.ericsson.com
>>
>> On fre, 2018-07-06 at 11:18 +0100, Thouraya TH wrote:
>> > Hi,
>> > Thank you so much for answers.
>> >
>> > Please, can you explain more what's metric libraries ? and give me some
>> examples ?
>> >
>> > Using nodetool status, to generate the history of my data center, i
>> intend to proceed as follows:
>> >
>> > From a node A:
>> >
>> > For i  1 ..24 hours  (every 2 minutes do)
>> >
>> > ./nodetool status >> file.txt
>> >
>> > End For
>> >
>> >
>> > is it a good idea?
>> >
>> > Thanks a lot.
>> > Kind regards.
>> >
>> > 2018-07-05 1:30 GMT+01:00 Anthony Grasso :
>> > > Hi,
>> > >
>> > > Yes, you can use nodetool status to inspect the health/status of the
>> cluster. Using nodetool status  will show the cluster
>> health/status as well as the amount of data that each node has
>> > > for the specified .  Using nodetool status without the
>>  argument will only show the cluster health/status.
>> > >
>> > > Unless there is a special reason for using nodetool to capture
>> history, you may want to consider using metric libraries to capture and
>> push information about each node to a metric server. It is
>> > > much easier to view the data captured on the metric server as there
>> are tools already made for this. Using metrics libraries will save you time
>> creating and maintaining a parser for the nodetool
>> > > output. It also makes monitoring the health of cluster very easy.
>> > >
>> > > Regards,
>> > > Anthony
>> > >
>> > > On Sun, 1 Jul 2018 at 20:19, Thouraya TH 
>> wrote:
>> > > > Hi,
>> > > > Thank you so much for answer.
>> > > > Please, is it possible to use this command ?
>> > > > nodetool status mykeyspace
>> > > >
>> > > > Datacenter: datacenter1
>> > > > ===
>> > > > Status=Up/Down
>> > > > |/ State=Normal/Leaving/Joining/Moving
>> > > > --  AddressLoad   Tokens  OwnsHost
>> ID   Rack
>> > > > UN  127.0.0.1  47.66 KB   1   33.3%   aaa1b7c1-
>> 6049-4a08-ad3e-3697a0e30e10  rack1
>> > > > UN  127.0.0.2  47.67 KB   1   33.3%   1848c369-
>> 4306-4874-afdf-5c1e95b8732e  rack1
>> > > > UN
>> > > > Thank you so much.
>> > > > Kind regards.
>> > > >
>> > > > 2018-06-29 1:40 GMT+01:00 Rahul Singh > >:
>> > > > >
>> > > > >
>> > > > > When you run TPstats or Tablestats subcommands in nodetool you
>> are actually accessing data inside Cassandra via JMX.
>> > > > >
>> > > > > You can start there at first.
>> > > > >
>> > > > > Rahul
>> > > > > On Jun 28, 2018, 10:55 AM -0500, Thouraya TH <
>> thouray...@gmail.com>, wrote:
>> > > > > > Hi,
>> > > > > >
>> > > > > > Please, how can check the health of my cluster / data center
>> using cassandra ?
>> > > > > > In fact i'd like to generate a hitory of the state of each
>> node. an history about the failure of my cluster ( 20% of failure in a day,
>> 40% of failure in a day etc...)
>> > > > > >
>> > > > > > Thank you so much.
>> > > > > > Kind regards.
>> > > >
>>
>
>

Re: Write Time of a Row in Multi DC Cassandra Cluster

[Jeff Jirsa]

For tracking delay in very recent versions, this exists: 
https://issues.apache.org/jira/browse/CASSANDRA-13289



-- 
Jeff Jirsa


> On Jul 11, 2018, at 1:05 AM, Simon Fontana Oscarsson 
>  wrote:
> 
> I thought you just wanted to test how big delay you have, that's why I 
> suggested trace.
> 
> Your best option is to write with EACH_QUORUM as Alain said.
> That way you will get a response when the write is successful on all dcs.
> The downside is that the request will fail if one dc is down.
> As usual it's up to you to decide if that's acceptable or not.
> You could follow up with a LOCAL_QUORUM request if the EACH_QUORUM fails and 
> rely on hints and repair to get consistency eventually.
> 
> -- 
> SIMON FONTANA OSCARSSON
> Software Developer
> 
> Ericsson
> Ölandsgatan 1
> 37133 Karlskrona, Sweden
> simon.fontana.oscars...@ericsson.com
> www.ericsson.com
> 
>> On tis, 2018-07-10 at 16:32 +, Saladi Naidu wrote:
>> Simon,
>> Trace would be significant burden on the cluster and it has to be on all the 
>> time. I am trying to find a way to know when a row is written on demand 
>> basis, is there a way to determine that?
>>  
>> Naidu Saladi 
>> 
>> 
>> On Tuesday, July 10, 2018 2:24 AM, Simon Fontana Oscarsson 
>>  wrote:
>> 
>> 
>> Have you tried trace?
>> -- 
>> SIMON FONTANA OSCARSSON
>> Software Developer
>> 
>> Ericsson
>> Ölandsgatan 1
>> 37133 Karlskrona, Sweden
>> simon.fontana.oscars...@ericsson.com
>> www.ericsson.com
>> 
>>> On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote:
>>> Cassandra is an eventual consistent DB, how to find when a row is actually 
>>> written in multi DC environment? Here is the problem I am trying to solve 
>>>  
>>> - I have multi DC (3 DC's) Cassandra cluster/ring - One of the application 
>>> wrote a row to DC1(using Local Quorum)  and within span of 50 ms, it tried 
>>> to read same row from DC2 and could not find
>> the
>>> row. Our both DC's have sub milli second latency at network level, usually 
>>> <2 ms. We promised 20 ms consistency. In this case Application could not 
>>> find the row in DC2 in 50 ms
>>>  
>>> I tried to use "select WRITETIME(authorizations_json) from 
>>> token_authorizations where " to find  when the Row is written in each 
>>> DC, but both DC's returned same Timestamp. After further
>> research
>>> I found that Client V3 onwards Timestamp is supplied at Client level so 
>>> WRITETIME does not help 
>>> "https://docs.datastax.com/en/developer/java-driver/3.4/manual/query_timestamps/;
>>>  
>>> So how to determine when the row is actually written in each DC?
>>>  
>>>  
>>> Naidu Saladi 
>> 

Re: Inconsistent Quorum Read after Quorum Write

[Mick Semb Wever]

Li,


I’ve confirmed that the inconsistency issues disappeared after repair
> finished.
>
> Anything changed with repair in 3.11.1? One difference I noticed is that
> the validation step during repair could turn down the node upon large
> tables, which never happen in 3.10. I had to throttle validation requests
> to let it pass. Also I switched back to -pr instead of incremental repair
> which is a resource killer and often hangs for the first node to be
> repaired.
>


When you switched back to non-incremental did you set `repairedAt` on all
sstables (on all nodes) back to zero (or unrepaired state)?
This should have been done with `sstablerepairedset --is-unrepaired … `
while the node is stopped.



> To address the inconsistency issue, I could do Write All and Read One by
> giving up availability and stop running repair. Any comments on that?
>


You loose availability doing this, and at the number of reads you're doing
I would not recommend it.
You could think about using a fallback strategy that initially tries CL.ALL
and falls back to CL.QUORUM. But this is a hack, could overload your
cluster, and if there's any correlation to dropped messages or flapping
nodes won't help.

I'd also be prepared to upgrade to 3.11.3, when it does get released.

regards,
Mick

-- 
Mick Semb Wever
Australia

The Last Pickle
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Check Cluster Health

[Thouraya TH]

  Hi all;

Please, i have a history of the state of each node in my data center. an
history about the failure of my cluster (UN: node up, DN: node down).

Here is some lines of the history:

   08:51:36 UN 127.0.0.1
   08:51:36 UN 127.0.0.2
   08:51:36 UN 127.0.0.3
   08:53:50 DN 127.0.0.1
   08:53:50 DN 127.0.0.2
   08:53:50 DN 127.0.0.3

I'd like from this history, deduce the failure rate of each node. How can i
do that please ? i have for example, to use AI technologies ML or i have to
sum UN of each node and divide it on the number of line.

Thank you so much for help.

Kind regards.


2018-07-06 16:01 GMT+01:00 Thouraya TH :

>
>
> 2018-07-06 13:04 GMT+01:00 Simon Fontana Oscarsson <
> simon.fontana.oscars...@ericsson.com>:
>
>> Running nodetool status is okay if you want the simplest solution.
>> But it generates a lot of output and creates a new JMX connection for
>> every execution.
>> Cassandra uses JMX to expose metrics via mbeans.
>> Read this to get a first understanding: https://docs.da
>> tastax.com/en/cassandra/2.1/cassandra/operations/ops_monitoring_c.html
>> Use Jconsole to explore the different metrics. Use documentation as
>> reference: https://cassandra.apache.org/doc/latest/operating/metrics.html
>>
>> As for your solution I recommend one of the following:
>> * Create a simple JMX client and add your beans. You can do some simple
>> logging with Logback or log4j. You can get some help by googling.
>> * Use a monitoring system such as Prometheus. This is the best solution
>> but most time consuming.
>>
>
> If i use  Prometheus , i have to install it on each node on my data center
> ? It will give me details about all nodes connections as does nodetool
> status?
>
>>
>> --
>> SIMON FONTANA OSCARSSON
>> Software Developer
>>
>> Ericsson
>> Ölandsgatan 1
>> 37133 Karlskrona, Sweden
>> simon.fontana.oscars...@ericsson.com
>> www.ericsson.com
>>
>> On fre, 2018-07-06 at 11:18 +0100, Thouraya TH wrote:
>> > Hi,
>> > Thank you so much for answers.
>> >
>> > Please, can you explain more what's metric libraries ? and give me some
>> examples ?
>> >
>> > Using nodetool status, to generate the history of my data center, i
>> intend to proceed as follows:
>> >
>> > From a node A:
>> >
>> > For i  1 ..24 hours  (every 2 minutes do)
>> >
>> > ./nodetool status >> file.txt
>> >
>> > End For
>> >
>> >
>> > is it a good idea?
>> >
>> > Thanks a lot.
>> > Kind regards.
>> >
>> > 2018-07-05 1:30 GMT+01:00 Anthony Grasso :
>> > > Hi,
>> > >
>> > > Yes, you can use nodetool status to inspect the health/status of the
>> cluster. Using nodetool status  will show the cluster
>> health/status as well as the amount of data that each node has
>> > > for the specified .  Using nodetool status without the
>>  argument will only show the cluster health/status.
>> > >
>> > > Unless there is a special reason for using nodetool to capture
>> history, you may want to consider using metric libraries to capture and
>> push information about each node to a metric server. It is
>> > > much easier to view the data captured on the metric server as there
>> are tools already made for this. Using metrics libraries will save you time
>> creating and maintaining a parser for the nodetool
>> > > output. It also makes monitoring the health of cluster very easy.
>> > >
>> > > Regards,
>> > > Anthony
>> > >
>> > > On Sun, 1 Jul 2018 at 20:19, Thouraya TH 
>> wrote:
>> > > > Hi,
>> > > > Thank you so much for answer.
>> > > > Please, is it possible to use this command ?
>> > > > nodetool status mykeyspace
>> > > >
>> > > > Datacenter: datacenter1
>> > > > ===
>> > > > Status=Up/Down
>> > > > |/ State=Normal/Leaving/Joining/Moving
>> > > > --  AddressLoad   Tokens  OwnsHost
>> ID   Rack
>> > > > UN  127.0.0.1  47.66 KB   1   33.3%   aaa1b7c1-
>> 6049-4a08-ad3e-3697a0e30e10  rack1
>> > > > UN  127.0.0.2  47.67 KB   1   33.3%   1848c369-
>> 4306-4874-afdf-5c1e95b8732e  rack1
>> > > > UN
>> > > > Thank you so much.
>> > > > Kind regards.
>> > > >
>> > > > 2018-06-29 1:40 GMT+01:00 Rahul Singh > >:
>> > > > >
>> > > > >
>> > > > > When you run TPstats or Tablestats subcommands in nodetool you
>> are actually accessing data inside Cassandra via JMX.
>> > > > >
>> > > > > You can start there at first.
>> > > > >
>> > > > > Rahul
>> > > > > On Jun 28, 2018, 10:55 AM -0500, Thouraya TH <
>> thouray...@gmail.com>, wrote:
>> > > > > > Hi,
>> > > > > >
>> > > > > > Please, how can check the health of my cluster / data center
>> using cassandra ?
>> > > > > > In fact i'd like to generate a hitory of the state of each
>> node. an history about the failure of my cluster ( 20% of failure in a day,
>> 40% of failure in a day etc...)
>> > > > > >
>> > > > > > Thank you so much.
>> > > > > > Kind regards.
>> > > >
>>
>
>

Re: Tuning Replication Factor - All, Consistency ONE

[Jürgen Albersdorfer]

And by all means, do not treat Cassandra as a relational Database. - Beware
of the limitations of CQL in contrast to SQL.
I don't want to argue angainst Cassandra because I like it for what it was
primarly designed - horizontal scalability for HUGE amounts of data.
It is good to access your Data by Key, but not for searching. High
Availibility is a nice giveaway here.
If you end having only one Table in C*, maybe something like Redis would
work for your needs, too.

Some hints from my own expierience with it - if you choose to use Cassandra:
Have at least as much Racks as Replication Factor - Replication Factor of 3
means you want to have at least 3 Racks.
Choose your Partitioning wisely - it starts becoming relevant from 10 mio.
records onwards.

regards,
Jürgen

Am Di., 10. Juli 2018 um 18:18 Uhr schrieb Jeff Jirsa :

>
>
> On Tue, Jul 10, 2018 at 8:29 AM, Code Wiget  wrote:
>
>> Hi,
>>
>> I have been tasked with picking and setting up a database with the
>> following characteristics:
>>
>>- Ultra-high availability - The real requirement is uptime - our
>>whole platform becomes inaccessible without a “read” from the database. We
>>need the read to authenticate users. Databases will never be spread across
>>multiple networks.
>>
>>
> Sooner or later life will happen and you're going to have some
> unavailability - may be worth taking the time to make it fail gracefully
> (cache auth responses, etc).
>
>>
>>- Reasonably quick access speeds
>>- Very low data storage - The data storage is very low - for 10
>>million users, we would have around 8GB of storage total.
>>
>> Having done a bit of research on Cassandra, I think the optimal approach
>> for my use-case would be to replicate the data on *ALL* nodes possible,
>> but require reads to only have a consistency level of one. So, in the case
>> that a node goes down, we can still read/write to other nodes. It is not
>> very important that a read be unanimously agreed upon, as long as Cassandra
>> is eventually consistent, within around 1s, then there shouldn’t be an
>> issue.
>>
>
> Seems like a reasonably good fit, but there's no 1s guarantee - it'll
> USUALLY happen within milliseconds, but the edge cases don't have a strict
> guarantee at all (imagine two hosts in adjacent racks, the link between the
> two racks goes down, but both are otherwise functional - a query at ONE in
> either rack would be able to read and write data, but it would diverge
> between the two racks for some period of time).
>
>
>>
>> When I go to set up the database though, I am required to set a
>> replication factor to a number - 1,2,3,etc. So I can’t just say “ALL” and
>> have it replicate to all nodes.
>>
>
> That option doesn't exist. It's been proposed (and exists in Datastax
> Enterprise, which is a proprietary fork), but reportedly causes quite a bit
> of pain when misused, so people have successfully lobbied against it's
> inclusion in OSS Apache Cassandra. You could (assuming some basic java
> knowledge) extend NetworkTopologyStrategy to have it accomplish this, but I
> imagine you don't REALLY want this unless you're frequently auto-scaling
> nodes in/out of the cluster. You should probably just pick a high RF and
> you'll be OK with it.
>
>
>> Right now, I have a 2 node cluster with replication factor 3. Will this
>> cause any issues, having a RF > #nodes? Or is there a way to just have it
>> copy to *all* nodes?
>>
>
> It's obviously not the intended config, but I don't think it'll cause many
> problems.
>
>
>> Is there any way that I can tune Cassandra to be more read-optimized?
>>
>>
> Yes - definitely use leveled compaction instead of STCS (the default), and
> definitely take the time to tune the JVM args - read path generates a lot
> of short lived java objects, so a larger eden will help you (maybe up to
> 40-50% of max heap size).
>
>
>> Finally, I have some misgivings about how well Cassandra fits my
>> use-case. Please, if anyone has a suggestion as to why or why not it is a
>> good fit, I would really appreciate your input! If this could be done with
>> a simple SQL database and this is overkill, please let me know.
>>
>> Thanks for your input!
>>
>>
>

Re: Write Time of a Row in Multi DC Cassandra Cluster

[Simon Fontana Oscarsson]

I thought you just wanted to test how big delay you have, that's why I 
suggested trace.

Your best option is to write with EACH_QUORUM as Alain said.
That way you will get a response when the write is successful on all dcs.
The downside is that the request will fail if one dc is down.
As usual it's up to you to decide if that's acceptable or not.
You could follow up with a LOCAL_QUORUM request if the EACH_QUORUM fails and 
rely on hints and repair to get consistency eventually.

-- 
SIMON FONTANA OSCARSSON
Software Developer

Ericsson
Ölandsgatan 1
37133 Karlskrona, Sweden
simon.fontana.oscars...@ericsson.com
www.ericsson.com

On tis, 2018-07-10 at 16:32 +, Saladi Naidu wrote:
> Simon,
> Trace would be significant burden on the cluster and it has to be on all the 
> time. I am trying to find a way to know when a row is written on demand 
> basis, is there a way to determine that?
>  
> Naidu Saladi 
> 
> 
> On Tuesday, July 10, 2018 2:24 AM, Simon Fontana Oscarsson 
>  wrote:
> 
> 
> Have you tried trace?
> -- 
> SIMON FONTANA OSCARSSON
> Software Developer
> 
> Ericsson
> Ölandsgatan 1
> 37133 Karlskrona, Sweden
> simon.fontana.oscars...@ericsson.com
> www.ericsson.com
> 
> On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote:
> > Cassandra is an eventual consistent DB, how to find when a row is actually 
> > written in multi DC environment? Here is the problem I am trying to solve 
> > 
> > - I have multi DC (3 DC's) Cassandra cluster/ring - One of the application 
> > wrote a row to DC1(using Local Quorum)  and within span of 50 ms, it tried 
> > to read same row from DC2 and could not find
> the
> > row. Our both DC's have sub milli second latency at network level, usually 
> > <2 ms. We promised 20 ms consistency. In this case Application could not 
> > find the row in DC2 in 50 ms
> > 
> > I tried to use "select WRITETIME(authorizations_json) from 
> > token_authorizations where " to find  when the Row is written in each 
> > DC, but both DC's returned same Timestamp. After further
> research
> > I found that Client V3 onwards Timestamp is supplied at Client level so 
> > WRITETIME does not help 
> > "https://docs.datastax.com/en/developer/java-driver/3.4/manual/query_timestamps/;
> > 
> > So how to determine when the row is actually written in each DC?
> > 
> >  
> > Naidu Saladi 
> 
> 

smime.p7s
Description: S/MIME cryptographic signature