High IO Util using TimeWindowCompaction

[Kurtis Norwood]

I've been testing out cassandra 3.11 (currently using 3.7) and have been
observing really high io util occasionally that sometimes results in
temporary flatlining at 100% io util for an extended period. I think my use
case is pretty simple and currently only testing part of it on this new
version so looking for advice on what might be going wrong.

Use Case: I am using cassandra as basically a large "set", my table schema
is incredibly simple, just a primary key. Records are all written with the
same TTL (7 days). Only queries are inserting a key (which we expect to
only happen once) and checking whether that key exists in the table. In my
3.7 cluster I am using DateTieredCompaction and running on c3.4xlarge (x30)
in AWS. I've been experimenting with i3.4xlarge and wanted to also try
TimeWindowCompaction to see if we could get better performance when adding
machines to the cluster, that was always a really painful experience in 3.7
with DateTieredCompaction and the docs say TimeWindowCompaction is ideal
for my use case.

Right now I am running a new cluster with 3.11 and TimeWindowCompaction
alongside the old cluster and doing writes to both. Only reads go to the
old cluster while I go through this preliminary testing. So the 3.11
cluster receives between 90K to 150K writes/second and no reads. This
morning for a period of about 30 minutes the cluster was at 100% ioutil and
eventually recovered from this state. At that time it was only receiving
~100K writes/second. I don't see anything interesting in the logs that
indicate what is going on, and I don't think a sudden compaction is the
issue since I have limits on compaction throughput.

Staying on 3.7 would be a major bummer so looking for advice.

Some information that might be useful:

compaction throughput - 16MB/s
concurrent compactors - 4
machine type - i3.4xlarge (x20)
disk - RAID0 across 2 NVMe SSDs

Table Schema looks like this:

CREATE TABLE prod_dedupe.event_hashes (

app int,

hash_value blob,

PRIMARY KEY ((app, hash_value))

) WITH bloom_filter_fp_chance = 0.01

AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

AND comment = 'For deduping'

AND compaction = {'class': 'org.apache.cassandra.db.compa
ction.TimeWindowCompactionStrategy', 'compaction_window_size': '4',
'compaction_window_unit': 'HOURS', 'max_threshold': '64', 'min_threshold':
'4'}

AND compression = {'chunk_length_in_kb': '4', 'class': '
org.apache.cassandra.io.compress.LZ4Compressor'}

AND crc_check_chance = 1.0

AND dclocal_read_repair_chance = 0.0

AND default_time_to_live = 0

AND gc_grace_seconds = 3600

AND max_index_interval = 2048

AND memtable_flush_period_in_ms = 0

AND min_index_interval = 128

AND read_repair_chance = 0.0

AND speculative_retry = 'NONE';


Thanks,
Kurt

Re: Node Failure Scenario

[Anthony Grasso]

Hi Anshu,

To add to Erick's comment, remember to remove the *replace_address* method
from the *cassandra-env.sh* file once the node has rejoined successfully.
The node will fail the next restart otherwise.

Alternatively, use the *replace_address_first_boot* method which works
exactly the same way as *replace_address* the only difference is there is
no need to remove it from the *cassandra-env.sh* file.

Kind regards,
Anthony

On 13 November 2017 at 14:59, Erick Ramirez  wrote:

> Use the replace_address method with its own IP address. Make sure you
> delete the contents of the following directories:
> - data/
> - commitlog/
> - saved_caches/
>
> Forget rejoining with repair -- it will just cause more problems. Cheers!
>
> On Mon, Nov 13, 2017 at 2:54 PM, Anshu Vajpayee 
> wrote:
>
>> Hi All ,
>>
>> There was a node failure in one of production cluster due to disk
>> failure.  After h/w recovery that node is noew ready be part of cluster,
>> but it doesn't has any data due to disk crash.
>>
>>
>>
>> I can think of following option :
>>
>>
>>
>> 1. replace the node with same. using replace_address
>>
>> 2. Set bootstrap=false , start the node and run the repair to stream the
>> data.
>>
>>
>>
>> Please suggest if both option are good and which is  best as per your
>> experience. This is live production cluster.
>>
>>
>> Thanks,
>>
>>
>> --
>> *C*heers,*
>> *Anshu V*
>>
>>
>>
>

Re: running C* on AWS EFS storage ...

[Subroto Barua]

From our experience, the ebs remount process was quite painful 

Subroto 

> On Nov 12, 2017, at 4:18 PM, kurt greaves  wrote:
> 
> What's wrong with just detaching the EBS volume and then attaching it to the 
> new node?​ Assuming you have a separate mount for your C* data (which you 
> probably should).


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: best practice for repair

[Jon Haddad]

We (The Last Pickle) maintain Reaper, an open source repair tool, specifically 
to address all the complexity around repairs.

http://cassandra-reaper.io/ 

Jon

> On Nov 13, 2017, at 3:18 AM, Peng Xiao <2535...@qq.com> wrote:
> 
> sub-range repair is much like primary range repair, except that each 
> sub-range repair operation focuses even smaller subset of data.
> 
> repair is a tough process.any advise?
> 
> Thanks
> 
> -- Original --
> From:  "我自己的邮箱";<2535...@qq.com>;
> Date:  Mon, Nov 13, 2017 06:51 PM
> To:  "user";
> Subject:  best practice for repair
> 
> Hi there,
> 
> we need to repair a huge CF,just want to clarify
> 1.nodetool repair -pr keyspace cf
> 2.nodetool repair -st -et -dc
> which will be better? or any other advice?
> 
> Thanks,
> Peng Xiao
> 

Re: STCS leaving sstables behind

[Nicolas Guyomar]

Quick follow up : triggering a repair did the trick, sstables are starting
to get compacted.

Thank you

On 13 November 2017 at 15:53, Nicolas Guyomar 
wrote:

> Hi,
>
> I'm using default "nodetool repair" in 3.0.13 which I believe is full by
> default. I'm not using subrange repair
>
> Jeff you're right, "Nov 11 01:23 mc-6474-big-Data.db" is not yet marked as
> repaired, my repair routine is broken  (sorry Alexander I'm not using
> repear yet ;)  )
>
> I'm gonna fix my repair script and see if it unlock this situation
>
> Thank you
>
>
> On 13 November 2017 at 15:41, Alexander Dejanovski  > wrote:
>
>> And actually, full repair with 3.0/3.x would have the same effect
>> (anticompation) unless you're using subrange repair.
>>
>> On Mon, Nov 13, 2017 at 3:28 PM Jeff Jirsa  wrote:
>>
>>> Running incremental repair puts sstables into a “repaired” set (and an
>>> unrepaired set), which results in something similar to what you’re
>>> describing.
>>>
>>> Were you running / did you run incremental repair ?
>>>
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Nov 13, 2017, at 5:04 AM, Nicolas Guyomar 
>>> wrote:
>>>
>>> Hi everyone,
>>>
>>> I'm facing quite a strange behavior on STCS on 3.0.13, the strategy
>>> seems to have "forgotten" about old sstables, and started a completely new
>>> cycle from scratch, leaving the old sstables on disk untouched :
>>>
>>> Something happened on Nov 10 on every node, which resulted in all those
>>> sstables left behind :
>>>
>>> -rw-r--r--.  8 cassandra cassandra   15G Nov  9 22:22 mc-4828-big-Data.db
>>> -rw-r--r--.  8 cassandra cassandra  4.8G Nov 10 01:39 mc-4955-big-Data.db
>>> -rw-r--r--.  8 cassandra cassandra  2.4G Nov 10 01:45 mc-4957-big-Data.db
>>> -rw-r--r--.  8 cassandra cassandra  662M Nov 10 01:47 mc-4959-big-Data.db
>>> -rw-r--r--.  8 cassandra cassandra  2.8G Nov 10 03:46 mc-5099-big-Data.db
>>> -rw-r--r--.  8 cassandra cassandra  4.6G Nov 10 03:58 mc-5121-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra   53M Nov 10 08:45 mc-5447-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  219M Nov 10 08:46 mc-5454-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  650M Nov 10 08:46 mc-5452-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  1.2G Nov 10 08:48 mc-5458-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  1.5G Nov 10 08:50 mc-5465-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  504M Nov 10 09:39 mc-5526-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra   57M Nov 10 09:40 mc-5527-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  101M Nov 10 09:41 mc-5532-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra   86M Nov 10 09:41 mc-5533-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  134M Nov 10 09:42 mc-5537-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  3.9G Nov 10 09:54 mc-5538-big-Data.db
>>> *-rw-r--r--.  7 cassandra cassandra  1.3G Nov 10 09:57
>>> mc-5548-big-Data.db*
>>> -rw-r--r--.  6 cassandra cassandra   16G Nov 11 01:23 mc-6474-big-Data.db
>>> -rw-r--r--.  4 cassandra cassandra   17G Nov 12 06:44 mc-7898-big-Data.db
>>> -rw-r--r--.  3 cassandra cassandra  8.2G Nov 12 13:45 mc-8226-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  6.8G Nov 12 22:38 mc-8581-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  6.1G Nov 13 03:10 mc-8937-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  3.1G Nov 13 04:12 mc-9019-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  3.0G Nov 13 05:56 mc-9112-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:14 mc-9138-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  1.1G Nov 13 06:27 mc-9159-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:46 mc-9182-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra  1.9G Nov 13 07:18 mc-9202-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra  353M Nov 13 07:22 mc-9207-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra  120M Nov 13 07:22 mc-9208-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra  100M Nov 13 07:23 mc-9209-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra   67M Nov 13 07:25 mc-9210-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra   51M Nov 13 07:25 mc-9211-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra   73M Nov 13 07:27 mc-9212-big-Data.db
>>>
>>>
>>> TRACE logs for the Compaction Manager shows that sstables before Nov 10
>>> are grouped in different buckets than the one after Nov 10.
>>>
>>> At first I thought off some coldness behavior that would filter those
>>> "old" sstables, but looking at the code https://github.com/apache/cass
>>> andra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/
>>> compaction/SizeTieredCompactionStrategy.java#L237 I don't see any
>>> coldness or time pattern used to create bucket.
>>>
>>> I tried restarting the node but the buckets are still grouping in 2
>>> "groups" splitted around Nov 10
>>>
>>> I may have missed sthg from the logs, but they are clear from error/warn
>>> at that Nov 

Re: Securing Cassandra database

[DuyHai Doan]

You can pass in login/password from the client side and encrypt the client
/ cassandra connection...

Le 13 nov. 2017 12:16, "Mokkapati, Bhargav (Nokia - IN/Chennai)" <
bhargav.mokkap...@nokia.com> a écrit :

Hi Team,



We are using Apache Cassandra 3.0.13 version.



As part of Cassandra database security, we have created database super user
authentication, but from driver side we have default cql connection syntax
as “cqlsh ” not like “cqlsh  -u username and -p
password”. So cqlsh connection failing from application side.



So we have choosen a firewall method to limit the access to Cassandra
database with system IP address ranges.



Suggest us If any other better method than IP address firewall to create a
security  for Cassandra.



Thanks,

Bhargav

Re: STCS leaving sstables behind

[Nicolas Guyomar]

Hi,

I'm using default "nodetool repair" in 3.0.13 which I believe is full by
default. I'm not using subrange repair

Jeff you're right, "Nov 11 01:23 mc-6474-big-Data.db" is not yet marked as
repaired, my repair routine is broken  (sorry Alexander I'm not using
repear yet ;)  )

I'm gonna fix my repair script and see if it unlock this situation

Thank you


On 13 November 2017 at 15:41, Alexander Dejanovski 
wrote:

> And actually, full repair with 3.0/3.x would have the same effect
> (anticompation) unless you're using subrange repair.
>
> On Mon, Nov 13, 2017 at 3:28 PM Jeff Jirsa  wrote:
>
>> Running incremental repair puts sstables into a “repaired” set (and an
>> unrepaired set), which results in something similar to what you’re
>> describing.
>>
>> Were you running / did you run incremental repair ?
>>
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Nov 13, 2017, at 5:04 AM, Nicolas Guyomar 
>> wrote:
>>
>> Hi everyone,
>>
>> I'm facing quite a strange behavior on STCS on 3.0.13, the strategy seems
>> to have "forgotten" about old sstables, and started a completely new cycle
>> from scratch, leaving the old sstables on disk untouched :
>>
>> Something happened on Nov 10 on every node, which resulted in all those
>> sstables left behind :
>>
>> -rw-r--r--.  8 cassandra cassandra   15G Nov  9 22:22 mc-4828-big-Data.db
>> -rw-r--r--.  8 cassandra cassandra  4.8G Nov 10 01:39 mc-4955-big-Data.db
>> -rw-r--r--.  8 cassandra cassandra  2.4G Nov 10 01:45 mc-4957-big-Data.db
>> -rw-r--r--.  8 cassandra cassandra  662M Nov 10 01:47 mc-4959-big-Data.db
>> -rw-r--r--.  8 cassandra cassandra  2.8G Nov 10 03:46 mc-5099-big-Data.db
>> -rw-r--r--.  8 cassandra cassandra  4.6G Nov 10 03:58 mc-5121-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra   53M Nov 10 08:45 mc-5447-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  219M Nov 10 08:46 mc-5454-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  650M Nov 10 08:46 mc-5452-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  1.2G Nov 10 08:48 mc-5458-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  1.5G Nov 10 08:50 mc-5465-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  504M Nov 10 09:39 mc-5526-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra   57M Nov 10 09:40 mc-5527-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  101M Nov 10 09:41 mc-5532-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra   86M Nov 10 09:41 mc-5533-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  134M Nov 10 09:42 mc-5537-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  3.9G Nov 10 09:54 mc-5538-big-Data.db
>> *-rw-r--r--.  7 cassandra cassandra  1.3G Nov 10 09:57
>> mc-5548-big-Data.db*
>> -rw-r--r--.  6 cassandra cassandra   16G Nov 11 01:23 mc-6474-big-Data.db
>> -rw-r--r--.  4 cassandra cassandra   17G Nov 12 06:44 mc-7898-big-Data.db
>> -rw-r--r--.  3 cassandra cassandra  8.2G Nov 12 13:45 mc-8226-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  6.8G Nov 12 22:38 mc-8581-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  6.1G Nov 13 03:10 mc-8937-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  3.1G Nov 13 04:12 mc-9019-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  3.0G Nov 13 05:56 mc-9112-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:14 mc-9138-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  1.1G Nov 13 06:27 mc-9159-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:46 mc-9182-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra  1.9G Nov 13 07:18 mc-9202-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra  353M Nov 13 07:22 mc-9207-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra  120M Nov 13 07:22 mc-9208-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra  100M Nov 13 07:23 mc-9209-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra   67M Nov 13 07:25 mc-9210-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra   51M Nov 13 07:25 mc-9211-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra   73M Nov 13 07:27 mc-9212-big-Data.db
>>
>>
>> TRACE logs for the Compaction Manager shows that sstables before Nov 10
>> are grouped in different buckets than the one after Nov 10.
>>
>> At first I thought off some coldness behavior that would filter those
>> "old" sstables, but looking at the code https://github.com/apache/
>> cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/
>> SizeTieredCompactionStrategy.java#L237 I don't see any coldness or time
>> pattern used to create bucket.
>>
>> I tried restarting the node but the buckets are still grouping in 2
>> "groups" splitted around Nov 10
>>
>> I may have missed sthg from the logs, but they are clear from error/warn
>> at that Nov 10 time
>>
>> For what it's worth, restarting the node fixed nodetool status from
>> reporting a wrong Load (nearly 2TB per node instead à 300Gb) => we are
>> loading some data for a week now, it seems that this can happen sometimes
>>
>> If anyone ever experienced that kind of behavior I'd be 

Re: STCS leaving sstables behind

[Alexander Dejanovski]

And actually, full repair with 3.0/3.x would have the same effect
(anticompation) unless you're using subrange repair.

On Mon, Nov 13, 2017 at 3:28 PM Jeff Jirsa  wrote:

> Running incremental repair puts sstables into a “repaired” set (and an
> unrepaired set), which results in something similar to what you’re
> describing.
>
> Were you running / did you run incremental repair ?
>
>
> --
> Jeff Jirsa
>
>
> On Nov 13, 2017, at 5:04 AM, Nicolas Guyomar 
> wrote:
>
> Hi everyone,
>
> I'm facing quite a strange behavior on STCS on 3.0.13, the strategy seems
> to have "forgotten" about old sstables, and started a completely new cycle
> from scratch, leaving the old sstables on disk untouched :
>
> Something happened on Nov 10 on every node, which resulted in all those
> sstables left behind :
>
> -rw-r--r--.  8 cassandra cassandra   15G Nov  9 22:22 mc-4828-big-Data.db
> -rw-r--r--.  8 cassandra cassandra  4.8G Nov 10 01:39 mc-4955-big-Data.db
> -rw-r--r--.  8 cassandra cassandra  2.4G Nov 10 01:45 mc-4957-big-Data.db
> -rw-r--r--.  8 cassandra cassandra  662M Nov 10 01:47 mc-4959-big-Data.db
> -rw-r--r--.  8 cassandra cassandra  2.8G Nov 10 03:46 mc-5099-big-Data.db
> -rw-r--r--.  8 cassandra cassandra  4.6G Nov 10 03:58 mc-5121-big-Data.db
> -rw-r--r--.  7 cassandra cassandra   53M Nov 10 08:45 mc-5447-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  219M Nov 10 08:46 mc-5454-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  650M Nov 10 08:46 mc-5452-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  1.2G Nov 10 08:48 mc-5458-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  1.5G Nov 10 08:50 mc-5465-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  504M Nov 10 09:39 mc-5526-big-Data.db
> -rw-r--r--.  7 cassandra cassandra   57M Nov 10 09:40 mc-5527-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  101M Nov 10 09:41 mc-5532-big-Data.db
> -rw-r--r--.  7 cassandra cassandra   86M Nov 10 09:41 mc-5533-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  134M Nov 10 09:42 mc-5537-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  3.9G Nov 10 09:54 mc-5538-big-Data.db
> *-rw-r--r--.  7 cassandra cassandra  1.3G Nov 10 09:57 mc-5548-big-Data.db*
> -rw-r--r--.  6 cassandra cassandra   16G Nov 11 01:23 mc-6474-big-Data.db
> -rw-r--r--.  4 cassandra cassandra   17G Nov 12 06:44 mc-7898-big-Data.db
> -rw-r--r--.  3 cassandra cassandra  8.2G Nov 12 13:45 mc-8226-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  6.8G Nov 12 22:38 mc-8581-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  6.1G Nov 13 03:10 mc-8937-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  3.1G Nov 13 04:12 mc-9019-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  3.0G Nov 13 05:56 mc-9112-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:14 mc-9138-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  1.1G Nov 13 06:27 mc-9159-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:46 mc-9182-big-Data.db
> -rw-r--r--.  1 cassandra cassandra  1.9G Nov 13 07:18 mc-9202-big-Data.db
> -rw-r--r--.  1 cassandra cassandra  353M Nov 13 07:22 mc-9207-big-Data.db
> -rw-r--r--.  1 cassandra cassandra  120M Nov 13 07:22 mc-9208-big-Data.db
> -rw-r--r--.  1 cassandra cassandra  100M Nov 13 07:23 mc-9209-big-Data.db
> -rw-r--r--.  1 cassandra cassandra   67M Nov 13 07:25 mc-9210-big-Data.db
> -rw-r--r--.  1 cassandra cassandra   51M Nov 13 07:25 mc-9211-big-Data.db
> -rw-r--r--.  1 cassandra cassandra   73M Nov 13 07:27 mc-9212-big-Data.db
>
>
> TRACE logs for the Compaction Manager shows that sstables before Nov 10
> are grouped in different buckets than the one after Nov 10.
>
> At first I thought off some coldness behavior that would filter those
> "old" sstables, but looking at the code
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java#L237
> I don't see any coldness or time pattern used to create bucket.
>
> I tried restarting the node but the buckets are still grouping in 2
> "groups" splitted around Nov 10
>
> I may have missed sthg from the logs, but they are clear from error/warn
> at that Nov 10 time
>
> For what it's worth, restarting the node fixed nodetool status from
> reporting a wrong Load (nearly 2TB per node instead à 300Gb) => we are
> loading some data for a week now, it seems that this can happen sometimes
>
> If anyone ever experienced that kind of behavior I'd be glad to know
> whether it is OK or not, I'd like to avoid manually triggering JMX
> UserDefinedCompaction ;)
>
> Thank you
>
> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: STCS leaving sstables behind

[Jeff Jirsa]

Running incremental repair puts sstables into a “repaired” set (and an 
unrepaired set), which results in something similar to what you’re describing.

Were you running / did you run incremental repair ?

-- 
Jeff Jirsa


> On Nov 13, 2017, at 5:04 AM, Nicolas Guyomar  
> wrote:
> 
> Hi everyone,
> 
> I'm facing quite a strange behavior on STCS on 3.0.13, the strategy seems to 
> have "forgotten" about old sstables, and started a completely new cycle from 
> scratch, leaving the old sstables on disk untouched : 
> 
> Something happened on Nov 10 on every node, which resulted in all those 
> sstables left behind : 
> 
> -rw-r--r--.  8 cassandra cassandra   15G Nov  9 22:22 mc-4828-big-Data.db
> -rw-r--r--.  8 cassandra cassandra  4.8G Nov 10 01:39 mc-4955-big-Data.db
> -rw-r--r--.  8 cassandra cassandra  2.4G Nov 10 01:45 mc-4957-big-Data.db
> -rw-r--r--.  8 cassandra cassandra  662M Nov 10 01:47 mc-4959-big-Data.db
> -rw-r--r--.  8 cassandra cassandra  2.8G Nov 10 03:46 mc-5099-big-Data.db
> -rw-r--r--.  8 cassandra cassandra  4.6G Nov 10 03:58 mc-5121-big-Data.db
> -rw-r--r--.  7 cassandra cassandra   53M Nov 10 08:45 mc-5447-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  219M Nov 10 08:46 mc-5454-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  650M Nov 10 08:46 mc-5452-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  1.2G Nov 10 08:48 mc-5458-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  1.5G Nov 10 08:50 mc-5465-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  504M Nov 10 09:39 mc-5526-big-Data.db
> -rw-r--r--.  7 cassandra cassandra   57M Nov 10 09:40 mc-5527-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  101M Nov 10 09:41 mc-5532-big-Data.db
> -rw-r--r--.  7 cassandra cassandra   86M Nov 10 09:41 mc-5533-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  134M Nov 10 09:42 mc-5537-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  3.9G Nov 10 09:54 mc-5538-big-Data.db
> -rw-r--r--.  7 cassandra cassandra  1.3G Nov 10 09:57 mc-5548-big-Data.db
> -rw-r--r--.  6 cassandra cassandra   16G Nov 11 01:23 mc-6474-big-Data.db
> -rw-r--r--.  4 cassandra cassandra   17G Nov 12 06:44 mc-7898-big-Data.db
> -rw-r--r--.  3 cassandra cassandra  8.2G Nov 12 13:45 mc-8226-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  6.8G Nov 12 22:38 mc-8581-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  6.1G Nov 13 03:10 mc-8937-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  3.1G Nov 13 04:12 mc-9019-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  3.0G Nov 13 05:56 mc-9112-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:14 mc-9138-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  1.1G Nov 13 06:27 mc-9159-big-Data.db
> -rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:46 mc-9182-big-Data.db
> -rw-r--r--.  1 cassandra cassandra  1.9G Nov 13 07:18 mc-9202-big-Data.db
> -rw-r--r--.  1 cassandra cassandra  353M Nov 13 07:22 mc-9207-big-Data.db
> -rw-r--r--.  1 cassandra cassandra  120M Nov 13 07:22 mc-9208-big-Data.db
> -rw-r--r--.  1 cassandra cassandra  100M Nov 13 07:23 mc-9209-big-Data.db
> -rw-r--r--.  1 cassandra cassandra   67M Nov 13 07:25 mc-9210-big-Data.db
> -rw-r--r--.  1 cassandra cassandra   51M Nov 13 07:25 mc-9211-big-Data.db
> -rw-r--r--.  1 cassandra cassandra   73M Nov 13 07:27 mc-9212-big-Data.db
> 
> 
> TRACE logs for the Compaction Manager shows that sstables before Nov 10 are 
> grouped in different buckets than the one after Nov 10.
> 
> At first I thought off some coldness behavior that would filter those "old" 
> sstables, but looking at the code 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java#L237
>  I don't see any coldness or time pattern used to create bucket.
> 
> I tried restarting the node but the buckets are still grouping in 2 "groups" 
> splitted around Nov 10
> 
> I may have missed sthg from the logs, but they are clear from error/warn at 
> that Nov 10 time
> 
> For what it's worth, restarting the node fixed nodetool status from reporting 
> a wrong Load (nearly 2TB per node instead à 300Gb) => we are loading some 
> data for a week now, it seems that this can happen sometimes
> 
> If anyone ever experienced that kind of behavior I'd be glad to know whether 
> it is OK or not, I'd like to avoid manually triggering JMX 
> UserDefinedCompaction ;) 
> 
> Thank you  
> 

STCS leaving sstables behind

[Nicolas Guyomar]

Hi everyone,

I'm facing quite a strange behavior on STCS on 3.0.13, the strategy seems
to have "forgotten" about old sstables, and started a completely new cycle
from scratch, leaving the old sstables on disk untouched :

Something happened on Nov 10 on every node, which resulted in all those
sstables left behind :

-rw-r--r--.  8 cassandra cassandra   15G Nov  9 22:22 mc-4828-big-Data.db
-rw-r--r--.  8 cassandra cassandra  4.8G Nov 10 01:39 mc-4955-big-Data.db
-rw-r--r--.  8 cassandra cassandra  2.4G Nov 10 01:45 mc-4957-big-Data.db
-rw-r--r--.  8 cassandra cassandra  662M Nov 10 01:47 mc-4959-big-Data.db
-rw-r--r--.  8 cassandra cassandra  2.8G Nov 10 03:46 mc-5099-big-Data.db
-rw-r--r--.  8 cassandra cassandra  4.6G Nov 10 03:58 mc-5121-big-Data.db
-rw-r--r--.  7 cassandra cassandra   53M Nov 10 08:45 mc-5447-big-Data.db
-rw-r--r--.  7 cassandra cassandra  219M Nov 10 08:46 mc-5454-big-Data.db
-rw-r--r--.  7 cassandra cassandra  650M Nov 10 08:46 mc-5452-big-Data.db
-rw-r--r--.  7 cassandra cassandra  1.2G Nov 10 08:48 mc-5458-big-Data.db
-rw-r--r--.  7 cassandra cassandra  1.5G Nov 10 08:50 mc-5465-big-Data.db
-rw-r--r--.  7 cassandra cassandra  504M Nov 10 09:39 mc-5526-big-Data.db
-rw-r--r--.  7 cassandra cassandra   57M Nov 10 09:40 mc-5527-big-Data.db
-rw-r--r--.  7 cassandra cassandra  101M Nov 10 09:41 mc-5532-big-Data.db
-rw-r--r--.  7 cassandra cassandra   86M Nov 10 09:41 mc-5533-big-Data.db
-rw-r--r--.  7 cassandra cassandra  134M Nov 10 09:42 mc-5537-big-Data.db
-rw-r--r--.  7 cassandra cassandra  3.9G Nov 10 09:54 mc-5538-big-Data.db
*-rw-r--r--.  7 cassandra cassandra  1.3G Nov 10 09:57 mc-5548-big-Data.db*
-rw-r--r--.  6 cassandra cassandra   16G Nov 11 01:23 mc-6474-big-Data.db
-rw-r--r--.  4 cassandra cassandra   17G Nov 12 06:44 mc-7898-big-Data.db
-rw-r--r--.  3 cassandra cassandra  8.2G Nov 12 13:45 mc-8226-big-Data.db
-rw-r--r--.  2 cassandra cassandra  6.8G Nov 12 22:38 mc-8581-big-Data.db
-rw-r--r--.  2 cassandra cassandra  6.1G Nov 13 03:10 mc-8937-big-Data.db
-rw-r--r--.  2 cassandra cassandra  3.1G Nov 13 04:12 mc-9019-big-Data.db
-rw-r--r--.  2 cassandra cassandra  3.0G Nov 13 05:56 mc-9112-big-Data.db
-rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:14 mc-9138-big-Data.db
-rw-r--r--.  2 cassandra cassandra  1.1G Nov 13 06:27 mc-9159-big-Data.db
-rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:46 mc-9182-big-Data.db
-rw-r--r--.  1 cassandra cassandra  1.9G Nov 13 07:18 mc-9202-big-Data.db
-rw-r--r--.  1 cassandra cassandra  353M Nov 13 07:22 mc-9207-big-Data.db
-rw-r--r--.  1 cassandra cassandra  120M Nov 13 07:22 mc-9208-big-Data.db
-rw-r--r--.  1 cassandra cassandra  100M Nov 13 07:23 mc-9209-big-Data.db
-rw-r--r--.  1 cassandra cassandra   67M Nov 13 07:25 mc-9210-big-Data.db
-rw-r--r--.  1 cassandra cassandra   51M Nov 13 07:25 mc-9211-big-Data.db
-rw-r--r--.  1 cassandra cassandra   73M Nov 13 07:27 mc-9212-big-Data.db


TRACE logs for the Compaction Manager shows that sstables before Nov 10 are
grouped in different buckets than the one after Nov 10.

At first I thought off some coldness behavior that would filter those "old"
sstables, but looking at the code
https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java#L237
I don't see any coldness or time pattern used to create bucket.

I tried restarting the node but the buckets are still grouping in 2
"groups" splitted around Nov 10

I may have missed sthg from the logs, but they are clear from error/warn at
that Nov 10 time

For what it's worth, restarting the node fixed nodetool status from
reporting a wrong Load (nearly 2TB per node instead à 300Gb) => we are
loading some data for a week now, it seems that this can happen sometimes

If anyone ever experienced that kind of behavior I'd be glad to know
whether it is OK or not, I'd like to avoid manually triggering JMX
UserDefinedCompaction ;)

Thank you

Re: best practice for repair

[Peng Xiao]

sub-range repair is much like primary range repair, except that each sub-range 
repair operation focuses even smaller subset of data.


repair is a tough process.any advise?


Thanks


-- Original --
From:  "";<2535...@qq.com>;
Date:  Mon, Nov 13, 2017 06:51 PM
To:  "user";

Subject:  best practice for repair



Hi there,

we need to repair a huge CF,just want to clarify
1.nodetool repair -pr keyspace cf 
2.nodetool repair -st -et -dc 
which will be better? or any other advice?


Thanks,
Peng Xiao

Securing Cassandra database

[Mokkapati, Bhargav (Nokia - IN/Chennai)]

Hi Team,

We are using Apache Cassandra 3.0.13 version.

As part of Cassandra database security, we have created database super user 
authentication, but from driver side we have default cql connection syntax as 
"cqlsh " not like "cqlsh  -u username and -p password". 
So cqlsh connection failing from application side.

So we have choosen a firewall method to limit the access to Cassandra database 
with system IP address ranges.

Suggest us If any other better method than IP address firewall to create a 
security  for Cassandra.

Thanks,
Bhargav

best practice for repair

[Peng Xiao]

Hi there,

we need to repair a huge CF,just want to clarify
1.nodetool repair -pr keyspace cf 
2.nodetool repair -st -et -dc 
which will be better? or any other advice?


Thanks,
Peng Xiao

Re: Warning for large batch sizes with a small number of statements

[Erick Ramirez]

You can increase it if you're sure that it fits your use case. For an
explanation of why batch size vs number of statements, see the discussion
in CASSANDRA-6487. Cheers!

On Mon, Nov 13, 2017 at 6:31 PM, Tim Moore  wrote:

> Hi,
>
> I'm trying to understand some of the details of the
> batch_size_warn_threshold_in_kb/batch_size_fail_threshold_in_kb settings.
>
> Specifically, why are the thresholds measured in kb rather than the number
> of partitions affected?
>
> We have run into the limit in a situation where there is a batch with
> writes to two tables (because we wanted to ensure atomicity of the writes
> in the case of a failure). In some situations, the data inserted into one
> of these tables can be large enough to push the total batch size over the
> limit.
>
> In this specific case, we were able to rewrite things so that it could be
> split into separate statement executions with the application handling
> retry on failure so atomicity is not needed. It left us wondering, however,
> whether executing this as a batch was really problematic, or if the warning
> from Cassandra was spurious in this case.
>
> Is it really the total data size that matters, or just the number of
> affected partitions?
>
> Thanks,
> Tim
>
> --
> Tim Moore
> *Senior Engineer, Lagom, Lightbend, Inc.*
> tim.mo...@lightbend.com
> +61 420 981 589 <+61%20420%20981%20589>
> Skype: timothy.m.moore
>
> 
>

Re: Cassandra Query

[Erick Ramirez]

That is one of the most asked questions that it prompted me to write a blog
post last month --
https://academy.datastax.com/support-blog/counting-keys-might-well-be-counting-stars

On Mon, Nov 13, 2017 at 6:55 PM, Hareesh Veduraj 
wrote:

> Hi Team,
>
> I have a new requirement , where I need to copy all the rows from one
> table to another table in Cassandra, where the second table contains one
> extra column.
>
> I have written a python script, which reads each row and inserts . But the
> problem is in stage environment I'm observing the select count query is
> returning different result each time I execute, so the row count is
> varying.
>
> The base  table contains 6 lakhs rows.The stage cluster is of 5 instances
> with a replication factor of 3.
>
> I'm able to successfully run the query in dev cluster with the same data,
> where 3 instances are there and replication factor is 3.
>
> Can anyone help me out what is the best way to achieve this?Also, what is
> the reason for the variation in select count query?
>
> Below is the Python script snippet:
>
> cql='select key, column1, column2, column3, column4, value from table1
> limit 100';
> insert_table2_query = session.prepare('INSERT INTO table2 (key, column1,
> column2, column3, column4, column5, value) VALUES (?, ?, ?, ?, ?, ?, ?)');
>
> def insert_into_table2():
> counter =1;
> for row in session.execute(cql):
> try:
> val1=row.column4;
> val2 = row.value;
> if(len(val1)==0 or val1 == None) :
>  val1="";
> session.execute(insert_table2_query,(row.key,row.column1,row
> .column2,row.column3,val1,val2,val2))
> print("processed row count : "+str(counter));
> counter = counter + 1;
> except Exception as e:
> counter = counter + 1;
> print 'failed to insert the row ';
>
>
> Appreciating any help.
>
> Thanks,
> Hareesh.A.V
> Mob: +7022643519
>
>