Re: Why compacting process uses more data that is expected

[Alexander Dejanovski]

Indeed, nodetool compactionstats shows uncompressed sizes.
As Oleksandr suggests, use the table compression ratio to compute the
actual size on disk.

It would actually be a great improvement for ops if we could add a switch
to compactionstats in order to have the compression ratio applied
automatically.

On Thu, Jan 5, 2017 at 7:22 AM Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On Jan 4, 2017 17:58, "Jean Carlo"  wrote:
>
> Hello guys
>
> I have a table with 34Gb of data in sstables (including tmp). And I can
> see cassandra is doing some compactions on it. What surprissed me is that
> nodetool compactionstats says he is compacting  138.66GB
>
>
> root@node001 /root # nodetool compactionstats -H
> pending tasks: 103
> *   compaction typekeyspace  table
> completed   total unit   progress*
> Compaction keyspace1   table_02   112.74 GB
> 138.66 GB   bytes 81.31%
> Active compaction remaining time :   0h03m27s
>
> So My question is, from where those 138.66GB come if my table has only
> 34GB of data.
>
>
> Hello,
>
> I believe that output of compactionstats shows you the size of
> *uncompressed* data. Can you check (with nodetool tablestats) your
> compression ratio?
>
> --
> Alex
>
> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Why compacting process uses more data that is expected

[Oleksandr Shulgin]

On Jan 4, 2017 17:58, "Jean Carlo"  wrote:

Hello guys

I have a table with 34Gb of data in sstables (including tmp). And I can see
cassandra is doing some compactions on it. What surprissed me is that
nodetool compactionstats says he is compacting  138.66GB


root@node001 /root # nodetool compactionstats -H
pending tasks: 103
*   compaction typekeyspace  table  completed
total unit   progress*
Compaction keyspace1   table_02   112.74 GB
138.66 GB   bytes 81.31%
Active compaction remaining time :   0h03m27s

So My question is, from where those 138.66GB come if my table has only 34GB
of data.


Hello,

I believe that output of compactionstats shows you the size of
*uncompressed* data. Can you check (with nodetool tablestats) your
compression ratio?

--
Alex

Re: Cipher Suite Cassandra 2.1.14 Encryption

[Nate McCall]

Is AES-GCM supported in python by default? I have a vague recollection that
it is not (certainly possible my knowledge is outdated as well).

On Wed, Dec 21, 2016 at 10:21 AM, Jacob Shadix 
wrote:

> I was testing client encryption w/cqlsh and get the following error when
> using TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 as the cipher. Any ideas why?
>
> Last error: _ssl.c:492: EOF occurred in violation of protocol")})
> -- Jacob Shadix
>



-- 
-
Nate McCall
Wellington, NZ
@zznate

CTO
Apache Cassandra Consulting
http://www.thelastpickle.com

Odd behavior when querying using two SASI indexes

[Voytek Jarnot]

Seeing queries return 0 rows incorrectly, running 3.9

Setup:

create table test1(id1 text PRIMARY KEY, val1 text, val2 text);

create custom index test1_idx_val1 on test1(val1) using
'org.apache.cassandra.index.sasi.SASIIndex';
create custom index test1_idx_val2 on test1(val2) using
'org.apache.cassandra.index.sasi.SASIIndex';

insert into test1(id1, val1, val2) values ('1', '1val1', '1val2');
insert into test1(id1, val1, val2) values ('2', '~~', '2val2');

Queries:

(1) select * from test1 where val1 = '~~';
(2) select * from test1 where val1 < '~~' allow filtering;
(3) select * from test1 where val2 = '1val2';
(4) select * from test1 where val1 < '~~' and val2 = '1val2' allow
filtering;

1, 2, and 3 all work correctly.  4 does not work.  2, 3, and 4 should
return the same row (id1='1'); 2 and 3 do, 4 returns 0 rows.

Weird, because you'd think that if 2 works fine, then 4 ought to as well.

Anyone else run into this?  Not sure if I'm breaking a where-clause rule,
or if I'm running into a bug.

Thanks.

Is Repair complete?

[Shannon Carey]

Can anyone give a better answer to the question "How do I know if a "nodetool 
repair" is finished?" on 
http://stackoverflow.com/questions/25064717/how-do-i-know-if-nodetool-repair-is-finished

I am trying to prepare to upgrade to DSE 5. I am migrating to incremental 
repairs first. So I disabled autocompaction and I kicked off a full sequential 
repair ("nodetool repair") on one node in DSE 4.8.5, and more than 24 hours 
later, I still often (but not always) see "Repair of …" showing up in the 
Activities in OpsCenter, and often (but not always) "nodetool compactionstats" 
shows a "Validation" compaction under "pending tasks". Is the repair done? How 
can I tell for certain?

Also, the node is an AWS EC2 i2.xlarge and only contains ~311GB… is >24 hr a 
reasonable/expected duration for a single-node full sequential repair?

Thanks,
Shannon

Re: weird jvm metrics

[Edward Capriolo]

The metric-reporter is actually leveraged from another project.

https://github.com/addthis/metrics-reporter-config

Check the version of metric-reporter (in cassandra/lib) and see if it has
changed from your old version to your new version.




On Wed, Jan 4, 2017 at 12:02 PM, Mike Torra  wrote:

> Just bumping - has anyone seen this before?
>
> http://stackoverflow.com/questions/41446352/cassandra-
> 3-9-jvm-metrics-have-bad-name
>
> From: Mike Torra 
> Reply-To: "user@cassandra.apache.org" 
> Date: Wednesday, December 28, 2016 at 4:49 PM
> To: "user@cassandra.apache.org" 
> Subject: weird jvm metrics
>
> Hi There -
>
> I recently upgraded from cassandra 3.5 to 3.9 (DDC), and I noticed that
> the "new" jvm metrics are reporting with an extra '.' character in them.
> Here is a snippet of what I see from one of my nodes:
>
> ubuntu@ip-10-0-2-163:~$ sudo tcpdump -i eth0 -v dst port 2003 -A | grep
> 'jvm'
>
> tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size
> 65535 bytes
>
> .Je..l>.pi.cassandra.us-east-1.cassy-node1.jvm.buffers..direct.capacity
> 762371494 1482960946
>
> pi.cassandra.us-east-1.cassy-node1.jvm.buffers..direct.count 3054
> 1482960946
>
> pi.cassandra.us-east-1.cassy-node1.jvm.buffers..direct.used 762371496
> 1482960946
>
> pi.cassandra.us-east-1.cassy-node1.jvm.buffers..mapped.capacity
> 515226631134 1482960946
>
> pi.cassandra.us-east-1.cassy-node1.jvm.buffers..mapped.count 45572
> 1482960946
>
> pi.cassandra.us-east-1.cassy-node1.jvm.buffers..mapped.used 515319762610
> 1482960946
>
> pi.cassandra.us-east-1.cassy-node1.jvm.fd.usage 0.00 1482960946
>
> My metrics.yaml looks like this:
>
> graphite:
>   -
> period: 60
> timeunit: 'SECONDS'
> prefix: 'pi.cassandra.us-east-1.cassy-node1'
> hosts:
>  - host: '#RELAY_HOST#'
>port: 2003
> predicate:
>   color: "white"
>   useQualifiedName: true
>   patterns:
> - "^org.+"
> - "^jvm.+"
> - "^java.lang.+"
>
> All the org.* metrics come through fine, and the jvm.fd.usage metric
> strangely comes through fine, too. The rest of the jvm.* metrics have this
> extra '.' character that causes them to not show up in graphite.
>
> Am I missing something silly here? Appreciate any help or suggestions.
>
> - Mike
>

Re: Why compacting process uses more data that is expected

[Jonathan Haddad]

What's the total size of your sstables on disk?

ls -lah /path/to/table/data
On Wed, Jan 4, 2017 at 8:58 AM Jean Carlo  wrote:

> Hello guys
>
> I have a table with 34Gb of data in sstables (including tmp). And I can
> see cassandra is doing some compactions on it. What surprissed me is that
> nodetool compactionstats says he is compacting  138.66GB
>
>
> root@node001 /root # nodetool compactionstats -H
> pending tasks: 103
> *   compaction typekeyspace  table
> completed   total unit   progress*
> Compaction keyspace1   table_02   112.74 GB
> 138.66 GB   bytes 81.31%
> Active compaction remaining time :   0h03m27s
>
> So My question is, from where those 138.66GB come if my table has only
> 34GB of data.
>
>
> Best greetings
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>

Re: weird jvm metrics

[Mike Torra]

Just bumping - has anyone seen this before?

http://stackoverflow.com/questions/41446352/cassandra-3-9-jvm-metrics-have-bad-name

From: Mike Torra >
Reply-To: "user@cassandra.apache.org" 
>
Date: Wednesday, December 28, 2016 at 4:49 PM
To: "user@cassandra.apache.org" 
>
Subject: weird jvm metrics

Hi There -

I recently upgraded from cassandra 3.5 to 3.9 (DDC), and I noticed that the 
"new" jvm metrics are reporting with an extra '.' character in them. Here is a 
snippet of what I see from one of my nodes:


ubuntu@ip-10-0-2-163:~$ sudo tcpdump -i eth0 -v dst port 2003 -A | grep 'jvm'

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 
bytes

.Je..l>.pi.cassandra.us-east-1.cassy-node1.jvm.buffers..direct.capacity 
762371494 1482960946

pi.cassandra.us-east-1.cassy-node1.jvm.buffers..direct.count 3054 1482960946

pi.cassandra.us-east-1.cassy-node1.jvm.buffers..direct.used 762371496 1482960946

pi.cassandra.us-east-1.cassy-node1.jvm.buffers..mapped.capacity 515226631134 
1482960946

pi.cassandra.us-east-1.cassy-node1.jvm.buffers..mapped.count 45572 1482960946

pi.cassandra.us-east-1.cassy-node1.jvm.buffers..mapped.used 515319762610 
1482960946

pi.cassandra.us-east-1.cassy-node1.jvm.fd.usage 0.00 1482960946

My metrics.yaml looks like this:

graphite:
  -
period: 60
timeunit: 'SECONDS'
prefix: 'pi.cassandra.us-east-1.cassy-node1'
hosts:
 - host: '#RELAY_HOST#'
   port: 2003
predicate:
  color: "white"
  useQualifiedName: true
  patterns:
- "^org.+"
- "^jvm.+"
- "^java.lang.+"

All the org.* metrics come through fine, and the jvm.fd.usage metric strangely 
comes through fine, too. The rest of the jvm.* metrics have this extra '.' 
character that causes them to not show up in graphite.

Am I missing something silly here? Appreciate any help or suggestions.

- Mike

Why compacting process uses more data that is expected

[Jean Carlo]

Hello guys

I have a table with 34Gb of data in sstables (including tmp). And I can see
cassandra is doing some compactions on it. What surprissed me is that
nodetool compactionstats says he is compacting  138.66GB


root@node001 /root # nodetool compactionstats -H
pending tasks: 103
*   compaction typekeyspace  table  completed
total unit   progress*
Compaction keyspace1   table_02   112.74 GB
138.66 GB   bytes 81.31%
Active compaction remaining time :   0h03m27s

So My question is, from where those 138.66GB come if my table has only 34GB
of data.


Best greetings

Jean Carlo

"The best way to predict the future is to invent it" Alan Kay

RE: Cassandra CPU perfomance

[DE VITO Dominique]

Hi,

A hint : depending on your data set size + your request rate per second, 8 GB 
of RAM may be too low.
And then, CPU might be high due to too frequent GC.

More RAM may bring:

· More space for OS FS to cache the SSTable files in memory.

· A greater heap size, and then, less frequent object promotion 
(young=>old space) and then, less major GC.


Dominique



[@@ THALES GROUP INTERNAL @@]

De : D. Salvatore [mailto:dd.salvat...@gmail.com]
Envoyé : mercredi 4 janvier 2017 16:28
À : user@cassandra.apache.org
Objet : Cassandra CPU perfomance


Hi,

I deployed a Cassandra 2.2 ring composed by 4 nodes in the cloud with 8 vCPU 
and 8GB of ram. I am running some tests now with cassandra-stress and YCSB 
tools to test its performance. I am mainly interested in read requests with a 
small amount of write requests (95%/5%).

Running the experiments, I noticed that even setting a high number of threads 
(or clients) the CPU (and disk) does not saturate, but still always around the 
60% of utilisation.

I am trying to figure out where is the bottleneck in my system. From the 
hardware point of view it seems all ok to me.

I also looked into the Cassandra configuration file to see if there are some 
tuning parameters to increase the system throughput. I increase the value of 
concurrent_read/write parameter, but it doesn't increase the performance. The 
log file also does not contain any warning.

What it could be that is limiting my system?

Thanks

Re: Reaper repair seems to "hang"

[Alexander Dejanovski]

Actually, the problem is related to CASSANDRA-11430
.

Before 2.2.6, the notification service did not work with newly deprecated
repair methods, on which Reaper still currently relies.
C* 2.2.6 and onwards are not affected by this problem and work fine with
Reaper.

We're working on switching to the new repair method for 2.2 and 3.0/3.x,
which should be ready in a few days/weeks.

When using incremental repair, watch out for CASSANDRA-11696 which was
fixed in C* 2.1.15, 2.2.7, 3.0.8 and 3.8. In prior versions, unrepaired
SSTables can be marked as repaired, and thus never be repaired.

Cheers,



On Wed, Jan 4, 2017 at 6:09 AM Bhuvan Rawal  wrote:

> Hi Daniel,
>
> Looks like yours is a different case. If you're running incremental repair
> for the first time it make take long time esp. if table is large. And
> repair may seem to stuck even when things are working.
>
> You can try nodetool compactionstats when repair appears stuck, you'll
> find a validation compaction happening if that's indeed the case.
>
> For the first incremental repair you can follow this doc, in further
> repairs incremental repair should encounter very few sstables:
>
> https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepairNodesMigration.html
>
> Regards,
> Bhuvan
>
>
>
> On Jan 4, 2017 3:52 AM, "Daniel Kleviansky"  wrote:
>
> Hi Bhuvan,
>
> Thank you so very much for your detailed reply.
> Just to ensure everyone is across the same information, and responses are
> not duplicated across two different forums, I thought I'd share with the
> mailing list that I've created a GitHub issue at:
> https://github.com/thelastpickle/cassandra-reaper/issues/39
>
> Kind regards,
> Daniel
>
> On Wed, Jan 4, 2017 at 6:31 AM, Bhuvan Rawal  wrote:
>
> Hi Daniel,
>
> We faced a similar issue during repair with reaper. We ran repair with
> more repair threads than number of cassandra nodes. But on and off repair
> was getting stuck and we had to do rolling restart of cluster or wait for
> lock time to expire (~1hr).
>
> We had a look at the stuck repair, threadpools were getting stuck at
> AntiEntropy stage. From the synchronized block in repair code it appeared
> that per node at max 1 concurrent repair session per node is possible.
>
> According to
> https://medium.com/@mlowicki/cassandra-reaper-introduction-ed73410492bf#.f0erygqpk
>  :
>
> Segment runner has protection mechanism to avoid overloading nodes using
> two simple rules to postpone repair if:
>
> 1. Number of pending compactions is greater than *MAX_PENDING_COMPACTIONS* (20
> by default)
> *2. Node is already running repair job*
>
> We tried running reaper with number of threads less than number of nodes
> (assuming reaper will not submit multiple segments to single cassandra
> node) but still it was observed that multiple repair segments were going to
> same node concurrently and threfore chances of nodes getting stuck in that
> state was possible. Finally we settled with single repair thread in reaper
> settings. Although takes a slightly more time but has completed
> successfully numerous times.
>
> Thread Dump of cassandra server when repair was getting stuck:
>
> "*AntiEntropyStage:1" #159 daemon prio=5 os_prio=0 tid=0x7f0fa16226a0
> nid=0x3c82 waiting for monitor entry [0x7ee9eabaf000*]
>java.lang.Thread.State: BLOCKED (*on object monitor*)
> at
> org.apache.cassandra.service.ActiveRepairService.removeParentRepairSession(ActiveRepairService.java:392)
> - waiting to lock <0x00067c083308> (a
> org.apache.cassandra.service.ActiveRepairService)
> at
> org.apache.cassandra.service.ActiveRepairService.doAntiCompaction(ActiveRepairService.java:417)
> at org.apache.cassandra.repair
> .RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:145)
> at org.apache.cassandra.net
> .MessageDeliveryTask.run(MessageDeliveryTask.java:67)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> Hope it helps!
>
> Regards,
> Bhuvan
>
> According to
> https://medium.com/@mlowicki/cassandra-reaper-introduction-ed73410492bf#.f0erygqpk
>  :
>
> Segment runner has protection mechanism to avoid overloading nodes using
> two simple rules to postpone repair if:
>
> 1. Number of pending compactions is greater than *MAX_PENDING_COMPACTIONS* (20
> by default)
> 2. Node is already running repair job
>
>
> On Tue, Jan 3, 2017 at 11:16 AM, Alexander Dejanovski <
> a...@thelastpickle.com> wrote:
>
> Hi Daniel,
>
> could you file a bug in the issue tracker ?
> https://github.com/thelastpickle/cassandra-reaper/issues
>
> We'll figure out what's wrong and get your repairs running.
>
> Thanks !
>
> On Tue, Jan 3, 2017 at 12:35 

Cassandra CPU perfomance

[D. Salvatore]

Hi,

I deployed a Cassandra 2.2 ring composed by 4 nodes in the cloud with 8
vCPU and 8GB of ram. I am running some tests now with cassandra-stress and
YCSB tools to test its performance. I am mainly interested in read requests
with a small amount of write requests (95%/5%).

Running the experiments, I noticed that even setting a high number of
threads (or clients) the CPU (and disk) does not saturate, but still always
around the 60% of utilisation.

I am trying to figure out where is the bottleneck in my system. From the
hardware point of view it seems all ok to me.

I also looked into the Cassandra configuration file to see if there are
some tuning parameters to increase the system throughput. I increase the
value of concurrent_read/write parameter, but it doesn't increase the
performance. The log file also does not contain any warning.

What it could be that is limiting my system?

Thanks