Re: Work in Progress - Bringing it all together in one "Awesome Cassandra" README

[Srinath C]

This is great! Thank you.

On Fri, Jul 27, 2018 at 2:20 AM Rahul Singh 
wrote:

> Hope you all are are having an amazing week.
>
> I recently updated  
> *https://github.com/Anant/awesome-cassandra*/. I've been working on this
> while organizing the initial repository of links for a "Cassandra Hub"
> (Planet Cassandra 2.0) as I organize links on Cassandra and distributed
> computing (e.g. Kafka, Spark, Akka, Kubernetes, etc.) .
>
> I've got about ~120 or so resources organized in this
> *Readme*, and I have a queue
> of another 100 or so. Please feel free to send me any focused Cassandra
> blogs related to development, architecture, or devops.
>
> Thanks,
>
> Rahul Singh
> Chief Executive Officer | Internet Architecture
> m 202.905.2818 | http://anant.us
>
> 1010 Wisconsin Ave NW, Suite 250
> Washington, D.C. 20007
>
> To empower people through the Internet to create a better world.
>
> How are we doing? Please take our survey.
>
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Anant Corporation. If you are not the intended recipient
> of this email, you must neither take any action based upon its contents,
> nor copy or show it to anyone. Please contact the sender if you believe you
> have received this email in error.
>

Work in Progress - Bringing it all together in one "Awesome Cassandra" README

[Rahul Singh]

Hope you all are are having an amazing week.

I recently updated https://github.com/Anant/awesome-cassandra/. I've been 
working on this while organizing the initial repository of links for a 
"Cassandra Hub" (Planet Cassandra 2.0) as I organize links on Cassandra and 
distributed computing (e.g. Kafka, Spark, Akka, Kubernetes, etc.) .

I've got about ~120 or so resources organized in this Readme, and I have a 
queue of another 100 or so. Please feel free to send me any focused Cassandra 
blogs related to development, architecture, or devops.

Thanks,

Rahul Singh
Chief Executive Officer | Internet Architecture
m 202.905.2818 | http://anant.us

1010 Wisconsin Ave NW, Suite 250
Washington, D.C. 20007

To empower people through the Internet to create a better world.

How are we doing? Please take our survey.

This email and any attachments to it may be confidential and are intended 
solely for the use of the individual to whom it is addressed. Any views or 
opinions expressed are solely those of the author and do not necessarily 
represent those of Anant Corporation. If you are not the intended recipient of 
this email, you must neither take any action based upon its contents, nor copy 
or show it to anyone. Please contact the sender if you believe you have 
received this email in error.

RE: [EXTERNAL] optimization to cassandra-env.sh

[Durity, Sean R]

This is a very good explanation of CMS tuning for Cassandra:
http://thelastpickle.com/blog/2018/04/11/gc-tuning.html
(author Jon Haddad has extensive Cassandra experience – a super star in our 
field)

Sean Durity
From: Durity, Sean R 
Sent: Thursday, July 26, 2018 2:08 PM
To: user@cassandra.apache.org
Subject: RE: [EXTERNAL] optimization to cassandra-env.sh

Check the archives for CMS or G1 (whichever garbage collector you are using). 
There has been significant and good advice on both. In general, though, G1 has 
one basic number to set and does very well in our use cases. CMS has lots of 
black art/science tuning and configuration, but you can test options on a 
“canary” node and tweak until it runs well.

If you need more help after looking back in the archives, we would need the 
Cassandra version, JVM type and version, jvm.options (or cassandra-env.sh) and 
what kinds of errors/gc you are seeing (GC logs can be helpful).


Sean Durity
From: R1 J1 mailto:rjsoft...@gmail.com>>
Sent: Thursday, July 26, 2018 1:28 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] optimization to cassandra-env.sh

Any one has tried to optimize or change cassandra-env.sh in an server 
installation to make it use more heap size for garbage collection ?
Any ideas ? We are having some oom issues and thinking if we have options other 
than increasing RAM for that node.

Regards




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: [EXTERNAL] optimization to cassandra-env.sh

[Durity, Sean R]

Check the archives for CMS or G1 (whichever garbage collector you are using). 
There has been significant and good advice on both. In general, though, G1 has 
one basic number to set and does very well in our use cases. CMS has lots of 
black art/science tuning and configuration, but you can test options on a 
“canary” node and tweak until it runs well.

If you need more help after looking back in the archives, we would need the 
Cassandra version, JVM type and version, jvm.options (or cassandra-env.sh) and 
what kinds of errors/gc you are seeing (GC logs can be helpful).


Sean Durity
From: R1 J1 
Sent: Thursday, July 26, 2018 1:28 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] optimization to cassandra-env.sh

Any one has tried to optimize or change cassandra-env.sh in an server 
installation to make it use more heap size for garbage collection ?
Any ideas ? We are having some oom issues and thinking if we have options other 
than increasing RAM for that node.

Regards




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

optimization to cassandra-env.sh

[R1 J1]

Any one has tried to optimize or change cassandra-env.sh in an server
installation to make it use more heap size for garbage collection ?
Any ideas ? We are having some oom issues and thinking if we have options
other than increasing RAM for that node.

Regards

Re: cassandro nodes restarts

[R1 J1]

Thanks for your prompt replies. No the same node is not bouncing over. When
you say it is about to tip over: What can we do to stop that ?

Also about that error : you guys are correct: it is  a warning and might
not be contributing to the node bounce issue and it can be removed by
changing batch_size_warn_threshold_in_kb: 5

R1J1

On Wed, Jul 25, 2018 at 10:32 PM, R1 J1  wrote:

> cassandro nodes restarts
>
>
>
> we see errors typically like these
>
>
> WARN  [Native-Transport-Requests-3] 2018-07-25 20:51:38,520
> BatchStatement.java:301 - Batch for "keyspace.table"
>  is of size 19.386KiB, exceeding specified threshold of 5.000KiB by
> 14.386KiB.
>
>
> Regards
> R1J1
>

Lost counter updates during Cassandra upgrade 2.2.11 to 3.11.2

[Konrad]

Hi,

During rolling upgrade of our cluster we noticed that some updates on table 
with counters were not being applied. It looked as if it depended on whether 
coordinator handling request was already upgraded or not. I observed similar 
behavior while using cqlsh and executing queries manually. Sometimes it took 
several retries to see counter updated. There were no errors/warns in neither 
application nor Cassandra logs. The updates started working reliably once again 
when all nodes in dc have been upgraded. However, the lost updates did not 
reappear. 

Our setup:
2 dc cluster, 5 + 5 nodes. However, only one is used for queries as client 
application is co-located in one region. I believe 1 dc is enough to reproduce 
it. 
Replication factor 3+2
Consistency level LOCAL_QUORUM
Upgrading 2.2.11 to 3.11.2

I haven't found any report of similar issue on the internet. Has anyone heard 
about such behavior? 

Thanks, 
Konrad

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Infinite loop of single SSTable compactions

[Rahul Singh]

Few questions


What is your maximumcompactedbytes across the cluster for this table ?
What’s your TTL ?
What does your data model look like as in what’s your PK?

Rahul
On Jul 25, 2018, 1:07 PM -0400, James Shaw , wrote:
> nodetool compactionstats  --- see compacting which table
> nodetool cfstats keyspace_name.table_name  --- check partition side, 
> tombstones
>
> go the data file directories:  look the data file size, timestamp,  --- 
> compaction will write to new temp file with _tmplink...,
>
> use sstablemetadata ...    look the largest or oldest one first
>
> of course, other factors may be,  like disk space, etc
> also what are compaction_throughput_mb_per_sec in cassandra.yaml
>
> Hope it is helpful.
>
> Thanks,
>
> James
>
>
>
>
> > On Wed, Jul 25, 2018 at 4:18 AM, Martin Mačura  wrote:
> > > Hi,
> > > we have a table which is being compacted all the time, with no change in 
> > > size:
> > >
> > > Compaction History:
> > > compacted_at            bytes_in    bytes_out   rows_merged
> > > 2018-07-25T05:26:48.101 57248063878 57248063878 {1:11655}
> > >
> > >                   2018-07-25T01:09:47.346 57248063878 57248063878
> > > {1:11655}
> > >                                          2018-07-24T20:52:48.652
> > > 57248063878 57248063878 {1:11655}
> > >
> > > 2018-07-24T16:36:01.828 57248063878 57248063878 {1:11655}
> > >
> > >                   2018-07-24T12:11:00.026 57248063878 57248063878
> > > {1:11655}
> > >                                          2018-07-24T07:28:04.686
> > > 57248063878 57248063878 {1:11655}
> > >
> > > 2018-07-24T02:47:15.290 57248063878 57248063878 {1:11655}
> > >
> > >                   2018-07-23T22:06:17.410 57248137921 57248063878
> > > {1:11655}
> > >
> > > We tried setting unchecked_tombstone_compaction to false, had no effect.
> > >
> > > The data is a time series, there will be only a handful of cell
> > > tombstones present. The table has a TTL, but it'll be least a month
> > > before it takes effect.
> > >
> > > Table properties:
> > >    AND compaction = {'class':
> > > 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
> > > 'compaction_window_size': '1', 'compaction_window_unit': 'DAYS',
> > > 'max_threshold': '32', 'min_threshold': '4',
> > > 'unchecked_tombstone_compaction': 'false'}
> > >    AND compression = {'chunk_length_in_kb': '64', 'class':
> > > 'org.apache.cassandra.io.compress.LZ4Compressor'}
> > >    AND crc_check_chance = 1.0
> > >    AND dclocal_read_repair_chance = 0.0
> > >    AND default_time_to_live = 63072000
> > >    AND gc_grace_seconds = 10800
> > >    AND max_index_interval = 2048
> > >    AND memtable_flush_period_in_ms = 0
> > >    AND min_index_interval = 128
> > >    AND read_repair_chance = 0.0
> > >    AND speculative_retry = 'NONE';
> > >
> > > Thanks for any help
> > >
> > >
> > > Martin
> > >
> > > -
> > > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: user-h...@cassandra.apache.org
> > >
>

Re: cassandro nodes restarts

[Rahul Singh]

Do the same nodes reboot or is it arbitrary? I’m wondering if it’s an isolated 
incident related to dat / traffic skew or could happen on any coordinator

Rahul
On Jul 26, 2018, 12:31 AM -0400, Jeff Jirsa , wrote:
> It’s a warning, but probably not causing you problems
>
> A 20kB batch is a hint that your batches are larger than Cassandra expects, 
> but the 5k limit for that logger was somewhat arbitrary, and I would be 
> shocked if 20kB batches were a problem unless you were already close to 
> tipping your cluster
>
> If I were you I’d disable that warning (or set it much higher).
>
> --
> Jeff Jirsa
>
>
> > On Jul 25, 2018, at 7:32 PM, R1 J1  wrote:
> >
> > cassandro nodes restarts
> >
> >
> >
> > we see errors typically like these
> >
> >
> > WARN [Native-Transport-Requests-3] 2018-07-25 20:51:38,520 
> > BatchStatement.java:301 - Batch for "keyspace.table"
> > is of size 19.386KiB, exceeding specified threshold of 5.000KiB by 
> > 14.386KiB.
> >
> >
> > Regards
> > R1J1
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>

Re: Why and How is Cassandra using all my ram ?

[Léo FERLIN SUTTON]

Hello again,

> It's possible that glibc is creating too many memory arenas. Are you
> setting/exporting MALLOC_ARENA_MAX to something sane before calling
> the JVM? You can check that in /proc//environ.

I checked and we have the default value of 4.

> I would also turn on -XX:NativeMemoryTracking=summary and use jcmd to
> check out native memory usage from the JVM's perspective.
>
> -Mark

I have turned on `-XX:NativeMemoryTracking=summary` and have more
information to share.

Right now one of my cassandra nodes is using `22467424` of RSS memory
(around 22GB) according to `ps`.

`nodetool info` tells me it should be using around 17GB :
```
Heap Memory (MB)   : 3129.20 / 12288.00
Off Heap Memory (MB)   : 5165.15
```

Finally this is the output of the jcmd :

```
sudo -u cassandra jcmd 30952 VM.native_memory summary.diff
30952:

Native Memory Tracking:

Total: reserved=15534476KB +402527KB, committed=14271664KB +418143KB
- Java Heap (reserved=12582912KB, committed=12582912KB)
(mmap: reserved=12582912KB, committed=12582912KB)
- Class (reserved=1120624KB +30810KB,
committed=75960KB +29914KB)
(classes #7034 +231)
(malloc=31088KB +26714KB #166677 +123037)
(mmap: reserved=1089536KB +4096KB,
committed=44872KB +3200KB)
-Thread (reserved=184281KB +37799KB,
committed=184281KB +37799KB)
(thread #627 +144)
(stack: reserved=182728KB +37440KB,
committed=182728KB +37440KB)
(malloc=818KB +190KB #3142 +720)
(arena=735KB +169 #1253 +288)
-  Code (reserved=260864KB +3698KB,
committed=67296KB +20210KB)
(malloc=11264KB +3698KB #14716 +4048)
(mmap: reserved=249600KB,
committed=56032KB +16512KB)
-GC (reserved=676609KB +121116KB,
committed=676609KB +121116KB)
(malloc=176897KB +121116KB #163106 +113361)
(mmap: reserved=499712KB, committed=499712KB)
-  Compiler (reserved=2017KB +728KB, committed=2017KB +728KB)
(malloc=1887KB +728KB #3621 +1189)
(arena=131KB #3)
-  Internal (reserved=654807KB +203186KB,
committed=654803KB +203186KB)
(malloc=654803KB +203186KB #284904 +219680)
(mmap: reserved=4KB, committed=0KB)
-Symbol (reserved=16381KB +140KB, committed=16381KB +140KB)
(malloc=14582KB +140KB #65637 +989)
(arena=1798KB #1)
-Native Memory Tracking (reserved=11060KB +7254KB,
committed=11060KB +7254KB)
(malloc=63KB +14KB #685 +148)
(tracking overhead=10997KB +7240KB)
-   Arena Chunk (reserved=312KB -2205KB, committed=312KB -2205KB)
(malloc=312KB -2205KB)
-   Unknown (reserved=24608KB, committed=32KB)
(mmap: reserved=24608KB, committed=32KB)
```

I made a baseline shortly after restarting the cassandra node.

Unless I am mistaken somewhere this result is very surprising, the jvm
is using memory that the native memory tracking doesn't see ?

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org