update cassandra.yaml file on number of cluster nodes

[ZAIDI, ASAD]

Hello Folks,

Can you guys please suggest tool or approach  to update  cassandra.yaml file in 
multi-dc environment with large number of nodes efficiently.

Thank you.
Asad

RE: Cassandra scale-out with no traffic on newly joined nodes

[ZAIDI, ASAD]

Can you share please what replica placement strategy ( in keyspace definition) 
and   partitioner is used across nodes ?
~Asad



From: Sandeep Nethi 
Sent: Tuesday, September 8, 2020 1:12 AM
To: user@cassandra.apache.org
Subject: Re: Cassandra scale-out with no traffic on newly joined nodes

That would be my last option to add a new host as contant point but as per my 
understanding cassandra should auto discover newly joined nodes and server or 
load balance connections automatically but it's not happening. So, i'm trying 
to understand what could be the root cause here.

Another problem that I have noticed is, the number of native client connections 
on newly joined nodes vs old nodes is 1:7 and client connections are not 
balanced across nodes (old nodes are overloaded compared to newly joined 
nodes). Since i have a 6 node cluster with 3 racks, is it mandatory to have 
rack awareness in driver load balancer?

On Tue, Sep 8, 2020 at 6:02 PM manish khandelwal 
mailto:manishkhandelwa...@gmail.com>> wrote:
Can you add new host as contact points and see if traffic lands on them or not?
Also you can verify new nodes are added in system.peers of host name which you 
are giving as contact points

On Tue, Sep 8, 2020 at 11:27 AM Sandeep Nethi 
mailto:nethisande...@gmail.com>> wrote:
Yes, all nodes are UN and no issues identified. Infact i could see some client 
connections on new nodes with telnet but not seeing any traffic.

Cassandra version: 3.11.6
Load Balancing policy used is default with no custom policies.

Thanks,

On Tue, Sep 8, 2020 at 5:52 PM Erick Ramirez 
mailto:erick.rami...@datastax.com>> wrote:
That shouldn't be a problem for the control connection.

I would suggest looking into the load-balancing policy configured on the 
driver. Also, are all the new nodes fully up and fully joined the cluster?

RE: Bootstraping is failing

[ZAIDI, ASAD]

heck if  [streaming_socket_timeout_in_ms ] setting in Cassandra.yaml file if 
that sufficient enough before streaming is interrupted ?
~Asad




From: Surbhi Gupta [mailto:surbhi.gupt...@gmail.com]
Sent: Thursday, May 7, 2020 3:09 PM
To: user@cassandra.apache.org
Subject: Re: Bootstraping is failing


[root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_time

300

[root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_intvl

30

[root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_probes

9

On Thu, 7 May 2020 at 12:32, Adam Scott 
mailto:adam.c.sc...@gmail.com>> wrote:
Maybe a firewall killing a connection?

What does the following show?
cat /proc/sys/net/ipv4/tcp_keepalive_time
cat /proc/sys/net/ipv4/tcp_keepalive_intvl
cat /proc/sys/net/ipv4/tcp_keepalive_probes

On Thu, May 7, 2020 at 10:31 AM Surbhi Gupta 
mailto:surbhi.gupt...@gmail.com>> wrote:
Hi,

We are trying to expand a datacenter and trying to add nodes but when node is 
bootstrapping , it goes half way through and then fail with below error, We 
have increased stremthroughput from 200 to 400 when we were trying for the 2nd 
time but still it failed. We are on 3.11.0 , using G1GC with 31GB heap.


ERROR [MessagingService-Incoming-/10.X.X.X] 2020-05-07 09:42:38,933 
CassandraDaemon.java:228 - Exception in thread 
Thread[MessagingService-Incoming-/10.X.X.X,main]

java.io.IOError: java.io.EOFException: Stream ended prematurely

at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:227)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:215)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:839)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:814)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:425)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:434)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123) 
~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:192)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:180)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

Caused by: java.io.EOFException: Stream ended prematurely

at 
net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218) 
~[lz4-1.3.0.jar:na]

at 
net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150) 
~[lz4-1.3.0.jar:na]

at 
net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117) 
~[lz4-1.3.0.jar:na]

at java.io.DataInputStream.readFully(DataInputStream.java:195) 
~[na:1.8.0_242]

at java.io.DataInputStream.readFully(DataInputStream.java:169) 
~[na:1.8.0_242]

at 
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:437) 
~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:245) 
~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.db.rows.UnfilteredSerializer.readComplexColumn(UnfilteredSerializer.java:665)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$deserializeRowBody$1(UnfilteredSerializer.java:606)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242) 
~[apache-cassandra-3.11.0.jar:3.11.0]

at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197) 
~[apache-cassandra-3.11.0.jar:3.11.0]

at org.apache.cassandra.db.Columns.apply(Columns.java:377) 
~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:600)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at 

RE: New seed node in the cluster immediately UN without passing for UJ state

[ZAIDI, ASAD]

Seed node doesn’t bootstrap so  if new node were to act as seed node, official 
recommendations are to boot strap ‘new’ node first , only after that list that 
node as seed.

Seed nodes are usually same across cluster nodes. You can designate 2 nodes as 
seed per dc in order to mitigate network latency for bootstrapping  node ie.   
Say you have 2dcs you designate 4 nodes (in total) as seed – 2 from each dc.

~Asad


From: Sergio [mailto:lapostadiser...@gmail.com]
Sent: Tuesday, February 25, 2020 6:19 PM
To: user@cassandra.apache.org
Cc: erik.rami...@datastax.com
Subject: Re: New seed node in the cluster immediately UN without passing for UJ 
state

Hi Erick!

Just follow up to your statement:

Limiting the seeds to 2 per DC means :

A) Each node in a DC has at least 2 seeds and those seeds belong to the same DC
or
B) Each node in a DC has at least 2 seeds even across different DC


Thanks,

Sergio

Il giorno gio 13 feb 2020 alle ore 19:46 Erick Ramirez 
mailto:erick.rami...@datastax.com>> ha scritto:
Not a problem. And I've just responded on the new thread. Cheers! 

Is CompactionStrategyClass mising in Cassandra 3.11.4 ?

[ZAIDI, ASAD]

Hello Folks,

I'm trying to modify tables compaction strategy using jmx term - one node at a 
time but unable to retrieve/set CompactionStrategyClass. I don't see option 
available in 3.11.4 version.  I'm wondering what I'm here ?

Thank/Asad


java -jar jmxterm-1.0.1-uber.jar --url localhost:7199
$>domain org.apache.cassandra.db

$>bean 
org.apache.cassandra.db:columnfamily=apc_table_tng_nrt_2_ws,keyspace=enterprise,type=ColumnFamilies
#bean is set to 
org.apache.cassandra.db:columnfamily=apc_table_tng_nrt_2_ws,keyspace=enterprise,type=ColumnFamilies

$>info
#mbean = 
org.apache.cassandra.db:columnfamily=apc_table_tng_nrt_2_ws,keyspace=enterprise,type=ColumnFamilies
#class name = org.apache.cassandra.db.ColumnFamilyStore
# attributes
  %0   - AutoCompactionDisabled (boolean, r)
  %1   - BuiltIndexes (java.util.List, r)
  %2   - ColumnFamilyName (java.lang.String, r)
  %3   - CompactionDiskSpaceCheckEnabled (boolean, r)
  %4   - CompactionParameters (java.util.Map, rw)
  %5   - CompactionParametersJson (java.lang.String, rw)
  %6   - CompressionParameters (java.util.Map, rw)
  %7   - CrcCheckChance (double, w)
  %8   - DroppableTombstoneRatio (double, r)
  %9   - LevelFanoutSize (int, r)
  %10  - MaximumCompactionThreshold (int, rw)
  %11  - MinimumCompactionThreshold (int, rw)
  %12  - SSTableCountPerLevel ([I, r)
  %13  - TableName (java.lang.String, r)
  %14  - UnleveledSSTables (int, r)
# operations
  %0   - void beginLocalSampling(java.lang.String p1,int p2)
  %1   - void compactionDiskSpaceCheck(boolean p1)
  %2   - long estimateKeys()
  %3   - javax.management.openmbean.CompositeData 
finishLocalSampling(java.lang.String p1,int p2)
  %4   - void forceCompactionForTokenRange(java.util.Collection p1)
  %5   - void forceMajorCompaction(boolean p1)
  %6   - java.util.List getSSTablesForKey(java.lang.String p1,boolean p2)
  %7   - java.util.List getSSTablesForKey(java.lang.String p1)
  %8   - void loadNewSSTables()
  %9   - void setCompactionThresholds(int p1,int p2)
  %10  - long trueSnapshotsSize()
#there's no notifications

$>get CompactionStrategyClass
#mbean = 
org.apache.cassandra.db:columnfamily=apc_table_tng_nrt_2_ws,keyspace=enterprise,type=ColumnFamilies:
$>

RE: Keyspace Clone in Existing Cluster

[ZAIDI, ASAD]

If you’re planning to restore snapshot to target keyspace in same cluster – you 
can:


1.  Take snapshot and copy snapshots to shared volume like NFS share so 
later you can load sstables using sstables loader from single node.

2.  Make sure you create target keyspace and tables (without data) only 
structure BEFORE loading with sstalbes. Usually when you take snapshot, a file 
named schema.cql is created .  you can create schema object with this script.

3.  The name of target kespace/tables are import – they have  to match with 
sstableloader input arguments.

4.  Then run sstableloader – you’ll get data/sstables loaded in bulk and 
streamed to all the nodes in cluster.

Thanks/Asad


From: Ankit Gadhiya [mailto:ankitgadh...@gmail.com]
Sent: Tuesday, October 29, 2019 1:20 PM
To: user@cassandra.apache.org
Subject: Re: Keyspace Clone in Existing Cluster

Thanks folks for your responses but still haven't found concrete solution for 
this.

Thanks & Regards,
Ankit Gadhiya


On Tue, Oct 29, 2019 at 2:15 PM Sergio Bilello 
mailto:lapostadiser...@gmail.com>> wrote:
Rolling bounce = Rolling repair per node? Would not it be easy to be scheduled 
with Cassandra Reaper?
On 2019/10/29 15:35:42, Paul Carlucci 
mailto:paul.carlu...@gmail.com>> wrote:
> Copy the schema from your source keyspace to your new target keyspace,
> nodetool snapshot on your source keyspace, copy the SSTable files over, do
> a rolling bounce, repair, enjoy.  In my experience a rolling bounce is
> easier than a nodetool refresh.
>
> It's either that or just copy it with Spark.
>
> On Tue, Oct 29, 2019, 11:19 AM Ankit Gadhiya 
> mailto:ankitgadh...@gmail.com>> wrote:
>
> > Thanks Alex. So How do I copy SSTables from 1.0 to 2.0? (Same
> > SSTableLoader or any other approach?)
> > Also since I've multi-node cluster - I'll have to do this on every single
> > node - is there any tool or better way to execute this just from a single
> > node?
> >
> > *Thanks & Regards,*
> > *Ankit Gadhiya*
> >
> >
> >
> > On Tue, Oct 29, 2019 at 11:16 AM Alex Ott 
> > mailto:alex...@gmail.com>> wrote:
> >
> >> You can create all tables in new keyspace, copy SSTables from 1.0 to 2.0
> >> tables & use nodetool refresh on tables in KS 2.0 to say Cassandra about
> >> them.
> >>
> >> On Tue, Oct 29, 2019 at 4:10 PM Ankit Gadhiya 
> >> mailto:ankitgadh...@gmail.com>>
> >> wrote:
> >>
> >>> Hello Folks,
> >>>
> >>> Greetings!.
> >>>
> >>> I've a requirement in my project to setup Blue-Green deployment for
> >>> Cassandra. E.x. Say My current active schema (application pointing to) is
> >>> Keyspace V1.0 and for my next release I want to setup Keysapce 2.0 (with
> >>> some structural changes) and all testing/validation would happen on it and
> >>> once successful , App would switch connection to keyspace 2.0 - This would
> >>> be generic release deployment for our project.
> >>>
> >>> One of the approach we thought of would be to Create keyspace 2.0 as
> >>> clone from Keyspace 1.0 including data using sstableloader but this would
> >>> be time consuming, also being a multi-node cluster (6+6 in each DC) - it
> >>> wouldn't be very feasible to do this manually on all the nodes for 
> >>> multiple
> >>> tables part of that keyspace. Was wondering if we have any other creative
> >>> way to suffice this requirement.
> >>>
> >>> Appreciate your time on this.
> >>>
> >>>
> >>> *Thanks & Regards,*
> >>> *Ankit Gadhiya*
> >>>
> >>>
> >>
> >> --
> >> With best wishes,Alex Ott
> >> http://alexott.net/
> >> Twitter: alexott_en (English), alexott (Russian)
> >>
> >
>

-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: 
user-h...@cassandra.apache.org

RE: Duplicates columns which are backed by LIST collection types

[ZAIDI, ASAD]

Guess TTL’ed data is lurking around?  If so , you can try get rid of tombstones 
(by reducing gc_grace_seconds to zero? ) and let compaction take care of 
tombstone before sstable  migration. Do keep an eye on hinted handoffs  because 
of  zero’ed gc_grace_second property.


From: Muralikrishna Gutha [mailto:muralikgu...@gmail.com]
Sent: Thursday, October 24, 2019 10:39 AM
To: user@cassandra.apache.org
Subject: Re: Duplicates columns which are backed by LIST collection types

Thanks, List datatype has been in-use for this table almost over a few years 
now and never had issues. We ran into this issue recently when we did the 
keyspace migration.

Thanks,
Murali

On Thu, Oct 24, 2019 at 11:36 AM ZAIDI, ASAD 
mailto:az1...@att.com>> wrote:
Have you chosen correct datatype to begin with, if you don’t want duplicates?

Generally speaking:

A set and a list both represent multiple values but do so differently.
A set doesn’t save ordering and values are sorted in ascending order. No 
duplicates are allowed.

A list saves ordering where you append or prepend the value into the list. A 
list allows duplicates.



From: Muralikrishna Gutha 
[mailto:muralikgu...@gmail.com<mailto:muralikgu...@gmail.com>]
Sent: Thursday, October 24, 2019 10:27 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Cc: Muralikrishna Gutha mailto:muralikgu...@gmail.com>>
Subject: Duplicates columns which are backed by LIST collection types

Hello Guys,

We started noticing strange behavior after we migrated one keyspace from 
existing cluster to new cluster.

We expanded our source cluster from 18 node to 36 nodes and Didn't run 
"nodetool cleanup".
We took sstable backups on source cluster and restored which has duplicate data 
and restored (sstableloader) it on to new cluster. Apparently applications 
started seeing duplicate data mostly on list backed columns. Below is 
sstable2json output for one of the list backed columns.

Clustering Column1:Clustering Column2:mods (List collection type
ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233

 
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b050eac811e9ab2729ea208ce219","eb25d0b13a6611e980b22102e728a233",1570648383445000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b051eac811e9ab2729ea208ce219","eb26bb113a6611e980b22102e728a233",1570648383445000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b052eac811e9ab2729ea208ce219","a4fcf1f1eac811e99664732b9302ab46",1570648383445000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973560ead811e98bf68711844fec13","eb25d0b13a6611e980b22102e728a233",1570654999478000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973561ead811e98bf68711844fec13","eb26bb113a6611e980b22102e728a233",1570654999478000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973562ead811e98bf68711844fec13","a4fcf1f1eac811e99664732b9302ab46",1570654999478000],

Below is the select statement i would expect Cassandra to return data with 
latest timestamp rather it returns duplicate values.

select mods from keyspace.table where partition_key ='1117302' and 
type='ModifierList' and id=eb26e221-3a66-11e9-80b2-2102e728a233;

[image.png]

Any help or guidance is greatly appreciated.

--
Thanks & Regards
  Murali K Gutha


--
Thanks & Regards
  Murali K Gutha

RE: Duplicates columns which are backed by LIST collection types

[ZAIDI, ASAD]

Have you chosen correct datatype to begin with, if you don’t want duplicates?

Generally speaking:

A set and a list both represent multiple values but do so differently.
A set doesn’t save ordering and values are sorted in ascending order. No 
duplicates are allowed.

A list saves ordering where you append or prepend the value into the list. A 
list allows duplicates.



From: Muralikrishna Gutha [mailto:muralikgu...@gmail.com]
Sent: Thursday, October 24, 2019 10:27 AM
To: user@cassandra.apache.org
Cc: Muralikrishna Gutha 
Subject: Duplicates columns which are backed by LIST collection types

Hello Guys,

We started noticing strange behavior after we migrated one keyspace from 
existing cluster to new cluster.

We expanded our source cluster from 18 node to 36 nodes and Didn't run 
"nodetool cleanup".
We took sstable backups on source cluster and restored which has duplicate data 
and restored (sstableloader) it on to new cluster. Apparently applications 
started seeing duplicate data mostly on list backed columns. Below is 
sstable2json output for one of the list backed columns.

Clustering Column1:Clustering Column2:mods (List collection type
ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233

 
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b050eac811e9ab2729ea208ce219","eb25d0b13a6611e980b22102e728a233",1570648383445000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b051eac811e9ab2729ea208ce219","eb26bb113a6611e980b22102e728a233",1570648383445000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b052eac811e9ab2729ea208ce219","a4fcf1f1eac811e99664732b9302ab46",1570648383445000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973560ead811e98bf68711844fec13","eb25d0b13a6611e980b22102e728a233",1570654999478000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973561ead811e98bf68711844fec13","eb26bb113a6611e980b22102e728a233",1570654999478000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973562ead811e98bf68711844fec13","a4fcf1f1eac811e99664732b9302ab46",1570654999478000],

Below is the select statement i would expect Cassandra to return data with 
latest timestamp rather it returns duplicate values.

select mods from keyspace.table where partition_key ='1117302' and 
type='ModifierList' and id=eb26e221-3a66-11e9-80b2-2102e728a233;

[image.png]

Any help or guidance is greatly appreciated.

--
Thanks & Regards
  Murali K Gutha

RE: Constant blocking read repair for such a tiny table

[ZAIDI, ASAD]

Wondering if you’ve  disabled  otc_coalescing_strategy  
CASSANDRA-12676 since 
you’ve upgraded from 2.x?  also if you found luck by  increasing 
native_transport_max_threads  to address blocked NTRs (CASSANDRA-11363)?
~Asad



From: Patrick Lee [mailto:patrickclee0...@gmail.com]
Sent: Wednesday, October 16, 2019 12:22 PM
To: user@cassandra.apache.org
Subject: Re: Constant blocking read repair for such a tiny table

haven't really figured this out yet.  it's not a big problem but it is annoying 
for sure! the cluster was upgraded from 2.1.16 to 3.11.4.  now my only thing is 
i'm not sure if had this type of behavior before the upgrade.  i'm leaning 
toward a no based on my data but i'm just not 100% sure.

just 1 table, out of all the ones on the cluster has this behavior. repair has 
been run few times via reaper.  even did a nodetool compact on the nodes (since 
this table is like 1GB per node..) . just don't see why there would be any 
inconsistency that would trigger read repair.

any insight you may have would be appreciated!  the real thing that started 
this digging into the cluster was during some stress test application team 
complained about high latency (30ms at p98).  this cluster is oversized already 
for this use case with only 14GB of data per node, there is more than enough 
ram so all the data is basically cached in ram.  the only thing that stands out 
is this crazy read repair.  so this read repair may not be my root issue but 
definitely shouldn't be happening like this.

the vm's..
12 cores
82GB ram
1.2TB local ephemeral ssd's

attached the info from 1 of the nodes.

On Tue, Oct 15, 2019 at 2:36 PM Alain RODRIGUEZ 
mailto:arodr...@gmail.com>> wrote:
Hello Patrick,

Still in trouble with this? I must admit I'm really puzzled by your issue. I 
have no real idea of what's going on. Would you share with us the output of:

- nodetool status 
- nodetool describecluster
- nodetool gossipinfo
- nodetool tpstats

Also you said the app is running for a long time, with no changes. What about 
Cassandra? Any recent operations?

I hope that with this information we might be able to understand better and 
finally be able to help.

---
Alain Rodriguez - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

Le ven. 4 oct. 2019 à 00:25, Patrick Lee 
mailto:patrickclee0...@gmail.com>> a écrit :
this table was actually leveled compaction before, just changed it to size 
tiered yesterday while researching this.

On Thu, Oct 3, 2019 at 4:31 PM Patrick Lee 
mailto:patrickclee0...@gmail.com>> wrote:
its not really time series data.   and it's not updated very often, it would 
have some updates but pretty infrequent. this thing should be super fast, on 
avg it's like 1 to 2ms p99 currently but if they double - triple the traffic on 
that table latencies go upward to 20ms to 50ms.. the only odd thing i see is 
just that there are constant read repairs that follow the same traffic pattern 
on the reads, which shows constant writes on the table (from the read repairs), 
which after read repair or just normal full repairs (all full through reaper, 
never ran any incremental repair) i would expect it to not have any mismatches. 
 the other 5 tables they use on the cluster can have the same level traffic all 
very simple select from table by partition key which returns a single record

On Thu, Oct 3, 2019 at 4:21 PM John Belliveau 
mailto:belliveau.j...@gmail.com>> wrote:
Hi Patrick,

Is this time series data? If so, I have run into issues with repair on time 
series data using the SizeTieredCompactionStrategy. I have had better luck 
using the TimeWindowCompactionStrategy.

John

Sent from 
Mail
 for Windows 10

From: Patrick Lee
Sent: Thursday, October 3, 2019 5:14 PM
To: user@cassandra.apache.org
Subject: Constant blocking read repair for such a tiny table

I have a cluster that is running 3.11.4 ( was upgraded a while back from 2.1.16 
).  what I see is a steady rate of read repair which is about 10% constantly on 
only this 1 table.  Repairs have been run (actually several times).  The table 
does not have a lot of writes to it so after repair, or even after a read 
repair I would expect it to be fine.  the reason i'm having to dig into this so 
much is for the fact that under a much large traffic load than their normal 
traffic, 

RE: Cassandra JVM configuration

[ZAIDI, ASAD]

Every use case is unique  so as such jvm configs go with it. 8G may or may not 
be sufficient depending on  live data you keep in, or fetch to memory. You can 
opt using G1GC,  that is easy to start with. 
Some good suggestions are made if you want to try G1GC or stick with CMS.  Take 
a look at [ https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html ]



-Original Message-
From: p...@xvalheru.org [mailto:p...@xvalheru.org] 
Sent: Thursday, September 05, 2019 9:05 AM
To: User 
Subject: Cassandra JVM configuration

Hi,

sorry to bring such question, but I want to ask what are the best JVM options 
for Cassandra node? In solution I'm implementing the Cassandra serves as 
read-only storage (of course populated at beginning) - the records are not 
changed in time. Currently each  Cassandra node's VM has this configuration 
16CPUs and 64GB of RAM. I've set these JVM options: 
-Xms4G and -Xmx40G; JDK 1.8.0_221. My question is what's the best size of 
memory for Cassandra node and if there's any relation between number of CPUs 
and memory size. When I've searched for an answer I've found that suggested 
size for node is 8GB of RAM, but I have doubts.

Thanks

Pat


Freehosting PIPNI - 
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pipni.cz_=DwICAg=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=H2t7NfxXR_c8S_gVHKbkqoJYt2uJNvBvRcQBiEk2beI=OOXUbYzda5jQw9DZVUA4yExJv1gDNPzsxS0yqPlrDqY=


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

RE: Datafile Corruption

[ZAIDI, ASAD A]

Did you check if packets are NOT being dropped for network interfaces Cassandra 
instances are consuming (ifconfig –a) internode compression is set for all 
endpoint – may be network is playing any role here?
is this corruption limited so certain keyspace/table | DCs or is that wide 
spread – the log snippet you shared it looked like only specific keyspace/table 
is affected – is that correct?
When you remove corrupted sstable of a certain table, I guess you verifies all 
nodes for corrupted sstables for same table (may be with with nodetool scrub 
tool) so to limit spread of corruptions – right?
Just curious to know – you’re not using lz4/default compressor for all tables 
there must be some reason for it.



From: Philip Ó Condúin [mailto:philipocond...@gmail.com]
Sent: Thursday, August 08, 2019 6:20 AM
To: user@cassandra.apache.org
Subject: Re: Datafile Corruption

Hi All,

Thank you so much for the replies.

Currently, I have the following list that can potentially cause some sort of 
corruption in a Cassandra cluster.

  *   Sudden Power cut  -  We have had no power cuts in the datacenters
  *   Network Issues - no network issues from what I can tell
  *   Disk full - I don't think this is an issue for us, see disks below.
  *   An issue in Casandra version like Cassandra-13752 - couldn't find any 
Jira issues similar to ours.
  *   Bit Flips - we have compression enabled so I don't think this should be 
an issue.
  *   Repair during upgrade has caused corruption too - we have not upgraded
  *   Dropping and adding columns with the same name but a different type - I 
will need to ask the apps team how they are using the database.


Ok, let me try and explain the issue we are having, I am under a lot of 
pressure from above to get this fixed and I can't figure it out.

This is a PRE-PROD environment.

  *   2 datacenters.
  *   9 physical servers in each datacenter
  *   4 Cassandra instances on each server
  *   72 Cassandra instances across the 2 data centres, 36 in site A, 36 in 
site B.

We also have 2 Reaper Nodes we use for repair.  One reaper node in each 
datacenter each running with its own Cassandra back end in a cluster together.

OS Details [Red Hat Linux]
cass_a@x 0 10:53:01 ~ $ uname -a
Linux x 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 
x86_64 x86_64 GNU/Linux

cass_a@x 0 10:57:31 ~ $ cat /etc/*release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"

Storage Layout
cass_a@xx 0 10:46:28 ~ $ df -h
Filesystem Size  Used Avail Use% Mounted on
/dev/mapper/vg01-lv_root20G  2.2G   18G  11% /
devtmpfs63G 0   63G   0% /dev
tmpfs   63G 0   63G   0% /dev/shm
tmpfs   63G  4.1G   59G   7% /run
tmpfs   63G 0   63G   0% /sys/fs/cgroup
>> 4 cassandra instances
/dev/sdd   1.5T  802G  688G  54% /data/ssd4
/dev/sda   1.5T  798G  692G  54% /data/ssd1
/dev/sdb   1.5T  681G  810G  46% /data/ssd2
/dev/sdc   1.5T  558G  932G  38% /data/ssd3

Cassandra load is about 200GB and the rest of the space is snapshots

CPU
cass_a@x 127 10:58:47 ~ $ lscpu | grep -E '^Thread|^Core|^Socket|^CPU\('
CPU(s):64
Thread(s) per core:2
Core(s) per socket:16
Socket(s): 2

Description of problem:
During repair of the cluster, we are seeing multiple corruptions in the log 
files on a lot of instances.  There seems to be no pattern to the corruption.  
It seems that the repair job is finding all the corrupted files for us.  The 
repair will hang on the node where the corrupted file is found.  To fix this we 
remove/rename the datafile and bounce the Cassandra instance.  Our hardware/OS 
team have stated there is no problem on their side.  I do not believe it the 
repair causing the corruption.

We have maintenance scripts that run every night running compactions and 
creating snapshots, I decided to turn these off, fix any corruptions we 
currently had and ran major compaction on all nodes, once this was done we had 
a "clean" cluster and we left the cluster for a few days.  After the process we 
noticed one corruption in the cluster, this datafile was created after I turned 
off the maintenance scripts so my theory of the scripts causing the issue was 
wrong.  We then kicked off another repair and started to find more corrupt 
files created after the maintenance script was turned off.


So let me give you an example of a corrupted file and maybe someone might be 
able to work through it with me?

When this corrupted file was reported in the log it looks like it was the 
repair that found it.

$ journalctl -u cassmeta-cass_b.service --since "2019-08-07 22:25:00" --until 
"2019-08-07 22:45:00"

Aug 07 22:30:33 cassandra[34611]: INFO  21:30:33 Writing 

RE: Repair failed and crash the node, how to bring it back?

[ZAIDI, ASAD A]

I don’t think anyone can predict with certainty if instance won’t crash but 
there are good chances it will -  unless you take remedial actions.
If you are not doing subrange repair, a lot of merkle tree data can potentially 
be scanned/streamed taking toll on memory resources – that , taking  account of 
all other running operations , easily bust available memory.

You can do few things like – as short term measure – increase allotted heap 
size along with running subrange repair with 
script or by using 
reaper tool.
You may also want to check partition sizes of tables (nodetool tablestats) if 
they’re bloated. See if table scans  are infested with lots of tombstones which 
in turn also tax on heap consumption. My $.002 cents for the moment.



From: Martin Xue [mailto:martin...@gmail.com]
Sent: Wednesday, July 31, 2019 5:05 PM
To: user@cassandra.apache.org
Subject: Re: Repair failed and crash the node, how to bring it back?

Hi Alex,

Thanks for your reply. The disk space was around 80%. The crash happened during 
repair, primary range full repair on 1TB keyspace.

Would that crash again?

Thanks
Regards
Martin

On Thu., 1 Aug. 2019, 12:04 am Alexander Dejanovski, 
mailto:a...@thelastpickle.com>> wrote:
It looks like you have a corrupted hint file.
Did the node run out of disk space while repair was running?

You might want to move the hint files off their current directory and try to 
restart the node again.
Since you'll have lost mutations then, you'll need... to run repair ¯\_(ツ)_/¯

-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com


On Wed, Jul 31, 2019 at 3:51 PM Martin Xue 
mailto:martin...@gmail.com>> wrote:
Hi,

I am running repair on production, started with one of 6 nodes in the cluster 
(3 nodes in each of two DC). Cassandra version 3.0.14.

running: repair -pr --full keyspace on node 1, 1TB data, takes two days, and 
crash,

error shows:
3202]] finished (progress: 3%)
Exception occurred during clean-up. 
java.lang.reflect.UndeclaredThrowableException
Cassandra has shutdown.
error: [2019-07-31 20:19:20,797] JMX connection closed. You should check server 
log for repair status of keyspace keyspace_masked (Subsequent keyspaces are not 
going to be repaired).
-- StackTrace --
java.io.IOException: [2019-07-31 20:19:20,797] JMX connection closed. You 
should check server log for repair status of keyspace keyspace_masked keyspaces 
are not going to be repaired).
at 
org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:97)
at 
org.apache.cassandra.tools.RepairRunner.handleConnectionClosed(RepairRunner.java:91)
at 
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:90)
at 
javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
at 
javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
at 
javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
at 
javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
at 
javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
at javax.management.remote.rmi.RMIConnector.close(RMIConnector.java:533)
at 
javax.management.remote.rmi.RMIConnector.access$1300(RMIConnector.java:121)
at 
javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1534)
at 
javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1352)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchOneNotif(ClientNotifForwarder.java:655)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:607)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:471)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)

system.log shows
INFO  [Service Thread] 2019-07-31 20:19:08,579 GCInspector.java:284 - G1 Young 
Generation GC in 2915ms.  G1 Eden Space: 914358272 -> 0; G1 Old Gen: 
19043999248 -> 20219035248;
INFO  [Service Thread] 2019-07-31 20:19:08,579 StatusLogger.java:52 - Pool Name 
   Active   Pending  Completed   Blocked  All Time Blocked
INFO  [Service 

RE: Repair / compaction for 6 nodes, 2 DC cluster

[ZAIDI, ASAD A]

Did you get chance to look at tlp reaper tool i.e. http://cassandra-reaper.io/
It is pretty awesome – Thanks to TLP team.



From: Martin Xue [mailto:martin...@gmail.com]
Sent: Wednesday, July 31, 2019 12:09 AM
To: user@cassandra.apache.org
Subject: Repair / compaction for 6 nodes, 2 DC cluster

Hello,

Good day. This is Martin.

Can someone help me with the following query regarding Cassandra repair and 
compaction?

Currently we have a large keyspace (keyspace_event) with 1TB of data (in 
/var/lib/cassandra/data/keyspace_event);
There is a cluster with Datacenter 1 contains 3 nodes, Data center 2 containing 
3 nodes; All together 6 nodes;

As part of maintenance, I run the repair on this keyspace with the following 
command:

nodetool repair -pr --full keyspace_event;

now it has been run for 2 days. yes 2 days, when doing nodetool tpstats, it 
shows there is a compaction running:

CompactionExecutor1 15783732 0  
   0

nodetool compactionstats shows:

pending tasks: 6
id   compaction type
   keyspace  table   completed   
totalunit   progress
  249ec5f1-b225-11e9-82bd-5b36ef02cadd   Anticompaction after repair   
keyspace_event table_event   1916937740948   2048931045927   bytes 93.56%


Now my questions are:
1. why running repair (with primary range option, -pr, as I want to limit the 
repair node by node), triggered the compaction running on other nodes?
2. when I run the repair on the second node with nodetool repair -pr --full 
keyspace_event; will the subsequent compaction run again on all the 6 nodes?

I want to know what are the best option to run the repair (full repair) as we 
did not run it before, especially if it can take less time (in current speed it 
will take 2 weeks to finish all).

I am running Cassandra 3.0.14

Any suggestions will be appreciated.

Thanks
Regards
Martin

RE: Jmx metrics shows node down

[ZAIDI, ASAD A]

Another way to purge gossip info  from each node is to:


1.  Gracefully stop cassandra i.e. nodetool drain; kill Casandra PID

2.  Move/delete files from $DATADIR/system/peers/

3.  Add JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false" in jvm.options 
file

4.  Restart Cassandra service.

5.  Repeat above steps on each nodes in dc/cluster.

6.  Once gossip info is purged, remove jvm option added in step 3 and restart 
instance again.

Depending on cluster,load size you may get this done swiftly.

~Asad


From: yuping wang [mailto:yupingwyp1...@gmail.com]
Sent: Monday, July 29, 2019 10:56 AM
To: user@cassandra.apache.org
Subject: Re: Jmx metrics shows node down

Is there workaround to shorten 72 hours to something shorter?(you said by 
default, wondering if one can set a non-default value?)

Thanks,
Yuping

On Jul 29, 2019, at 7:28 AM, Oleksandr Shulgin 
mailto:oleksandr.shul...@zalando.de>> wrote:
On Mon, Jul 29, 2019 at 1:21 PM Rahul Reddy 
mailto:rahulreddy1...@gmail.com>> wrote:

Decommissioned 2 nodes from cluster nodetool status doesn't  list the nodes as 
expected but jmx metrics shows still those 2 nodes has down. Nodetool gossip 
shows the 2 nodes in Left state. Why does my jmx still shows those nodes down 
even after 24 hours. Cassandra version 3.11.3 ?

AFAIK, the nodes are not removed from gossip for 72 hours by default.

Anything else need to be done?

Wait another 48 hours? ;-)

--
Alex

RE: Performance impact with ALLOW FILTERING clause.

[ZAIDI, ASAD A]

Thank you all for your insights.

When spark-connector adds allows filtering to a query, it makes the query to 
just ‘run’ no matter if it is expensive for larger table OR  not so expensive 
for table with fewer rows.
In my particular case, nodes are reaching 2TB/per node load in 50 node cluster. 
When bunch of such queries run ,  causes impact on server resources.

Since allow filtering is an expensive operation - I’m trying find knobs which 
if I turn, mitigate the impact.

What I think , correct me if I am wrong , is – it is query design itself which 
is not optimized per table design  - that in turn causing connector to add 
allow filtering implicitly.  I’m not thinking to add secondary indexes on 
tables because they’ve their own overheads.  kindly share if there are  other 
means which we can use to influence connector not to use allow filtering.

Thanks again.
Asad



From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: Thursday, July 25, 2019 10:24 AM
To: cassandra 
Subject: Re: Performance impact with ALLOW FILTERING clause.

"unpredictable" is such a loaded word. It's quite predictable, but it's often 
mispredicted by users.

"ALLOW FILTERING" basically tells the database you're going to do a query that 
will require scanning a bunch of data to return some subset of it, and you're 
not able to provide a WHERE clause that's sufficiently fine grained to avoid 
the scan. It's a loose equivalent of doing a full table scan in SQL databases - 
sometimes it's a valid use case, but it's expensive, you're ignoring all of the 
indexes, and you're going to do a lot more work.

It's predictable, though - you're probably going to walk over some range of 
data. Spark is grabbing all of the data to load into RDDs, and it probably does 
it by slicing up the range, doing a bunch of range scans.

It's doing that so it can get ALL of the data and do the filtering / joining / 
searching in-memory in spark, rather than relying on cassandra to do the 
scanning/searching on disk.

On Thu, Jul 25, 2019 at 6:49 AM ZAIDI, ASAD A 
mailto:az1...@att.com>> wrote:
Hello Folks,

I was going thru documentation and saw at many places saying ALLOW FILTERING 
causes performance unpredictability.  Our developers says ALLOW FILTERING 
clause is implicitly added on bunch of queries by spark-Cassandra  connector 
and they cannot control it; however at the same time we see unpredictability in 
application performance – just as documentation says.

I’m trying to understand why would a connector add a clause in query when this 
can cause negative impact on database/application performance. Is that data 
model that is driving connector make its decision and add allow filtering to 
query automatically or if there are other reason this clause is added to the 
code. I’m not a developer though I want to know why developer don’t have any 
control on this to happen.

I’ll appreciate your guidance here.

Thanks
Asad

Performance impact with ALLOW FILTERING clause.

[ZAIDI, ASAD A]

Hello Folks,

I was going thru documentation and saw at many places saying ALLOW FILTERING 
causes performance unpredictability.  Our developers says ALLOW FILTERING 
clause is implicitly added on bunch of queries by spark-Cassandra  connector 
and they cannot control it; however at the same time we see unpredictability in 
application performance – just as documentation says.

I’m trying to understand why would a connector add a clause in query when this 
can cause negative impact on database/application performance. Is that data 
model that is driving connector make its decision and add allow filtering to 
query automatically or if there are other reason this clause is added to the 
code. I’m not a developer though I want to know why developer don’t have any 
control on this to happen.

I’ll appreciate your guidance here.

Thanks
Asad

RE: Rebooting one Cassandra node caused all the application nodes go down

[ZAIDI, ASAD A]

https://lore.kernel.org/patchwork/patch/884501/
according to this link , it sound like having patched kernel is real solution 
to bug



From: Rahul Reddy [mailto:rahulreddy1...@gmail.com]
Sent: Friday, July 19, 2019 11:48 AM
To: user@cassandra.apache.org
Subject: Re: Rebooting one Cassandra node caused all the application nodes go 
down

Raj,

No that was not the case in system.log I see the started listening to call 
client at 16:42 but some how it still unreachable to 16:50 below grafana 
dashboard shows it. Once everything up in logs why would it still show down in 
nodetool status and grafana.

Zaidi,

In latest aws Linux Ami they took care of this bug . And also changing the Ami 
needs rebuild of all the nodes so didn't took that route.

On Fri, Jul 19, 2019, 12:32 PM ZAIDI, ASAD A 
mailto:az1...@att.com>> wrote:
“aws asked to set nvme_timeout to higher number in etc/grub.conf.”

Did you ask AWS if setting higher value is real solution to bug - Is there not 
any patch available to address the bug?   - just curios to know

From: Rahul Reddy 
[mailto:rahulreddy1...@gmail.com<mailto:rahulreddy1...@gmail.com>]
Sent: Friday, July 19, 2019 10:49 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Rebooting one Cassandra node caused all the application nodes go down

Here ,

We have 6 nodes each in 2 data centers us-east-1 and us-west-2  . We have RF 3 
and  cl set to local quorum. And gossip snitch. All our instance are c5.2xlarge 
and data files and comit logs are stored in gp2 ebs.  C5 instance type had a 
bug which aws asked to set nvme_timeout to higher number in etc/grub.conf. 
after setting the parameter and did run nodetool drain and reboot the node in 
east

Instance cameup but Cassandra didn't come up normal had to start the Cassandra. 
Cassandra cameup but it shows other instances down. Even though didn't reboot 
the other node down same was observed in one other node. How could that happen 
and don't any errors in system.log which is set to info.
Without any intervention gossip settled in 10 mins entire cluster became normal.

Tried same thing West it happened again



I'm concerned how to check what caused it and if a reboot happens again how to 
avoid this.
 If I just  STOP Cassandra instead of reboot I don't see this issue.

RE: Rebooting one Cassandra node caused all the application nodes go down

[ZAIDI, ASAD A]

“aws asked to set nvme_timeout to higher number in etc/grub.conf.”

Did you ask AWS if setting higher value is real solution to bug - Is there not 
any patch available to address the bug?   - just curios to know

From: Rahul Reddy [mailto:rahulreddy1...@gmail.com]
Sent: Friday, July 19, 2019 10:49 AM
To: user@cassandra.apache.org
Subject: Rebooting one Cassandra node caused all the application nodes go down

Here ,

We have 6 nodes each in 2 data centers us-east-1 and us-west-2  . We have RF 3 
and  cl set to local quorum. And gossip snitch. All our instance are c5.2xlarge 
and data files and comit logs are stored in gp2 ebs.  C5 instance type had a 
bug which aws asked to set nvme_timeout to higher number in etc/grub.conf. 
after setting the parameter and did run nodetool drain and reboot the node in 
east

Instance cameup but Cassandra didn't come up normal had to start the Cassandra. 
Cassandra cameup but it shows other instances down. Even though didn't reboot 
the other node down same was observed in one other node. How could that happen 
and don't any errors in system.log which is set to info.
Without any intervention gossip settled in 10 mins entire cluster became normal.

Tried same thing West it happened again



I'm concerned how to check what caused it and if a reboot happens again how to 
avoid this.
 If I just  STOP Cassandra instead of reboot I don't see this issue.

node re-start delays , busy Deleting mc-txn-compaction/ Adding log file replica

[ZAIDI, ASAD A]

I’m on environment with  apache Cassandra 3.11.1 with  java 1.8.0_144.

One Node went OOM and crashed. Re-starting this crashed node is taking long 
time. Trace level debug log is showing messages like:


Debug.log trace excerpt:


TRACE [main] 2019-06-18 21:30:43,449 LogTransaction.java:217 - Deleting 
/cassandra/data/enterprise/device_connection_ws-f65649e0aea011e7baeb8166fa28890a/mc-9337720-big-CompressionInfo.db
TRACE [main] 2019-06-18 21:30:43,449 LogTransaction.java:217 - Deleting 
/cassandra/data/enterprise/device_connection_ws-f65649e0aea011e7baeb8166fa28890a/mc-9337720-big-Filter.db
TRACE [main] 2019-06-18 21:30:43,449 LogTransaction.java:217 - Deleting 
/cassandra/data/enterprise/device_connection_ws-f65649e0aea011e7baeb8166fa28890a/mc-9337720-big-TOC.txt
TRACE [main] 2019-06-18 21:30:43,455 LogTransaction.java:217 - Deleting 
/cassandra/data/enterprise/device_connection_ws-f65649e0aea011e7baeb8166fa28890a/mc_txn_compaction_642976c0-91c3-11e9-97bb-6b1dee397c3f.log
TRACE [main] 2019-06-18 21:30:43,458 LogReplicaSet.java:67 - Added log file 
replica 
/cassandra/data/enterprise/device_connection_ws-f65649e0aea011e7baeb8166fa28890a/mc_txn_compaction_5a6c8c90-91cc-11e9-97bb-6b1dee397c3f.log


Above messages are repeated for unique [mc--* ] files. Such messages are 
repeating constantly.

I’m seeking help here to find out what may be going on here , any hint to root 
cause and how I can quickly start the node. Thanks in advance.

Regards/asad

RE: postmortem on 2.2.13 scale out difficulties

[ZAIDI, ASAD A]

Adding one node at a time – is that successful?
Check value of streaming_socket_timeout_in_ms parameter in cassandra.yaml and 
increase if needed.
Have you tried Nodetool bootstrap resume & jvm option i.e. JVM_OPTS="$JVM_OPTS 
-Dcassandra.consistent.rangemovement=false"  ?


From: Carl Mueller [mailto:carl.muel...@smartthings.com.INVALID]
Sent: Wednesday, June 12, 2019 11:35 AM
To: user@cassandra.apache.org
Subject: Re: postmortem on 2.2.13 scale out difficulties

We only were able to scale out four nodes and then failures started occurring, 
including multiple instances of nodes joining a cluster without streaming.

Sigh.

On Tue, Jun 11, 2019 at 3:11 PM Carl Mueller 
mailto:carl.muel...@smartthings.com>> wrote:
We had a three-DC (asia-tokyo/europe/us) cassandra 2.2.13 cluster, AWS, IPV6

Needed to scale out the asia datacenter, which was 5 nodes, europe and us were 
25 nodes

We were running into bootstrapping issues where the new node failed to 
bootstrap/stream, it failed with

"java.lang.RuntimeException: A node required to move the data consistently is 
down"

...even though they were all up based on nodetool status prior to adding the 
node.

First we increased the phi_convict_threshold to 12, and that did not help.

CASSANDRA-12281 appeared similar to what we had problems with, but I don't 
think we hit that. Somewhere in there someone wrote

"For us, the workaround is either deleting the data (then bootstrap again), or 
increasing the ring_delay_ms. And the larger the cluster is, the longer 
ring_delay_ms is needed. Based on our tests, for a 40 nodes cluster, it 
requires ring_delay_ms to be >50seconds. For a 70 nodes cluster, >100seconds. 
Default is 30seconds."
Given the WAN nature or our DCs, we used ring_delay_ms to 100 seconds and it 
finally worked.

side note:

During the rolling restarts for setting phi_convict_threshold we observed quite 
a lot of status map variance between nodes (we have a program to poll all of a 
datacenter or cluster's view of the gossipinfo and statuses. AWS appears to 
have variance in networking based on the phi_convict_threshold advice, I'm not 
sure if our difficulties were typical in that regard and/or if our IPV6 and/or 
globally distributed datacenters were exacerbating factors.

We could not reproduce this in loadtest, although loadtest is only eu and us 
(but is IPV6)

RE: Bootstrapping to Replace a Dead Node vs. Adding a New Node:Consistency Guarantees

[ZAIDI, ASAD A]

The article you mentioned here clearly says  “For new users to Cassandra, the 
safest way to add multiple nodes into a cluster is to add them one at a time. 
Stay tuned as I will be following up with another post on bootstrapping.”

When extending cluster it is indeed recommended to go slow & serially. 
Optionally you can use cassandra.consistent.rangemovement=false but you can run 
in getting over streamed data.  Since you’re using release way newer when fixed 
introduced , I assumed you won’t see same behavior as described for the version 
which fix addresses. After adding node , if you won’t get  consistent data, you 
query consistency level should be able to pull consistent data , given you can 
tolerate bit latency until your repair is complete – if you go by 
recommendation i.e. to add one node at a time – you’ll avoid all these nuances .


From: Fd Habash [mailto:fmhab...@gmail.com]
Sent: Wednesday, May 01, 2019 3:12 PM
To: user@cassandra.apache.org
Subject: RE: Bootstrapping to Replace a Dead Node vs. Adding a New 
Node:Consistency Guarantees

Probably, I needed to be clearer in my inquiry ….

I’m investigating a situation where our diagnostic data is telling us that C* 
has lost some of the application data. I mean, getsstables for the data returns 
zero on all nodes in all racks.

The last pickle article below & Jeff Jirsa had described a situation where 
bootstrapping a node to extend the cluster can loose data if this new node 
bootstraps from a stale SECONDARY replica (node that was offline > hinted 
had-off window). This was fixed in cassandra-2434. 
http://thelastpickle.com/blog/2017/05/23/auto-bootstrapping-part1.html

The article & the Jira above describe bootstrapping when extending a cluster.

I understand replacing a dead node does not involve range movement, but will 
the above Jira fix prevent the bootstrap process when a replacing a dead node 
from using secondary replica?

Thanks


Thank you

From: Fred Habash
Sent: Wednesday, May 1, 2019 6:50 AM
To: user@cassandra.apache.org
Subject: Re: Bootstrapping to Replace a Dead Node vs. Adding a New 
Node:Consistency Guarantees

Thank you.

Range movement is one reason this is enforced when adding a new node. But, what 
about forcing a consistent bootstrap i.e. bootstrapping from primary owner of 
the range and not a secondary replica.

How’s consistent bootstrap enforced when replacing a dead node.

-
Thank you.

On Apr 30, 2019, at 7:40 PM, Alok Dwivedi 
mailto:alok.dwiv...@instaclustr.com>> wrote:
When a new node joins the ring, it needs to own new token ranges. This should 
be unique to the new node and we don’t want to end up in a situation where two 
nodes joining simultaneously can own same range (and ideally evenly 
distributed). Cassandra has this 2 minute wait rule for gossip state to 
propagate before a node is added.  But this on its does not guarantees that 
token ranges can’t overlap. See this ticket for more details 
https://issues.apache.org/jira/browse/CASSANDRA-7069
 To overcome this  issue, the approach was to only allow one node joining at a 
time.

When you replace a dead node the new token range selection does not applies as 
the replacing node just owns the token ranges of the dead node. I think that’s 
why the restriction of only replacing one node at a time does not applies in 
this case.


Thanks
Alok Dwivedi
Senior Consultant
https://www.instaclustr.com/platform/





From: Fd Habash mailto:fmhab...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Wednesday, 1 May 2019 at 06:18
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Bootstrapping to Replace a Dead Node vs. Adding a New Node: 
Consistency Guarantees

Reviewing the documentation &  based on my testing, using C* 2.2.8, I was not 
able to extend the cluster by adding multiple nodes simultaneously. I got an 
error message …

Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while 
cassandra.consistent.rangemovement is true

I understand this is to force a node to bootstrap from the former owner of the 

RE: cassandra node was put down with oom error

[ZAIDI, ASAD A]

Is there any chance partition size has grown over time and taking much 
allocated memory - if yes,  that could also affect compaction thread as they'll 
too take more heap and kept in heap longer - leaving less for other processes . 
You can check partition size if they are manageable using nodetool tablestats - 
ideally size  should be even across nodes. you can check if # of concurrent 
compactor (nodetool) are optimal and if throughput is capped/ throttled (with 
nodetool utility). See if repair is unusually  running longer  taking much 
resources  i.e. cpu/heap  etc.  check if storage is not acting up (using iostat 
-x , look at await column). See if there is bursty workload /batches are 
hitting nodes tipping over the instance  using nodetool tpstats  (look at 
native-transport-requests - all time blocked column) .. above should give some 
clue what is going around



-Original Message-
From: Mia [mailto:yeomii...@gmail.com] 
Sent: Wednesday, May 01, 2019 5:47 AM
To: user@cassandra.apache.org
Subject: Re: cassandra node was put down with oom error

Hello, Ayub.

I'm using apache cassandra, not dse edition. So I have never used the dse 
search feature.
In my case, all the nodes of the cluster have the same problem. 

Thanks.

On 2019/05/01 06:13:06, Ayub M  wrote: 
> Do you have search on the same nodes or is it only cassandra. In my 
> case it was due to a memory leak bug in dse search that consumed more 
> memory resulting in oom.
> 
> On Tue, Apr 30, 2019, 2:58 AM yeomii...@gmail.com 
> 
> wrote:
> 
> > Hello,
> >
> > I'm suffering from similar problem with OSS cassandra version3.11.3.
> > My cassandra cluster have been running for longer than 1 years and 
> > there was no problem until this year.
> > The cluster is write-intensive, consists of 70 nodes, and all rows 
> > have 2 hr TTL.
> > The only change is the read consistency from QUORUM to ONE. (I 
> > cannot revert this change because of the read latency) Below is my 
> > compaction strategy.
> > ```
> > compaction = {'class':
> > 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
> > 'compaction_window_size': '3', 'compaction_window_unit': 'MINUTES',
> > 'enabled': 'true', 'max_threshold': '32', 'min_threshold': '4',
> > 'tombstone_compaction_interval': '60', 'tombstone_threshold': '0.2',
> > 'unchecked_tombstone_compaction': 'false'} ``` I've tried rolling 
> > restarting the cluster several times, but the memory usage of 
> > cassandra process always keeps going high.
> > I also tried Native Memory Tracking, but it only measured less 
> > memory usage than the system mesaures (RSS in 
> > /proc/{cassandra-pid}/status)
> >
> > Is there any way that I could figure out the cause of this problem?
> >
> >
> > On 2019/01/26 20:53:26, Jeff Jirsa  wrote:
> > > You’re running DSE so the OSS list may not be much help. Datastax 
> > > May
> > have more insight
> > >
> > > In open source, the only things offheap that vary significantly 
> > > are
> > bloom filters and compression offsets - both scale with disk space, 
> > and both increase during compaction. Large STCS compaction can cause 
> > pretty meaningful allocations for these. Also, if you have an 
> > unusually low compression chunk size or a very low bloom filter FP 
> > ratio, those will be larger.
> > >
> > >
> > > --
> > > Jeff Jirsa
> > >
> > >
> > > > On Jan 26, 2019, at 12:11 PM, Ayub M  wrote:
> > > >
> > > > Cassandra node went down due to OOM, and checking the 
> > > > /var/log/message
> > I see below.
> > > >
> > > > ```
> > > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: java invoked oom-killer:
> > gfp_mask=0x280da, order=0, oom_score_adj=0
> > > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: java cpuset=/ 
> > > > mems_allowed=0 
> > > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Node 0 DMA: 1*4kB (U) 
> > > > 0*8kB
> > 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 
> > 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
> > > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Node 0 DMA32: 
> > > > 1294*4kB (UM)
> > 932*8kB (UEM) 897*16kB (UEM) 483*32kB (UEM) 224*64kB (UEM) 114*128kB 
> > (UEM) 41*256kB (UEM) 12*512kB (UEM) 7*1024kB (UE
> > > > M) 2*2048kB (EM) 35*4096kB (UM) = 242632kB Jan 23 20:07:17 
> > > > ip-xxx-xxx-xxx-xxx kernel: Node 0 Normal: 5319*4kB
> > (UE) 3233*8kB (UEM) 960*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 
> > 0*512kB 0*1024kB 0*2048kB 0*4096kB = 62500kB
> > > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Node 0 
> > > > hugepages_total=0
> > hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
> > > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Node 0 
> > > > hugepages_total=0
> > hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> > > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: 38109 total pagecache 
> > > > pages Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: 0 pages in swap 
> > > > cache Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Swap cache 
> > > > stats: add 0,
> > delete 0, find 0/0
> > > > Jan 23 20:07:17 

RE: Decommissioning a new node when the state is JOINING

[ZAIDI, ASAD A]

Just stop the server/kill C* process  as node never fully joined the cluster 
yet – that should be enough. You can safely remove the data i.e. streamed-in on 
new node so you can use the node for other new cluster.


From: Akshay Bhardwaj [mailto:akshay.bhardwaj1...@gmail.com]
Sent: Tuesday, April 30, 2019 6:35 AM
To: user@cassandra.apache.org
Subject: Decommissioning a new node when the state is JOINING

Hi Experts,

I have a cassandra cluster running with 5 nodes. For some reason, I was 
creating a new cassandra cluster, but one of the nodes intended for new cluster 
had the same cassandra.yml file as the existing cluster. This resulted in the 
new node joining the existing cluster, making total no. of nodes as 6.

As of now in "nodetool status" command, I see that the state of the new node is 
JOINING, and also rebalancing data with other nodes.
What is the best way to decommission the node?

  1.  Can I execute "nodetool decommission" immediately for the new node?
  2.  Should I wait for the new node to finish sync, and decommission only 
after that?
  3.  Any other quick approach without data loss for existing cluster?

Thanks in advance!

Akshay Bhardwaj
+91-97111-33849

RE: SSTable Compression Ratio -1.0

[ZAIDI, ASAD A]

Compression ratio is ratio of compression to its original size - smaller is 
better; see it like compressed/uncompressed 
1 would mean no change in size after compression!



-Original Message-
From: Vitaliy Semochkin [mailto:vitaliy...@gmail.com] 
Sent: Tuesday, August 28, 2018 12:03 PM
To: user@cassandra.apache.org
Subject: SSTable Compression Ratio -1.0

Hello,

nodetool tablestats my_kespace
returns SSTable Compression Ratio -1.0

Can someone explain, what does -1.0 mean?

Regards,
Vitaliy

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

RE: Upgrade from 2.1 to 3.11

[ZAIDI, ASAD A]

You may want to check if coincidentally you’re having expired cells in heap. GC 
log should be able to tell you OR look for tombstones in system.log file. See 
your compactions are under control and normal.  This may not be related to 
upgrade at all!


From: Pradeep Chhetri [mailto:prad...@stashaway.com]
Sent: Tuesday, August 28, 2018 3:32 AM
To: user@cassandra.apache.org
Subject: Re: Upgrade from 2.1 to 3.11

You may want to try upgrading to 3.11.3 instead which has some memory leaks 
fixes.

On Tue, Aug 28, 2018 at 9:59 AM, Mun Dega 
mailto:mundeg...@gmail.com>> wrote:
I am surprised that no one else ran into any issues with this version.  GC 
can't catch up fast enough and there is constant Full GC taking place.

The result? unresponsive nodes makeing entire cluster unusable.

Any insight on this issue from anyone that is using this version would be 
appreciated.

Ma

On Fri, Aug 24, 2018, 04:30 Mohamadreza Rostami 
mailto:mohamadrezarosta...@gmail.com>> wrote:
You have very large heap,it’s take most of  cpu time in GC stage.you should in 
maximum set heap on 12GB and enable row cache to your cluster become faster.

On Friday, 24 August 2018, Mun Dega 
mailto:mundeg...@gmail.com>> wrote:
120G data
28G heap out of 48 on system
9 node cluster, RF3

On Thu, Aug 23, 2018, 17:19 Mohamadreza Rostami 
mailto:mohamadrezarosta...@gmail.com>> wrote:
Hi,
How much data do you have? How much RAM do your servers have? How much do you 
have a heep?
On Thu, Aug 23, 2018 at 10:14 PM Mun Dega 
mailto:mundeg...@gmail.com>> wrote:
Hello,

We recently upgraded from Cassandra 2.1 to 3.11.2 on one cluster.  The process 
went OK including upgradesstable but we started to experience high latency for 
r/w, occasional OOM and long GC pause after.

For the same cluster with 2.1, we didn't have any issues like this.  We also 
kept server specs, heap, all the same in post upgrade

Has anyone else had similar issues going to 3.11 and what are the major changes 
that could have such a major setback in the new version?

Ma Dega

Cassandra 3.11 with NFS mounted system

[ZAIDI, ASAD A]

Hello Folks,

I've an virtualized environment running with VMware  where Cassandra is humming 
on NFS mounted storage. As the application load increases ,they increase number 
of nodes in data center however writes are getting slower, nodes are flapping 
and application complains in write performance. I can see compaction is getting 
behind, flush writers are struggling - symptoms that points to storage system 
but system Admins says they have expensive, topnotch SSDs under data volume.

I know having nfs is not recommended though getting totally new h/w is not 
possible at this time. I'm wondering if  there is better way to utilize 
available SAN resources to run with Cassandra ? I'll much appreciate your 
insight here.

Thanks/Asad


Nfsiostat
===
$ nfsiostat /cassandra/data
172.XX.XX.16:/vol/cns_mount_0020_data mounted on /cassandra/data:

   op/s rpc bklog
839.330.00
read: ops/skB/s   kB/op retrans 
avg RTT (ms)avg exe (ms)
236.611 13471.72956.936   20 (0.0%) 
  6.125  13.236
write:ops/skB/s   kB/op retrans 
avg RTT (ms)avg exe (ms)
302.627 19048.74462.9453 (0.0%) 
 23.154 253.329

nodetool tpstats
==
$ nodetool tpstats
Pool Name Active   Pending  Completed   Blocked  
All time blocked
ReadStage  0 03474362 0 
0
MiscStage  0 0  0 0 
0
CompactionExecutor12   337  47130 0 
0
MutationStage256238996  313857751 0 
0
MemtableReclaimMemory  0 0  27081 0 
0
PendingRangeCalculator 0 0 37 0 
0
GossipStage0 0 173852 0 
0
SecondaryIndexManagement   0 0  0 0 
0
HintsDispatcher0 0  0 0 
0
RequestResponseStage   0 0  337132828 0 
0
Native-Transport-Requests  1 0  120206509 0 
  7073177
ReadRepairStage0 0   8934 0 
0
CounterMutationStage   0 0  0 0 
0
MigrationStage 1 2102 0 
0
MemtablePostFlush  1   223  27753 0 
0
PerDiskMemtableFlushWriter_0   2 2  27095 0 
0
ValidationExecutor 0 0  0 0 
0
Sampler0 0  0 0 
0
MemtableFlushWriter5   175  27093 0 
0
InternalResponseStage  0 0 46 0 
0
ViewMutationStage  0 0  0 0 
0
AntiEntropyStage   0 0  0 0 
0
CacheCleanupExecutor   0 0  0 0 
0

Message type   Dropped
READ 0
RANGE_SLICE  0
_TRACE   0
HINT944297
MUTATION  15707984
COUNTER_MUTATION 0
BATCH_STORE  0
BATCH_REMOVE 0
REQUEST_RESPONSE 0
PAGED_RANGE  0
READ_REPAIR   3195

RE: Maximum SSTable size

[ZAIDI, ASAD A]

With  Leveled compaction 
strategy  
, you can set target size with sstable_size_in_mb attribute however actual size 
can still be larger than target given large partition size.

Thanks/Asad

From: Lucas Benevides [mailto:lu...@maurobenevides.com.br]
Sent: Wednesday, June 27, 2018 7:02 AM
To: user@cassandra.apache.org
Subject: Maximum SSTable size

Hello Community,

Is there a maximum SSTable Size?
If there is not, does it go up to the maximum Operational System values?

Thanks in advance,
Lucas Benevides

RE: Problem with dropped mutations

[ZAIDI, ASAD A]

Are you also seeing time-outs on certain Cassandra operations?? If yes, you may 
have to tweak *request_timeout parameter in order to get rid of dropped 
mutation messages if application data model is not upto mark!

You can also check if network isn't dropping packets (ifconfig  -a tool) +  
storage (dstat tool) isn't reporting too slow disks.

Cheers/Asad


-Original Message-
From: Hannu Kröger [mailto:hkro...@gmail.com] 
Sent: Tuesday, June 26, 2018 9:49 AM
To: user 
Subject: Problem with dropped mutations

Hello,

We have a cluster with somewhat heavy load and we are seeing dropped mutations 
(variable amount and not all nodes have those).

Are there some clear trigger which cause those? What would be the best 
pragmatic approach to start debugging those? We have already added more memory 
which seemed to help somewhat but not completely.

Cheers,
Hannu



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Error creating pool to / Pool was closed during initialization

[ZAIDI, ASAD A]

Hello Folks,

I’m looking for possible reasons and solution for these frequently appearing 
warning messages I’m seeing in spark <>cassandra job’s log file. The message 
suggest Cassandra host server machine is acting up and throwing messages like:

18/06/25 14:07:44 WARN Session: Error creating pool to /XXX.XXX.30.54:9042
com.datastax.driver.core.exceptions.ConnectionException: [/XXX.XXX.30.54:9042] 
Pool was closed during initialization


Can you folks please help point me to cause/solution so I can get rid of these 
annoying warnings. I’m on apache-cassandra version 3.11.1 , Spark Version 1.6.3

Thank you,
Asad




Error Stack:
==
18/06/25 14:07:44 INFO CassandraConnector: Connected to Cassandra cluster: 
ProdSNA Cluster
18/06/25 14:07:44 WARN Session: Error creating pool to /XXX.XXX.30.54:9042
com.datastax.driver.core.exceptions.ConnectionException: [/XXX.XXX.30.54:9042] 
Pool was closed during initialization
at 
com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:148)
at 
com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:134)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.Futures$6.run(Futures.java:1319)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.Futures$CombinedFuture.setOneValue(Futures.java:1764)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.Futures$CombinedFuture.access$400(Futures.java:1608)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.Futures$CombinedFuture$2.run(Futures.java:1686)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.Futures$FallbackFuture$1$1.onSuccess(Futures.java:479)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.Futures$6.run(Futures.java:1319)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.Futures$ImmediateFuture.addListener(Futures.java:106)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.Futures.addCallback(Futures.java:1322)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.Futures$FallbackFuture$1.onFailure(Futures.java:476)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.Futures$FallbackFuture$1$1.onFailure(Futures.java:487)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:170)
at 
shade.com.datastax.spark.connector.google.common.util.concurrent.Futures.addCallback(Futures.java:1322)
at 

RE: Jon Haddad on Diagnosing Performance Problems in Production

[ZAIDI, ASAD A]

Perhaps Mr. Hadad himself  will share it again somewhere; he was kind enough to 
share it once at datastax!


From: Kenneth Brotman [mailto:kenbrot...@yahoo.com.INVALID]
Sent: Tuesday, February 27, 2018 10:39 AM
To: user@cassandra.apache.org
Subject: RE: Jon Haddad on Diagnosing Performance Problems in Production

Nicolas,

I think you had the link to the other version I was thinking of.  I couldn’t 
find it.  I think it might have gotten taken down; a lot of other stuff seems 
to be gone too.  Maybe it will be back.  Maybe they are just redoing stuff.  
Either way, it’s another sign of Mom and Dad drifting apart – I’m not sure 
who’s Mom and who’s Dad: DataStax or ASF.  Hopefully, for the sake of everyone 
in the family they will reconcile.

It’s gems like that presentation that will keep us vital.

Kenneth Brotman

From: Nicolas Guyomar [mailto:nicolas.guyo...@gmail.com]
Sent: Tuesday, February 27, 2018 8:21 AM
To: user@cassandra.apache.org
Subject: Re: Jon Haddad on Diagnosing Performance Problems in Production

Is Jon blog post 
https://academy.datastax.com/planet-cassandra/blog/cassandra-summit-recap-diagnosing-problems-in-production
 was relocated somewhere ?

On 27 February 2018 at 16:34, Kenneth Brotman 
> wrote:
One presentation that I hope can get updated is Jon Haddad’s very thorough 
presentation on Diagnosing Performance Problems in Production.  I’ve seen 
another version somewhere where I believe he says something like “This should 
help you fix 99% of the problems you see.”  Seems right.

I’m sure it will be well attended and well viewed for some time.  Here’s the 
version I found: 
https://www.youtube.com/watch?v=2JlUpgsEdN8

If Jon did a new version I’d probably stop and watch it three times right now.

If we started with that video inline on the Apache Cassandra web site in the 
troubleshooting section, that would help a lot of people because of the quality 
of the content and the density of the content.

Kenneth Brotman

RE: Cassandra Daemon not coming up

[ZAIDI, ASAD A]

Can you check if you’ve enough disk space available ?
~Asad

From: mahesh rajamani [mailto:rajamani.mah...@gmail.com]
Sent: Tuesday, February 27, 2018 10:11 AM
To: user@cassandra.apache.org
Subject: Cassandra Daemon not coming up

I am using Cassandra 3.0.9 version on a 12 node cluster. I have multiple node 
down after a restart. The cassandra VM is not coming up with an asset error as 
below. On running in debug mode it is failing while doing operation on " 
resource_role_permissons_index" in system_auth keyspace. Please let me know how 
to bring the cassandra running from this state.

Logs from system.log

INFO  [main] 2018-02-27 15:43:24,005 ColumnFamilyStore.java:389 - Initializing 
system_schema.columns
INFO  [main] 2018-02-27 15:43:24,012 ColumnFamilyStore.java:389 - Initializing 
system_schema.triggers
INFO  [main] 2018-02-27 15:43:24,019 ColumnFamilyStore.java:389 - Initializing 
system_schema.dropped_columns
INFO  [main] 2018-02-27 15:43:24,029 ColumnFamilyStore.java:389 - Initializing 
system_schema.views
INFO  [main] 2018-02-27 15:43:24,038 ColumnFamilyStore.java:389 - Initializing 
system_schema.types
INFO  [main] 2018-02-27 15:43:24,049 ColumnFamilyStore.java:389 - Initializing 
system_schema.functions
INFO  [main] 2018-02-27 15:43:24,061 ColumnFamilyStore.java:389 - Initializing 
system_schema.aggregates
INFO  [main] 2018-02-27 15:43:24,072 ColumnFamilyStore.java:389 - Initializing 
system_schema.indexes
ERROR [main] 2018-02-27 15:43:24,127 CassandraDaemon.java:709 - Exception 
encountered during startup
java.lang.AssertionError: null
at 
org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103)
 ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:311) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.config.CFMetaData.(CFMetaData.java:288) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:366) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:954) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891)
 ~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868)
 ~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856)
 ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:239) 
[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568) 
[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:696) 
[apache-cassandra-3.0.9.jar:3.0.9]

--
Regards,
Mahesh Rajamani

RE: Old tombstones not being cleaned up

[ZAIDI, ASAD A]

No it doesn’t. unchecked_tombstone_compaction sub property is common in all 
STCS, DTCS & LCS. Though you can also use jmxterm tool and invoke different 
compaction on single node if you desire.


From: Bo Finnerup Madsen [mailto:bo.gunder...@gmail.com]
Sent: Thursday, February 01, 2018 3:17 PM
To: user@cassandra.apache.org
Subject: Re: Old tombstones not being cleaned up

I, almost, tried that today :) I ran a repair, changed the compaction algorithm 
from leveled to sizetierd and back. This definitely forced a compaction, but 
the tombstones are still there.
Will setting the unchecked_tombstone_compaction force another type of 
compaction?
tor. 1. feb. 2018 kl. 19.37 skrev ZAIDI, ASAD A 
<az1...@att.com<mailto:az1...@att.com>>:
Make data consistent (run repair), reduce gc_grace_seconds (try set it to 0 
temporarily though careful as this can affect hinted handoff!)  and set table’s 
compaction sub-property i.e. unchecked_tombstone_compaction to true. Compaction 
 will  take care of tombstones!


From: Jonathan Haddad [mailto:j...@jonhaddad.com<mailto:j...@jonhaddad.com>]
Sent: Thursday, February 01, 2018 11:29 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Old tombstones not being cleaned up

Changing the defaul TTL doesn’t change the TTL on the existing data, only new 
data. It’s only set if you don’t supply one yourself.

On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen 
<bo.gunder...@gmail.com<mailto:bo.gunder...@gmail.com>> wrote:
Hi,

We are running a small 9 node Cassandra v2.1.17 cluster. The cluster generally 
runs fine, but we have one table that are causing OOMs because an enormous 
amount of tombstones.
Looking at the data in the table (sstable2json), the first of the tombstones 
are almost a year old. The table was initially created with a gc_grace_period 
of 10 days, but I have now lowered it to 1 hour.
I have run a full repair of the table across all nodes. I have forced several 
major compactions of the table by using "nodetool compact", and also tried to 
switch from LeveledCompaction to SizeTierCompaction and back.

What could cause cassandra to keep these tombstones?

sstable2json:
{"key": "foo",
 "cells": 
[["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
   
["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
   
["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
   
["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
   
["22cc-d69c-4596-89e5-3e976c0cb9a8:_","22cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],
   
["2777-4b1a-4267-8efc-c43054e63170:_","2777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],
   
["61e8-f48b-4484-96f1-f8b6a3ed8f9f:_","61e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],
   
["63da-f165-449b-b65d-2b7869368734:_","63da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],
   
["656f-f8b5-472b-93ed-1a893002f027:_","656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
...
{"key": "bar",
 "metadata": {"deletionInfo": 
{"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}},
 "cells": 
[["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],
   
["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],
   
["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],


sstablemetadata:
stablemetadata 
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db
SSTable: 
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.10
Minimum timestamp: 1488976211688000
Maximum timestamp: 1517468644066000
SSTable max local deletion time: 2147483647
Compression ratio: 0.5121956624389545
Estimated droppable tombstones: 18.00161766553587
SSTable Level: 0
Repaired at: 0
ReplayPosition(segmentId=1517168739626, 
position=22690189<tel:22%2069%2001%2

RE: Converting from Apache Cassandra 2.2.6 to 3.x

[ZAIDI, ASAD A]

You may want to upgrade python and java/JDK version with Cassandra upgrade. 
please refer to CHANGES.txt for all updates & improvement made in your selected 
3.x version.



From: William Boutin [mailto:william.bou...@ericsson.com]
Sent: Thursday, February 01, 2018 4:49 PM
To: user@cassandra.apache.org
Cc: William Boutin 
Subject: Converting from Apache Cassandra 2.2.6 to 3.x

We are converting our product from Apache Cassandra 2.2.6 to 3.x. What issues 
may we run into when we convert?


[Image removed by sender. 
Ericsson]
WILLIAM L. BOUTIN
Engineer IV - Sftwr
BMDA PADB DSE DU CC NGEE

Ericsson
1 Ericsson Drive, US PI06 1.S747
Piscataway, NJ, 08854, USA
Phone (913) 241-5574
Mobile (732) 213-1368
Emergency (732) 354-1263
william.bou...@ericsson.com
www.ericsson.com
[Image removed by sender. 
http://www.ericsson.com/current_campaign]
Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242. This 
Communication is Confidential. We only send and receive email on the basis of 
the terms set out at 
www.ericsson.com/email_disclaimer

RE: Old tombstones not being cleaned up

[ZAIDI, ASAD A]

Make data consistent (run repair), reduce gc_grace_seconds (try set it to 0 
temporarily though careful as this can affect hinted handoff!)  and set table’s 
compaction sub-property i.e. unchecked_tombstone_compaction to true. Compaction 
 will  take care of tombstones!


From: Jonathan Haddad [mailto:j...@jonhaddad.com]
Sent: Thursday, February 01, 2018 11:29 AM
To: user@cassandra.apache.org
Subject: Re: Old tombstones not being cleaned up

Changing the defaul TTL doesn’t change the TTL on the existing data, only new 
data. It’s only set if you don’t supply one yourself.

On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen 
> wrote:
Hi,

We are running a small 9 node Cassandra v2.1.17 cluster. The cluster generally 
runs fine, but we have one table that are causing OOMs because an enormous 
amount of tombstones.
Looking at the data in the table (sstable2json), the first of the tombstones 
are almost a year old. The table was initially created with a gc_grace_period 
of 10 days, but I have now lowered it to 1 hour.
I have run a full repair of the table across all nodes. I have forced several 
major compactions of the table by using "nodetool compact", and also tried to 
switch from LeveledCompaction to SizeTierCompaction and back.

What could cause cassandra to keep these tombstones?

sstable2json:
{"key": "foo",
 "cells": 
[["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
   
["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
   
["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
   
["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
   
["22cc-d69c-4596-89e5-3e976c0cb9a8:_","22cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],
   
["2777-4b1a-4267-8efc-c43054e63170:_","2777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],
   
["61e8-f48b-4484-96f1-f8b6a3ed8f9f:_","61e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],
   
["63da-f165-449b-b65d-2b7869368734:_","63da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],
   
["656f-f8b5-472b-93ed-1a893002f027:_","656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
...
{"key": "bar",
 "metadata": {"deletionInfo": 
{"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}},
 "cells": 
[["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],
   
["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],
   
["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],


sstablemetadata:
stablemetadata 
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db
SSTable: 
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.10
Minimum timestamp: 1488976211688000
Maximum timestamp: 1517468644066000
SSTable max local deletion time: 2147483647
Compression ratio: 0.5121956624389545
Estimated droppable tombstones: 18.00161766553587
SSTable Level: 0
Repaired at: 0
ReplayPosition(segmentId=1517168739626, position=22690189)
Estimated tombstone drop times:%n
1488976211: 1
1489906506:  4706
1490174752:  6111
1490449759:  6554
1490735410:  6559
1491016789:  6369
1491347982: 10216
1491680214: 13502
...

desc:
CREATE TABLE xxx.yyy (
ti text,
uuid text,
json_data text,
PRIMARY KEY (ti, uuid)
) WITH CLUSTERING ORDER BY (uuid ASC)
AND bloom_filter_fp_chance = 0.1
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'class': 
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'sstable_compression': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 3600
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';

jmx props(picture):
[image.png]

RE: [announce] Release of Cassandra Prometheus metrics exporter

[ZAIDI, ASAD A]

Hi Romain Gérard - Thanks for sharing Cassandra-exporter.

Since you're   already monitoring tons of Cassandra instances nodes - can you 
please let us know if you can  share Prometheus dashboards/json code along with 
Cassandra-exporter.

Thanks/Asad



-Original Message-
From: Romain Gerard [mailto:romain.ger...@erebe.eu] 
Sent: Wednesday, January 10, 2018 9:06 AM
To: user@cassandra.apache.org
Subject: [announce] Release of Cassandra Prometheus metrics exporter

Hello C*,

A little mail to announce that we released today our internal tool at Criteo to 
monitor Cassandra nodes with Prometheus[1].
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_criteo_cassandra-5Fexporter=DwIDaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=jf3zVfMnJpPJbsjnF3FyFzYuEN0ayq393EwVCYgeEto=OM9PZ3fFgHrar-8ASL4x69WKoY1dSULUUyNu9foTe10=
 

The application is production ready as we use it internally to monitor our > 
100 Cassandra nodes.

I hope it can be useful to you too !
Feel free to send feedbacks/contributions/questions.

[1] 
https://urldefense.proofpoint.com/v2/url?u=https-3A__prometheus.io_=DwIDaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=jf3zVfMnJpPJbsjnF3FyFzYuEN0ayq393EwVCYgeEto=LRtrxtND7d8TLlmTdy-eYyQTOUEHZvnpHguyQ_SRJO4=
 

Regards,
Romain Gérard

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

RE: Connection refused - 127.0.0.1-Gossip

[ZAIDI, ASAD A]

Throwing my $ .002

Rpc_address  defaults to localhosts that if not set, picks value from 
dns/hostname file. May be you can try setting rpc_address:  
- see if this helps!


From: Marek Kadek -T (mkadek - CONSOL PARTNERS LTD at Cisco) 
[mailto:mka...@cisco.com]
Sent: Wednesday, December 06, 2017 2:19 AM
To: user@cassandra.apache.org
Subject: Re: Connection refused - 127.0.0.1-Gossip

Thanks for any ideas/hints, any straw is worth checking at this point ☺

Well, the clusters “work”, data is correctly stored and queries. I’m interested 
in why it tries to open a gossip to localhost, and what kind of (performance) 
impact could this have on clusters.
The env vars are correctly passed, and cassandra yaml seems to be correctly 
set. We are using Cassandra docker image.

listen_address: 100.110.253.6 (correct pod ip)
# listen_interface: eth0
# listen_interface_prefer_ipv6: false
broadcast_rpc_address: 100.110.253.6

It’s also observable with minikube and single C* node on local machine.




From: Lerh Chuan Low >
Reply-To: "user@cassandra.apache.org" 
>
Date: Tuesday, December 5, 2017 at 11:14 PM
To: "user@cassandra.apache.org" 
>
Subject: Re: Connection refused - 127.0.0.1-Gossip

I think as Jeff mentioned it sounds like a configuration issue, are you sure 
you are using the same configmap/however it's being passed in and just throwing 
out ideas, maybe the pods are behind a http proxy and you may have forgotten to 
pass in the env vars?

On 6 December 2017 at 08:45, Jeff Jirsa 
> wrote:
I don't have any k8 clusters to test with, but do you know how your yaml 
translates to cassandra.yaml ? What are the listen/broadcast addresses being 
set?


On Tue, Dec 5, 2017 at 6:09 AM, Marek Kadek -T (mkadek - CONSOL PARTNERS LTD at 
Cisco) > wrote:

We are experiencing following issues with Cassandra on our kubernetes clusters:

```

@ kubectl exec -it cassandra-cassandra-0 -- tail /var/log/cassandra/debug.log

DEBUG [MessagingService-Outgoing-localhost/127.0.0.1-Gossip] 2017-12-05 
09:02:06,560 OutboundTcpConnection.java:545 - Unable to connect to 
localhost/127.0.0.1

java.net.ConnectException: Connection refused

at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_131]

at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_131]

at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_131]

at 
sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_131]

at 
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:132)
 ~[apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:433)
 [apache-cassandra-3.11.0.jar:3.11.0]

at 
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:262)
 [apache-cassandra-3.11.0.jar:3.11.0]

```



Basically, it’s tons and tons of the same message over and over (on all 
clusters, all C* nodes). It tries roughly 4-5 times a second to open a tcp 
connection to localhost (?) for gossiping.



What we know:

- does not happen on Cassandra 3.0.15, but happen on 3.11.1 (same 
configuration).

- does happen even on minikube-single-Cassandra “cluster”.

- does not happen on docker-compose Cassandra cluster, only on kubernetes one.



Our configuration is pretty much this helm chart: 

table repair question

[ZAIDI, ASAD A]

Hello folk,

I'm wondering if there is way to find out list of table(s) which need repair OR 
if there Is way to find out what %age of data would need to be repaired on a 
table?  Is such information available from Cassandra  db engine through some 
other means?

TIA~ Asad

detail of compactionstats, pending tasks

[ZAIDI, ASAD A]

Hello Folks,

Is it possible to findout detail of those 275 pending tasks compactionstats 
command output is showing?
I’ve bumpedup concurrent_compactors to 25 though not all threads are 
compacting, only 8 threads are being used so i’m wondering how can I utilize 
all configured concurrent compactors when there are still pending tasks?



[cassandra@server]$ nodetool compactionstats -H
pending tasks: 275
 id   compaction type keyspace  
  table   completed   totalunit   progress
   64c55e50-9f17-11e7-94ce-0fe8b18d4e6cCompaction   corporate  
usrcnt_location_by_hour70.47 MB   178.29 MB   bytes 39.52%
   65930e90-9f17-11e7-94ce-0fe8b18d4e6cCompaction   corporate   
   visi_det_rollup87.46 MB   208.54 MB   bytes 41.94%
   73c34250-9f17-11e7-94ce-0fe8b18d4e6cCompaction   corporate   
   visi_det_rollup  3.1 MB   122.41 MB   bytes  2.53%
   d4e2cde0-9f16-11e7-94ce-0fe8b18d4e6cCompaction   corporate   
 timing_by_account 1.64 GB 5.96 GB   bytes 27.59%
   6a430210-9f17-11e7-94ce-0fe8b18d4e6cCompaction   corporate   
   stream_host_etl   127.05 MB   162.78 MB   bytes 78.05%
   6c997530-9f17-11e7-94ce-0fe8b18d4e6cCompaction   corporate   
det_rollup40.44 MB   131.33 MB   bytes 30.79%
   59fc4a60-9f17-11e7-94ce-0fe8b18d4e6cCompaction   corporate   
det_rollup   180.48 MB   250.69 MB   bytes 72.00%
   607739e0-9f17-11e7-94ce-0fe8b18d4e6cCompaction   corporate   
det_rollup   131.36 MB   256.18 MB   bytes 51.28%
   659817a0-9f17-11e7-94ce-0fe8b18d4e6cCompaction   corporate   
  station_data  108 MB   267.68 MB   bytes 40.35%
Active compaction remaining time :n/a

==

[cassandra@server]$ nodetool tpstats

Pool NameActive   Pending  Completed   Blocked  All 
time blocked
MutationStage 1 0  200764219 0  
   0
ReadStage 0 0  0 0  
   0
RequestResponseStage  0 0 37 0  
   0
ReadRepairStage   0 0  0 0  
   0
CounterMutationStage  0 0  0 0  
   0
HintedHandoff 0 0 16 0  
   0
MiscStage 0 0  0 0  
   0
CompactionExecutor8 8  19129 0  
   0
MemtableReclaimMemory 0 0  14586 0  
   0
PendingRangeCalculator0 0  6 0  
   0
GossipStage   0 0  75457 0  
   0
MigrationStage0 0   2558 0  
   0
MemtablePostFlush 1 1  15908 0  
   0
ValidationExecutor0 0  0 0  
   0
Sampler   0 0  0 0  
   0
MemtableFlushWriter   1 1  14573 0  
   0
InternalResponseStage 0 0  0 0  
   0
AntiEntropyStage  0 0  0 0  
   0
CacheCleanupExecutor  0 0  0 0  
   0

Message type   Dropped
READ 0
RANGE_SLICE  0
_TRACE   0
MUTATION 0
COUNTER_MUTATION 0
REQUEST_RESPONSE 0
PAGED_RANGE  0
READ_REPAIR  0



Thanks/Asad

MemtablePostFlush pending

[ZAIDI, ASAD A]

Hello Folks,

I’m using Cassandra 2.2 on 14 node cluster.

Now a days, I’m observing memtablepostflush pending number going high , this 
happens intermittently. I’m looking if  Is there way to ‘tune’ 
memtablepostflush stage?

Thanks/ASad

RE: rebuild constantly fails, 3.11

[ZAIDI, ASAD A]

Without exact failure text, it is really hard to guess what may be going-on -  
can you please share logfile excerpt detailing the failure error so we can have 
better idea of the nature failure.
Adjusting phi_convict_threshold may yet be another shot in the dark when we 
don’t know what is causing the failure and network is supposedly stable.

~Asad



-Original Message-
From: Micha [mailto:mich...@fantasymail.de] 
Sent: Tuesday, August 08, 2017 8:35 AM
To: user@cassandra.apache.org; ZAIDI, ASAD A <az1...@att.com>; 
user@cassandra.apache.org
Subject: Re: rebuild constantly fails, 3.11

no, I have left it at the default value of 24hours.

I've read about adjusting phi_convict_threshold, but I haven't done this yet as 
the network is stable. maybe I set this to 10.


On 08.08.2017 15:24, ZAIDI, ASAD A wrote:
> Is there any chance you've set streaming_socket_timeout_in_ms parameter set 
> too low on failing node?
> 
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

RE: rebuild constantly fails, 3.11

[ZAIDI, ASAD A]

Is there any chance you've set streaming_socket_timeout_in_ms parameter set too 
low on failing node?


-Original Message-
From: Micha [mailto:mich...@fantasymail.de] 
Sent: Tuesday, August 08, 2017 3:01 AM
To: user@cassandra.apache.org; d...@cassandra.apache.org
Subject: rebuild constantly fails, 3.11

Hi,

it seems I'm not able to add add 3 node dc to a 3 node dc. After starting the 
rebuild on a new node, nodetool netstats show it will receive 1200 files from 
node-1 and 5000 from node-2. The stream from
node-1 completes but the stream from node-2 allways fails, after sending ca 
4000 files.

After restarting the rebuild it again starts to send the 5000 files.
The whole cluster is connected via one switch only , no firewall between, the 
networks shows no errors.
The machines have 8 cores, 32GB RAM and two 1TB discs as raid0.
the logs show no errors. The size of the data is ca 1TB.


Any help is really welcome,

cheers
 Michael






The error is:

Cassandra has shutdown.
error: null
-- StackTrace --
java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267)
at
sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at
javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
at
javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020)
at
javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298)
at com.sun.proxy.$Proxy7.rebuild(Unknown Source)
at org.apache.cassandra.tools.NodeProbe.rebuild(NodeProbe.java:1190)
at
org.apache.cassandra.tools.nodetool.Rebuild.execute(Rebuild.java:58)
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:254)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:168)

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Cassandra upgrade from 2.2.8 to 3.10

[ZAIDI, ASAD A]

Hi folks, I’ve question on upgrade method I’m thinking to execute.

I’m  planning from apache-Cassandra 2.2.8 to release 3.10.

My Cassandra cluster is configured like one rack with two Datacenters like:


1.   DC1 has 4 nodes

2.   DC2 has 16 nodes

We’re adding another 12 nodes and would eventually need to remove those 4 nodes 
in DC1.

I’m thinking to add another third data center with like DC3 with 12 nodes 
having apache Cassandra 3.10 installed. Then, I start upgrading seed nodes 
first in DC1 & DC2 – once all 20nodes in ( DC1 plus DC2) upgraded – I can 
safely remove 4 DC1 nodes,
can you guys please let me know if this approach would work? I’m concerned if 
having mixed version on Cassandra nodes may  cause any issues like in streaming 
 data/sstables from existing DC to newly created third DC with version 3.10 
installed, will nodes in DC3 join the cluster with data without issues?

Thanks/Asad

RE: Different data size between datacenters

[ZAIDI, ASAD A]

Are you using same number of token/vnodes in both data centers?

From: Chuck Reynolds [mailto:creyno...@ancestry.com]
Sent: Monday, August 07, 2017 1:51 PM
To: user@cassandra.apache.org
Subject: Different data size between datacenters

I have a cluster that spans two datacenters running Cassandra 2.1.12.  135 
nodes in my data center and about 185 in AWS.

The size of the second data center (AWS) is quite a bit smaller.  Replication 
is the same in both datacenters.  Is there a logical explanation for this?

thanks

RE: MUTATION messages were dropped in last 5000 ms for cross node timeout

[ZAIDI, ASAD A]

Hi Akhil,

Thank you for your reply.

I kept testing different timeout numbers over last week and eventually settled 
at setting *_request_timeout_in_ms parameters at 1.5minutes for coordinator 
wait time. That is the number where I donot see any dropped mutations.

Also asked developers to tweak data model where we saw bunch of tables with 
really large partition size , some are ranging  Partition-key size around 
~6.6GB.. we’re now working to reduce the partition size of the tables. I am 
hoping corrected data model will help reduce coordinator wait time (get back to 
default number!)  again.

Thank again/Asad

From: Akhil Mehra [mailto:akhilme...@gmail.com]
Sent: Friday, July 21, 2017 4:24 PM
To: user@cassandra.apache.org
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
timeout

Hi Asad,

The 5000 ms is not configurable 
(https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/net/MessagingService.java#L423<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_cassandra_blob_8b3a60b9a7dbefeecc06bace617279612ec7092d_src_java_org_apache_cassandra_net_MessagingService.java-23L423=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=dp_TvjXTbUtu3Iu43aZ83eHl1fgW6l4P4PSQglF855g=USbrEM6jaGFIRKSUhJBx3VAkSSrXzid0db6TDV1vrDs=>).
 This just the time after which the number of dropped messages are reported. 
Thus dropped messages are reported every 5000ms.

If you are looking to tweak the number of ms after which a message is 
considered dropped then you need to use the write_request_timeout_in_ms.  The 
write_request_timeout_in_ms 
(http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.datastax.com_en_cassandra_2.1_cassandra_configuration_configCassandra-5Fyaml-5Fr.html=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=dp_TvjXTbUtu3Iu43aZ83eHl1fgW6l4P4PSQglF855g=ab1NW9WoAXIlxT2kWjsiYFVaVidEnC_MB770pwTtqLs=>)
 can be used to increase the mutation timeout. By default it is set to 2000ms.

I hope that helps.

Regards,
Akhil


On 22/07/2017, at 2:46 AM, ZAIDI, ASAD A 
<az1...@att.com<mailto:az1...@att.com>> wrote:

Hi Akhil,

Thank you for your reply. Previously, I did ‘tune’ various timeouts – basically 
increased them a bit but none of those parameter listed in the link matches 
with that “were dropped in last 5000 ms”.
I was wondering from where that [5000ms] number is coming from when,  like I 
mentioned before, none of any timeout parameter settings matches that #!

Load is intermittently high but again cpu queue length never goes beyond medium 
depth. I wonder if there is some internal limit that I’m still not aware of.

Thanks/Asad


From: Akhil Mehra [mailto:akhilme...@gmail.com]
Sent: Thursday, July 20, 2017 3:47 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
timeout

Hi Asad,

http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped<https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra.apache.org_doc_latest_faq_index.html-23why-2Dmessage-2Ddropped=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=WcHuHKcjg2YCsAbw2NR_0-CiHr9JNxtCzYikia16mpo=0_0pQfoOZLuswpQ_lE-AU2bTMFLgRbR4k4Kh8vEOZSk=>

As mentioned in the link above this is a load shedding mechanism used by 
Cassandra.

Is you cluster under heavy load?

Regards,
Akhil


On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A 
<az1...@att.com<mailto:az1...@att.com>> wrote:

Hello Folks –

I’m using apache-cassandra 2.2.8.

I see many messages like below in my system.log file. In Cassandra.yaml file [ 
cross_node_timeout: true] is set and NTP server is also running correcting 
clock drift on 16node cluster. I do not see pending or blocked HintedHandoff  
in tpstats output though there are bunch of MUTATIONS dropped observed.


INFO  [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - 
MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 
2152 for cross node timeout


I’m seeking help here if you please let me know what I need to check in order 
to address these cross node timeouts.

Thank you,
Asad

RE: MUTATION messages were dropped in last 5000 ms for cross node timeout

[ZAIDI, ASAD A]

Hi Akhil,

Thank you for your reply. Previously, I did ‘tune’ various timeouts – basically 
increased them a bit but none of those parameter listed in the link matches 
with that “were dropped in last 5000 ms”.
I was wondering from where that [5000ms] number is coming from when,  like I 
mentioned before, none of any timeout parameter settings matches that #!

Load is intermittently high but again cpu queue length never goes beyond medium 
depth. I wonder if there is some internal limit that I’m still not aware of.

Thanks/Asad


From: Akhil Mehra [mailto:akhilme...@gmail.com]
Sent: Thursday, July 20, 2017 3:47 PM
To: user@cassandra.apache.org
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
timeout

Hi Asad,

http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped<https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra.apache.org_doc_latest_faq_index.html-23why-2Dmessage-2Ddropped=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=WcHuHKcjg2YCsAbw2NR_0-CiHr9JNxtCzYikia16mpo=0_0pQfoOZLuswpQ_lE-AU2bTMFLgRbR4k4Kh8vEOZSk=>

As mentioned in the link above this is a load shedding mechanism used by 
Cassandra.

Is you cluster under heavy load?

Regards,
Akhil


On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A 
<az1...@att.com<mailto:az1...@att.com>> wrote:

Hello Folks –

I’m using apache-cassandra 2.2.8.

I see many messages like below in my system.log file. In Cassandra.yaml file [ 
cross_node_timeout: true] is set and NTP server is also running correcting 
clock drift on 16node cluster. I do not see pending or blocked HintedHandoff  
in tpstats output though there are bunch of MUTATIONS dropped observed.


INFO  [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - 
MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 
2152 for cross node timeout


I’m seeking help here if you please let me know what I need to check in order 
to address these cross node timeouts.

Thank you,
Asad

RE: MUTATION messages were dropped in last 5000 ms for cross node timeout

[ZAIDI, ASAD A]

Thanks for your reply Subroto – I’ll try you suggestions to see if this help. 
I’ll revert with results.


From: Subroto Barua [mailto:sbarua...@yahoo.com.INVALID]
Sent: Thursday, July 20, 2017 12:22 PM
To: user@cassandra.apache.org
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
timeout

In a cloud environment, cross_node_timeout = true can cause issues; we had this 
issue in our environment and it is set to false now.
Dropped messages is an another issue

Subroto

On Jul 20, 2017, at 8:27 AM, ZAIDI, ASAD A 
<az1...@att.com<mailto:az1...@att.com>> wrote:
Hello Folks –

I’m using apache-cassandra 2.2.8.

I see many messages like below in my system.log file. In Cassandra.yaml file [ 
cross_node_timeout: true] is set and NTP server is also running correcting 
clock drift on 16node cluster. I do not see pending or blocked HintedHandoff  
in tpstats output though there are bunch of MUTATIONS dropped observed.


INFO  [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - 
MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 
2152 for cross node timeout


I’m seeking help here if you please let me know what I need to check in order 
to address these cross node timeouts.

Thank you,
Asad

RE: MUTATION messages were dropped in last 5000 ms for cross node timeout

[ZAIDI, ASAD A]

Thank you for your reply. I’ll increase memTable_flush_writes and report back 
if it helps.

Is there any formula that we can use to arrive at correct number of 
memTable_flush_writers ? or the exercise would windup be like “try and error” 
taking much time to arrive at some number that may not be optimal.

Thank you again.



From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in]
Sent: Thursday, July 20, 2017 12:17 PM
To: ZAIDI, ASAD A <az1...@att.com>; user@cassandra.apache.org
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
timeout

Hi Asad,

You can do following things:

1.Increase memtable_flush_writers especially if you have a write heavy load.

2.Make sure there are no big gc pauses on your nodes. If yes,  go for heap 
tuning.


Please let us know whether above measures fixed your problem or not.


Thanks
Anuj

Sent from Yahoo Mail on 
Android<https://urldefense.proofpoint.com/v2/url?u=https-3A__overview.mail.yahoo.com_mobile_-3F.src-3DAndroid=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=wu-cVspapPKFg21Ul4vRDSw9hpc-s7H1OxPMTPTEApE=YCtcGLTbnUV3ZWX_ZFNGe-DCoXPuokN0NCCwS_ahD2I=>

On Thu, 20 Jul 2017 at 20:57, ZAIDI, ASAD A
<az1...@att.com<mailto:az1...@att.com>> wrote:

Hello Folks –



I’m using apache-cassandra 2.2.8.



I see many messages like below in my system.log file. In Cassandra.yaml file [ 
cross_node_timeout: true] is set and NTP server is also running correcting 
clock drift on 16node cluster. I do not see pending or blocked HintedHandoff  
in tpstats output though there are bunch of MUTATIONS dropped observed.





INFO  [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - 
MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 
2152 for cross node timeout





I’m seeking help here if you please let me know what I need to check in order 
to address these cross node timeouts.



Thank you,

Asad

MUTATION messages were dropped in last 5000 ms for cross node timeout

[ZAIDI, ASAD A]

Hello Folks -

I'm using apache-cassandra 2.2.8.

I see many messages like below in my system.log file. In Cassandra.yaml file [ 
cross_node_timeout: true] is set and NTP server is also running correcting 
clock drift on 16node cluster. I do not see pending or blocked HintedHandoff  
in tpstats output though there are bunch of MUTATIONS dropped observed.


INFO  [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - 
MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 
2152 for cross node timeout


I'm seeking help here if you please let me know what I need to check in order 
to address these cross node timeouts.

Thank you,
Asad

reduced num_token = improved performance ??

[ZAIDI, ASAD A]

Hi Folks,

Pardon me if I’m missing  something obvious.  I’m still using apache-cassandra 
2.2 and planning for upgrade to  3.x.
I came across this jira [https://issues.apache.org/jira/browse/CASSANDRA-7032] 
that suggests reducing num_token may improve general performance of Cassandra 
like having  num_token=16 instead of 256   may help!

Can you please suggests if having less num_token would provide real performance 
benefits or if  it comes with any downsides that we should also consider? I’ll 
much appreciate your insights.

Thank you
Asad

RE: READ Queries timing out.

[ZAIDI, ASAD A]

>> I analysed the GC logs not having any issues with major GC's
If you don’t have issues on GC , than why do you want to [tune] GC 
parameters ?
Instead focus on why select queries are taking time.. may be take a look on 
their trace?


From: Pranay akula [mailto:pranay.akula2...@gmail.com]
Sent: Friday, July 07, 2017 9:27 AM
To: user@cassandra.apache.org
Subject: READ Queries timing out.

Lately i am seeing some select queries timing out, data modelling to blame for 
but not in a situation to redo it.

Does increasing heap will help ??

currently using 1GB new_heap, I analysed the GC logs not having any issues with 
major GC's .

Using G1GC , does increasing new_heap will help ??

currently using JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500", even if i 
increase heap to lets say 2GB is that effective b/c young GC's will kick in 
more frequently  to complete in 500ms right ??


Thanks
Pranay.

RE: Understanding of cassandra metrics

[ZAIDI, ASAD A]

What exactly does mean CoordinatorScanLatency for example
CoordinatorScanLatency  is a timer metric that present coordinator range scan 
latency for  table.
Is it latency on full table scan or maybe range scan by clustering key?
It is range scan.. clustering key is used to only store data in 
sorted fashion – partition key along with chosen partitioner helps in range 
scan of data.
Can anybody write into partition while locked?
Writes are atomic – it depends on your chosen consistency level 
to determine if writes will fail or succeed.

From: Павел Сапежко [mailto:amelius0...@gmail.com]
Sent: Friday, July 07, 2017 8:23 AM
To: user@cassandra.apache.org
Subject: Re: Understanding of cassandra metrics

Are you really think that I don't read docs? Do you have enough information in 
the documentation? I think no. What exactly does mean CoordinatorScanLatency 
for example? Is it latency on full table scan or maybe range scan by clustering 
key? What exactly mean ViewLockAcquireTime? What is "partition lock"? Can 
anybody write into partition while locked? Etc.
пт, 7 июл. 2017 г. в 13:01, Ivan Iliev 
>:
1st result on google returns:

http://cassandra.apache.org/doc/latest/operating/metrics.html

On Fri, Jul 7, 2017 at 12:16 PM, Павел Сапежко 
> wrote:
Hello, I have several question about cassandra metrics. What does exactly mean 
the next metrics:

  *   CoordinatorReadLatency
  *   CoordinatorScanLatency
  *   ReadLatency
  *   RangeLatency
  *   ViewLockAcquireTime
  *   ViewReadTime
--

С уважением,

Павел Сапежко

skype: p.sapezhko

--

С уважением,

Павел Сапежко

skype: p.sapezhko

RE: Node failure Due To Very high GC pause time

[ZAIDI, ASAD A]

>>   here my is doubt is that does all the deleted 3.3Million row will be 
>> loaded in my on-heap memory? if not what will be object that occupying those 
>> memory ?

  It depends on your queries what data they’re fetching from your 
database.   Assuming you’re using CMS garbage collector and you’ve enabled GC 
logs with PrintGCDetails, PrintClassHistogramBeforeFullGC, 
PrintClassHistogramAfterFullGC – your logs should tell you what java classes 
occupies most of your  heap memory.

System.log file can also give you some clue  like if you see references to your 
tables with [tombstones],  A quick [grep –i tombstone /path/to/system.log] 
command would tell you what objects are suffering with tombstones!


From: Karthick V [mailto:karthick...@zohocorp.com]
Sent: Monday, July 03, 2017 11:47 AM
To: user 
Subject: Re: Node failure Due To Very high GC pause time

Hi Bryan,

Thanks for your quick response.  We have already tuned our memory 
and GC based on our hardware specification and it was working fine until 
yesterday, i.e before facing the below specified delete request. As you 
specified we will once again look into our GC & memory configuration.

FYKI :  We are using memtable_allocation_typ as offheap_objects.

Consider the following table

CREATE TABLE  EmployeeDetails (
branch_id text,
department_id  text,
emp_id bigint,
emp_details text,
PRIMARY KEY (branch, department, emp_id)
) WITH CLUSTERING ORDER BY (department ASC, emp_id ASC)


In this table I have 10 million records for the a particular branch_id and 
department_id . And following are the list of operation which I perform in C* 
in chronological order

  1.  Deleting 5 million records, from the start, in batches of 500 records per 
request for the particular branch_id (say 'xxx' ) and department_id (say 'yyy')
  2.  Read the next 500 records as soon the above delete operation is being 
completed ( Select * from EmployeeDetails where branch_id='xxx' and 
department_id = 'yyy' and emp_id >5000 limit 500 )

It's only after executing the above read request there was a spike in memory 
and within few minutes the node has been marked down.

So my question here is , will the above read request will load all the deleted 
5 million records in my memory before it starts fetching or will it jump 
directly to the offset of 5001 record (since we have specified the greater 
than condition) ? If its going to the former case then for sure the read 
request will keep the data in main memory and performs merge operation before 
it delivers the data as per this wiki( 
https://wiki.apache.org/cassandra/ReadPathForUsers
 ). If not let me know how the above specified read request will provide me the 
data .


Note : And also while analyzing my heap dump its clear that majority of the 
memory is being held my Tombstone threads.


Thanks in advance
-- karthick



 On Mon, 03 Jul 2017 20:40:10 +0530 Bryan Cheng 
> wrote 

This is a very antagonistic use case for Cassandra :P I assume you're familiar 
with Cassandra and deletes? (eg. 
http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html,
 
http://docs.datastax.com/en/cassandra/2.1/cassandra/dml/dml_about_deletes_c.html)

That being said, are you giving enough time for your tables to flush to disk? 
Deletes generate markers which can and will consume memory until they have a 
chance to be flushed, after which they will impact query time and performance 
(but should relieve memory pressure). If you're saturating the capability of 
your nodes your tables will have difficulty flushing. See 
http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_memtable_thruput_c.html.

This could also be a heap/memory configuration issue as well or a GC tuning 
issue (although unlikely if you've left those at 

Cassandra Cluster Expansion Criteria

[ZAIDI, ASAD A]

Hello Folks,

I’m on Cassandra 2.2.8 cluster with 14 nodes , each with around 2TB of data 
volume. I’m looking for a criteria /or data points that can help me decide when 
or  if I should add more nodes to the cluster and by how many nodes.

I’ll really appreciate if you guys can share your insights.

Thanks/Asad

RE: COUNT

[ZAIDI, ASAD A]

Is it possible for you to share tracing info for the query? You can enable 
tracing at cqlsh prompt with command

Cqlsh > TRACING ON
Cqlsh> run your query
Tracing session info should be printed on screen

Tracing will enable us to know where most of the time is spent!

From: web master [mailto:socketman2...@gmail.com]
Sent: Wednesday, June 21, 2017 1:44 AM
To: user@cassandra.apache.org
Subject: COUNT

I have this schema

CREATE TABLE IF NOT EXISTS "inbox" (
  "groupId"   BIGINT,
  "createTime"   TIMEUUID,
  "mailId"   BIGINT,
  "body" TEXT,
  PRIMARY KEY ("groupId","createTime","mailId")
)WITH CLUSTERING ORDER BY ("createTime" DESC);

This table is frequency updated (250K per second) and each between 10-1000 new 
record is inserted in each "groupId" per day

The problem is I want to count `Unread mails` that based on a TIMEUUID compare, 
that means I want to count

SELECT count(1) FROM inbox WHERE "groupId"=123456 AND "createTime"> 
specificTimeUUID

But this query is inefficiend and slow sometimes

If we have <1000 unread message there is no problem but when we have 50K+ 
unread message we have huge issue

What is the best solution for the problem?

RE: Best practice to add(bootstrap) multiple nodes to cluster at once

[ZAIDI, ASAD A]

adding multiple nodes at once tax system more and caused me issues on existing 
nodes. I prefer to add one node at a time …


From: techpyaasa . [mailto:techpya...@gmail.com]
Sent: Tuesday, June 20, 2017 9:32 AM
To: user@cassandra.apache.org
Subject: Best practice to add(bootstrap) multiple nodes to cluster at once

Hi,

What is the best practice to add(bootstrap) multiple nodes at once to c* 
cluster.

Using c*-2.1.17 , 2 DCs , 3 groups in each DC

Thanks
TechPyaasa

RE: Secondary Index

[ZAIDI, ASAD A]

Hey there –

Like other suggested before adding more index , look for opportunity to 
de-normalize your data model OR create composite keys for your primary index – 
if that works for you.
Secondary index are there so you can leverage them they come with cost. They’re 
difficult to manage , as you repair data  your secondary index will NOT be 
automatically repaired so you’ll need to maintain them
On each cluster node. Depending on size of your cluster that could be a 
significant effort. Be prepared to rebuild your new index (nodetool 
rebuild_index) as often as you change the data . performance will eventually 
get a hit cause index rebuilding is expensive operation on CPU ..

See please http://docs.datastax.com/en/cql/3.1/cql/ddl/ddl_when_use_index_c.html



From: techpyaasa . [mailto:techpya...@gmail.com]
Sent: Tuesday, June 20, 2017 2:30 AM
To: ZAIDI, ASAD A <az1...@att.com>
Cc: user@cassandra.apache.org
Subject: Re: Secondary Index

Hi ZAIDI,

Thanks for reply.
Sorry I didn't get your line
"You can get away the potential situation by leveraging composite key, if that 
is possible for you?"

How can I get through it??

Like I have a table as below
CREATE TABLE ks1.cf1 (id1 bigint, id2 bigint, resp text, status int, PRIMARY 
KEY (id1, id2)
) WITH CLUSTERING ORDER BY (id2 ASC)

'status' will have values of 0/1/2/3/4 (4 possible values) , insertions to 
table(partition) will happen based on id2 i.e values(id1,id2,resp,status)

I want to have a filtering/criteria applied on 'status' column too like
select * from ks1.cf1 where id1=123 and status=0;

How can I achieve this w/o secondary index (on 'status' column )??

On Tue, Jun 20, 2017 at 12:09 AM, ZAIDI, ASAD A 
<az1...@att.com<mailto:az1...@att.com>> wrote:
If you’re only creating index so that your query work, think again!  You’ll be 
storing secondary index on each node , queries involving index could create 
issues (slowness!!) down the road the when index on multiple node Is involved 
and  not maintained!  Tables involving a lot of inserts/delete could easily 
ruin index performance.

You can get away the potential situation by leveraging composite key, if that 
is possible for you?


From: techpyaasa . [mailto:techpya...@gmail.com<mailto:techpya...@gmail.com>]
Sent: Monday, June 19, 2017 1:01 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Secondary Index

Hi,

I want to create Index on already existing table which has more than 3 GB/node.
We are using c*-2.1.17 with 2 DCs , each DC with 3 groups and each group has 7 
nodes.(Total 42 nodes in cluster)

So is it ok to create Index on this table now or will it have any problem?
If its ok , how much time it would take for this process?


Thanks in advance,
TechPyaasa

RE: Adding nodes and cleanup

[ZAIDI, ASAD A]

I think the token ranges that are clean/completed and potentially streamed down 
to additional node ,  won’t be cleaned again  so potentially you’ll need to run 
cleanup once again.
Can you can stop cleanup, add additional node and start cleanup over again so 
to get nodes clean in single shot!


From: Mark Furlong [mailto:mfurl...@ancestry.com]
Sent: Monday, June 19, 2017 2:28 PM
To: user@cassandra.apache.org
Subject: Adding nodes and cleanup

I have added a few nodes and now am running some cleanups. Can I add an 
additional node while these cleanups are running? What are the ramifications of 
doing this?

Mark Furlong

Sr. Database Administrator

mfurl...@ancestry.com
M: 801-859-7427
O: 801-705-7115
1300 W Traverse Pkwy
Lehi, UT 84043





​[http://c.mfcreative.com/mars/email/shared-icon/sig-logo.gif]

RE: Partition range incremental repairs

[ZAIDI, ASAD A]

Few options that you can consider to improve repair time are:

§  Un-throttle streamthroughput & interdcstreamthroughput , at least for the 
duration of repair.

§  Increase number of job threads i.e. to use –j option

§  Use subrange repair options

§  implement jumbo frames on your internode- communication network cards on 
your C* host machines.

§  If possible, reduce # of vnodes!

From: Chris Stokesmore [mailto:chris.elsm...@demandlogic.co]
Sent: Monday, June 19, 2017 4:50 AM
To: anujw_2...@yahoo.co.in
Cc: user@cassandra.apache.org
Subject: Re: Partition range incremental repairs

Anyone have anymore thoughts on this at all? Struggling to understand it..


On 9 Jun 2017, at 11:32, Chris Stokesmore 
> wrote:

Hi Anuj,

Thanks for the reply.

1). We are using Cassandra 2.2.8, and our repair commands we are comparing are
"nodetool repair --in-local-dc --partitioner-range” and
"nodetool repair --in-local-dc”
Since 2.2 I believe inc repairs are the default - that seems to be confirmed in 
the logs that list the repair details when a repair starts.

2) From looks at a few runsr, on average:
with -pr repairs, each node is approx 6.5 - 8 hours, so a total over the 7 
nodes of 53 hours
With just inc repairs, each node ~26 - 29 hours, so a total of 193

3) we currently have two DCs in total, the ‘production’ ring with 7 nodes and 
RF=3, and a testing ring with one single node and RF=1 for our single keyspace 
we currently use.

4) Yeah that number came from the Cassandra repair logs from an inc repair, I 
can share the number reports when using a pr repair later this evening when the 
currently running repair has completed.


Many thanks for the reply again,

Chris


On 6 Jun 2017, at 17:50, Anuj Wadehra 
> wrote:

Hi Chris,

Can your share following info:

1. Exact repair commands you use for inc repair and pr repair

2. Repair time should be measured at cluster level for inc repair. So, whats 
the total time it takes to run repair on all nodes for incremental vs pr 
repairs?

3. You are repairing one dc DC3. How many DCs are there in total and whats the 
RF for keyspaces? Running pr on a specific dc would not repair entire data.

4. 885 ranges? From where did you get this number? Logs? Can you share the 
number ranges printed in logs for both inc and pr case?


Thanks
Anuj

Sent from Yahoo Mail on 
Android

On Tue, Jun 6, 2017 at 9:33 PM, Chris Stokesmore
> wrote:
Thank you for the excellent and clear description of the different versions of 
repair Anuj, that has cleared up what I expect to be happening.

The problem now is in our cluster, we are running repairs with options 
(parallelism: parallel, primary range: false, incremental: true, job threads: 
1, ColumnFamilies: [], dataCenters: [DC3], hosts: [], # of ranges: 885) and 
when we do our repairs are taking over a day to complete when previously when 
running with the partition range option they were taking more like 8-9 hours.

As I understand it, using incremental should have sped this process up as all 
three sets of data on each repair job should be marked as repaired however this 
does not seem to be the case. Any ideas?

Chris

On 6 Jun 2017, at 16:08, Anuj Wadehra 
> wrote:

Hi Chris,

Using pr with incremental repairs does not make sense. Primary range repair is 
an optimization over full repair. If you run full repair on a n node cluster 
with RF=3, you would be repairing each data thrice.
E.g. in a 5 node cluster with RF=3, a range may exist on node A,B and C . When 
full repair is run on node A, the entire data in that range gets synced with 
replicas on node B and C. Now, when you run full repair on nodes B and C, you 
are wasting resources on repairing data which is already repaired.

Primary range repair ensures that when you run repair on a node, it ONLY 
repairs the data which is owned by the node. Thus, no node repairs data which 
is not owned by it and must be repaired by other node. Redundant work is 
eliminated.

Even in pr, each time you run pr on all nodes, you repair 100% of data. Why to 
repair complete data in each cycle?? ..even data which has not even changed 
since the last repair cycle?

This is where Incremental repair comes as an improvement. Once repaired, a data 
would be marked repaired so that the next repair cycle could just focus on 
repairing the delta. Now, lets go back to the example of 5 node cluster with RF 
=3.This time we run incremental repair on all nodes. When you repair entire 
data on node A, all 3 replicas are 

RE: Secondary Index

[ZAIDI, ASAD A]

If you’re only creating index so that your query work, think again!  You’ll be 
storing secondary index on each node , queries involving index could create 
issues (slowness!!) down the road the when index on multiple node Is involved 
and  not maintained!  Tables involving a lot of inserts/delete could easily 
ruin index performance.

You can get away the potential situation by leveraging composite key, if that 
is possible for you?


From: techpyaasa . [mailto:techpya...@gmail.com]
Sent: Monday, June 19, 2017 1:01 PM
To: user@cassandra.apache.org
Subject: Secondary Index

Hi,

I want to create Index on already existing table which has more than 3 GB/node.
We are using c*-2.1.17 with 2 DCs , each DC with 3 groups and each group has 7 
nodes.(Total 42 nodes in cluster)

So is it ok to create Index on this table now or will it have any problem?
If its ok , how much time it would take for this process?


Thanks in advance,
TechPyaasa

RE: Question: Large partition warning

[ZAIDI, ASAD A]

Check partition size of your table’s partitions (  nodetool tablehistograms 
tsg_ae logs_by_user).
You may need to reduce partition size of your table using guidelines given in [ 
http://docs.datastax.com/en/archived/cassandra/2.2/cassandra/planning/planPlanningPartitionSize.html]
 & 
[https://stackoverflow.com/questions/20512710/cassandra-has-a-limit-of-2-billion-cells-per-partition-but-whats-a-partition]




From: Thakrar, Jayesh [mailto:jthak...@conversantmedia.com]
Sent: Wednesday, June 14, 2017 3:14 PM
To: User 
Subject: Question: Large partition warning

We are on Cassandra 2.2.5 and I am constantly seeing warning messages about 
large partitions in system.log even though our setting for partition warning 
threshold is set to 4096 (MB).

WARN  [CompactionExecutor:43180] 2017-06-14 20:02:13,189 
BigTableWriter.java:184 - Writing large partition 
tsg_ae/logs_by_user:114303419784957147 (17187 bytes)
WARN  [CompactionExecutor:43180] 2017-06-14 20:02:13,190 
BigTableWriter.java:184 - Writing large partition 
tsg_ae/logs_by_user:820590487870502244 (2613 bytes)
WARN  [CompactionExecutor:43180] 2017-06-14 20:02:13,191 
BigTableWriter.java:184 - Writing large partition 
tsg_ae/logs_by_user:421586605444858333 (54923 bytes)

Here's what we have in cassandra.yaml.

# Log a warning when compacting partitions larger than this value
compaction_large_partition_warning_threshold_mb: 4096

So wondering if such a warning is normal or is there something wrong in our 
configuration or is it a bug?

I have looked at 
https://issues.apache.org/jira/browse/CASSANDRA-9643.

Any pointers will be greatly appreciated.

Thanks,
Jayesh

Apache Cassandra - Memory usage on server

[ZAIDI, ASAD A]

Hi folks,

I’m using apache Cassandra 2.2.

Instance is configured with max_heap_size set at 16G, memtable_allocation_type 
is  offheap_objects – total available memory is 62G on the server.
There is nothing but Cassandra is running on my Linux server.

My Cassandra instance is consuming all available memory on my machine!

$ free -h
 total   used   free sharedbuffers cached
Mem:   62G62G   776M   184K35M42G
-/+ buffers/cache:19G43G
Swap: 4.0G 0B   4.0G

Can you guys  please suggest how I can limit memory usage of my Cassandra 
instance and why C* may be using all available memory?
I’ll much appreciate your suggestions.

Thanks/Asad

RE: Data in multi disks is not evenly distributed

[ZAIDI, ASAD A]

Check status of load with nodetool status command. Make sure your there isn’t 
huge number of pending compactions for your tables. Ideally speaking data 
distribution should be even across your nodes.

you should have reserved extra 15% of free space relative to your maximum size 
of your table i.e. candidate for compaction so compaction goes comfortably. 
Some documents suggest you should reserve 50% of free space for worst case 
scenario though I think that is bit aggressive#.




From: Xihui He [mailto:xihu...@gmail.com]
Sent: Wednesday, June 07, 2017 5:16 AM
To: user@cassandra.apache.org
Subject: Data in multi disks is not evenly distributed

Dear All,

We are using multiple disks per node and find the data is not evenly 
distributed (data01 uses 1.1T, but data02 uses 353G). Is this expected? If 
data01 becomes full, would the node be still writable? We are using 2.2.6.

Thanks,
Xihui

data_file_directories:
- /data00/cassandra
- /data01/cassandra
- /data02/cassandra
- /data03/cassandra
- /data04/cassandra

df
/dev/sde1   1.8T  544G  1.2T  32% /data03
/dev/sdc1   1.8T  1.1T  683G  61% /data01
/dev/sdf1   1.8T  491G  1.3T  29% /data04
/dev/sdd1   1.8T  353G  1.4T  21% /data02
/dev/sdb1   1.8T  285G  1.5T  17% /data00

root@n9-016-015:~# du -sh /data01/cassandra/album_media_feature/*
143M   
/data01/cassandra/album_media_feature/media_feature_blur-066e5700c41511e5beacf197ae340934
4.4G
/data01/cassandra/album_media_feature/media_feature_c1-dbadf930c41411e5974743d3a691d887
56K 
/data01/cassandra/album_media_feature/media_feature_duplicate-09d4b380c41511e58501e9aa37be91a5
16K 
/data01/cassandra/album_media_feature/media_feature_emotion-b8570470054d11e69fb88f073bab8267
240M   
/data01/cassandra/album_media_feature/media_feature_exposure-f55449c0c41411e58f5c9b66773b60c3
649M   
/data01/cassandra/album_media_feature/media_feature_group-f8de0cc0c41411e5827b995f709095c8
22G 
/data01/cassandra/album_media_feature/media_feature_multi_class-cf3bb72006c511e69fb88f073bab8267
44K 
/data01/cassandra/album_media_feature/media_feature_pool5-1185b200c41511e5b7d8757e25e34d67
15G 
/data01/cassandra/album_media_feature/media_feature_poster-fcf45850c41411e597bb1507d1856305
8.0K
/data01/cassandra/album_media_feature/media_feature_quality-155d9500c41511e5974743d3a691d887
17G 
/data01/cassandra/album_media_feature/media_feature_quality_rc-51babf50dba811e59fb88f073bab8267
8.7G
/data01/cassandra/album_media_feature/media_feature_scene-008a5050c41511e59ebcc3582d286c8d
8.0K
/data01/cassandra/album_media_feature/media_region_features_v4-29a0cd10150611e6bd3e3f41faa2612a
971G   
/data01/cassandra/album_media_feature/media_region_features_v5-1b805470a3d711e68121757e9ac51b7b

root@n9-016-015:~# du -sh /data02/cassandra/album_media_feature/*
1.6G
/data02/cassandra/album_media_feature/media_feature_blur-066e5700c41511e5beacf197ae340934
44G 
/data02/cassandra/album_media_feature/media_feature_c1-dbadf930c41411e5974743d3a691d887
64K 
/data02/cassandra/album_media_feature/media_feature_duplicate-09d4b380c41511e58501e9aa37be91a5
75G 
/data02/cassandra/album_media_feature/media_feature_emotion-b8570470054d11e69fb88f073bab8267
2.0G
/data02/cassandra/album_media_feature/media_feature_exposure-f55449c0c41411e58f5c9b66773b60c3
21G 
/data02/cassandra/album_media_feature/media_feature_group-f8de0cc0c41411e5827b995f709095c8
336M   
/data02/cassandra/album_media_feature/media_feature_multi_class-cf3bb72006c511e69fb88f073bab8267
44K 
/data02/cassandra/album_media_feature/media_feature_pool5-1185b200c41511e5b7d8757e25e34d67
2.0G
/data02/cassandra/album_media_feature/media_feature_poster-fcf45850c41411e597bb1507d1856305
8.0K
/data02/cassandra/album_media_feature/media_feature_quality-155d9500c41511e5974743d3a691d887
17G 
/data02/cassandra/album_media_feature/media_feature_quality_rc-51babf50dba811e59fb88f073bab8267
141M   
/data02/cassandra/album_media_feature/media_feature_scene-008a5050c41511e59ebcc3582d286c8d
8.0K
/data02/cassandra/album_media_feature/media_region_features_v4-29a0cd10150611e6bd3e3f41faa2612a
93G 
/data02/cassandra/album_media_feature/media_region_features_v5-1b805470a3d711e68121757e9ac51b7b

root@n9-016-015:~# du -sh /data03/cassandra/album_media_feature/*
4.3G
/data03/cassandra/album_media_feature/media_feature_blur-066e5700c41511e5beacf197ae340934
19G 
/data03/cassandra/album_media_feature/media_feature_c1-dbadf930c41411e5974743d3a691d887
72K 
/data03/cassandra/album_media_feature/media_feature_duplicate-09d4b380c41511e58501e9aa37be91a5
2.8G
/data03/cassandra/album_media_feature/media_feature_emotion-b8570470054d11e69fb88f073bab8267
105M   
/data03/cassandra/album_media_feature/media_feature_exposure-f55449c0c41411e58f5c9b66773b60c3
15G 
/data03/cassandra/album_media_feature/media_feature_group-f8de0cc0c41411e5827b995f709095c8
23G 

RE: Local_serial >> Adding nodes

[ZAIDI, ASAD A]

Please share exact timeout error message text to get idea what type of timeout 
you're facing.


From: Nitan Kainth [mailto:ni...@bamlabs.com]
Sent: Wednesday, June 07, 2017 7:24 AM
To: vasu gunja 
Cc: user@cassandra.apache.org
Subject: Re: Local_serial >> Adding nodes

What is in system log?
Does it show new tokens allocated?
nodetool netstats

On Jun 7, 2017, at 6:50 AM, vasu gunja 
> wrote:

Yeah all of them are in UJ state..

Thanks,
Vasu

On Jun 7, 2017, at 6:32 AM, Nitan Kainth 
> wrote:
Is the streaming still going on?
Timeouts are in reads or writes?

Sent from my iPhone

On Jun 6, 2017, at 9:50 PM, vasu gunja 
> wrote:
Hi All,

We are having 2 DC setup each consists of 20 odd nodes and recently we decided 
to add 6 more nodes to DC1.  We are using LWT's,  application dirvers are 
configuared to use LOCAL_SERIAL.
As we are adding multiple nodes at a time we used option  
"-Dcassandra.consistent.rangemovement=false" we added all nodes with gap of 10 
mins each

We are facing lot of timeouts more 30k transactions over 8 hours of period . is 
anyone ran into same issue ? are we doing something.wrong ?



Thanks,
vasu

RE: Convert single node C* to cluster (rebalancing problem)

[ZAIDI, ASAD A]

Did you make sure auto_bootstrap property is indeed set to [true] when you 
added the node?

From: Junaid Nasir [mailto:jna...@an10.io]
Sent: Monday, June 05, 2017 6:29 AM
To: Akhil Mehra 
Cc: Vladimir Yudovin ; user@cassandra.apache.org
Subject: Re: Convert single node C* to cluster (rebalancing problem)

not evenly, i have setup a new cluster with subset of data (around 5gb). using 
the configuration above I am getting these results


Datacenter: datacenter1

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address  Load   Tokens   Owns (effective)  Host ID Rack

UN  10.128.2.1   4.86 GiB   256  44.9% 
e4427611-c247-42ee-9404-371e177f5f17  rack1

UN  10.128.2.10  725.03 MiB  256 55.1% 
690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1
is there anything else I can tweak/check to make the distribution even?

On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra 
> wrote:
So now the data is evenly balanced in both nodes?

Refer to the following documentation to get a better understanding of the 
roc_address and the broadcast_rpc_address 
https://www.instaclustr.com/demystifying-cassandras-broadcast_address/.
 I am surprised that your node started up with rpc_broadcast_address set as 
this is an unsupported property. I am assuming you are using Cassandra version 
3.10.


Regards,
Akhil

On 2/06/2017, at 11:06 PM, Junaid Nasir > 
wrote:

I am able to get it working. I added a new node with following changes

#rpc_address:0.0.0.0

rpc_address: 10.128.1.11

#rpc_broadcast_address:10.128.1.11
rpc_address was set to 0.0.0.0, (I ran into a problem previously regarding 
remote connection and made these changes 
https://stackoverflow.com/questions/12236898/apache-cassandra-remote-access)

should it be happening?

On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin 
> wrote:
Did you run "nodetool cleanup" on first node after second was bootstrapped? It 
should clean rows not belonging to node after tokens changed.

Best regards, Vladimir Yudovin,
Winguzone
 - Cloud Cassandra Hosting


 On Wed, 31 May 2017 03:55:54 -0400 Junaid Nasir 
> wrote 

Cassandra ensure that adding or removing nodes are very easy and that load is 
balanced between nodes when a change is made. but it's not working in my case.
I have a single node C* deployment (with 270 GB of data) and want to load 
balance the data on multiple nodes, I followed this 
guide
`nodetool status` shows 2 nodes but load is not balanced between them

Datacenter: dc1

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address  Load   Tokens   Owns (effective)  Host IDRack

UN  10.128.0.7   270.75 GiB  256  48.6%
1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1

UN  10.128.0.14  414.36 KiB  256  51.4%
66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
I also ran 'nodetool repair' on new node but result is same. any pointers would 
be appreciated :)

conf file of new node

cluster_name: 'cluster1'

 - seeds: "10.128.0.7"
num_tokens: 256

endpoint_snitch: GossipingPropertyFileSnitch
Thanks,
Junaid

MemtablePostFlush pending

[ZAIDI, ASAD A]

Hello Folks,

I'm adding another node on my 14 node open source apache cassandra 2.2.8 
cluster.  New node is taking long time to join the cluster.
I see there are bunch of pending [memtablepostflush] threads. I did increase 
memtable_flush_writers from 8 to 24 , though it is not helping with situation.

Can you guys please share your experience or advise me what else I can look so 
to  remediate the situation.


nodetool tpstats
Pool NameActive   Pending  Completed   Blocked  All 
time blocked
MutationStage 0 08543301 0  
   0
ReadStage 0 0  0 0  
   0
RequestResponseStage  0 0  0 0  
   0
ReadRepairStage   0 0  0 0  
   0
CounterMutationStage  0 0  0 0  
   0
GossipStage   0 0  0 0  
   0
MigrationStage0 0  0 0  
   0
MemtablePostFlush 1   388 15 0  
   0
ValidationExecutor0 0  0 0  
   0
Sampler   0 0  0 0  
   0
MiscStage 0 0  0 0  
   0
MemtableFlushWriter   3 3133 0  
   0
InternalResponseStage 0 0  0 0  
   0
CompactionExecutor0 0 18 0  
   0
MemtableReclaimMemory 0 0133 0  
   0
AntiEntropyStage  0 0  0 0  
   0
CacheCleanupExecutor  0 0  0 0  
   0

Message type   Dropped
READ 0
RANGE_SLICE  0
_TRACE   0
MUTATION 0
COUNTER_MUTATION 0
REQUEST_RESPONSE 0
PAGED_RANGE  0
READ_REPAIR  0

Cassandra Node Density thresholds

[ZAIDI, ASAD A]

Hello Folks -
I'm using open source apache Cassandra 2.2 .My cluster is spread over 14 nodes 
in cluster in two data centers.

My DC1 data center nodes are reaching 2TB of consumed volume. we don't have 
much space left on disk.
I am wondering if there is guideline available that can point me to certain 
best practice that describe when we should add more nodes to the cluster.  
should we add more storage or add more nodes. I guess we should scale Cassandra 
horizontally so adding node may be better option.. i am looking for a criteria 
that describes node density thresholds, if there are any.
Can you guys please share your thoughts , experience. I'll much appreciate your 
reply. Thanks/Asad

Apache Cassandra - Configuration Management

[ZAIDI, ASAD A]

Good Morning Folks –

I’m running 14 nodes Cassandra cluster in two data centers , each node is has 
roughly 1.5TB. we’re anticipating more load therefore we’ll be expanding 
cluster with additional nodes.
At this time, I’m kind of struggling to keep consistent cassandra.yaml file on 
each server – at this time, I’m maintaining yaml file manually. The only tool 
I’ve is splunk  which is only to ‘monitor‘ threads.

Would you guy please suggest  open source tool that can help maintain the 
cluster. I’ll really appreciate your reply – Thanks/Asad

RE: Slow writes and Frequent timeouts

[ZAIDI, ASAD A]

Would you please share what changes you made that proves helpful for you?


From: Akshay Suresh [mailto:akshay.sur...@unotechsoft.com]
Sent: Tuesday, April 25, 2017 7:33 AM
To: user@cassandra.apache.org
Subject: Re: Slow writes and Frequent timeouts

Hi
It turned out to be an issue in my yaml file - Each record had a size in GB. I 
modified my yaml and I am getting
good results.

Thanks for the support.
Cheers.

On Tue, Apr 18, 2017 at 11:45 AM, Akshay Suresh 
> wrote:
Hi

- Hardware Specs
RAM: 14GB
Cores: 8
Disk: 440GB SSD

- I have deployed 7 instances ( Earlier, I mentioned 8 but I had to 
decommission 1 due to low disk space ) on AWS.
- Every node is in the same datacenter.
.
- I have created a custom schema with 28 columns. Created a simple YAML file 
for the stress tool.
The command is as follows:
./cassandra-stress user profile=foo.yml duration=180m ops\(insert=1\) cl=one 
-rate threads=850
- I have used a heap space of 7GB. I am using the default CMS for GC.
I am clearing all the data and logs. I will try to replicate the issue and post 
the exact metrics.



Thanks in advance.


On Mon, Apr 17, 2017 at 10:28 PM, Jon Haddad 
> wrote:
What are your hardware specs?  Where are you running the cluster?  Is every 
node in the same physical datacenter?  What command are you using to run stress?

On Apr 17, 2017, at 9:57 AM, Akshay Suresh 
> wrote:

Hi
I have not done much. Just created a schema with SimpleStrategy and a 
replication factor of 3.
Then I created a yaml file
 Now I am running the cassandra stress tool invoking the yaml file - with 
10,000 records ( no warmup ) with 10 concurrent threads. I am just running 
writes ( no reads )

When I keep running this script again and again eventually my nodes start going 
down and I start getting timeouts and out of memory errors.


On Mon, Apr 17, 2017 at 9:23 PM, Hannu Kröger 
> wrote:
It would help to know what kind queries are slow.

Hannu

On 17 Apr 2017, at 18:42, Akshay Suresh 
> wrote:


Hi

I have set up a cassandra cluster of 8 nodes.

I am using Apache Cassandra 3.9

While using cassandra-stress tool for load testing, I am getting really slow 
writes ( low upto few 10-20 writes per second ) along with frequent timeouts 
and out of heap space errors.

Kindly let me know how do I resolve this issue.

Disclaimer : The contents of this e-mail and attachment(s) thereto are 
confidential and intended for the named recipient(s) only. It shall not attach 
any liability on the originator or Unotech Software Pvt. Ltd. or its 
affiliates. Any views or opinions presented in this email are solely those of 
the author and may not necessarily reflect the opinions of Unotech Software 
Pvt. Ltd. or its affiliates. Any form of reproduction, dissemination, copying, 
disclosure, modification, distribution and / or publication of this message 
without the prior written consent of the author of this e-mail is strictly 
prohibited. If you have received this email in error please delete it and 
notify the sender immediately.




--
Regards,


Akshay Suresh
Unotech Software Pvt. Ltd

[https://docs.google.com/uc?export=download=0B5gpZneW6JQpUU5Ob3lDQVA0NUk=0B5gpZneW6JQpVVNBKzhhRzBqenc0M2RRdmFZQTdPQUgxUEE4PQ]

M

:

+91 99309 80509

O

:

+91 (22) 2687 9402

A

:

 D Wing, 7th floor, 32 Corporate Avenue, Off Mahakali Caves Road, Andheri (E), 
Mumbai, 400093

W

:

www.unotechsoft.com


Disclaimer : The contents of this e-mail and attachment(s) thereto are 
confidential and intended for the named recipient(s) only. It shall not attach 
any liability on the originator or Unotech Software Pvt. Ltd. or its 
affiliates. Any views or opinions presented in this email are solely those of 
the author and may not necessarily reflect the opinions of Unotech Software 
Pvt. Ltd. or its affiliates. Any form of reproduction, dissemination, copying, 
disclosure, modification, distribution and / or publication of this message 
without the prior written consent of the author of this e-mail is strictly 
prohibited. If you have received this email in error please delete it and 
notify the sender immediately.




--
Regards,


Akshay Suresh
Unotech Software Pvt. Ltd