RE: MUTATION messages were dropped in last 5000 ms for cross node timeout

lt it is set to 2000ms. I hope that helps. Regards, Akhil On 22/07/2017, at 2:46 AM, ZAIDI, ASAD A <az1...@att.com<mailto:az1...@att.com>> wrote: Hi Akhil, Thank you for your reply. Previously, I did ‘tune’ various timeouts – basically increased them a bit but none of those para

MemtablePostFlush pending

Hello Folks, I’m using Cassandra 2.2 on 14 node cluster. Now a days, I’m observing memtablepostflush pending number going high , this happens intermittently. I’m looking if Is there way to ‘tune’ memtablepostflush stage? Thanks/ASad

reduced num_token = improved performance ??

Hi Folks, Pardon me if I’m missing something obvious. I’m still using apache-cassandra 2.2 and planning for upgrade to 3.x. I came across this jira [https://issues.apache.org/jira/browse/CASSANDRA-7032] that suggests reducing num_token may improve general performance of Cassandra like

RE: Understanding of cassandra metrics

What exactly does mean CoordinatorScanLatency for example CoordinatorScanLatency is a timer metric that present coordinator range scan latency for table. Is it latency on full table scan or maybe range scan by clustering key? It is range scan.. clustering key is used to only

RE: READ Queries timing out.

>> I analysed the GC logs not having any issues with major GC's If you don’t have issues on GC , than why do you want to [tune] GC parameters ? Instead focus on why select queries are taking time.. may be take a look on their trace? From: Pranay akula

MUTATION messages were dropped in last 5000 ms for cross node timeout

Hello Folks - I'm using apache-cassandra 2.2.8. I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is set and NTP server is also running correcting clock drift on 16node cluster. I do not see pending or blocked HintedHandoff in tpstats

RE: MUTATION messages were dropped in last 5000 ms for cross node timeout

not be optimal. Thank you again. From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in] Sent: Thursday, July 20, 2017 12:17 PM To: ZAIDI, ASAD A <az1...@att.com>; user@cassandra.apache.org Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout Hi Asad, You can do fol

RE: MUTATION messages were dropped in last 5000 ms for cross node timeout

for cross node timeout In a cloud environment, cross_node_timeout = true can cause issues; we had this issue in our environment and it is set to false now. Dropped messages is an another issue Subroto On Jul 20, 2017, at 8:27 AM, ZAIDI, ASAD A <az1...@att.com<mailto:az1...@att.com>>

RE: MUTATION messages were dropped in last 5000 ms for cross node timeout

under heavy load? Regards, Akhil On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A <az1...@att.com<mailto:az1...@att.com>> wrote: Hello Folks – I’m using apache-cassandra 2.2.8. I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is

RE: Node failure Due To Very high GC pause time

>> here my is doubt is that does all the deleted 3.3Million row will be >> loaded in my on-heap memory? if not what will be object that occupying those >> memory ? It depends on your queries what data they’re fetching from your database. Assuming you’re using CMS garbage

Cassandra Cluster Expansion Criteria

Hello Folks, I’m on Cassandra 2.2.8 cluster with 14 nodes , each with around 2TB of data volume. I’m looking for a criteria /or data points that can help me decide when or if I should add more nodes to the cluster and by how many nodes. I’ll really appreciate if you guys can share your

RE: Slow writes and Frequent timeouts

Would you please share what changes you made that proves helpful for you? From: Akshay Suresh [mailto:akshay.sur...@unotechsoft.com] Sent: Tuesday, April 25, 2017 7:33 AM To: user@cassandra.apache.org Subject: Re: Slow writes and Frequent timeouts Hi It turned out to be an issue in my yaml file

RE: rebuild constantly fails, 3.11

Is there any chance you've set streaming_socket_timeout_in_ms parameter set too low on failing node? -Original Message- From: Micha [mailto:mich...@fantasymail.de] Sent: Tuesday, August 08, 2017 3:01 AM To: user@cassandra.apache.org; d...@cassandra.apache.org Subject: rebuild

RE: rebuild constantly fails, 3.11

is causing the failure and network is supposedly stable. ~Asad -Original Message- From: Micha [mailto:mich...@fantasymail.de] Sent: Tuesday, August 08, 2017 8:35 AM To: user@cassandra.apache.org; ZAIDI, ASAD A <az1...@att.com>; user@cassandra.apache.org Subject: Re: rebuild constantly

RE: Different data size between datacenters

Are you using same number of token/vnodes in both data centers? From: Chuck Reynolds [mailto:creyno...@ancestry.com] Sent: Monday, August 07, 2017 1:51 PM To: user@cassandra.apache.org Subject: Different data size between datacenters I have a cluster that spans two datacenters running Cassandra

Cassandra upgrade from 2.2.8 to 3.10

Hi folks, I’ve question on upgrade method I’m thinking to execute. I’m planning from apache-Cassandra 2.2.8 to release 3.10. My Cassandra cluster is configured like one rack with two Datacenters like: 1. DC1 has 4 nodes 2. DC2 has 16 nodes We’re adding another 12 nodes and would

Apache Cassandra - Configuration Management

Good Morning Folks – I’m running 14 nodes Cassandra cluster in two data centers , each node is has roughly 1.5TB. we’re anticipating more load therefore we’ll be expanding cluster with additional nodes. At this time, I’m kind of struggling to keep consistent cassandra.yaml file on each server

RE: COUNT

Is it possible for you to share tracing info for the query? You can enable tracing at cqlsh prompt with command Cqlsh > TRACING ON Cqlsh> run your query Tracing session info should be printed on screen Tracing will enable us to know where most of the time is spent! From: web master

RE: Secondary Index

. performance will eventually get a hit cause index rebuilding is expensive operation on CPU .. See please http://docs.datastax.com/en/cql/3.1/cql/ddl/ddl_when_use_index_c.html From: techpyaasa . [mailto:techpya...@gmail.com] Sent: Tuesday, June 20, 2017 2:30 AM To: ZAIDI, ASAD A <az1...@att.com&

RE: Best practice to add(bootstrap) multiple nodes to cluster at once

adding multiple nodes at once tax system more and caused me issues on existing nodes. I prefer to add one node at a time … From: techpyaasa . [mailto:techpya...@gmail.com] Sent: Tuesday, June 20, 2017 9:32 AM To: user@cassandra.apache.org Subject: Best practice to add(bootstrap) multiple nodes

RE: Adding nodes and cleanup

I think the token ranges that are clean/completed and potentially streamed down to additional node , won’t be cleaned again so potentially you’ll need to run cleanup once again. Can you can stop cleanup, add additional node and start cleanup over again so to get nodes clean in single shot!

RE: Secondary Index

If you’re only creating index so that your query work, think again! You’ll be storing secondary index on each node , queries involving index could create issues (slowness!!) down the road the when index on multiple node Is involved and not maintained! Tables involving a lot of inserts/delete

RE: Partition range incremental repairs

Few options that you can consider to improve repair time are: § Un-throttle streamthroughput & interdcstreamthroughput , at least for the duration of repair. § Increase number of job threads i.e. to use –j option § Use subrange repair options § implement jumbo frames on your internode-

Cassandra Node Density thresholds

Hello Folks - I'm using open source apache Cassandra 2.2 .My cluster is spread over 14 nodes in cluster in two data centers. My DC1 data center nodes are reaching 2TB of consumed volume. we don't have much space left on disk. I am wondering if there is guideline available that can point me to

MemtablePostFlush pending

Hello Folks, I'm adding another node on my 14 node open source apache cassandra 2.2.8 cluster. New node is taking long time to join the cluster. I see there are bunch of pending [memtablepostflush] threads. I did increase memtable_flush_writers from 8 to 24 , though it is not helping with

RE: Convert single node C* to cluster (rebalancing problem)

Did you make sure auto_bootstrap property is indeed set to [true] when you added the node? From: Junaid Nasir [mailto:jna...@an10.io] Sent: Monday, June 05, 2017 6:29 AM To: Akhil Mehra Cc: Vladimir Yudovin ; user@cassandra.apache.org Subject: Re:

RE: Local_serial >> Adding nodes

Please share exact timeout error message text to get idea what type of timeout you're facing. From: Nitan Kainth [mailto:ni...@bamlabs.com] Sent: Wednesday, June 07, 2017 7:24 AM To: vasu gunja Cc: user@cassandra.apache.org Subject: Re: Local_serial >> Adding nodes What

RE: Data in multi disks is not evenly distributed

Check status of load with nodetool status command. Make sure your there isn’t huge number of pending compactions for your tables. Ideally speaking data distribution should be even across your nodes. you should have reserved extra 15% of free space relative to your maximum size of your table

Apache Cassandra - Memory usage on server

Hi folks, I’m using apache Cassandra 2.2. Instance is configured with max_heap_size set at 16G, memtable_allocation_type is offheap_objects – total available memory is 62G on the server. There is nothing but Cassandra is running on my Linux server. My Cassandra instance is consuming all

RE: Question: Large partition warning

Check partition size of your table’s partitions ( nodetool tablehistograms tsg_ae logs_by_user). You may need to reduce partition size of your table using guidelines given in [ http://docs.datastax.com/en/archived/cassandra/2.2/cassandra/planning/planPlanningPartitionSize.html] &

detail of compactionstats, pending tasks

Hello Folks, Is it possible to findout detail of those 275 pending tasks compactionstats command output is showing? I’ve bumpedup concurrent_compactors to 25 though not all threads are compacting, only 8 threads are being used so i’m wondering how can I utilize all configured concurrent

table repair question

Hello folk, I'm wondering if there is way to find out list of table(s) which need repair OR if there Is way to find out what %age of data would need to be repaired on a table? Is such information available from Cassandra db engine through some other means? TIA~ Asad

RE: Connection refused - 127.0.0.1-Gossip

Throwing my $ .002 Rpc_address defaults to localhosts that if not set, picks value from dns/hostname file. May be you can try setting rpc_address: - see if this helps! From: Marek Kadek -T (mkadek - CONSOL PARTNERS LTD at Cisco) [mailto:mka...@cisco.com] Sent: Wednesday, December 06, 2017

RE: Problem with dropped mutations

Are you also seeing time-outs on certain Cassandra operations?? If yes, you may have to tweak *request_timeout parameter in order to get rid of dropped mutation messages if application data model is not upto mark! You can also check if network isn't dropping packets (ifconfig -a tool) +

RE: Maximum SSTable size

With Leveled compaction strategy , you can set target size with sstable_size_in_mb attribute however actual size can still be larger than target given large partition size. Thanks/Asad From: Lucas Benevides

Error creating pool to / Pool was closed during initialization

Hello Folks, I’m looking for possible reasons and solution for these frequently appearing warning messages I’m seeing in spark <>cassandra job’s log file. The message suggest Cassandra host server machine is acting up and throwing messages like: 18/06/25 14:07:44 WARN Session: Error creating

RE: Old tombstones not being cleaned up

Make data consistent (run repair), reduce gc_grace_seconds (try set it to 0 temporarily though careful as this can affect hinted handoff!) and set table’s compaction sub-property i.e. unchecked_tombstone_compaction to true. Compaction will take care of tombstones! From: Jonathan Haddad

RE: Old tombstones not being cleaned up

ting the unchecked_tombstone_compaction force another type of compaction? tor. 1. feb. 2018 kl. 19.37 skrev ZAIDI, ASAD A <az1...@att.com<mailto:az1...@att.com>>: Make data consistent (run repair), reduce gc_grace_seconds (try set it to 0 temporarily though careful as this can affect hinted handoff!)

RE: Converting from Apache Cassandra 2.2.6 to 3.x

You may want to upgrade python and java/JDK version with Cassandra upgrade. please refer to CHANGES.txt for all updates & improvement made in your selected 3.x version. From: William Boutin [mailto:william.bou...@ericsson.com] Sent: Thursday, February 01, 2018 4:49 PM To:

RE: [announce] Release of Cassandra Prometheus metrics exporter

Hi Romain Gérard - Thanks for sharing Cassandra-exporter. Since you're already monitoring tons of Cassandra instances nodes - can you please let us know if you can share Prometheus dashboards/json code along with Cassandra-exporter. Thanks/Asad -Original Message- From: Romain

RE: Upgrade from 2.1 to 3.11

You may want to check if coincidentally you’re having expired cells in heap. GC log should be able to tell you OR look for tombstones in system.log file. See your compactions are under control and normal. This may not be related to upgrade at all! From: Pradeep Chhetri

Cassandra 3.11 with NFS mounted system

Hello Folks, I've an virtualized environment running with VMware where Cassandra is humming on NFS mounted storage. As the application load increases ,they increase number of nodes in data center however writes are getting slower, nodes are flapping and application complains in write

RE: SSTable Compression Ratio -1.0

Compression ratio is ratio of compression to its original size - smaller is better; see it like compressed/uncompressed 1 would mean no change in size after compression! -Original Message- From: Vitaliy Semochkin [mailto:vitaliy...@gmail.com] Sent: Tuesday, August 28, 2018 12:03 PM

RE: Cassandra Daemon not coming up

Can you check if you’ve enough disk space available ? ~Asad From: mahesh rajamani [mailto:rajamani.mah...@gmail.com] Sent: Tuesday, February 27, 2018 10:11 AM To: user@cassandra.apache.org Subject: Cassandra Daemon not coming up I am using Cassandra 3.0.9 version on a 12 node cluster. I have

RE: Jon Haddad on Diagnosing Performance Problems in Production

Perhaps Mr. Hadad himself will share it again somewhere; he was kind enough to share it once at datastax! From: Kenneth Brotman [mailto:kenbrot...@yahoo.com.INVALID] Sent: Tuesday, February 27, 2018 10:39 AM To: user@cassandra.apache.org Subject: RE: Jon Haddad on Diagnosing Performance

node re-start delays , busy Deleting mc-txn-compaction/ Adding log file replica

I’m on environment with apache Cassandra 3.11.1 with java 1.8.0_144. One Node went OOM and crashed. Re-starting this crashed node is taking long time. Trace level debug log is showing messages like: Debug.log trace excerpt: TRACE [main] 2019-06-18 21:30:43,449

RE: postmortem on 2.2.13 scale out difficulties

Adding one node at a time – is that successful? Check value of streaming_socket_timeout_in_ms parameter in cassandra.yaml and increase if needed. Have you tried Nodetool bootstrap resume & jvm option i.e. JVM_OPTS="$JVM_OPTS -Dcassandra.consistent.rangemovement=false" ? From: Carl Mueller

RE: cassandra node was put down with oom error

Is there any chance partition size has grown over time and taking much allocated memory - if yes, that could also affect compaction thread as they'll too take more heap and kept in heap longer - leaving less for other processes . You can check partition size if they are manageable using

RE: Bootstrapping to Replace a Dead Node vs. Adding a New Node:Consistency Guarantees

The article you mentioned here clearly says “For new users to Cassandra, the safest way to add multiple nodes into a cluster is to add them one at a time. Stay tuned as I will be following up with another post on bootstrapping.” When extending cluster it is indeed recommended to go slow &

RE: Decommissioning a new node when the state is JOINING

Just stop the server/kill C* process as node never fully joined the cluster yet – that should be enough. You can safely remove the data i.e. streamed-in on new node so you can use the node for other new cluster. From: Akshay Bhardwaj [mailto:akshay.bhardwaj1...@gmail.com] Sent: Tuesday, April

RE: Cassandra JVM configuration

Every use case is unique so as such jvm configs go with it. 8G may or may not be sufficient depending on live data you keep in, or fetch to memory. You can opt using G1GC, that is easy to start with. Some good suggestions are made if you want to try G1GC or stick with CMS. Take a look at [

RE: Jmx metrics shows node down

Another way to purge gossip info from each node is to: 1. Gracefully stop cassandra i.e. nodetool drain; kill Casandra PID 2. Move/delete files from $DATADIR/system/peers/ 3. Add JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false" in jvm.options file 4. Restart Cassandra service. 5.

RE: Datafile Corruption

Did you check if packets are NOT being dropped for network interfaces Cassandra instances are consuming (ifconfig –a) internode compression is set for all endpoint – may be network is playing any role here? is this corruption limited so certain keyspace/table | DCs or is that wide spread – the

RE: Repair / compaction for 6 nodes, 2 DC cluster

Did you get chance to look at tlp reaper tool i.e. http://cassandra-reaper.io/ It is pretty awesome – Thanks to TLP team. From: Martin Xue [mailto:martin...@gmail.com] Sent: Wednesday, July 31, 2019 12:09 AM To: user@cassandra.apache.org Subject: Repair / compaction for 6 nodes, 2 DC cluster

RE: Repair failed and crash the node, how to bring it back?

I don’t think anyone can predict with certainty if instance won’t crash but there are good chances it will - unless you take remedial actions. If you are not doing subrange repair, a lot of merkle tree data can potentially be scanned/streamed taking toll on memory resources – that , taking

RE: Rebooting one Cassandra node caused all the application nodes go down

“aws asked to set nvme_timeout to higher number in etc/grub.conf.” Did you ask AWS if setting higher value is real solution to bug - Is there not any patch available to address the bug? - just curios to know From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] Sent: Friday, July 19, 2019 10:49

RE: Rebooting one Cassandra node caused all the application nodes go down

and grafana. Zaidi, In latest aws Linux Ami they took care of this bug . And also changing the Ami needs rebuild of all the nodes so didn't took that route. On Fri, Jul 19, 2019, 12:32 PM ZAIDI, ASAD A mailto:az1...@att.com>> wrote: “aws asked to set nvme_timeout to higher number in etc/gru

Performance impact with ALLOW FILTERING clause.

Hello Folks, I was going thru documentation and saw at many places saying ALLOW FILTERING causes performance unpredictability. Our developers says ALLOW FILTERING clause is implicitly added on bunch of queries by spark-Cassandra connector and they cannot control it; however at the same time

RE: Performance impact with ALLOW FILTERING clause.

range, doing a bunch of range scans. It's doing that so it can get ALL of the data and do the filtering / joining / searching in-memory in spark, rather than relying on cassandra to do the scanning/searching on disk. On Thu, Jul 25, 2019 at 6:49 AM ZAIDI, ASAD A mailto:az1...@att.com>&

RE: Constant blocking read repair for such a tiny table

Wondering if you’ve disabled otc_coalescing_strategy CASSANDRA-12676 since you’ve upgraded from 2.x? also if you found luck by increasing native_transport_max_threads to address blocked NTRs (CASSANDRA-11363)? ~Asad From: Patrick

RE: Duplicates columns which are backed by LIST collection types

Have you chosen correct datatype to begin with, if you don’t want duplicates? Generally speaking: A set and a list both represent multiple values but do so differently. A set doesn’t save ordering and values are sorted in ascending order. No duplicates are allowed. A list saves ordering where

RE: Duplicates columns which are backed by LIST collection types

into this issue recently when we did the keyspace migration. Thanks, Murali On Thu, Oct 24, 2019 at 11:36 AM ZAIDI, ASAD mailto:az1...@att.com>> wrote: Have you chosen correct datatype to begin with, if you don’t want duplicates? Generally speaking: A set and a list both represent multiple

RE: Keyspace Clone in Existing Cluster

If you’re planning to restore snapshot to target keyspace in same cluster – you can: 1. Take snapshot and copy snapshots to shared volume like NFS share so later you can load sstables using sstables loader from single node. 2. Make sure you create target keyspace and tables (without

Is CompactionStrategyClass mising in Cassandra 3.11.4 ?

Hello Folks, I'm trying to modify tables compaction strategy using jmx term - one node at a time but unable to retrieve/set CompactionStrategyClass. I don't see option available in 3.11.4 version. I'm wondering what I'm here ? Thank/Asad java -jar jmxterm-1.0.1-uber.jar --url localhost:7199

RE: New seed node in the cluster immediately UN without passing for UJ state

Seed node doesn’t bootstrap so if new node were to act as seed node, official recommendations are to boot strap ‘new’ node first , only after that list that node as seed. Seed nodes are usually same across cluster nodes. You can designate 2 nodes as seed per dc in order to mitigate network

RE: Bootstraping is failing

heck if [streaming_socket_timeout_in_ms ] setting in Cassandra.yaml file if that sufficient enough before streaming is interrupted ? ~Asad From: Surbhi Gupta [mailto:surbhi.gupt...@gmail.com] Sent: Thursday, May 7, 2020 3:09 PM To: user@cassandra.apache.org Subject: Re: Bootstraping is

RE: Cassandra scale-out with no traffic on newly joined nodes

Can you share please what replica placement strategy ( in keyspace definition) and partitioner is used across nodes ? ~Asad From: Sandeep Nethi Sent: Tuesday, September 8, 2020 1:12 AM To: user@cassandra.apache.org Subject: Re: Cassandra scale-out with no traffic on newly joined nodes That

update cassandra.yaml file on number of cluster nodes

Hello Folks, Can you guys please suggest tool or approach to update cassandra.yaml file in multi-dc environment with large number of nodes efficiently. Thank you. Asad