Re: DataStax OpsCenter with Apache Cassandra

2016-05-03 Thread Otis Gospodnetić
For what it's worth, people have mentioned SPM for Cassandra as one of the OpsCenter alternatives. I know somebody requested "Number of SSTables per read" metric to be added to SPM 2 days ago and it was just added today. Otis --

Re: tombstone_failure_threshold being ignored?

2016-05-03 Thread Rick Gunderson
I would have thought that a RangeSliceReply (which is the parent object that seems to "own" the ArrayList) would have contained only those objects related to the corresponding query. The hierarchy of objects appears to be: org.apache.cassandra.net.OutboundTcpConnection$QueuedMessage

Re: Decommissioned node shows up in the gossip log

2016-05-03 Thread Joel Knighton
The fix versions for CASSANDRA-10371 are 2.1.14, 2.2.6, 3.0.4, and 3.4 because we have several long-lived branches. If you're running datastax-ddc-3.3.0, that's on the 3.x branch before this was fixed and you could in fact be hitting CASSANDRA-10371. On Tue, May 3, 2016 at 1:26 PM, Zhang, Charles

RE: Decommissioned node shows up in the gossip log

2016-05-03 Thread Zhang, Charles
The version I’m running is datastax-ddc-3.3.0. It seems not an affected version… From: Joel Knighton [mailto:joel.knigh...@datastax.com] Sent: Tuesday, May 03, 2016 1:31 PM To: user@cassandra.apache.org Subject: Re: Decommissioned node shows up in the gossip log What version of Cassandra do you

MigrationManager.java:164 - Migration task failed to complete

2016-05-03 Thread Zhang, Charles
I have seen a bunch of them in the log files of some newly joined nodes. I did a search in google and it seems increasing the countdown latch timeout can solve this problem. But I assume it only resolves it for future nodes when joining happens? For the existing nodes, anything needs to be

Re: Decommissioned node shows up in the gossip log

2016-05-03 Thread Joel Knighton
What version of Cassandra do you have deployed? It sounds like you might be hitting https://issues.apache.org/jira/browse/CASSANDRA-10371 - you can check the fix version to see if you are running an affected version. If so, the comments on that ticket may help explain and resolve your situation.

Re: Decommissioned node shows up in the gossip log

2016-05-03 Thread Russell Bradberry
The impact is that it is still in gossip and may still be in your peers. Node tool status pulls from the snitch, not gossip, so since it was decommissioned it will not show up there. The only way to remove it from gossip would be to unsafeAssasinate the endpoint. From: "Zhang, Charles"

Decommissioned node shows up in the gossip log

2016-05-03 Thread Zhang, Charles
I decommissioned a seed node from the cluster but now it still shows up in the log files but does not show up in "nodetool status". The steps I took to decommission this seed node was first I added another node to the seed node list and remove this node from the seed node list and then

Bloom filter memory usage disparity

2016-05-03 Thread Kai Wang
Hi, I have a table on 3-node cluster. I notice bloom filter memory usage are very different on one of the node. For a given table, I checked CassandraMetricsRegistry$JmxGauge.[table]_BloomFilterOffHeapMemoryUsed.Value. 2 of 3 nodes show 1.5GB while the other shows 2.5 GB. What could be the

Re: tombstone_failure_threshold being ignored?

2016-05-03 Thread Oleksandr Petrov
If I understand the problem correctly, tombstone_failure_theshold is never reached because the ~2M objects might have been collected for different queries running in parallel, not for one query. Every separate query never reached the threshold although all together they contributed to the OOM.