Thanks for the detailed answer Alexander.
We'll look into your suggestions, it's definitely helpful. We have plans
to reduce tombstones and remove the table with the big partitions,
hopefully after we've done that the cluster will be stable again.
Thanks again.
On Tue, Nov 22, 2016, at
Hi Vincent,
Here are a few pointers for disabling swap :
-
https://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html
-
http://stackoverflow.com/questions/22988824/why-swap-needs-to-be-turned-off-in-datastax-cassandra
Tombstones are definitely the kind of object
Thanks for your answer Alexander.
We're writing constantly to the table, we estimate it's something like
1.5k to 2k writes per second. Some of these requests update a bunch of
fields, some update fields + append something to a set.
We don't read constantly from it but when we do it's a lot of
Vincent,
only the 2.68GB partition is out of bounds here, all the others (<256MB)
shouldn't be much of a problem.
It could put pressure on your heap if it is often read and/or compacted.
But to answer your question about the 1% harming the cluster, a few big
partitions can definitely be a big
@Vladimir
We tried with 12Gb and 16Gb, the problem appeared eventually too.
In this particular cluster we have 143 tables across 2 keyspaces.
@Alexander
We have one table with a max partition of 2.68GB, one of 256 MB, a bunch
with the size varying between 10MB to 100MB ~. Then there's
Hi Vincent,
one of the usual causes of OOMs is very large partitions.
Could you check your nodetool cfstats output in search of large partitions
? If you find one (or more), run nodetool cfhistograms on those tables to
get a view of the partition sizes distribution.
Thanks
On Mon, Nov 21, 2016
Did you try any value in the range 8-20 (e.g. 60-70% of physical memory).
Also how many tables do you have across all keyspaces? Each table can consume
minimum 1M of Java heap.
Best regards, Vladimir Yudovin,
Winguzone - Hosted Cloud Cassandra
Launch your cluster in minutes.
On
