Hi Bowen,
> "how many tables are being written frequently" - there are normally
about less then 10 tables being written concurrently.
Regarding the "num_token", unfortunately we can change it now since it
would require rebuilding all rings/nodes of the cluster. Actually we
used to add a ring with num_token: 4 in this cluster but later remove
it due to some other issue. We have started using num_token: 16 as
standard for any new clusters.
Thanks,
Jiayong Sun
On Monday, August 16, 2021, 02:46:24 AM PDT, Bowen Song <bo...@bso.ng>
wrote:
Hello Jiayong,
/>//There is only one major table taking 90% of writes. /
In your case, what matters isn't the 90% writes, but the remaining
10%. If the remaining 10% writes are spread over 100 different tables,
your Cassandra node will end up flushing over 600 times per minute.
Therefore, "how many tables are being written frequently" is still a
valid question, even vast majority of the writes are in just one table.
/> Currently we don't set the "commitlog_total_space_in_mb" value so
it should be using default of 8192 MB (8 GB). What do you suggest for
this parameter's value?/
For the sake of saving the time spent on troubleshooting, increase the
commitlog_total_space_in_mb as much as your free disk space allows,
and definitely no less than 32GB (that's only ~100 minutes worth of
commit log). If your free disk space is very large, set it to ~230GB
(~12 hour worth of commit log) is a good starting point.
/> Since our reaper repair is not working effectively as now, we still
need to rely on the hints to reply writes cross nodes and DC. Do you
have any suggestion for how old hints could be safely removed without
impacting data consistency? I think this question may be depending on
many factors but I was wondering if there is any kind of rule of thumb?/
Unfortunately, there is no safe way to remove not-yet-delivered
hintswithout impacting data consistency if you don't run repair
periodically. Cassandra will remove the delivered hints automatically,
so you are only removing the not-yet-delivered hints if you are
manually removing them. My recommendation is to reduce the num_tokens
in all nodes in order to "fix" the repair. This, of course, will
require a lots of planning, and the actual process will involve moving
data around servers for quiet a few times (or a large number of new
servers, depending on your plan).
Cheers,
Bowen
On 16/08/2021 05:48, Jiayong Sun wrote:
Hi Bowen,
There is only one major table taking 90% of writes.
I will try to increase the "commitlog_segment_size_in_mb" value to 1
GB and set "max_mutation_size_in_kb" to 16MB.
Currently we don't set the "commitlog_total_space_in_mb" value so it
should be using default of 8192 MB (8 GB). What do you suggest for
this parameter's value?
By the way we don't back up the commit log.
Since our reaper repair is not working effectively as now, we still
need to rely on the hints to reply writes cross nodes and DC. Do you
have any suggestion for how old hints could be safely removed without
impacting data consistency? I think this question may be depending on
many factors but I was wondering if there is any kind of rule of thumb?
Thanks,
Jiayong Sun
On Sunday, August 15, 2021, 05:58:11 AM PDT, Bowen Song <bo...@bso.ng>
<mailto:bo...@bso.ng> wrote:
Hi Jiayong,
Based on this statement:
/> //We see the commit logs switched about 10 times per minutes/
I'd also like to know, roughly speaking, how many tables in Cassandra
are being written frequently? I'm asking this because the commit log
segments are being created (and recycled) so frequently (~ every 6
seconds), and I suspect that a lots of tables are involved in each
commit log segment, and that leads to many SSTable flushes.
You could try to drain a node, and then remove all commit logs from
the node and increase the "commitlog_segment_size_in_mb" value to
something much larger (say, 1GB), and also increase the
commitlog_total_space_in_mb accordingly on this node, and see if this
helps improving the situation. Note that you may also want to manually
set the "max_mutation_size_in_kb" to 16MB (the default value is half
of the commit segment size) to prevent unexpected extra large sized
mutations get accepted on this node and then failing on other nodes.
Please also note that this may interfere with some backup tools which
backs up the commit log segments.
In addition to that, if you periodically purge the hints, you probably
are better off by just disabling hinted handoff and make sure you
always run repair within the gc_grace_seconds.
Cheers,
Bowen
On 14/08/2021 03:33, Jiayong Sun wrote:
Hi Bowen,
Thanks for digging into source code so deep.
Here are answers to your questions:
* Does your application changes the table schema frequently?
Specifically: alter table, drop table and drop keyspace. - No,
either admin or apps doesn't frequently alter/drop/create table
schema in run-time.
* Do you have the memtable_flush_period_in_ms table property set to
non-zero on any table at all? - all tables use
"memtable_flush_period_in_ms = 0" for default.
* Is the timing of frequent small SSTable flushes coincident with
streaming activities? - The repair job is paused and don't see
streaming occurring in system.log
* What's your commitlog_segment_size_in_mb and
commitlog_total_space_in_mb in cassandra.yaml and what's your free
space size on the disk where commit log is located? -
"commitlog_segment_size_in_mb: 32". "commitlog_total_space_in_mb"
is not set. The commit logs have separate disk and no chance it'd
be filled up. Data disk is about 50% used.
* How fast do you fill up a commit log segment? I.e.: how fast are
you writing to Cassandra? - We see the commit logs switched about
10 times per minutes, and lots of hints cumulated on disk and
replayed constantly. This could be due to many nodes unresponsive
due to this ongoing issue.
* Anything invoking the "nodetool flush" or "nodetool drain"
command? - no, we don't issue these commands unless restarting a
node manually or automatically through some monitoring mechanism
which is not happening frequesntly.
I doubt if this could be related with huge amount of hints replay. The
cluster is stressed by heave writes through spark jobs and there are
some hot-spot partitions. Huge amount of hints (e.g. >50GB per node)
are not uncommon in this cluster especially since this issue has been
occurring causing many node lost gossip. We have to set up a daily
cron job to clear the older hints from disk, but not sure if this
would hurt data inconsistency among nodes and DCs.
Thoughts?
Thanks,
Jiayong Sun
On Friday, August 13, 2021, 03:39:44 PM PDT, Bowen Song <bo...@bso.ng>
<mailto:bo...@bso.ng> wrote:
Hi Jiayong,
I'm sorry to hear that. I did not know many nodes were/are
experiencing the same issue. A bit of dig in the source code indicates
the log below comes from the ColumnFamilyStore.logFlush() method.
DEBUG [NativePoolCleaner] <timestamp> ColumnFamilyStore.java:932 -
Enqueuing flush of sstable_activity: 0.408KiB (0%) on-heap,
0.154KiB (0%) off-heap
The ColumnFamilyStore.logFlush() method is a private method and the
only place referencing to it is the ColumnFamilyStore.switchmemtable()
method in the same file, and that has been referenced in two places -
ColumnFamilyStore.reload() and
ColumnFamilyStore.switchMemtableIfCurrent(). Ignoring secondary index
and MV, the former is only being called during the node starts up and
on table schema changes, and it's unlikely our suspect (unless you are
frequently changing the schema or restarting the node). The later is
being referenced in two methods: ColumnFamilyStore.forceFlush() and
ColumnFamilyStore.FlushLargestColumnFamily.run(). The
ColumnFamilyStore.FlushLargestColumnFamily.run() method is only called
by the MemtableCleanerThread, and we have pretty much ruled that out
in the previous conversations. The forceFlush() method is invoked if
the table property memtable_flush_period_in_ms is set, when Cassandra
is preparing for sending/receiving files via streaming, when an old
segment of commit log is recycled,on "nodetool drain", and again, on
schema changes (drop keyspace/table).
So, my questions would be:
* Does your application changes the table schema frequently?
Specifically: alter table, drop table and drop keyspace.
* Do you have the memtable_flush_period_in_ms table property set to
non-zero on any table at all?
* Is the timing of frequent small SSTable flushes coincident with
streaming activities?
* What's your commitlog_segment_size_in_mb and
commitlog_total_space_in_mb in cassandra.yaml and what's your free
space size on the disk where commit log is located?
* How fast do you fill up a commit log segment? I.e.: how fast are
you writing to Cassandra?
* Anything invoking the "nodetool flush" or "nodetool drain" command?
I hope the above questions will help you find the root cause.
Cheers,
Bowen
On 13/08/2021 22:47, Jiayong Sun wrote:
Hi Bowen,
There are many nodes having this issue and some of them repeatedly
having it.
Replacing a node by wiping out everything and streaming in good shape
of sstables would work, but if we don't know the root cause the node
would be in the bad shape again.
Yes, we know the reaper repair running so long like weeks is not good
which most likely due to the multiple DC with large size of rings. We
are planning to upgrade to newer version of reaper to see if that helps.
We do have debug.log turned on but didn't catch anything helpful other
than those constant enqueuing/flashing/deleting of memtable and
sstables (I listed a few examples messages at beginning of this email
thread).
Thanks for all your thoughts and I really appreciate.
Thanks,
Jiayong Sun
On Friday, August 13, 2021, 01:36:21 PM PDT, Bowen Song <bo...@bso.ng>
<mailto:bo...@bso.ng> wrote:
Hi Jiayong,
That doesn't really match the situation described in the SO question.
I suspected it was related to repairing a table with MV and large
partitions, but based on the information you've given, I was clearly
wrong.
A few hundreds MB partitions is not exactly unusual, I don't see that
alone could lead to frequent SSTable flushing. A repair session takes
weeks to complete is a bit worrying in terms of performance and
maintainability, but again it should not cause this issue.
Since we don't know the cause of it, I can see two possible solutions
- either replace the "broken" node, or dig into the logs (remember to
turn on the debug logs) and trying to identify the root cause. I
personally would recommend replacing the problematic node as a quick win.
Cheers,
Bowen
On 13/08/2021 20:31, Jiayong Sun wrote:
Hi Bowen,
We do have reaper repair job scheduled periodically and it can take
days even weeks to complete one round of repair due to large number of
rings/nodes. However, we have paused the repair since we are facing
this issue.
We do not use the MV in this cluster.
There is major table taking 95% of disk storage and workload but its
Partition Size is around 30 MB. There are a couple small tables with
the Max Partition Size over several hundreds of MB but their total
data size just about a few GB.
Any thoughts?
Thanks,
Jiayong
On Friday, August 13, 2021, 03:32:45 AM PDT, Bowen Song <bo...@bso.ng>
<mailto:bo...@bso.ng> wrote:
Hi Jiayong,
Sorry I didn't make it clear in my previous email. When I commented on
the RAID0 setup, it was only a comment on the RAID0 setup vs JBOD, and
that was not in relation to the SSTable flushing issue. The part of my
previous email after the "On the frequent SSTable flush issue" line is
the part related to the SSTable flushing issue, and those two
questions at the end of it remain valid:
* Did you run repair?
* Do you use materialized views?
and, if I may, I'd also like to add another question:
* Do you have large (> 100 MB) partitions?
Those are the 3 things mentioned in the SO question. I'm trying to
find the connections between the issue you are experiencing and the
issue described in the SO question.
Cheers,
Bowen
On 13/08/2021 01:36, Jiayong Sun wrote:
Hello Bowen,
Thanks for your response.
Yes, we are aware of the theory that RAID0 vs individual JBOD, but all
of our clusters are using this RAID0 configuration through Azure,
while only on this cluster we see this issue so it's hardly to
conclude root cause to the disk. This is more like workload related,
and we are seeking feedback here for any other parameters in the yaml
that we could tune for this.
Thanks again,
Jiayong Sun
On Thursday, August 12, 2021, 04:55:51 AM PDT, Bowen Song
<bo...@bso.ng> <mailto:bo...@bso.ng> wrote:
Hello Jiayong,
Using multiple disks in a RAID0 for Cassandra data directory is not
recommended. You will get better fault tolerance and often better
performance too with multiple data directories, one on each disk.
If you stick with RAID0, it's not 4 disks, it's 1 from Cassandra's
point of view, because any read or write operation will have to touch
all 4 member disks. Therefore, 4 flush writers doesn't make much sense.
On the frequent SSTable flush issue, a quick internet search leads me to:
* an old bug in Cassandra 2.1 - CASSANDRA-8409
<https://issues.apache.org/jira/browse/CASSANDRA-8409> which
shouldn't affect 3.x at all
* a StackOverflow question
<https://stackoverflow.com/questions/61030392/cassandra-node-jvm-hang-during-node-repair-a-table-with-materialized-view>
may be related
Did you run repair? Do you use materialized views?
Regards,
Bowen
On 11/08/2021 15:58, Jiayong Sun wrote:
Hi Erick,
The nodes have 4 SSD (1TB for each but we only use 2.4TB of space.
Current disk usage is about 50%) with RAID0.
Based on number of disks we increased memtable_flush_writers: 4
instead of default of 2.
For the following we set:
- max heap size - 31GB
- memtable_heap_space_in_mb (use default)
- memtable_offheap_space_in_mb (use default)
In the logs, we also noticed system.sstable_activity table has
hundreds of MB or GB of data and constantly flushing:
DEBUG [NativePoolCleaner] <timestamp> ColumnFamilyStore.java:932 -
Enqueuing flush of sstable_activity: 0.293KiB (0%) on-heap, 0.107KiB
(0%) off-heap
DEBUG [NonPeriodicTasks:1] <timestamp> SSTable.java:105 - Deleting
sstable:
/app/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/md-103645-big
DEBUG [NativePoolCleaner] <timestamp> ColumnFamilyStore.java:1322 -
Flushing largest CFS(Keyspace='system',
ColumnFamily='sstable_activity') to free up room. Used total:
0.06/1.00, live: 0.00/0.00, flushing: 0.02/0.29, this: 0.00/0.00
Thanks,
Jiayong Sun
On Wednesday, August 11, 2021, 12:06:27 AM PDT, Erick Ramirez
<erick.rami...@datastax.com> <mailto:erick.rami...@datastax.com> wrote:
4 flush writers isn't bad since the default is 2. It doesn't make a
difference if you have fast disks (like NVMe SSDs) because only 1
thread gets used.
But if flushes are slow, the work gets distributed to 4 flush writers
so you end up with smaller flush sizes although it's difficult to tell
how tiny the SSTables would be without analysing the logs and overall
performance of your cluster.
Was there a specific reason you decided to bump it up to 4? I'm just
trying to get a sense of why you did it since it might provide some
clues. Out of curiosity, what do you have set for the following?
- max heap size
- memtable_heap_space_in_mb
- memtable_offheap_space_in_mb
