[
https://issues.apache.org/jira/browse/CASSANDRA-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mykyta Bozhenko updated CASSANDRA-19569:
----------------------------------------
Test and Documentation Plan:
SSTableRewriterTest.testOfflineCacheSkippingAcrossWriterSwitches: Byteman
records every range passed to NativeLibrary.trySkipCache while an offline
rewriter appends across three writers, and asserts one advice per preemptive
open interval. Without the fix, it fails with an extra advice on the partition
after the interval is first exceeded. Full SSTableRewriterTest (22 tests)
passes.
Throughput measured on the same offline path sstableupgrade uses
(LifecycleTransaction.offline plus SSTableRewriter, 50 MiB interval, ext4, JDK
11): 124 MB of ~1 KB partitions takes 6980 ms (17.0 MB/s) before and 1894 ms
(62.6 MB/s) after, median of three runs. With ~84 B partitions, the same
comparison is 5.8 MB/s before and 18.0 MB/s after, i.e. the cost is per
partition, not per byte. bpftrace on the 124 MB case: 69,394 fadvise64 calls
before versus 3 after, and 111,048 openat calls before versus 35,251, because
trySkipCache(String, long, long) reopens the data file on every call. Only 253
ms of the ~5 s difference is spent inside fadvise64 itself.
No documentation change.
> sstableupgrade is very slow
> ---------------------------
>
> Key: CASSANDRA-19569
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19569
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Compaction
> Reporter: Norbert Schultz
> Assignee: Mykyta Bozhenko
> Priority: Normal
> Fix For: 4.1.x
>
> Attachments: flamegraph_ok.png, flamegraph_sstableupgrade.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We are in the process of migrating cassandra from 3.11.x to 4.1.4 and
> upgrading the sstables using sstableupgrade from Cassandra V4.1.4, from `me-`
> to `nb-` Format
> Unfortunately, the process is very very slow (less than 0.5 MB/s).
> Some observations:
> - The process is only slow on (fast) SSDs, but not on ram disks.
> - The sstables consist of many partitions (this may be unrelated)
> - The upgrade process is fast, if we use `automatic_sstable_upgrade` instead
> of the sstableupgradetool.
> - We give enough RAM (export MAX_HEAP_SIZE=8g)
> On profiling, we found out, that sstableupgrade is burning most CPU time on
> {{posix_fadvise}} (see flamegraph_sstableupgrade.png ).
> My naive interpretation of the whole {{maybeReopenEarly}} to
> {{posix_fadvise}} chain is, that the process just informs the linux kernel,
> that the written data should not be cached. If we comment out the call to
> {{NativeLibrary.trySkipCache}}, the conversion is running at expected 10MB/s
> (see flamegraph_ok.png )
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
