[
https://issues.apache.org/jira/browse/CASSANDRA-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456489#comment-17456489
]
Joey Lynch commented on CASSANDRA-14898:
----------------------------------------
*trunk* - [Java 11
CI|https://app.circleci.com/pipelines/github/jolynch/cassandra/67/workflows/6a98bc38-3021-4763-906b-f84099157ba0],
[Java 8
CI|https://app.circleci.com/pipelines/github/jolynch/cassandra/67/workflows/06b843ac-a0f9-4123-b14a-965dc8f22755]
4 dtest failures in:
* (J8) test_bootstrap_with_reset_bootstrap_state - bootstrap_test.TestBootstrap
* (J8) test_compression_cql_options - compression_test.TestCompression
* (J11) test_multiple_repair -
repair_tests.incremental_repair_test.TestIncRepair
* (J11, marked flakey) test_bootstrap_with_reset_bootstrap_state -
bootstrap_test.TestBootstrap
1 JVM dtest failure:
* (J8): readWriteDuringBootstrapTest -
org.apache.cassandra.distributed.test.ring.BootstrapTest
*4.0* - [Java 11
CI|https://app.circleci.com/pipelines/github/jolynch/cassandra/68/workflows/2fe689eb-1938-42fd-b5cb-abab32cf0d1f]
[Java 8
CI|https://app.circleci.com/pipelines/github/jolynch/cassandra/68/workflows/7929a6c9-cc9c-4c5e-b271-cc276fc1d9a1]
1 JVM dtest failure:
* (J8, J11) readWriteDuringBootstrapTest -
org.apache.cassandra.distributed.test.ring.BootstrapTest
*3.11* [Java 8
CI|https://app.circleci.com/pipelines/github/jolynch/cassandra/65/workflows/6e86d386-4057-4d83-aab1-4d3731023f5d]
1 JVM dtest failure:
* (J8) bootstrapTest - org.apache.cassandra.distributed.test.BootstrapTest
*3.0* [Java 8
CI|https://app.circleci.com/pipelines/github/jolynch/cassandra/64/workflows/5f7095bf-5031-465f-9d82-d574991e21ac]
1 JVM dtest failure:
* (J8) bootstrapTest - org.apache.cassandra.distributed.test.BootstrapTest
Looking at the tests that failed I don't think any of these look related to the
change. A local run of IreadWriteDuringBootstrapTest passes on 4.0 and I'm
running a trunk
[CI|https://app.circleci.com/pipelines/github/jolynch/cassandra/69/workflows/58cdf28a-8e13-4920-94a2-870d0fe00790]
run to see if readWriteDuringBootstrapTest flakes on trunk as well without the
patch.
I believe this is ready to commit.
> Key cache loading is very slow when there are many SSTables
> -----------------------------------------------------------
>
> Key: CASSANDRA-14898
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14898
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
> Environment: AWS i3.2xlarge, 4 physical cores (8 threads), 60GB of
> RAM, loading about 8MB of KeyCache with 10k keys in it.
> Reporter: Joey Lynch
> Assignee: Venkata Harikrishna Nukala
> Priority: Low
> Labels: Performance, low-hanging-fruit
> Attachments: key_cache_load_slow.svg
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> While dealing with a production issue today where some 3.0.17 nodes had close
> to ~8k sstables on disk due to excessive write pressure, we had a few nodes
> crash due to OOM and then they took close to 17 minutes to load the key cache
> and recover. This excessive key cache load significantly increased the
> duration of the outage (to mitigate we just removed the saved key cache
> files). For example here is one example taking 17 minutes to load 10k keys,
> or about 10 keys per second (which is ... very slow):
> {noformat}
> INFO [pool-3-thread-1] 2018-11-15 21:50:21,885 AutoSavingCache.java:190 -
> reading saved cache /mnt/data/cassandra/saved_caches/KeyCache-d.db
> INFO [pool-3-thread-1] 2018-11-15 22:07:16,490 AutoSavingCache.java:166 -
> Completed loading (1014606 ms; 10103 keys) KeyCache cache
> {noformat}
> I've witnessed similar behavior in the past with large LCS clusters, and
> indeed it appears that any time the number of sstables is large, KeyCache
> loading takes a _really_ long time. Today I got a flame graph and I believe
> that I found the issue and I think it's reasonably easy to fix. From what I
> can tell the {{KeyCacheSerializer::deserialize}} [method
> |https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L445]
> which is called for every key is linear in the number of sstables due to the
> [call|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L459]
> to {{ColumnFamilyStore::getSSTables}} which ends up calling {{View::select}}
> [here|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/db/lifecycle/View.java#L139].
> The {{View::select}} call is linear in the number of sstables and causes a
> _lot_ of {{HashSet}}
> [resizing|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/db/lifecycle/View.java#L139]
> when the number of sstables is much greater than 16 (the default size of the
> backing {{HashMap}}).
> As we see in the attached flamegraph we spend 50% of our CPU time in these
> {{getSSTable}} calls, of which 36% is spent adding sstables to the HashSet in
> {{View::select}} and 17% is spent just iterating the sstables in the first
> place. A full 16% of CPU time is spent _just resizing the HashMap_. Then
> another 4% is spend calling {{CacheService::findDesc}} which does [a linear
> search|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/service/CacheService.java#L475]
> for the sstable generation.
> I believe that this affects at least Cassandra 3.0.17 and trunk, and could be
> pretty easily fixed by either caching the getSSTables call or at the very
> least pre-sizing the {{HashSet}} in {{View::select}} to be the size of the
> sstables map.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
