[
https://issues.apache.org/jira/browse/CASSANDRA-21681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116809#comment-18116809
]
Yifan Cai commented on CASSANDRA-21681:
---------------------------------------
+1
> Compression dictionary distribution can silently leave a node on a stale
> dictionary
> -----------------------------------------------------------------------------------
>
> Key: CASSANDRA-21681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21681
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Feature/Compression
> Reporter: Stefan Miklosovic
> Assignee: Stefan Miklosovic
> Priority: Normal
> Fix For: 6.x, 7.x
>
> Attachments: screenshot-1.png
>
>
> h2. Problem
>
> Dictionaries live in \{{system_distributed.compression_dictionaries}} and
> propagate via a
> fire-and-forget \{{CompressionDictionaryUpdateMessage}}. Every read/write used
> {\{ConsistencyLevel.ONE}} and every failure along the path is silent: the
> write may land on one replica ahead of the (unordered) notification; a
> recipient reading at ONE can hit a replica without the row yet and get
> nothing; \{{retrieveCompressionDictionary}} swallows exceptions as \{{null}}
> unlogged; and \{{CompressionDictionaryEventHandler}} feeds that \{{null}} to
> \{{cache.add()}} (a no-op).
>
> The node then keeps its old dictionary, silently, until the periodic refresh
> (\{{compression_dictionary_refresh_interval}}, default 3600s) - so after a
> workload shift it can write an hour of SSTables with a stale dictionary while
> its peers have moved on
>
> h2. Changes
>
> # *ONE -> QUORUM* for dictionary store/retrieve/delete/list in
> \{{SystemDistributedKeyspace}}: a QUORUM write cannot be missed by a later
> QUORUM read, removing the race rather than papering over
> it. (View-build queries and the \{{ANY}} write elsewhere are deliberately
> untouched.)
> # *Failed reads are logged*: the retrieves now WARN on a genuine read failure
> - otherwise an unavailable replica is indistinguishable from a table with no
> dictionary.
> # *No \{{null}} to the cache*: on notify-but-can't-read,
> \{{CompressionDictionaryEventHandler}} WARNs
> and returns instead of calling \{{cache.add(null)}}, stating that the node
> keeps its current
> dictionary until the next refresh.
>
> h2. Empty result != failure
>
> {\{UntypedResultSet.one()}} throws on an empty result set, which is the
> normal state for any table whose dictionary has not been stored yet. Routing
> that through (2)'s catch would fire the WARN on every refresh/eval tick and
> bury the real failures it exists to surface (measured: 7 spurious
> \{{IllegalStateException}} warnings in 36 min on one table). The retrieves
> therefore check \{{isEmpty()}} and return \{{null}} on the normal path,
> leaving the catch for genuine failures.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
