[
https://issues.apache.org/jira/browse/CASSANDRA-18433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17709516#comment-17709516
]
Huapeng Yuan commented on CASSANDRA-18433:
------------------------------------------
>From source code, 3.11.14 should still have same issue. Sorry, I don't have
>such test to reproduce the issue. It is a race condition and depends on some
>timing. In our production system, it happened 1 or 2 times per hour.
The line has issue:
In SinglePartitionReadCommand::getThroughCache:
CacheService.instance.rowCache.replace(key, sentinel, toCache);
Suppose it should insert only if the sentinel exists and matches. Should not
insert if the sentinel had been deleted by invalidation from another thread.
> Row cache inconsistency issue: A read can put stale data into row cache in a
> race condition
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-18433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18433
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Caching
> Reporter: Huapeng Yuan
> Priority: Normal
> Fix For: 3.11.x
>
>
> We found the issue in our production system which has the version 3.11.6.
> When we did an update and then read immediately after update successfully, we
> may read the stale data sometimes. Same issue for writeAll + readOne
> consistency and writeQuorm+readQuorum. The issue is gone once we disabled the
> row cache.
> The config for row cache:
> caching = \{'keys': 'ALL', 'rows_per_partition': 'ALL'}
>
> After some investigations, we think there is a race condition during
> read/write path. Problems:
> When two threads are reading and writing the same partition (for example, two
> rows with same partition key) at same time, the read thread may load the
> stale data into row cache for the row which is being updated.
> {{}}
> {panel:title=The steps of write-thread inserting a row to partition p}
> {{W-Step }}{{{}1{}}}{{{}: inserts the value v1 to memtable.{}}}
> {{W-Step }}{{{}2{}}}{{{}: invalidates the row cache using partition key.{}}}
> {panel}
> {{}}
> {panel:title=The steps of read-thread reading a row from partition p}
> {{R-Step }}{{{}1{}}}{{{}: Checks row cache and finds whether the row is not
> present in cache. If not, goes to '{}}}{{{}R-Step {}}}{{{}2'{}}}{{{}.{}}}
> {{R-Step }}{{{}2{}}}{{{}: Insert a sentinel (timestamp) as the row value into
> row cache to tell other read threads should skip the row cache.{}}}
> {{R-Step }}{{{}3{}}}{{{}: Read from storage layer and get value v0 which can
> be older than v1.{}}}
> {{R-Step }}{{{}4{}}}{{{}: Insert v0 to row cache {}}}{{for}} {{the row by
> checking }}{{if}} {{the row doesn't exist or it has the same sentinel. *The
> inconsistency is caused by this step. Should not insert the stale value if
> the sentinel doesn't exist in row cache any more.*}}
> {panel}
> {{}}
> {panel:title=The sequence to reproduce the issue}
> {{R-Step }}{{1}}
> {{R-Step }}{{2}}
> {{R-Step }}{{3}}
> {{W-Step }}{{1}}
> {{W-Step }}{{2}}
> {{R-Step }}{{4}}
> {panel}
> {{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
