[
https://issues.apache.org/jira/browse/CASSANDRA-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mykyta Bozhenko updated CASSANDRA-16047:
----------------------------------------
Test and Documentation Plan:
SSTableReaderTest.testIncrementalBackupPinsComponentsUntilLinked: a Mockito spy
releases the reader's own last reference the moment createLinks starts; then
every backed-up component is compared byte for byte with the original. Against
unpatched production, it fails with java.nio.file.NoSuchFileException on
backups/pa-1-big-Statistics.db, the same failure mode as the report; with the
patch it passes.
SSTableReaderTest.testCreateLinksSkipsRegenerableComponentButFailsOnMissingRequired:
a missing FILTER (regenerated on load) is skipped, and the remaining
components are linked, while a missing STATS throws UncheckedIOException with a
NoSuchFileException cause. Against unpatched production, the second half fails
("Expecting code to raise a throwable"), since today every missing component is
skipped.
Both tests were run on a clean full build (3278 sources, Java 11, Linux). No
documentation change.
> Potential race condition in creating hard link when incremental backup is
> turned on
> -----------------------------------------------------------------------------------
>
> Key: CASSANDRA-16047
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16047
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/SSTable
> Reporter: Wei Deng
> Assignee: Mykyta Bozhenko
> Priority: Urgent
> Attachments: incremental_backup_hardlink_exception.jpg,
> incremental_backup_hardlink_exception1.jpg
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It seems that there is a race condition in creating hard link if incremental
> backup is turned on.
> The following screenshot was captured in a production cluster running
> Cassandra 3.0.15 after turning on incremental backup. When this
> {{NoSuchFileException}} happens, due to the {{FSWriteError}} and the default
> disk failure policy, the JVM will be shutdown, so it's a pretty critical bug.
> !incremental_backup_hardlink_exception.jpg!
> Due to the risk of causing production database downtime (if similar issue
> happens on multiple nodes in a short time frame), and same exception causing
> JVM shutdown multiple times already, incremental backup had to be turned off
> for now, but this is not an ideal situation.
> !incremental_backup_hardlink_exception1.jpg!
> The deployment is on a public cloud environment with EBS-like disks that are
> backed by SSD with decent latency, throughput and IOPS, so it is hard to
> think the culprit being in the OS and IO layer. Based on the second
> screenshot above, this is a low flush traffic {{system.size_estimates}}
> table, so compaction of the source SSTable doesn't seem to be at play here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
