Caleb Rackliffe created CASSANDRA-17273:
-------------------------------------------
Summary: Lazy transaction log replica creation allows incorrect
replica content divergence during anticompaction
Key: CASSANDRA-17273
URL: https://issues.apache.org/jira/browse/CASSANDRA-17273
Project: Cassandra
Issue Type: Bug
Components: Consistency/Repair, Local/Compaction
Reporter: Caleb Rackliffe
Assignee: Caleb Rackliffe
Recently encountered this around compaction/anticompaction:
{noformat}
2022-01-13 10:18:24,325 ERROR [main]
org.apache.cassandra.db.lifecycle.LogTransaction - Unexpected disk state:
failed to read transaction log
[mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log in
.../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f,
.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f]
Files and contents follow:
.../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log
ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350438-big,0,8][2380834168]
REMOVE:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350435-big,1642049328006,8][2338829485]
REMOVE:[.../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350436-big,1642049366291,8][4248366924]
COMMIT:[,0,0][2613697770]
.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log
ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350437-big,0,8][4051162457]
***Does not match
<ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350438-big,0,8][2380834168]>
in first replica file
ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350438-big,0,8][2380834168]
REMOVE:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350435-big,1642049328006,8][2338829485]
REMOVE:[.../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350436-big,1642049366291,8][4248366924]
COMMIT:[,0,0][2613697770]
{noformat}
We have two data directories and two transaction log files, but one is missing
an ADD entry when the contents of the two log replicas should be identical. One
scenario that can cause this is the following:
1. Start anticompaction on a single file, in directory {{/tmp/d0}}.
2. Call {{trackNew()}} with 2 new files, both in a single directory, but in
directory {{/tmp/d1}}. This initializes the log file in {{/tmp/d1}}, but there
is still no log file in {{/tmp/d0}}.
3. Anticompaction only writes to one of the files in {{/tmp/d1}} (say all other
keys were outside the repaired range).
4. When anticompaction is done, the empty writer is aborted and we call
{{untrackNew()}}, which removes the added file from the registered log
“records" (BUT NOT FROM DISK in {{/tmp/d1}}).
5. The REMOVE record is added. This references {{/tmp/d0}}. We lazily create
the log file there by dumping all the records we have in memory to that file,
which does not include the aborted SSTable above.
6. Now the log files contain:
{noformat}
/tmp/d1/logfile.log:
ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-2-big,0,8][3268492367]
ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-3-big,0,8][2813724425]
REMOVE:[/tmp/d0/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-1-big,1642078019000,8][2401235379]
COMMIT:[,0,0][2613697770]
** /tmp/d0/logfile.log:
ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-3-big,0,8][2813724425]
REMOVE:[/tmp/d0/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-1-big,1642078019000,8][2401235379]
COMMIT:[,0,0][2613697770]
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
