[
https://issues.apache.org/jira/browse/CASSANDRA-17273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caleb Rackliffe updated CASSANDRA-17273:
----------------------------------------
Reviewers: Caleb Rackliffe
> Lazy transaction log replica creation allows incorrect replica content
> divergence during compaction
> ---------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-17273
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17273
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair, Local/Compaction
> Reporter: Caleb Rackliffe
> Assignee: Marcus Eriksson
> Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0.x, 4.x
>
>
> Recently encountered this around compaction/anticompaction:
> {noformat}
> 2022-01-13 10:18:24,325 ERROR [main]
> org.apache.cassandra.db.lifecycle.LogTransaction - Unexpected disk state:
> failed to read transaction log
> [mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log in
> .../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f,
> .../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f]
> Files and contents follow:
> .../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log
>
> ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350438-big,0,8][2380834168]
>
> REMOVE:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350435-big,1642049328006,8][2338829485]
>
> REMOVE:[.../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350436-big,1642049366291,8][4248366924]
> COMMIT:[,0,0][2613697770]
> .../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log
>
> ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350437-big,0,8][4051162457]
> ***Does not match
> <ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350438-big,0,8][2380834168]>
> in first replica file
>
> ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350438-big,0,8][2380834168]
>
> REMOVE:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350435-big,1642049328006,8][2338829485]
>
> REMOVE:[.../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350436-big,1642049366291,8][4248366924]
> COMMIT:[,0,0][2613697770]
> {noformat}
> We have two data directories and two transaction log files, but one is
> missing an ADD entry when the contents of the two log replicas should be
> identical. One scenario that can cause this is the following:
> 1. Start anticompaction on a single file, in directory {{/tmp/d0}}.
> 2. Call {{trackNew()}} with 2 new files, both in a single directory, but in
> directory {{/tmp/d1}}. This initializes the log file in {{/tmp/d1}}, but
> there is still no log file in {{/tmp/d0}}.
> 3. Anticompaction only writes to one of the files in {{/tmp/d1}} (say all
> other keys were outside the repaired range).
> 4. When anticompaction is done, the empty writer is aborted and we call
> {{untrackNew()}}, which removes the added file from the registered log
> “records" (BUT NOT FROM DISK in {{/tmp/d1}}).
> 5. The REMOVE record is added. This references {{/tmp/d0}}. We lazily create
> the log file there by dumping all the records we have in memory to that file,
> which does not include the aborted SSTable above.
> 6. Now the log files contain:
> {noformat}
> /tmp/d1/logfile.log:
> ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-2-big,0,8][3268492367]
> ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-3-big,0,8][2813724425]
> REMOVE:[/tmp/d0/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-1-big,1642078019000,8][2401235379]
> COMMIT:[,0,0][2613697770]
> ** /tmp/d0/logfile.log:
> ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-3-big,0,8][2813724425]
> REMOVE:[/tmp/d0/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-1-big,1642078019000,8][2401235379]
> COMMIT:[,0,0][2613697770]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
