Zerolzj opened a new pull request, #10582:
URL: https://github.com/apache/paimon/pull/10582
### Purpose
Prevent commit abort from deleting compaction input files that are reused by
a level upgrade.
A compaction can return the same physical file in both `compactBefore` and
`compactAfter`, with only its level changed. `FileStoreCommitImpl.abort()`
currently deletes every `compactAfter` file, including these reused files. This
can remove data that is still referenced by an existing snapshot.
Exclude compaction outputs whose resolved physical paths also occur in
`compactBefore`. Newly generated compaction outputs and their extra files are
still cleaned up. Comparing resolved paths also handles external data files,
including files with the same name in different directories.
The extra-file cleanup from #9586 is retained unchanged. This patch does not
add a file format or change the writer API.
### Tests
Add a parameterized regression test for internal and external reused files.
Each case checks that:
- The reused main file and its extra file survive abort, including a
repeated abort.
- A newly generated compaction output and its extra file are deleted.
Before the production change, both parameterized cases failed because the
reused file was deleted (2 failures, 0 errors).
After the change, the following command passed on JDK 17 (117 tests, 0
failures/errors/skips), including Checkstyle and Spotless checks:
```shell
mvn -B -ntp -Pflink1 -pl paimon-core -am \
-DwildcardSuites=none \
-Dtest=FileStoreCommitTest,SingleFileWriterTest,PrimaryKeyBlobFileWriterTest \
-DfailIfNoTests=false package
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
