sundapeng opened a new pull request, #8862:
URL: https://github.com/apache/paimon/pull/8862
### Purpose
Part of #8860. Split out of #8861 because it stands on its own and touches
the write path rather than
the listing.
`RenamingTwoPhaseOutputStream` stages a file at `<target
dir>/_temporary/.tmp.<uuid>` and renames it
into place on commit. `TempFileCommitter#clean` then deleted
**`tempPath.getParent()`** — the whole
`_temporary` directory — after committing a single file.
That directory is not private to one stream. It is the conventional staging
directory of the target
directory, so deleting it outright throws away work that is still in flight:
- two concurrent writers into the same directory delete each other's pending
files, and the loser
fails at `rename` with `Failed to rename ...`;
- a Hadoop `FileOutputCommitter` job staging into the same directory loses
its whole task output —
data loss for that job, not just a failed commit.
`clean()` now deletes this stream's own temporary file, and removes
`_temporary` only once nothing is
staged there any more. In the ordinary single-writer case the directory is
empty by then and is
removed exactly as before, so nothing is left behind; when a concurrent job
is staging, its files
survive.
### Tests
`RenamingTwoPhaseOutputStreamTest`
- `testCleanKeepsTheSharedStagingDirectory`: another writer's pending file,
and the directory
holding it, survive `clean()`. Fails on master.
- `testCleanRemovesTheStagingDirectoryOnceEmpty`: a plain commit still
leaves no `_temporary`
behind.
### API and Format
No format change. `RenamingTwoPhaseOutputStream` is `@Public`, but the
signature and the
post-conditions of an uncontended commit are unchanged: only the case where
someone else is staging
in the same directory behaves differently.
### Documentation
None.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
