sundapeng opened a new pull request, #8900:
URL: https://github.com/apache/paimon/pull/8900
### Problem
`RenamingTwoPhaseOutputStream` stages into
`<target-dir>/_temporary/.tmp.<uuid>` — the same directory name Hadoop's
`FileOutputCommitter` uses, and with it Hive, Spark and MapReduce writing to
the same location. `clean()` removed that directory recursively:
```java
public void clean(FileIO fileIO) {
fileIO.deleteDirectoryQuietly(tempPath.getParent()); //
<target-dir>/_temporary
}
```
`clean()` runs after every successful commit, so a Paimon write into a
directory where another job is mid-write deleted that job's staged files, and
the job then failed to commit its own output.
### Fix
Delete only the file this committer staged, then ask whether the directory
can go by attempting a non-recursive delete: it removes the directory when
nothing is staged there any more, and refuses while a concurrent writer still
has files in it. Removing it stays best-effort — a writer that stages here next
recreates it — so a failure is logged at debug and the commit is unaffected.
### Scope
paimon-common only, and in practice this is a format-table path: the sole
production caller of `FileIO.newTwoPhaseOutputStream` is
`FormatTableSingleFileWriter`, so managed tables never construct this stream.
`OSSFileIO`, `S3FileIO` and `JindoFileIO` override it with multipart upload and
do not stage under `_temporary` at all.
This is one of three PRs splitting up the original change; the other two are
the format-table read path (#8861) and the `INSERT OVERWRITE` delete path, both
in paimon-core and both independent of this one.
### Tests
`RenamingTwoPhaseOutputStreamTest`:
- `testCleanKeepsTheSharedStagingDirectory` — a foreign file staged in the
same `_temporary` survives `clean()`; fails against the previous recursive
delete
- `testCleanRemovesTheStagingDirectoryOnceEmpty` — nothing else staged, so
the directory does go, and a caller listing the target directory does not trip
over a leftover `_temporary`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
