gudladona opened a new issue, #20657:
URL: https://github.com/apache/hudi/issues/20657
### Bug Description
**What happened:**
With the row writer (`hoodie.datasource.write.row.writer.enable` /
`hoodie.streamer.write.row.writer.enable`), a write task attempt that Spark
does not use can upload Parquet files after the commit has completed. They are
not in the commit's write stats, but a storage listing (no metadata table)
counts them as data of the completed instant, so readers see duplicate rows and
later table services pick them up. In our case clustering replaced one of them,
so the duplicates moved into the clustered files.
The write loop in `HoodieDatasetBulkInsertHelper` (master `c32b64e98a3a`,
lines 213-221):
```scala
try {
iter.foreach(writer.write)
} catch {
case t: Throwable =>
writer.abort()
throw t
} finally {
writer.close()
}
```
`BulkInsertDataInternalWriterHelper.abort()` is empty (line 192), so on the
failure path `finally` closes every open `HoodieRowCreateHandle`. That finishes
the Parquet file and, on S3A, uploads it. A task Spark kills therefore still
publishes its partial files.
This is reachable without speculation. After a fetch failure Spark resubmits
the stage, and the earlier attempt's tasks that are still running keep going.
When the job finishes, `DAGScheduler` cancels them with reason `Stage
finished`, without interrupting the thread unless the job sets
`spark.job.interruptOnCancel`, so a task only notices at its next
`InterruptibleIterator` check. By then the commit has run
`reconcileAgainstMarkers`. Its list of paths comes from markers, so it does
include the zombie's open file, but the file is not on storage yet:
`deleteFile` returns false, `exists` is false, and nothing is deleted. The
zombie then lands in `catch`, and `finally { writer.close() }` uploads the file.
The row writer gives each attempt a random file ID prefix
(`BulkInsertDataInternalWriterHelper`, `fileIdPrefix = UUID.randomUUID()`), so
the zombie's file is a separate file group with the commit's instant time:
**duplicate rows**, not a shadowed copy. The non-row-writer insert path assigns
file IDs on the driver, so a zombie there writes into the winner's file group.
A second gap, independent of the kill path: the row writer writes straight
to final paths and never asks the driver's `OutputCommitCoordinator` whether
its attempt may commit, as Spark's own writers do
(`SparkHadoopMapRedUtil.commitTask`). An attempt that finishes after another
attempt was used, or after its stage completed, keeps its files.
**What you expected:**
An attempt that is killed, fails, or is not the one the job used leaves no
data files: `abort()` deletes what the attempt wrote, and a finished attempt
checks with the driver before returning its write statuses.
**Steps to reproduce:**
1. Row-writer bulk insert into a COPY_ON_WRITE table, metadata table
disabled, on S3.
2. During the write stage, lose an executor holding map output (decommission
or pod deletion), so some write tasks fail with `FetchFailed` and the stage is
resubmitted.
3. A write task of the first stage attempt that did not need the lost blocks
keeps running. The job finishes with the resubmitted attempt, the commit
completes, and the old attempt is cancelled with `TaskKilled (Stage finished)`.
4. List the partition: the old attempt's file (write token
`<partition>-<stage>-<its task id>`) is on storage, written after the commit's
completion time, and is not in the commit's write stats. Reading the partition
returns its rows twice.
Driver log from one occurrence (Spark 3.5.5, Hudi 1.1.1 based):
```
12:36:25 WARN TaskSetManager: Lost task 427.0 in stage 659.0 (TID 150911)
...: FetchFailed(BlockManagerId(34, ...), shuffleId=164, ...)
... 16 write tasks of stage 659.0 fail the same way; stage 659 is
resubmitted
12:36:50 INFO HoodieTable: Removing duplicate files created due to task
retries before committing. Paths=[...,
<partition>/e02198ad-...-1_406-659-150895_<instant>.parquet, ...]
12:36:51 commit completes
12:38:32 WARN TaskSetManager: Lost task 406.0 in stage 659.0 (TID 150895)
...: TaskKilled (Stage finished)
12:38:33 the file e02198ad-...-1_406-659-150895_<instant>.parquet appears on
S3
```
### Proposed fix
Small, in the row writer plus one hook in `HoodieWrapperFileSystem`:
1. `abort()` deletes every file the attempt wrote, closed or open; `close()`
after `abort()` does nothing. The write loop calls `close()` only on success,
not in `finally`.
2. An open file is aborted, not closed, where the storage allows it. On S3A
closing is the upload, so close-then-delete would still publish the whole file
first and leave it visible if the executor dies, or a listing runs, between the
two. `HoodieWrapperFileSystem` finds the open stream, unwraps it to the
storage's own stream, and calls Hadoop's `Abortable.abort()` (3.3.1+) when the
stream reports `fs.capability.outputstream.abortable`, so the object is never
published; the Parquet writer is then closed only to release its buffers.
Streams that cannot be aborted (local, HDFS) are closed and deleted.
3. A file the abort could not delete is logged at ERROR as an orphan risk
and attached as a suppressed exception to the failure that caused the abort. A
metric cannot carry it: Spark drops accumulator updates from failed and killed
tasks.
4. After a successful write loop, ask the driver's `OutputCommitCoordinator`
whether the attempt may commit (once per task attempt: the coordinator refuses
a second request even from the same attempt). If refused, or if the driver
cannot be reached, throw `CommitDeniedException` and abort.
I have this with tests (an abort leaves no files, sorted and unsorted; an
abortable stream is aborted, not closed, and a non-abortable one is closed and
deleted; an attempt the driver does not authorize leaves no files) and can open
a PR against master.
Related, broader: #18395 (stage files in the marker directory and promote at
finalize), #18273 (same trigger, GCS delete/write race), PR #9922 (task-side
cleanup of partial files on abort, which changes
`BulkInsertDataInternalWriterHelper` but not the `finally { writer.close() }`
in `HoodieDatasetBulkInsertHelper`).
### Environment
**Hudi version:** 1.1.1-based; the same code is on master (`c32b64e98a3a`)
**Query engine:** Spark 3.5.5 on Kubernetes, S3 (S3A)
**Relevant configs:** `hoodie.datasource.write.operation=bulk_insert`, row
writer enabled, `hoodie.metadata.enable=false`, `spark.speculation=false`
(default)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]