sam-1112 opened a new pull request, #6117:
URL: https://github.com/apache/datafusion-comet/pull/6117
## Which issue does this PR close?
Closes part of #5646 (native-to-JVM handoff cleanup).
## Rationale for this change
A native Iceberg writer can successfully return its task payload, then fail
on the JVM while decoding the manifest and building the task commit message. At
that point, no commit message reaches the driver, so the task's data files must
be deleted by its failure listener.
## What changes are included in this PR?
- Add a scoped test-only failpoint after `WrittenFileCleanup` owns native
`written_file_locations` and before manifest decoding.
- Add an end-to-end native Iceberg write regression test that injects a
post-native handoff failure.
- Verify all failed task attempts report non-empty native payload locations,
no snapshot is committed, and no failed-task Parquet file remains on disk or in
Iceberg metadata.
## How are these changes tested?
```bash
make core
./mvnw test -Pspark-3.5 -Dtest=none \
'-Dsuites=org.apache.comet.CometIcebergWriteActionSuite post-native
handoff failure'
./mvnw test -Dtest=none \
'-Dsuites=org.apache.comet.CometIcebergWriteActionSuite post-native
handoff failure'
./mvnw spotless:check -Pspark-3.5
```
The targeted regression passes on Spark 3.5 and the default Spark 4.1
profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
