sam-1112 opened a new pull request, #6601:
URL: https://github.com/apache/datafusion-comet/pull/6601
## Which issue does this PR close?
Close part of #5646(speculative execution and executor-loss scenarios for
native Iceberg writes).
## Rationale for this change
Local write tests do not exercise Spark's selection of speculative task
results or recovery after an executor is lost during a native write.
These tests assert both the visible table outcome and storage state,
including files that remain unreferenced and are therefore invisible to readers.
## What changes are included in this PR?
- Add a manual multi-worker integration suite covering:
- Original-attempt and speculative-attempt winner directions.
- Executor-loss task replacement.
- Shuffle-output-loss stage re-execution.
- Add explicitly scoped JVM/native probes with bounded gates and atomic
per-attempt markers. Executor termination occurs after native write progress
and physical file creation, before writer close or payload handoff.
- Record scheduler events, driver-accepted task-message file sets, and
successful commit markers.
- Assert the exact 48,000-row ID multiset, one new snapshot, manifest
equality with accepted winner files, and exclusion of rejected-attempt files
from manifest references.
- Compare physical files with manifest references and export orphan/missing
file audits. Executor-loss scenarios allow only orphans belonging to known
rejected attempts; unknown orphans and missing referenced files fail.
Speculation retains strict cleanup assertions.
- Add a local probe smoke test, a Linux/SSH executor-termination helper, and
documentation for the external-cluster requirements.
- Exclude the manual integration suite from ordinary single-host CI
discovery.
This PR adds test instrumentation and coverage; it does not implement an
orphan-cleanup mechanism.
## How are these changes tested?
Tests ran on Linux ARM64 through Docker Desktop on a Mac mini M4, using
Spark 4.1.3, Java 17, Iceberg 1.11.0, two Spark Standalone workers, and a
shared Docker volume.
- Both speculative winner-direction tests passed (2 passed, 0 failed).
- Both executor-loss tests passed (2 passed, 0 failed). Task replacement
recovered on another executor; stage re-execution recorded `FetchFailed` and an
increased writer stage attempt.
- The mid-write retry, post-native cleanup, commit-failure, post-native
retry-success, and scheduler-probe smoke tests each passed locally.
- The executor-loss tests verified rows, snapshot count, accepted files
against manifest references, and isolation of rejected-attempt files. Each
storage audit found one known rejected-attempt orphan, zero unknown orphans,
and zero missing referenced files. Storage evidence was captured before fixture
teardown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
