amogh-jahagirdar opened a new pull request, #18531:
URL: https://github.com/apache/iceberg/pull/18531
Repro for a case where a Spark Streaming data file ends up with different
contents than the manifest says (record count, bounds).
Sequence of events:
1. Epoch N runs. Tasks write files named
{partitionId}-{taskId}-{queryId}-{epochId}-{fileCount}.parquet.
2. Iceberg commits the snapshot for epoch N, with the files and their stats
in the manifest.
3. The driver dies before Spark writes the checkpoint commit log entry for N.
4. Driver restarts on the same checkpoint. Offset log has N, commit log
doesn't, so Spark replays N.
5. taskAttemptId is a per-SparkContext counter that restarts from 0 on the
new driver. If a replay task lands on the same counter value as the original
(e.g. the first batch of a scheduled AvailableNow run), it gets the same
partitionId and taskId, so the same file name.
6. The replay can assign different rows to that partition (shuffle, AQE
coalescing), and data writers use overwrite, so the committed file at that path
is replaced.
7. commit(N) sees epochId <= lastCommittedEpochId and skips. Nothing is
cleaned up or flagged.
8. Table now points at a file whose rows don't match its manifest entry.
The test does 1, 2 and 4 to 6 directly against the streaming writer, then
checks the committed file's row count against the manifest. It fails on current
code. Fix not included.
---
**AI Disclosure**
- Model: Claude Sonnet 5.5
- Human Oversight: Iterated a bunch, had it search for cases where spark
streamign can still clobber files because that's the only way this could
possibly be SS specific (if it was how stats were actually collected from
footer and put into metadata this issue would be way more prevalent)
- Prompt Summary: Investigate Spark Streaming data file / manifest stats
mismatch and write a minimal repro test for the replayed epoch overwrite
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
