allthingssecurity opened a new pull request, #18763: URL: https://github.com/apache/iceberg/pull/18763
`SparkTableUtil.importSparkPartitions`, used by `add_files`, `snapshot` and `migrate`, deleted the staged manifests on any failure of the append commit. With snapshot ID inheritance (the default for v2 and later) those manifests are committed as they are, so a `CommitStateUnknownException` for a commit that actually succeeded left the table pointing at deleted manifests and every read failed. This rethrows `CommitStateUnknownException` without cleanup and deletes only on a `CleanableFailure`, the same handling as `RewriteManifestsSparkAction`. Backports to other Spark versions will follow separately. Reported in #13443 (closed by the stale bot; still reproducible on 1.12.0). Test plan: new `TestSparkTableUtilImportCommitFailure` commits the append and then throws `CommitStateUnknownException`. On main the committed snapshot references a deleted staged manifest; with this change all manifests exist and the rows read back. `TestRewriteManifestsAction`, `TestIdentityPartitionData`, `checkstyleMain` and `checkstyleTest` pass for Spark 4.2; the full Spark module suite was not run locally. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: fully reviewed - Prompt Summary: Reproduce failure-handling bugs in Iceberg's Spark integration against released jars and prepare a minimal fix with a regression test. The fix, the test and this description are generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
