huan233usc opened a new pull request, #18896: URL: https://github.com/apache/iceberg/pull/18896
## Summary Continues #17953 on the rebased branch. Expose Iceberg copy-on-write changes through Spark 4.2's native SQL and DataFrame CDC APIs. Commit versions are Iceberg snapshot sequence numbers. Format v2+ supports raw changes; row-lineage post-processing removes carryovers, computes update images, and collapses net changes. Explicit `identifier-columns` enables batch business-key processing without requiring row lineage. Rebased onto Apache Iceberg main (`83bf0ffab`), retaining the merged initial-offset persistence and AvailableNow fixes. This update adds end-to-end option coverage and fixes two inconsistencies: - Reject ordinary time travel, branch, legacy incremental-range, and ordinary streaming-filter options instead of silently ignoring them; reject reversed CDC ranges. - Preserve nested field metadata through business-key row encoding so SQL and DataFrame reads expose the same complete schema. ## Validation Spark 4.2 targeted suites: **143 passed, 3 skipped, 0 failures** (146 discovered). The skipped cases are the non-session catalog slots for a session-only fallback test. - `TestSparkChangelog`: 32 cases, including the complete native option matrix. - `TestNativeChangelogCatalogs`: Hive, Hadoop, REST, and SparkSessionCatalog batch/streaming integration and session fallback. - `TestBusinessKeyChangelog`: 72 cases across all four catalogs, with strict full-schema and output parity. - `TestSparkChangelogOptionValidation`: SQL, DataFrame batch, and DataFrame streaming reject 13 incompatible options. - `TestChangelogReader`, `TestStreamingInitialOffsetStore`, and 18 AvailableNow regressions from `TestStructuredStreamingRead3`. - Spark 4.2 core/extensions Checkstyle main/test and Spotless checks. The streaming `netChanges + computeUpdates=true` case is verified by a separately named E2E test using the explicit Spark 4.2.0 workaround below; the other streaming combinations use the default empty-collection setting. The E2E matrix checks all three deduplication modes with both `computeUpdates` values (including rejected combinations), defaults, version/timestamp bounds and inclusivity, microsecond precision, SQL/DataFrame schemas and full results, real AvailableNow streaming, four catalog implementations, and explicit option/key errors. Expected events come from fixed insert/update/delete histories, including rewrite carryovers and transient rows. ## Scope and limitations Delete-file/MOR changelogs remain separate work. Business-key processing supports batch `dropCarryovers` only; without update computation, grouping values must be orderable. Streaming post-processing rejects commit timestamp sequences that can lose changes to Spark's watermark. Streaming `netChanges` requires RocksDB state storage and applies to buffered changes rather than retroactively collapsing previously emitted micro-batches. On Spark 4.2.0, streaming `netChanges + computeUpdates=true` additionally requires `spark.sql.legacy.createEmptyCollectionUsingStringType=true` to work around an analyzer type-resolution bug; this changes empty collection literal inference for the session. Iceberg does not enable it implicitly. The documentation describes enabling it explicitly and restoring it afterward. The option matrix and these contracts are documented in `docs/docs/spark-queries.md`. Related to #17949. The initial-offset persistence preparation was merged as #18092. --- **AI Disclosure** - Model: Grok (original implementation); GPT-6 (rebase, fixes, and E2E coverage) - Platform/Tool: Cursor (original implementation); Codex - Human Oversight: unreviewed for the latest agent changes - Prompt Summary: Reopen and rebase this PR, and verify every CDC option and supported combination with end-to-end tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
