moomindani commented on PR #17765: URL: https://github.com/apache/iceberg/pull/17765#issuecomment-5377460321
@huaxingao @szehon-ho would one of you be able to take a look? This revives #15594, which the stale bot closed in May after ten weeks without a review; GitHub offers no reopen on that one, so this is a fresh PR from the same branch. It is rebased onto current `main`, the `spark/v3.4` copies are dropped since #14122 removed Spark 3.4 support, and all eight `spark-tests` jobs are green (3.5 × 2.12/2.13, 4.0, 4.1, core and extensions). The parts that most need a reviewer's judgment are the SCD Type-2 semantics rather than the plumbing: - `NULL` as the open-ended sentinel for `_valid_to`, rather than `9999-12-31` - `_valid_from` / `_valid_to` naming, following dbt and Kimball rather than inventing Iceberg-specific names - dropping `UPDATE_BEFORE` rows, which are intermediate artifacts in an SCD Type-2 view - keeping DELETE rows with `_is_current = false`, so hard deletes stay visible If any of those should go the other way, I am happy to change them. The full rationale and a demo output are in the PR description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
