rajarshisarkar commented on PR #18655: URL: https://github.com/apache/iceberg/pull/18655#issuecomment-6523025543
Thanks @RussellSpitzer and @huan233usc for the feedback! I have updated the PR across Spark 3.5, 4.0, 4.1, and 4.2: 1. Resolved column names against the active scan schema (`expectedSchema()` in 3.5/4.0; `schema()` in 4.1/4.2) rather than the table snapshot schema. This correctly accounts for time travel and column renames where the snapshot ID remains unchanged. 2. Added a null check on `scanSchema.findColumnName(fieldId)` to safely skip fields that were dropped or are unrecognized in the scan schema. 3. Added unit tests in `TestSparkScan` across all 4 modules (`v3.5`, `v4.0`, `v4.1`, `v4.2`) covering column renaming, new column additions under the same snapshot, and dropped columns with CBO stats verification. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
