ewoodbury opened a new pull request, #3255: URL: https://github.com/apache/iceberg-rust/pull/3255
## Which issue does this PR close? - Closes #2617 - Revives #2647 (original work by @jordepic) ## Rationale for this change When a nested struct (or a struct nested in a list/map) gains a field after some Parquet files were written, `RecordBatchTransformer` promoted the column with a positional Arrow cast. Children after the gap no longer line up, so the read fails (for example `Casting from Utf8 to Struct(...)`, or `expected 3 got 2` when the new field is appended). Iceberg-Java projects nested children by field id and fills missing optional fields with nulls. The files are valid. This also shows up in Apache DataFusion Comet 1.0's native Iceberg scan (Spark 4 + Iceberg): vanilla Spark reads the evolved table, Comet's iceberg-rust path errors on the old files. ## What changes are included in this PR? This is @jordepic's #2647 rebased onto current `main`, plus one extra unit test for the append case. Replace the flat cast with a `PromotePlan` built once per file: nested struct children are matched by `PARQUET:field_id`, recursing through list / large-list / map. Per-batch apply is index lookups and array assembly. - Absent optional children → typed nulls - Absent required children without a default → error - Absent children with `initial-default` → `FeatureUnsupported` (nested defaults are not wired yet; do not silently null) - Source struct with children but no field ids → error (do not silently drop data) - Primitive promotions still use `cast` Review feedback on #2647 is already incorporated (`PromotePlan` cached per file, shared field-id helper, `try_new`, tests for rename / nested int→long / null parent rows / end-to-end `process_record_batch`). The production change is about 250 lines in the transformer; most of the diff is tests from that review. This is a revival of the already-reviewed design, not a new approach. Out of scope (same as #2647): recursive name-mapping for id-less nested columns, and nested `initial-default` values. ## Are these changes tested? Yes. Existing #2647 tests, plus `promote_struct_fills_appended_field_by_id` for `struct<a,b>` → `struct<a,b,c>`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
