moomindani opened a new pull request, #3630: URL: https://github.com/apache/iceberg-python/pull/3630
Closes #3628 (part of #1818) # Rationale for this change The V3 spec requires readers to read tables with unknown transforms, ignoring them. PyIceberg raised (`Multi argument transforms are not yet supported`) on partition or sort fields with more than one entry in `source-ids`, so loading such tables failed entirely. - Model `source-ids` on `PartitionField` / `SortField` and treat multi-argument transforms as `UnknownTransform`, which already gives the spec behavior (no partition pruning: `project` returns None). - Serialize per spec: `source-id` only for single-argument transforms, `source-ids` only for multi-argument ones, so specs round-trip faithfully instead of being silently rewritten on the next commit. - Fix two latent tolerance bugs found along the way: unknown transform names sharing a prefix with known ones (e.g. `bucketv2[4]`) failed parsing instead of becoming `UnknownTransform`, and `str(UnknownTransform)` returned `"unknown"` instead of the original name, corrupting the transform name when metadata is rewritten. Evaluating multi-argument transforms on write stays out of scope until the spec defines concrete ones. Note on other implementations: none currently satisfies the spec's tolerance requirement in full. Java has not implemented `source-ids` (its parser requires `source-id`), though its transform parsing already matches names exactly and `UnknownTransform.toString()` preserves the original name — the two tolerance fixes here align PyIceberg with that behavior. iceberg-rust and iceberg-cpp reject unrecognized transform names outright. The `source-ids` handling here follows the spec text directly (partition/sort field JSON: single-argument transforms write only `source-id`, multi-argument only `source-ids`). ## Are these changes tested? Yes: eight new tests covering multi-argument parse tolerance (partition and sort), round-trip serialization (`source-ids` preserved, `source-id` omitted, transform name intact), single-element normalization, `partition_type` resolution, prefix-colliding unknown transform names, and `str(UnknownTransform)` name preservation. All fail without the fix. No new failures in `tests/table`, `tests/catalog`, `tests/utils`. ## Are there any user-facing changes? Yes: tables using multi-argument transforms now load and scan (without pruning on those fields) instead of raising. This pull request and its description were written by Claude Fable 5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
