c-thiel commented on PR #2188:
URL: https://github.com/apache/iceberg-rust/pull/2188#issuecomment-4989081786
> 1. Could we attach the canonical Arrow extension name to variant fields in
ToArrowSchemaConverter::variant()? i.e. return the Struct{metadata, value}
field with ARROW:extension:name = arrow.parquet.variant (matches
parquet_variant_compute::VariantType::NAME in arrow-rs).
>
>
> Two reasons:
>
> Consumers can't distinguish a variant column from a regular
struct<metadata: binary, value: binary> in the converted schema. The extension
name is the standard signal downstream engines key on to decode the column as
variant instead of a plain struct. Consistency with file-derived schemas:
parquet-rs already attaches this extension when reading VARIANT-annotated
files, so schemas read from data files carry it while schema_to_arrow_schema
output doesn't. Anything that reconciles the two (e.g. RecordBatchTransformer)
ends up stripping the extension from emitted batches.
>
> 2. The shredded-variant rejection in collect_variant_leaves_in_field
is bypassed on the position-based fallback path.
>
>
> Trigger: a data file with no embedded Parquet field IDs (and no name
mapping assigning any), e.g. registered via add_files / migrated tables,
containing a shredded variant column (typed_value sub-field present).
**Extension name:** done — variant fields now get `ARROW:extension:name =
arrow.parquet.variant` (matches `parquet_variant_compute::VariantType::NAME`).
Attached at the field level in `field()` (not `variant()`, which only returns
the bare `Struct{metadata,value}`), so it covers top-level +
struct/list/map-nested variants, and `value` is nullable. So consumers can tell
it from a plain struct, and it lines up with what parquet-rs attaches to file
schemas.
**Shredded fallback bypass:** moot now — variant read is descoped.
`get_arrow_projection_mask` rejects anything touching a variant *before* the
fallback/field-id split, so both paths (incl. `add_files`/migrated, shredded or
not) error with `FeatureUnsupported`. No shredded-only guard left to slip past.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
