raghav-reglobe opened a new pull request, #2683:
URL: https://github.com/apache/iceberg-rust/pull/2683
## What
When reading a manifest, iceberg-rust parses the table schema and partition
spec from that manifest's *own* `schema` / `partition-spec` Avro key-value
metadata and uses them to decode entries — hard-failing if `schema` is not a
valid Iceberg schema. This PR derives the schema + spec from the authoritative
**table metadata** (by the manifest's `schema-id` / `partition-spec-id`)
instead, falling back to the manifest's own keys when no table metadata is
available.
## Why
1. **Redundant + brittle.** The scan already holds `TableMetadata`
(`ObjectCache::get_manifest_list` takes it;
`schema_by_id`/`partition_spec_by_id` exist). A manifest's embedded schema is a
redundant copy.
2. **Ecosystem alignment.** iceberg-java's `ManifestReader` takes
`specsById` from table metadata and has **deprecated** reading the schema from
manifest file metadata — the warning is literally *"Pass specsById to avoid
reading from file metadata"* (removed in 1.12.0). pyiceberg and iceberg-go
don't read it on the scan path either. iceberg-rust is currently the only
implementation that hard-depends on it.
3. **Observable impact.** Tables whose manifest `schema` key holds a
non-conformant value are readable by pyiceberg, Doris, and Spark (iceberg-java)
but **not** iceberg-rust. For example duckdb-iceberg serializes the
manifest_entry Avro schema there (Avro type names like `array`/`record`),
producing `data did not match any variant of untagged enum SchemaEnum`.
## How
`ObjectCache::get_manifest` now takes `&TableMetadataRef` and threads it
through `ManifestFile::load_manifest_with` →
`Manifest::try_from_avro_bytes_with` → `ManifestMetadata::parse_with`, which
prefers `table_metadata.{schema_by_id, partition_spec_by_id}`. The existing
public `parse` / `parse_avro` / `load_manifest` / `try_from_avro_bytes` are
preserved (they delegate with `None`), so behaviour is unchanged when no table
metadata is available.
## Tests
- New `test_manifest_metadata_parse_prefers_table_metadata_over_bad_schema`:
a manifest with a non-conformant `schema` key parses successfully via table
metadata, while the manifest-only path rejects it.
- Full lib suite green (1359 passed), clippy + fmt clean.
Closes #2682.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
