raghav-reglobe opened a new issue, #2682:
URL: https://github.com/apache/iceberg-rust/issues/2682
## What happens
`table.scan()...to_arrow()` fails on Iceberg tables whose manifests were
written by **DuckDB** (the `duckdb-iceberg` extension), with:
```
Failed to load manifest <...>.avro
→ DataInvalid: Fail to parse schema in manifest metadata
→ data did not match any variant of untagged enum SchemaEnum
```
The same tables read correctly in **pyiceberg, Apache Doris, and Spark
(iceberg-java)** — iceberg-rust is the only implementation that rejects them.
## Root cause
A manifest avro file carries the table schema as JSON in its `schema`
key-value metadata. `ManifestMetadata::parse` (`spec/manifest/metadata.rs`)
does:
```rust
serde_json::from_slice::<Schema>(meta.get("schema"))? // hard error
```
and `Manifest::try_from_avro_bytes` (`spec/manifest/mod.rs`) then uses that
parsed schema + spec to derive the partition type for decoding entries:
```rust
let partition_type =
metadata.partition_spec.partition_type(&metadata.schema)?;
```
So iceberg-rust **depends on the manifest's self-described
`schema`/`partition-spec` keys**. duckdb-iceberg writes a non-conformant
`schema` key (it serializes the manifest_entry Avro record schema there — using
Avro type names `array`/`record` instead of the Iceberg table schema's
`list`/`struct`), so the strict parse fails before any entry is decoded.
## Why the other implementations don't hit this
They don't read the manifest's `schema` key on the scan path:
- **iceberg-java** `ManifestReader` takes `specsById` (the partition specs
from table metadata): `spec = specsById.get(specId)`; `fileSchema =
DataFile.getType(spec.rawPartitionType())`. It only reads `meta["schema"]` in a
fallback that is **deprecated and slated for removal** (the warning literally
says *"Pass specsById to avoid reading from file metadata"*).
- **pyiceberg** decodes via its hardcoded `MANIFEST_ENTRY_SCHEMAS` + the
table-metadata schema.
- **iceberg-go** decodes via the Avro writer schema.
i.e. the manifest's `schema` key is treated as redundant with the
authoritative table metadata.
## Proposed fix
Don't hard-depend on the manifest's `schema` metadata key. Two options:
1. **Thread the table metadata's schema/spec into the scan's manifest
decode** (mirrors iceberg-java's `specsById`): the scan already holds
`TableMetadataRef` (`ObjectCache::get_manifest_list` takes it), and
`TableMetadata::{schema_by_id, partition_spec_by_id}` exist — use them for
`partition_type`, making the manifest's own `schema`/`partition-spec` a
best-effort fallback.
2. **Recover from the Avro writer schema** when the `schema` key fails to
parse (mirrors iceberg-go); `avro_schema_to_schema` already exists in
`crates/iceberg/src/avro/schema.rs`.
Option 1 aligns with the iceberg-java direction. Happy to open a PR —
flagging here first to confirm the preferred approach since it touches a public
path (`Manifest::parse_avro`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
