wroever opened a new issue, #1983: URL: https://github.com/apache/iceberg-go/issues/1983
### Apache Iceberg version main (development) ### Please describe the bug 🐞 ## Version Observed on `main` at `87789102577c` (2026-09-01) using DuckDB `v1.5.5` as writer. ## Summary Reading a table whose Avro manifests specify a lowercase `file_format` for data files (e.g. `parquet`) fails on the first data file with the following error: ``` not implemented: only parquet format is implemented, got parquet ``` This happens via the ordinary read path: `LoadTable` followed by `tbl.Scan()`. The Iceberg spec [uses lowercase strings as illustrative example values](https://iceberg.apache.org/spec/#data-file-fields) for `file_format`, so so I don't think this is reasonably characterized as a DuckDB quirk/bug. Meanwhile, unlike `iceberg-go`, both `pyiceberg` and the Java reference implementation appear to be case-agnostic with respect to `file_format` (Pyiceberg has a [test case](https://github.com/apache/iceberg-python/blob/d12f68042a459e5d4732bd6ee5a188e891b0dd44/tests/catalog/test_scan_planning_models.py#L208) specifically intended to ensure case-insensitivity when scanning). `iceberg-go` ought to behave similarly to ensure portability of tables written by other writers. <details><summary><b>Steps to reproduce</b></summary> <br/> **1. Start a REST catalog over a local directory.** ```bash mkdir -p /tmp/icewh docker run -d --name ice-rest -p 8181:8181 \ -v /tmp/icewh:/tmp/icewh \ -e CATALOG_WAREHOUSE=file:///tmp/icewh \ apache/iceberg-rest-fixture:latest ``` **2. Create and populate a table with DuckDB.** ```sql INSTALL iceberg; LOAD iceberg; ATTACH 'file:///tmp/icewh' AS cat ( TYPE ICEBERG, ENDPOINT 'http://localhost:8181', AUTHORIZATION_TYPE 'none' ); CREATE SCHEMA cat.ns; CREATE TABLE cat.ns.t (id BIGINT); INSERT INTO cat.ns.t VALUES (1); ``` **3. Confirm the manifest records a lowercase `file_format`** ```sql SELECT file_format FROM iceberg_metadata(cat.ns.t); -- "parquet" expected ``` **4. Scan the table with iceberg-go**: ```go cat, _ := rest.NewCatalog(ctx, "cat", "http://localhost:8181";, rest.WithWarehouseLocation("file:///tmp/icewh")) tbl, err := cat.LoadTable(ctx, table.Identifier{"ns", "t"}) // succeeds: table metadata and manifest lists parse fine _, recs, err := tbl.Scan().ToArrowRecords(ctx) for rec, err := range recs { if err != nil { log.Fatal(err) // not implemented: only parquet format is implemented, got parquet } _ = rec } ``` </details> <details><summary><b>Details on Java/Python behavior</b></summary> <br/> Both the Java and Python reference implementations avoid problems here by normalizing `file_format` casing on decode. `pyiceberg` has a test case specifically to ensure case-insensitivity. **Java** — `BaseFile.internalSet` routes the raw Avro value through `FileFormat.fromString`: ```java // core/src/main/java/org/apache/iceberg/BaseFile.java case 2: this.format = FileFormat.fromString(value.toString()); return; ``` ```java // api/src/main/java/org/apache/iceberg/FileFormat.java public static FileFormat fromString(String fileFormat) { Preconditions.checkArgument(null != fileFormat, "Invalid file format: null"); try { return FileFormat.valueOf(fileFormat.toUpperCase(Locale.ROOT)); } catch (IllegalArgumentException e) { throw new IllegalArgumentException( String.format("Invalid file format: %s", fileFormat), e); } } ``` **Python** — `FileFormat` is a `str` enum with a `_missing_` hook that uppercases before matching: ```python # pyiceberg/manifest.py class FileFormat(str, Enum): AVRO = "AVRO" PARQUET = "PARQUET" ORC = "ORC" PUFFIN = "PUFFIN" @classmethod def _missing_(cls, value: object) -> None | str: for member in cls: if member.value == str(value).upper(): return member return None ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
