dgokeeffe commented on PR #4419: URL: https://github.com/apache/iceberg-python/pull/4419#issuecomment-7008774690
I tested this branch against the table from #4417, using the PR head (`6862dd6`) rebased onto `main` at `f26689a`. The rebase applied without conflicts. The table is a Databricks Delta table with UniForm (`icebergCompatV2`), column mapping `name` and row tracking. I read it through the Unity Catalog Iceberg REST catalog with vended credentials (Azure, `adlfs`). The data file footer has six data columns named `col-<uuid>` with field IDs 1 to 6. It also has two int64 columns with no field ID, `_row-id-col-<uuid>` and `_row-commit-version-col-<uuid>`. `schema.name-mapping.default` maps IDs 1 to 6 to the logical names only. I checked each result against a Databricks SQL warehouse reading the same table, comparing the row count and the sum of a double column: | Query | `main` | This PR | |---|---|---| | Full scan | `ValueError: Could not find field with name: col-<uuid>` | 296,906 rows, matches | | Filter on a date column | same error | 9,905 rows, matches | | Three-column projection | same error | 296,906 rows, matches | I also read three other tables: a UniForm v3 table, a managed Iceberg v2 table and a managed Iceberg v3 table. Each one reads the same on `main` and on this branch, and all of them match the SQL warehouse. Two have deletion vectors. Local test results: - `pytest tests/io/test_pyarrow_visitor.py tests/io/test_pyarrow.py -m unit`: 320 passed, 3 skipped. - `test_projection_partial_field_ids` passes on this branch. Against `main`'s `pyiceberg/io/pyarrow.py` it fails with `Parquet file does not have field-ids and the Iceberg table does not have 'schema.name-mapping.default' defined`. - `pytest tests/ -m unit --ignore=tests/integration`: `main` and this branch give the same failures and errors. Most come from optional dependencies I didn't install (flask, s3fs, google-auth, thrift and others). The rest are two error-message assertions in `tests/expressions/test_parser.py` that fail with pyparsing 3.3.3. I used an AI coding assistant to run these checks and draft this comment. I reviewed the results before posting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
