dgokeeffe commented on PR #4419:
URL: https://github.com/apache/iceberg-python/pull/4419#issuecomment-7008774690

   I tested this branch against the table from #4417, using the PR head 
(`6862dd6`) rebased onto `main` at `f26689a`. The rebase applied without 
conflicts.
   
   The table is a Databricks Delta table with UniForm (`icebergCompatV2`), 
column mapping `name` and row tracking. I read it through the Unity Catalog 
Iceberg REST catalog with vended credentials (Azure, `adlfs`). The data file 
footer has six data columns named `col-<uuid>` with field IDs 1 to 6. It also 
has two int64 columns with no field ID, `_row-id-col-<uuid>` and 
`_row-commit-version-col-<uuid>`. `schema.name-mapping.default` maps IDs 1 to 6 
to the logical names only.
   
   I checked each result against a Databricks SQL warehouse reading the same 
table, comparing the row count and the sum of a double column:
   
   | Query | `main` | This PR |
   |---|---|---|
   | Full scan | `ValueError: Could not find field with name: col-<uuid>` | 
296,906 rows, matches |
   | Filter on a date column | same error | 9,905 rows, matches |
   | Three-column projection | same error | 296,906 rows, matches |
   
   I also read three other tables: a UniForm v3 table, a managed Iceberg v2 
table and a managed Iceberg v3 table. Each one reads the same on `main` and on 
this branch, and all of them match the SQL warehouse. Two have deletion vectors.
   
   Local test results:
   
   - `pytest tests/io/test_pyarrow_visitor.py tests/io/test_pyarrow.py -m 
unit`: 320 passed, 3 skipped.
   - `test_projection_partial_field_ids` passes on this branch. Against 
`main`'s `pyiceberg/io/pyarrow.py` it fails with `Parquet file does not have 
field-ids and the Iceberg table does not have 'schema.name-mapping.default' 
defined`.
   - `pytest tests/ -m unit --ignore=tests/integration`: `main` and this branch 
give the same failures and errors. Most come from optional dependencies I 
didn't install (flask, s3fs, google-auth, thrift and others). The rest are two 
error-message assertions in `tests/expressions/test_parser.py` that fail with 
pyparsing 3.3.3.
   
   I used an AI coding assistant to run these checks and draft this comment. I 
reviewed the results before posting.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to