gedaliakliger opened a new pull request, #4400: URL: https://github.com/apache/iceberg-python/pull/4400
Closes #4399 # Rationale for this change A scan whose `row_filter` tests a list or map column for null (`tags is null`, `props is not null`) fails in `to_arrow()` with `ValueError: Cannot explicitly project List or Map types`. `ArrowScan._projected_field_ids` leaves list and map IDs out of the projected schema's IDs, because `prune_columns` rebuilds those columns from their nested fields, but it adds the filter's field IDs as they are. A null test on a list or map column adds the column's own ID, and `prune_columns` rejects it. This change projects the nested fields of such a column instead, the same way a selected list or map column is read. ## Are these changes tested? Yes. `test_projection_null_filter_on_list_and_map` in `tests/io/test_pyarrow.py` runs `IsNull` and `NotNull` on a list, a map and a list of structs, with the column selected and with only `id` selected. It fails on `main` with the `ValueError` above and passes with this change. Ran locally: `pytest tests/io/test_pyarrow.py -m unit`, and the `ruff`, `ruff-format` and `mypy` hooks on the two changed files. The rest of the suite and the integration tests are left to CI. ## Are there any user-facing changes? `IS NULL` and `IS NOT NULL` filters on list and map columns return the matching rows instead of raising. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
