hcrosse opened a new pull request, #4278: URL: https://github.com/apache/iceberg-python/pull/4278
Closes https://github.com/apache/iceberg-python/issues/4276 # Rationale for this change `count()` includes NULL rows that `to_arrow()` drops in the issue's example: `p != 'a'` gives `count() == 9` versus `len(to_arrow()) == 4`. I made `count()` follow `to_arrow()` without changing residual evaluation or `to_arrow()`. The NULL convention question in https://github.com/apache/iceberg-python/pull/3918 stays open. `count()` now reads a file instead of using `record_count` when one of its partition fields is NULL and the row filter references that field's source column outside `IsNull` or `NotNull`. Null checks are ignored because they are exact on a NULL partition value. If the file's spec is missing from table metadata, `count()` reads the file when the filter has any such reference. Other files keep the existing behavior. Deletes are unchanged. ## Are these changes tested? I added two regression tests to `tests/catalog/test_catalog_behaviors.py`. Both run on the memory and SQL catalogs, and both fail on main. - One checks that `count()` equals `len(to_arrow())` across identity partitions, bucket partitions and an evolved spec with files written before and after removing a partition field. - The other checks which files remain metadata-only. It deletes the files that must not be read before calling `count()`, so an unexpected read fails. `make test` gives 4478 passed, 4 failed and 5 skipped. The same four Hive Kerberos tests fail on main because the `hive-kerberos` extra cannot be installed in the test environment. `prek run -a` passed. The tests use locally planned scans. I did not add a REST test, because the REST planning response rejects a NULL partition value. [`RESTContentFile.partition`](https://github.com/apache/iceberg-python/blob/e3564a54c10960307147377c578680276865ed18/pyiceberg/catalog/rest/scan_planning.py#L77) accepts only [`PrimitiveTypeValue`](https://github.com/apache/iceberg-python/blob/e3564a54c10960307147377c578680276865ed18/pyiceberg/catalog/rest/scan_planning.py#L36), which excludes `None`. The new check uses only the task's partition data and spec ID, so REST-planned tasks take the same path. ## Are there any user-facing changes? Yes. For these scans `count()` now returns the same number as `to_arrow()`, for example 4 instead of 9 in the issue's example. It reads the affected files instead of counting them from metadata, including some files whose count does not change. There is no API change. Please add the `changelog` label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
