linzch3 opened a new pull request, #4379: URL: https://github.com/apache/iceberg-python/pull/4379
# Rationale for this change When a REST server omits `residual-filter` from a file scan task, PyIceberg treats the task as unfiltered. For a Parquet file with `x = [1, 2, 1]`, `scan(row_filter="x != 1").count()` returns 3 while `to_arrow()` returns 1. Use the original scan filter as the fallback residual for both the initial plan and any fetched task batches, as required by the [Iceberg REST specification](https://github.com/apache/iceberg/blob/acea99d28a4be93a8c0dfc7bd8ece4a237324243/open-api/rest-catalog-open-api.yaml#L5853-L5858). Explicit server residuals still take precedence. This addresses the REST case reported in [#4278](https://github.com/apache/iceberg-python/pull/4278#issuecomment-6832465418). ## Are these changes tested? The regression uses a local Parquet file and exercises both the initial response and pagination. It fails on `main` and the current head of #4278, and passes with this change. The planning tests also verify that an explicit true residual is preserved and that an unfiltered request still gets an unfiltered task. `make test`: 4,685 passed, 5 skipped. `make lint` passed. ## Are there any user-facing changes? Filtered counts now agree with Arrow scans when a server omits the residual. Affected files may be read to apply the filter. ## AI assistance OpenAI Codex helped trace the REST planning and count paths, implement the fallback, and write and run the regression tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
