linzch3 opened a new pull request, #4379:
URL: https://github.com/apache/iceberg-python/pull/4379

   # Rationale for this change
   
   When a REST server omits `residual-filter` from a file scan task, PyIceberg 
treats the task as unfiltered. For a Parquet file with `x = [1, 2, 1]`, 
`scan(row_filter="x != 1").count()` returns 3 while `to_arrow()` returns 1.
   
   Use the original scan filter as the fallback residual for both the initial 
plan and any fetched task batches, as required by the [Iceberg REST 
specification](https://github.com/apache/iceberg/blob/acea99d28a4be93a8c0dfc7bd8ece4a237324243/open-api/rest-catalog-open-api.yaml#L5853-L5858).
 Explicit server residuals still take precedence.
   
   This addresses the REST case reported in 
[#4278](https://github.com/apache/iceberg-python/pull/4278#issuecomment-6832465418).
   
   ## Are these changes tested?
   
   The regression uses a local Parquet file and exercises both the initial 
response and pagination. It fails on `main` and the current head of #4278, and 
passes with this change. The planning tests also verify that an explicit true 
residual is preserved and that an unfiltered request still gets an unfiltered 
task.
   
   `make test`: 4,685 passed, 5 skipped. `make lint` passed.
   
   ## Are there any user-facing changes?
   
   Filtered counts now agree with Arrow scans when a server omits the residual. 
Affected files may be read to apply the filter.
   
   ## AI assistance
   
   OpenAI Codex helped trace the REST planning and count paths, implement the 
fallback, and write and run the regression tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to