carlosjourdan opened a new issue, #4104:
URL: https://github.com/apache/iceberg-python/issues/4104
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
Commit: main `b32392d` (includes #3724). The problem also occurs in 0.12.0.
With `scan-planning-mode=server`, `DataScan` sends only the projected
columns in the `select` field of the plan request. Then, it applies
`row_filter` again on the client when it reads the files.
The server can return tasks without a `residual-filter`. In this condition,
the specification tells the client to apply the original filter.
The server can also return files that contain only the selected columns.
Databricks Unity Catalog does this when it applies row filters and column masks
for external engines.
When these two conditions occur, PyIceberg applies the filter to a column
that is not in the file. The filter removes all rows. The scan returns an empty
result and shows **no error**.
Client-side planning does not have this problem, because it reads the filter
columns for the residual evaluation.
**Proposed fix:**
1. Add the columns from `row_filter` to the `select` field of the plan
request.
2. Remove these columns from the output.
Alternatively, do not apply the filter on the client when the residual of
the task is `true`.
**Procedure to show the problem:**
```python
cat = load_catalog("uc", type="rest", uri=..., token=..., warehouse=...,
**{"scan-planning-mode": "server"})
t = cat.load_table("ns.events") # partitioned by `data`, row filter
enforced server-side
t.scan(row_filter=EqualTo("data", "2026-01-05"), selected_fields=("regiao",
"documento"),
case_sensitive=False).to_arrow().num_rows # -> 0
t.scan(row_filter=EqualTo("data", "2026-01-05"), selected_fields=("regiao",
"documento", "data"),
case_sensitive=False).to_arrow().num_rows # -> 419812
```
The plan response shows 419,812 records for the two requests. The tasks do
not have a `residual-filter`.
For Unity Catalog, we change two values in the response before PyIceberg
reads it: `content: "DATA"` and `partition: {}`. We report these two
differences to Databricks in a different ticket.
### Willingness to contribute
- [X] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
