2162174202-sudo commented on issue #14995:
URL: https://github.com/apache/iceberg/issues/14995#issuecomment-7055935287
Confirming `@joyhaldar` and `@alexandrefimov`'s findings with a 3-engine
cross-check on the **same frozen Apache Iceberg 1.11.0 table snapshot** (REST
Catalog, format-version 2, generated via our multi-engine Lakehouse
differential fuzzer):
Query pattern (unescaped `_`, snake_case prefix):
```sql
SELECT id, city
FROM iceberg.default.t_main
WHERE city LIKE 'prefix_16_bytes_alpha%';
```
### Physical Plan Comparison (Same Frozen Iceberg Snapshot)
1. **Apache Spark 3.5.5 + Iceberg 1.11.0** — no prefix filter pushed to
`BatchScan` (only `IS NOT NULL`), falling back to a full scan:
```text
(1) BatchScan iceberg.default.t_main
iceberg.default.t_main (branch=null) [filters=city IS NOT NULL, groupedBy=]
...
Condition : (isnotnull(city) AND city LIKE prefix_16_bytes_alpha%)
```
2. **Trino 470** — extracts the literal prefix `'prefix'` before the first
`_` into a conservative range filter pushed to the Iceberg scan, while keeping
`$like(...)` as a residual check:
```text
ScanFilter[
table = iceberg:default.t_main$data@...,
filterPredicate = ((city >= varchar 'prefix') AND (city < varchar
'prefiy') AND $like(...))
]
```
3. **DuckDB 2.0.0 (`iceberg` extension `4c0c0874c4`)** — pushes `Filters:`
directly into `ICEBERG_SCAN`.
This confirms the gap is specifically in Spark Catalyst's
`LikeSimplification` / DSv2 pushdown path: even for **unescaped `_`** (not just
`\_` / `ESCAPE`), Spark does not derive a conservative auxiliary
`StartsWith(col, <literal_prefix_before_first_wildcard>)` for DSv2
`pushPredicates(...)` while keeping the original `LIKE` as a residual filter.
As `@alexandrefimov` suggested, filing a focused Spark Jira proposal
highlighting the DSv2 / Iceberg file-pruning impact (rather than replacing
`Like` in-place) seems like the cleanest path forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]