smaheshwar-pltr opened a new pull request, #14017:
URL: https://github.com/apache/iceberg/pull/14017
If you partition by a string column, on which you do `contains` filters,
such as:
```python
# Partitioned on 'url' column for ideally fast filters
french_sites _df = df.filter(F.col('url').contains('fr'))
```
file pruning based on partition values does not not occur during scan
planning, and queries are surprisingly slow, because Iceberg lacks contains /
not-contains expressions.
However, Spark actually exposes StringContains as a filter it's able to push
down to data sources (and Flink's `LIKE` can be sometimes mapped to the new
Iceberg contains expression), so this is actually achievable, although
controversial.
(There's also endsWith, that's potentially less contentious because it's
easier and more efficient to check, but there's still the question whether to
support expressions only useful for partition-based evaluation. There's also
the question of generalising to more predicates)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
