Rodrigo-Palma opened a new pull request, #3995:
URL: https://github.com/apache/iceberg-python/pull/3995
Closes #3982.
`rewrite_to_dnf` distributes every `AND` over the `OR`s below it, so a
filter built
from n two-branch `OR` groups expands to `2**n` terms. Nothing bounded the
result, so
a small filter could produce an expression no consumer can use.
Measured on `main` before this change, with `(a_i = x OR b_i = y)` groups
joined by `AND`:
| OR groups | DNF terms | time | peak memory |
|---|---|---|---|
| 12 | 4,096 | 0.31s | 3.7 MiB |
| 14 | 16,384 | 1.34s | 15.0 MiB |
| 16 | 65,536 | 5.77s | 59.8 MiB |
| 20 | 1,048,576 | 16.9s | ~1 GiB |
20 groups is about 40 predicates, a plausible size for a filter that an
application
builds from user input.
## Change
`visit_and` now checks the size of the product before building it and raises
when it
would pass `MAX_DNF_TERMS`. The same 20-group input fails in 0.22s instead
of spending
17s to build an unusable result.
The bound is applied where the amplification happens, the distribution step,
rather
than on the total term count, so an expression that is genuinely large as
written
(a long chain of `OR`s, which grows linearly) keeps working as before.
`MAX_DNF_TERMS` is `1 << 14`. Past that point the conversion already costs
more than a
second and tens of MiB, while staying far above any filter meant to be used
as a
predicate. Happy to move it if you prefer a different point.
The constant and the `ValueError` follow the same shape as
`MAX_DECOMPRESSED_BLOCK_SIZE`
in #3993.
## Tests
Two tests in `tests/expressions/test_visitors.py`:
- `test_to_dnf_at_expansion_limit`: an expression that expands to exactly
`MAX_DNF_TERMS`
still converts.
- `test_to_dnf_rejects_unbounded_expansion`: one group beyond the bound
raises.
Without the source change the second test fails. `make lint` and `make test`
are green
locally (4188 passed, 5 skipped).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
