surnaik opened a new pull request, #13531:
URL: https://github.com/apache/gluten/pull/13531
## What changes are proposed in this pull request?
Fixes #13489.
Currently, `BloomFilterMightContainJointRewriteRule` at `injectPre`
unconditionally rewrites every `BloomFilterAggregate` and
`BloomFilterMightContain` in the physical plan to `VeloxBloomFilterAggregate`
and `VeloxBloomFilterMightContain`. This causes failures or incompatible
serialized bloom filter bytes when:
1. `bloom_filter_agg` is called on a `StringType` column (or other
non-integral input types not supported by Velox's `bloom_filter_agg`).
2. `bloom_filter_agg` is called on a nullable column containing `NULL`
values (where Velox's `BloomFilterAggAggregate` throws `"First argument of
bloom_filter_agg cannot be null"` or emits a non-null empty bloom filter on
all-null inputs instead of `NULL`).
3. `might_contain` consumes a binary bloom filter expression that is not a
paired `ScalarSubquery(Aggregate(BloomFilterAggregate))` (e.g., constant
Spark-format bloom filter bytes or arbitrary subqueries).
4. A standalone SQL `bloom_filter_agg` query returns bloom filter bytes to
the caller without a paired `might_contain`.
This PR adds a Pre-CBO logical rule
`BloomFilterMightContainJointRewriteRule.Logical`:
- Jointly rewrites
`BloomFilterMightContain(ScalarSubquery(Aggregate(BloomFilterAggregate)))` to
`VeloxBloomFilterMightContain` and `VeloxBloomFilterAggregate` only when both
producer and consumer are verified to be a pair on a supported integral input
type (`ByteType`, `ShortType`, `IntegerType`, `LongType`), widening narrow
integral inputs to `LongType` and adding an `IsNotNull` aggregate filter when
the input is nullable.
- Tags unpaired or unsupported `BloomFilterMightContain` and
`BloomFilterAggregate` expressions with `KEEP_VANILLA_BLOOM_FILTER_TAG`
(`TreeNodeTag[Unit]`) so they remain on vanilla Spark.
- Retains the physical `BloomFilterMightContainJointRewriteRule` at
`injectPre` (checking `!isKeepVanilla`) so runtime bloom filters injected later
by Spark's `InjectRuntimeFilter` continue to offload natively and survive
whole-stage fallback (`GLUTEN-12013`).
## How was this patch tested?
- Added unit tests in `GlutenBloomFilterFallbackSuite` covering:
- `bloom_filter_agg` on `StringType` columns falling back to vanilla Spark
(standalone and paired with `might_contain`).
- `bloom_filter_agg` on nullable `LongType` columns (with `NULL` rows,
`FILTER (WHERE ...)` clauses, and all-`NULL` inputs).
- `might_contain` with non-`bloom_filter_agg` first arguments (binary
literals, non-aggregate subqueries, `first(bf)` aggregates, `GROUP BY`
subqueries, and CTE subqueries).
- Standalone SQL `bloom_filter_agg` returning Spark-readable bloom filter
bytes.
- `bloom_filter_agg` on `ByteType`, `ShortType`, and `IntegerType` columns
offloading natively.
- Verified existing suites `GlutenBloomFilterFallbackSuite`,
`GlutenBloomFilterAggregateQuerySuite`, and
`GlutenBloomFilterAggregateQuerySuiteCGOff`.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Gemini-4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]