jiangxt2 opened a new pull request, #58066:
URL: https://github.com/apache/spark/pull/58066
### What changes were proposed in this pull request?
This PR adds four scalar set operations for Spark's flat bitmap
representation:
- `bitmap_and`
- `bitmap_or`
- `bitmap_andnot`
- `bitmap_xor`
Each function operates on two `BINARY` values from the same row. The
implementation:
- accepts inputs up to 4096 bytes;
- treats missing trailing bytes as zero;
- returns a newly allocated, fixed-size 4096-byte bitmap;
- propagates `NULL` inputs;
- rejects oversized inputs with the structured `BITMAP_INPUT_TOO_LARGE`
error;
- does not mutate either input.
The functions are implemented as Catalyst-native binary expressions with
both interpreted
and whole-stage codegen paths. They are registered in the SQL function
registry and exposed
through the Scala API, PySpark classic, and Spark Connect.
This PR also adds SQL documentation, Python documentation, a structured
error definition,
Connect plan and schema golden files, and tests for the new APIs.
### Why are the changes needed?
Spark provides functions for constructing, aggregating, and counting flat
bitmaps, but it
does not provide scalar set operations for combining two bitmap values from
the same row.
Without native scalar functions, users need UDFs or external bitmap
processing to compute
intersection, union, difference, or symmetric difference between precomputed
bitmap columns.
Native Catalyst expressions provide consistent SQL semantics, structured
validation, and
code generation, and allow these operations to compose naturally with the
existing bitmap
aggregate and count functions.
### Does this PR introduce _any_ user-facing change?
Yes. It adds four new SQL, Scala, PySpark classic, and Spark Connect
functions:
```sql
SELECT bitmap_and(left_bitmap, right_bitmap);
SELECT bitmap_or(left_bitmap, right_bitmap);
SELECT bitmap_andnot(left_bitmap, right_bitmap);
SELECT bitmap_xor(left_bitmap, right_bitmap);
```
`bitmap_andnot(left, right)` is directional and returns the bits present in
`left` but not
in `right`.
No existing function behavior is changed.
### How was this patch tested?
The following targeted suites and build checks passed:
- `build/sbt "catalyst/testOnly
org.apache.spark.sql.catalyst.expressions.BitmapExpressionUtilsSuite"`
(10 tests)
- `build/sbt "sql/testOnly org.apache.spark.sql.BitmapExpressionsQuerySuite"`
(19 tests, including interpreted and codegen execution)
- `build/sbt "sql/testOnly org.apache.spark.sql.ExpressionsSchemaSuite"`
- `build/sbt "connect-client-jvm/testOnly
org.apache.spark.sql.PlanGenerationTestSuite -- -z 'function bitmap'"`
(9 tests)
- `build/sbt -Phive package`
- `python/run-tests --testnames "pyspark.sql.tests.test_functions
FunctionsTests.test_bitmap_scalar_functions"`
- `python/run-tests --testnames "pyspark.sql.tests.connect.test_connect_plan
SparkConnectPlanTests.test_bitmap_scalar_functions"`
- Targeted PySpark doctests for the four new functions (12 tests)
The tests cover variable-length and empty inputs, the 4096-byte boundary,
input immutability,
null propagation, invalid types, oversized inputs on either side,
interpreted and codegen
execution, aggregation composition, grouped queries, PySpark classic, and
Spark Connect.
Scalafmt, Ruff, JSON validation, golden-file consistency checks, and `git
diff --check` also
passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex and Claude AI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
