[
https://issues.apache.org/jira/browse/SPARK-58831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58831:
-----------------------------------
Labels: pull-request-available (was: )
> Add bitmap scalar set operation functions
> -----------------------------------------
>
> Key: SPARK-58831
> URL: https://issues.apache.org/jira/browse/SPARK-58831
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: jiangxintong
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> Spark's flat-bitmap infrastructure provides functions for constructing,
> aggregating, and counting
> bitmaps, including {{bitmap_construct_agg}}, {{bitmap_or_agg}},
> {{bitmap_and_agg}},
> {{bitmap_count}}, {{bitmap_bucket_number}}, and {{bitmap_bit_position}}.
> However, Spark does not provide scalar set operations for combining two
> precomputed bitmaps from
> the same row. Users currently need a UDF or application-side processing to
> calculate bitmap
> intersection, union, difference, or symmetric difference. A native
> implementation keeps these
> operations visible to Catalyst and whole-stage codegen and provides
> consistent SQL, Scala,
> PySpark, and Spark Connect APIs.
> h2. Proposed Functions
> ||Function||Signature||Description||
> |bitmap_and|bitmap_and(BINARY, BINARY) -> BINARY|Returns the intersection of
> two bitmaps.|
> |bitmap_or|bitmap_or(BINARY, BINARY) -> BINARY|Returns the union of two
> bitmaps.|
> |bitmap_andnot|bitmap_andnot(BINARY, BINARY) -> BINARY|Returns the bits
> present in the left bitmap but not in the right bitmap.|
> |bitmap_xor|bitmap_xor(BINARY, BINARY) -> BINARY|Returns the symmetric
> difference of two bitmaps.|
> h2. Semantics
> * Both arguments use Spark's existing flat-bitmap Binary representation, not
> a RoaringBitmap
> serialization.
> * Each input may contain between 0 and 4096 bytes.
> * Missing bytes in a shorter input are treated as zero.
> * The result is always a new 4096-byte Binary value.
> * If either input is NULL, the result is NULL.
> * An input longer than 4096 bytes raises the structured error
> {{BITMAP_INPUT_TOO_LARGE}}.
> * {{bitmap_andnot(left, right)}} is directional and computes
> {{left AND NOT right}}.
> * These are scalar functions that combine two bitmaps from the same row.
> Existing
> {{bitmap_*_agg}} functions continue to combine bitmaps across rows.
> h2. Examples
> {code:sql}
> SELECT substring(hex(bitmap_and(X 'F0', X '70')), 0, 2);
> -- 70
> SELECT substring(hex(bitmap_or(X '10', X '20')), 0, 2);
> -- 30
> SELECT substring(hex(bitmap_andnot(X 'F0', X '70')), 0, 2);
> -- 80
> SELECT substring(hex(bitmap_xor(X 'F0', X '70')), 0, 2);
> -- 80
> {code}
> h2. API Surface
> * Spark SQL functions: {{bitmap_and}}, {{bitmap_or}}, {{bitmap_andnot}}, and
> {{bitmap_xor}}.
> * Scala DataFrame functions under {{org.apache.spark.sql.functions}}.
> * PySpark classic and Spark Connect functions under {{pyspark.sql.functions}}.
> h2. Scope and Compatibility
> * The change is additive and does not alter the behavior of existing bitmap
> functions.
> * It does not introduce a new bitmap storage format or change bucket-number
> or bit-position
> mapping.
> * It does not add or modify aggregate bitmap functions.
> * Existing SQL queries and APIs are unaffected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
