[
https://issues.apache.org/jira/browse/SPARK-58186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58186:
-----------------------------------
Labels: pull-request-available (was: )
> Add bitmap_contains function
> ----------------------------
>
> Key: SPARK-58186
> URL: https://issues.apache.org/jira/browse/SPARK-58186
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: jiangxintong
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> Spark's flat-bitmap infrastructure provides functions for building and
> merging bitmaps (bitmap_construct_agg, bitmap_or_agg, bitmap_and_agg,
> bitmap_count, bitmap_bucket_number, bitmap_bit_position), but lacks a
> predicate to test whether a specific bit is set. Without this, precomputed
> bitmaps cannot be used for membership filtering — the primary use case of
> bitmap indexes in analytical databases.
> Other databases with this capability:
> - ClickHouse: bitmapContains(bitmap, value)
> - Doris: bitmap_contains(bitmap, value)
> - StarRocks: bitmap_contains(bitmap, value)
> h3. Proposed Function
> || Function || Signature || Description ||
> | bitmap_contains | bitmap_contains(BINARY, LONG) -> BOOLEAN | Returns true
> if the bit at the given position is set in the bitmap. Returns false for
> out-of-range positions. |
> h3. Examples
> {code:sql}
> > SELECT bitmap_contains(X '01', 0L);
> true
> > SELECT bitmap_contains(X '01', 1L);
> false
> > SELECT bitmap_contains(X '10', 4L);
> true
> -- Practical usage with bitmap_construct_agg
> SELECT e.*
> FROM events e
> JOIN segment_bitmaps b ON bitmap_bucket_number(e.user_id) = b.bucket
> WHERE bitmap_contains(b.bm, bitmap_bit_position(e.user_id));
> {code}
> h3. Semantics
> - Input: (BINARY bitmap, LONG bit_position)
> - Output: BOOLEAN
> - Returns false for out-of-range positions (negative or >= 32768), safe by
> default
> - NULL bitmap or NULL position → NULL result (standard SQL NULL propagation)
> - Works with bitmaps of any length (not limited to the standard 4096-byte
> bitmap)
> h3. Notes
> - This function operates on Spark's flat-bitmap format (fixed 4KB per
> bucket), not RoaringBitmap. The 2-step mapping (bucket_number + bit_position)
> is still required.
> - Completes the flat-bitmap expression family alongside the existing
> construct/merge/count functions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
