[
https://issues.apache.org/jira/browse/SPARK-58186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiangxintong updated SPARK-58186:
---------------------------------
Description:
h3. Problem
Spark's flat-bitmap infrastructure provides functions for building and merging
bitmaps (bitmap_construct_agg, bitmap_or_agg, bitmap_and_agg, bitmap_count,
bitmap_bucket_number, bitmap_bit_position), but lacks a predicate to test
whether a specific bit is set. Without this, precomputed bitmaps cannot be used
for membership filtering — the primary use case of bitmap indexes in analytical
databases.
Other databases with this capability:
- ClickHouse: bitmapContains(bitmap, value)
- Doris: bitmap_contains(bitmap, value)
- StarRocks: bitmap_contains(bitmap, value)
h3. Proposed Function
|| Function || Signature || Description ||
| bitmap_contains | bitmap_contains(BINARY, LONG) -> BOOLEAN | Returns true if
the bit at the given position is set in the bitmap. Returns false for
out-of-range positions. |
h3. Examples
{code:sql}
> SELECT bitmap_contains(X '01', 0L);
true
> SELECT bitmap_contains(X '01', 1L);
false
> SELECT bitmap_contains(X '10', 4L);
true
-- Practical usage with bitmap_construct_agg
SELECT e.*
FROM events e
JOIN segment_bitmaps b ON bitmap_bucket_number(e.user_id) = b.bucket
WHERE bitmap_contains(b.bm, bitmap_bit_position(e.user_id));
{code}
h3. Semantics
- Input: (BINARY bitmap, LONG bit_position)
- Output: BOOLEAN
- Returns false for out-of-range positions (negative or >= 32768), safe by
default
- NULL bitmap or NULL position → NULL result (standard SQL NULL propagation)
- Works with bitmaps of any length (not limited to the standard 4096-byte
bitmap)
h3. Notes
- This function operates on Spark's flat-bitmap format (fixed 4KB per bucket),
not RoaringBitmap. The 2-step mapping (bucket_number + bit_position) is still
required.
- Completes the flat-bitmap expression family alongside the existing
construct/merge/count functions.
was:
Spark's flat-bitmap infrastructure (bitmap_construct_agg, bitmap_or_agg,
bitmap_and_agg,
bitmap_count, bitmap_bucket_number, bitmap_bit_position) currently lacks a
predicate to
test whether a specific bit is set — limiting its use for membership filtering.
Add `bitmap_contains(bitmap, bit_position)` that returns true if the bit at the
given
position is set in the bitmap, false otherwise.
SELECT bitmap_contains(X '01', 0L); -- true
SELECT bitmap_contains(X '01', 1L); -- false
Motivation:
In ClickHouse and Doris, `bitmapContains` is the core predicate that enables
user-segment filtering, retention analysis, and crowd-selection queries. Adding
this
function completes the Spark flat-bitmap expression family, enabling patterns
like:
SELECT e.*
FROM events e
JOIN segment_bitmaps b ON bitmap_bucket_number(e.user_id) = b.bucket
WHERE bitmap_contains(b.bm, bitmap_bit_position(e.user_id));
Semantics:
- Input: (BINARY bitmap, LONG bit_position)
- Output: BOOLEAN — true if the bit is set, false otherwise
- Returns false for out-of-range positions (safe by default)
- NULL bitmap → NULL result (standard SQL NULL propagation)
> Add bitmap_contains function
> ----------------------------
>
> Key: SPARK-58186
> URL: https://issues.apache.org/jira/browse/SPARK-58186
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: jiangxintong
> Priority: Major
>
> h3. Problem
> Spark's flat-bitmap infrastructure provides functions for building and
> merging bitmaps (bitmap_construct_agg, bitmap_or_agg, bitmap_and_agg,
> bitmap_count, bitmap_bucket_number, bitmap_bit_position), but lacks a
> predicate to test whether a specific bit is set. Without this, precomputed
> bitmaps cannot be used for membership filtering — the primary use case of
> bitmap indexes in analytical databases.
> Other databases with this capability:
> - ClickHouse: bitmapContains(bitmap, value)
> - Doris: bitmap_contains(bitmap, value)
> - StarRocks: bitmap_contains(bitmap, value)
> h3. Proposed Function
> || Function || Signature || Description ||
> | bitmap_contains | bitmap_contains(BINARY, LONG) -> BOOLEAN | Returns true
> if the bit at the given position is set in the bitmap. Returns false for
> out-of-range positions. |
> h3. Examples
> {code:sql}
> > SELECT bitmap_contains(X '01', 0L);
> true
> > SELECT bitmap_contains(X '01', 1L);
> false
> > SELECT bitmap_contains(X '10', 4L);
> true
> -- Practical usage with bitmap_construct_agg
> SELECT e.*
> FROM events e
> JOIN segment_bitmaps b ON bitmap_bucket_number(e.user_id) = b.bucket
> WHERE bitmap_contains(b.bm, bitmap_bit_position(e.user_id));
> {code}
> h3. Semantics
> - Input: (BINARY bitmap, LONG bit_position)
> - Output: BOOLEAN
> - Returns false for out-of-range positions (negative or >= 32768), safe by
> default
> - NULL bitmap or NULL position → NULL result (standard SQL NULL propagation)
> - Works with bitmaps of any length (not limited to the standard 4096-byte
> bitmap)
> h3. Notes
> - This function operates on Spark's flat-bitmap format (fixed 4KB per
> bucket), not RoaringBitmap. The 2-step mapping (bucket_number + bit_position)
> is still required.
> - Completes the flat-bitmap expression family alongside the existing
> construct/merge/count functions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
