[
https://issues.apache.org/jira/browse/SPARK-58186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiangxintong updated SPARK-58186:
---------------------------------
Description:
Spark's flat-bitmap infrastructure (bitmap_construct_agg, bitmap_or_agg,
bitmap_and_agg,
bitmap_count, bitmap_bucket_number, bitmap_bit_position) currently lacks a
predicate to
test whether a specific bit is set — limiting its use for membership filtering.
Add `bitmap_contains(bitmap, bit_position)` that returns true if the bit at the
given
position is set in the bitmap, false otherwise.
SELECT bitmap_contains(X '01', 0L); -- true
SELECT bitmap_contains(X '01', 1L); -- false
Motivation:
In ClickHouse and Doris, `bitmapContains` is the core predicate that enables
user-segment filtering, retention analysis, and crowd-selection queries. Adding
this
function completes the Spark flat-bitmap expression family, enabling patterns
like:
SELECT e.*
FROM events e
JOIN segment_bitmaps b ON bitmap_bucket_number(e.user_id) = b.bucket
WHERE bitmap_contains(b.bm, bitmap_bit_position(e.user_id));
Semantics:
- Input: (BINARY bitmap, LONG bit_position)
- Output: BOOLEAN — true if the bit is set, false otherwise
- Returns false for out-of-range positions (safe by default)
- NULL bitmap → NULL result (standard SQL NULL propagation)
> Add bitmap_contains function
> ----------------------------
>
> Key: SPARK-58186
> URL: https://issues.apache.org/jira/browse/SPARK-58186
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: jiangxintong
> Priority: Major
>
> Spark's flat-bitmap infrastructure (bitmap_construct_agg, bitmap_or_agg,
> bitmap_and_agg,
> bitmap_count, bitmap_bucket_number, bitmap_bit_position) currently lacks a
> predicate to
> test whether a specific bit is set — limiting its use for membership
> filtering.
> Add `bitmap_contains(bitmap, bit_position)` that returns true if the bit at
> the given
> position is set in the bitmap, false otherwise.
> SELECT bitmap_contains(X '01', 0L); -- true
> SELECT bitmap_contains(X '01', 1L); -- false
> Motivation:
> In ClickHouse and Doris, `bitmapContains` is the core predicate that enables
> user-segment filtering, retention analysis, and crowd-selection queries.
> Adding this
> function completes the Spark flat-bitmap expression family, enabling patterns
> like:
> SELECT e.*
> FROM events e
> JOIN segment_bitmaps b ON bitmap_bucket_number(e.user_id) = b.bucket
> WHERE bitmap_contains(b.bm, bitmap_bit_position(e.user_id));
> Semantics:
> - Input: (BINARY bitmap, LONG bit_position)
> - Output: BOOLEAN — true if the bit is set, false otherwise
> - Returns false for out-of-range positions (safe by default)
> - NULL bitmap → NULL result (standard SQL NULL propagation)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
