[
https://issues.apache.org/jira/browse/SPARK-58186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiangxintong updated SPARK-58186:
---------------------------------
Description:
h3. Problem
Spark's flat-bitmap infrastructure provides functions for building and merging
bitmaps
({{bitmap_construct_agg}}, {{bitmap_or_agg}}, {{bitmap_and_agg}},
{{bitmap_count}},
{{bitmap_bucket_number}}, and {{bitmap_bit_position}}), but lacks a predicate
to test whether a
specific bit is set. Without this, precomputed bitmaps cannot be used directly
as membership
predicates in SQL queries.
Related bitmap membership operations exist in other analytical databases, but
their argument
contracts are not necessarily identical to Spark's flat-bitmap mapping. This
proposal is defined
around Spark's existing bucket-number and bit-position functions.
h3. Proposed Function
|| Function || Signature || Description ||
| bitmap_contains | bitmap_contains(BINARY, numeric) -> BOOLEAN | Returns true
if the given bit position is set in one bitmap bucket, and false otherwise. The
numeric position is explicitly cast to BIGINT before the lookup. |
h3. Examples
{code:sql}
SELECT bitmap_contains(X '01', 0L);
-- true
SELECT bitmap_contains(X '01', 1L);
-- false
SELECT bitmap_contains(X '10', 4L);
-- true
-- Practical usage with bitmap_construct_agg
SELECT e.*
FROM events e
JOIN segment_bitmaps b
ON bitmap_bucket_number(e.user_id) = b.bucket
WHERE bitmap_contains(b.bm, bitmap_bit_position(e.user_id));
{code}
h3. Semantics
* The first argument must be a BINARY bitmap. The second argument accepts Spark
numeric types and
is explicitly cast to BIGINT using standard Spark cast semantics.
* The function checks a bit position within one bucket. It does not accept a
raw business value;
callers should use {{bitmap_bucket_number}} and {{bitmap_bit_position}} when
mapping values to
Spark's flat-bitmap representation.
* For a non-NULL bitmap, the function returns false for a negative position, a
position greater
than or equal to 32768, or a position beyond the actual byte-array length. A
standard flat
bitmap bucket contains 4096 bytes and 32768 bits; shorter binary arrays are
read safely.
* NULL bitmap or position inputs, including untyped NULL literals accepted by
the function's input
contract, produce SQL NULL.
* Finite fractional positions are truncated toward zero by the BIGINT cast.
Invalid or
out-of-range numeric values follow Spark cast semantics; ANSI mode controls
whether a failing
cast raises a cast error. Non-numeric, non-NULL position types are rejected
during analysis.
h3. Notes
* This function operates on Spark's existing flat-bitmap representation. It
does not introduce a
new bitmap storage format, replace bucket calculation, or change the physical
join plan.
* The function completes the flat-bitmap expression family alongside the
existing construct,
merge, count, bucket-number, and bit-position functions.
was:
h3. Problem
Spark's flat-bitmap infrastructure provides functions for building and merging
bitmaps (bitmap_construct_agg, bitmap_or_agg, bitmap_and_agg, bitmap_count,
bitmap_bucket_number, bitmap_bit_position), but lacks a predicate to test
whether a specific bit is set. Without this, precomputed bitmaps cannot be used
for membership filtering — the primary use case of bitmap indexes in analytical
databases.
Other databases with this capability:
- ClickHouse: bitmapContains(bitmap, value)
- Doris: bitmap_contains(bitmap, value)
- StarRocks: bitmap_contains(bitmap, value)
h3. Proposed Function
|| Function || Signature || Description ||
| bitmap_contains | bitmap_contains(BINARY, LONG) -> BOOLEAN | Returns true if
the bit at the given position is set in the bitmap. Returns false for
out-of-range positions. |
h3. Examples
{code:sql}
> SELECT bitmap_contains(X '01', 0L);
true
> SELECT bitmap_contains(X '01', 1L);
false
> SELECT bitmap_contains(X '10', 4L);
true
-- Practical usage with bitmap_construct_agg
SELECT e.*
FROM events e
JOIN segment_bitmaps b ON bitmap_bucket_number(e.user_id) = b.bucket
WHERE bitmap_contains(b.bm, bitmap_bit_position(e.user_id));
{code}
h3. Semantics
- Input: (BINARY bitmap, LONG bit_position)
- Output: BOOLEAN
- Returns false for out-of-range positions (negative or >= 32768), safe by
default
- NULL bitmap or NULL position → NULL result (standard SQL NULL propagation)
- Works with bitmaps of any length (not limited to the standard 4096-byte
bitmap)
h3. Notes
- This function operates on Spark's flat-bitmap format (fixed 4KB per bucket),
not RoaringBitmap. The 2-step mapping (bucket_number + bit_position) is still
required.
- Completes the flat-bitmap expression family alongside the existing
construct/merge/count functions.
> Add bitmap_contains function
> ----------------------------
>
> Key: SPARK-58186
> URL: https://issues.apache.org/jira/browse/SPARK-58186
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: jiangxintong
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> Spark's flat-bitmap infrastructure provides functions for building and
> merging bitmaps
> ({{bitmap_construct_agg}}, {{bitmap_or_agg}}, {{bitmap_and_agg}},
> {{bitmap_count}},
> {{bitmap_bucket_number}}, and {{bitmap_bit_position}}), but lacks a predicate
> to test whether a
> specific bit is set. Without this, precomputed bitmaps cannot be used
> directly as membership
> predicates in SQL queries.
> Related bitmap membership operations exist in other analytical databases, but
> their argument
> contracts are not necessarily identical to Spark's flat-bitmap mapping. This
> proposal is defined
> around Spark's existing bucket-number and bit-position functions.
> h3. Proposed Function
> || Function || Signature || Description ||
> | bitmap_contains | bitmap_contains(BINARY, numeric) -> BOOLEAN | Returns
> true if the given bit position is set in one bitmap bucket, and false
> otherwise. The numeric position is explicitly cast to BIGINT before the
> lookup. |
> h3. Examples
> {code:sql}
> SELECT bitmap_contains(X '01', 0L);
> -- true
> SELECT bitmap_contains(X '01', 1L);
> -- false
> SELECT bitmap_contains(X '10', 4L);
> -- true
> -- Practical usage with bitmap_construct_agg
> SELECT e.*
> FROM events e
> JOIN segment_bitmaps b
> ON bitmap_bucket_number(e.user_id) = b.bucket
> WHERE bitmap_contains(b.bm, bitmap_bit_position(e.user_id));
> {code}
> h3. Semantics
> * The first argument must be a BINARY bitmap. The second argument accepts
> Spark numeric types and
> is explicitly cast to BIGINT using standard Spark cast semantics.
> * The function checks a bit position within one bucket. It does not accept a
> raw business value;
> callers should use {{bitmap_bucket_number}} and {{bitmap_bit_position}}
> when mapping values to
> Spark's flat-bitmap representation.
> * For a non-NULL bitmap, the function returns false for a negative position,
> a position greater
> than or equal to 32768, or a position beyond the actual byte-array length.
> A standard flat
> bitmap bucket contains 4096 bytes and 32768 bits; shorter binary arrays are
> read safely.
> * NULL bitmap or position inputs, including untyped NULL literals accepted by
> the function's input
> contract, produce SQL NULL.
> * Finite fractional positions are truncated toward zero by the BIGINT cast.
> Invalid or
> out-of-range numeric values follow Spark cast semantics; ANSI mode controls
> whether a failing
> cast raises a cast error. Non-numeric, non-NULL position types are rejected
> during analysis.
> h3. Notes
> * This function operates on Spark's existing flat-bitmap representation. It
> does not introduce a
> new bitmap storage format, replace bucket calculation, or change the
> physical join plan.
> * The function completes the flat-bitmap expression family alongside the
> existing construct,
> merge, count, bucket-number, and bit-position functions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
