Add the missing links in my previous email [1] https://github.com/flink-extended/flink-roaringbitmap [2] https://github.com/ChenHuajun/pg_roaringbitmap
On Sun, 19 Apr 2026 at 23:51, Jark Wu <[email protected]> wrote: > Hi Prajwal, > > Thanks for the proposal! RoaringBitmap SQL functions are a valuable > addition to Fluss. I have a few suggestions: > > *1. Defer BITMAP Type Introduction* > > I suggest excluding the BITMAP type (Layer 1) from this FIP and keeping > bitmap columns as BYTES. > > The primary benefit of a native BITMAP type — as proposed in Flink's > FLIP-556 — is that functions can pass Bitmap Java objects in-memory between > operators without serialization/deserialization at each step. However, > since FLIP-556 is still under discussion and not yet available in Flink, > the Flink UDFs we register can only operate on byte[] (BYTES). On the Fluss > server side, the merge engine aggregators always deserialize from byte[], > perform the OR, and serialize back — regardless of whether the logical type > is BYTES or BITMAP. > > Introducing the BITMAP type offers limited immediate benefits, while > ensuring compatibility is a complex issue that warrants separate > discussion. This includes addressing how type conversions between Fluss and > various engines bridge to BITMAP without breaking existing compatibility, > which is not currently covered in this FIP. Therefore, I propose deferring > the introduction of the BITMAP type to simplify and accelerate the approval > of the current FIP. > > The FIP can focus on what matters most: SQL functions operating on BYTES > columns that is roaringbitmap binary format underlying. We can revisit the > BITMAP type once FLIP-556 has been released in Flink. > > *2. Function Naming: Use "RB_" Prefix Instead of "BITMAP_"* > > I suggest using the "RB_" prefix rather than "BITMAP_". Reasons: > > (1) Intentional differentiation from Flink built-ins. When FLIP-556 > eventually lands, Flink will have its own BITMAP_* built-in functions > operating on a native BITMAP type. Our functions operate on BYTES and are > registered via FlussCatalog. Using RB_ avoids ambiguity and accidental > invocation of the wrong function at runtime. > (2) Consistent with Fluss conventions. Fluss already uses rbm32/rbm64 as > aggregation function identifiers in the merge engine. RB_ aligns naturally > with this existing naming. > (3) PostgreSQL protocol compatibility. Fluss has future plans for PG > protocol compatibility. The de facto standard for RoaringBitmap in > PostgreSQL (pg_roaringbitmap) uses rb_ prefix. Adopting this now avoids a > rename later. > > *3. Expand the Function Set* > > Referencing the function sets from flink-roaringbitmap[1] and > pg_roaringbitmap[2], I suggest the following additions: > > Phase 1 (add these — they are very commonly used): > > * rb_build(ARRAY<INT>) → BYTES — construct a bitmap from an array without > aggregation > * rb_contains(BYTES, INT) → BOOLEAN — check if a bitmap contains a value > * rb_to_array(BYTES) → ARRAY<INT> — convert bitmap back to array > (debugging, integration) > > Phase 2 (add these to complete set algebra): > > * rb_xor_agg(BYTES) → BYTES — XOR aggregation > * rb_xor(BYTES, BYTES) → BYTES — XOR of two bitmaps > * rb_andnot(BYTES, BYTES) → BYTES — difference (AND NOT) of two bitmaps > > Also, consider deferring "bitmap_build_agg_64" — the 64-bit RoaringBitmap > serialization format is not standardized, which limits cross-engine > interoperability. FLIP-556 explicitly scoped to 32-bit only for this reason. > > *4. Pushdown Optimization Needs Careful Scoping* > > The bitmap OR aggregation pushdown is the riskiest part of this proposal. > Unlike count(*) (metadata-only), bitmap aggregation pushdown requires the > TabletServer to scan all rows, deserialize each bitmap, and perform > OR-merge. Key concerns: > > * Full table scan on server: Could starve normal read/write operations on > the same tablet. > * Memory pressure: Large bitmaps + concurrent queries could cause OOM. No > memory budget or backpressure is described. > * Server is not a query engine: Currently, TabletServer is designed for > low-latency read/write, not OLAP-style full scans. > > Reading all the bytes and computing in the Flink/StarRocks may be more > efficient and stable for now. I suggest deferring pushdown to future work. > > Overall, this is a solid proposal. Narrowing the scope (skip BITMAP type, > use RB_ prefix, expand functions, carefully scope pushdown) would make it > more incremental and lower-risk. > > Best regards, > Jark Wu >
