Hi Prajwal, Thanks for the update. The new FIP looks good to me. +1 to start the vote.
It would be great if Yang Wang could take a final look, given his contributions to the RoaringBitmap aggregate merge engine and the related blog post. Best, Jark On Tue, 21 Apr 2026 at 13:58, Prajwal Banakar <[email protected]> wrote: > > Hi Jark, > > Thank you for the thorough and precise feedback. I agree with all four > points you raised and have updated the proposal accordingly. > > 1. On deferring the BITMAP type: > Agreed. Since FLIP-556 is not yet available in Flink, the in-memory > benefits of a native type cannot be realized as UDFs must operate on byte[] > regardless of the logical type. I have removed Layer 1 and will keep bitmap > columns as BYTES. We can revisit the BITMAP type in a dedicated FIP once > FLIP-556 lands. > > 2. On the RB_ prefix: > Agreed. Using the "RB_" prefix avoids future conflicts with Flink > built-ins, aligns with existing rbm32/64 naming in Fluss, and ensures > compatibility with pg_roaringbitmap conventions. All functions have been > renamed. > > 3. On expanding the function set: > Thank you for the references. I have added rb_build, rb_contains, and > rb_to_array to Phase 1, and rb_xor_agg, rb_xor, and rb_andnot to Phase 2. I > have also deferred rb_build_agg_64 due to the lack of standardized 64-bit > serialization formats, consistent with the scope of FLIP-556. > > 3. On deferring pushdown: > Agreed. The concerns regarding full table scans on the TabletServer and > potential memory pressure are valid. I have deferred Layer 3 to future work > and documented these architectural concerns. > > FIP-37 is now a focused single-layer proposal implementing 10 RoaringBitmap > SQL functions registered via FlussCatalog on BYTES columns. The updated > document here: > https://cwiki.apache.org/confluence/display/FLUSS/FIP-37%3A+Native+RoaringBitmap+Integration+for+Apache+Fluss > > Best regards, > Prajwal Banakar > > On Sun, 19 Apr 2026 at 21:23, Jark Wu <[email protected]> wrote: > > > Add the missing links in my previous email > > > > [1] https://github.com/flink-extended/flink-roaringbitmap > > [2] https://github.com/ChenHuajun/pg_roaringbitmap > > > > On Sun, 19 Apr 2026 at 23:51, Jark Wu <[email protected]> wrote: > > > > > Hi Prajwal, > > > > > > Thanks for the proposal! RoaringBitmap SQL functions are a valuable > > > addition to Fluss. I have a few suggestions: > > > > > > *1. Defer BITMAP Type Introduction* > > > > > > I suggest excluding the BITMAP type (Layer 1) from this FIP and keeping > > > bitmap columns as BYTES. > > > > > > The primary benefit of a native BITMAP type — as proposed in Flink's > > > FLIP-556 — is that functions can pass Bitmap Java objects in-memory > > between > > > operators without serialization/deserialization at each step. However, > > > since FLIP-556 is still under discussion and not yet available in Flink, > > > the Flink UDFs we register can only operate on byte[] (BYTES). On the > > Fluss > > > server side, the merge engine aggregators always deserialize from byte[], > > > perform the OR, and serialize back — regardless of whether the logical > > type > > > is BYTES or BITMAP. > > > > > > Introducing the BITMAP type offers limited immediate benefits, while > > > ensuring compatibility is a complex issue that warrants separate > > > discussion. This includes addressing how type conversions between Fluss > > and > > > various engines bridge to BITMAP without breaking existing compatibility, > > > which is not currently covered in this FIP. Therefore, I propose > > deferring > > > the introduction of the BITMAP type to simplify and accelerate the > > approval > > > of the current FIP. > > > > > > The FIP can focus on what matters most: SQL functions operating on BYTES > > > columns that is roaringbitmap binary format underlying. We can revisit > > the > > > BITMAP type once FLIP-556 has been released in Flink. > > > > > > *2. Function Naming: Use "RB_" Prefix Instead of "BITMAP_"* > > > > > > I suggest using the "RB_" prefix rather than "BITMAP_". Reasons: > > > > > > (1) Intentional differentiation from Flink built-ins. When FLIP-556 > > > eventually lands, Flink will have its own BITMAP_* built-in functions > > > operating on a native BITMAP type. Our functions operate on BYTES and are > > > registered via FlussCatalog. Using RB_ avoids ambiguity and accidental > > > invocation of the wrong function at runtime. > > > (2) Consistent with Fluss conventions. Fluss already uses rbm32/rbm64 as > > > aggregation function identifiers in the merge engine. RB_ aligns > > naturally > > > with this existing naming. > > > (3) PostgreSQL protocol compatibility. Fluss has future plans for PG > > > protocol compatibility. The de facto standard for RoaringBitmap in > > > PostgreSQL (pg_roaringbitmap) uses rb_ prefix. Adopting this now avoids a > > > rename later. > > > > > > *3. Expand the Function Set* > > > > > > Referencing the function sets from flink-roaringbitmap[1] and > > > pg_roaringbitmap[2], I suggest the following additions: > > > > > > Phase 1 (add these — they are very commonly used): > > > > > > * rb_build(ARRAY<INT>) → BYTES — construct a bitmap from an array without > > > aggregation > > > * rb_contains(BYTES, INT) → BOOLEAN — check if a bitmap contains a value > > > * rb_to_array(BYTES) → ARRAY<INT> — convert bitmap back to array > > > (debugging, integration) > > > > > > Phase 2 (add these to complete set algebra): > > > > > > * rb_xor_agg(BYTES) → BYTES — XOR aggregation > > > * rb_xor(BYTES, BYTES) → BYTES — XOR of two bitmaps > > > * rb_andnot(BYTES, BYTES) → BYTES — difference (AND NOT) of two bitmaps > > > > > > Also, consider deferring "bitmap_build_agg_64" — the 64-bit RoaringBitmap > > > serialization format is not standardized, which limits cross-engine > > > interoperability. FLIP-556 explicitly scoped to 32-bit only for this > > reason. > > > > > > *4. Pushdown Optimization Needs Careful Scoping* > > > > > > The bitmap OR aggregation pushdown is the riskiest part of this proposal. > > > Unlike count(*) (metadata-only), bitmap aggregation pushdown requires the > > > TabletServer to scan all rows, deserialize each bitmap, and perform > > > OR-merge. Key concerns: > > > > > > * Full table scan on server: Could starve normal read/write operations on > > > the same tablet. > > > * Memory pressure: Large bitmaps + concurrent queries could cause OOM. No > > > memory budget or backpressure is described. > > > * Server is not a query engine: Currently, TabletServer is designed for > > > low-latency read/write, not OLAP-style full scans. > > > > > > Reading all the bytes and computing in the Flink/StarRocks may be more > > > efficient and stable for now. I suggest deferring pushdown to future > > work. > > > > > > Overall, this is a solid proposal. Narrowing the scope (skip BITMAP type, > > > use RB_ prefix, expand functions, carefully scope pushdown) would make it > > > more incremental and lower-risk. > > > > > > Best regards, > > > Jark Wu > > > > >
