Hi Jark, Thank you for the thorough and precise feedback. I agree with all four points you raised and have updated the proposal accordingly.
1. On deferring the BITMAP type: Agreed. Since FLIP-556 is not yet available in Flink, the in-memory benefits of a native type cannot be realized as UDFs must operate on byte[] regardless of the logical type. I have removed Layer 1 and will keep bitmap columns as BYTES. We can revisit the BITMAP type in a dedicated FIP once FLIP-556 lands. 2. On the RB_ prefix: Agreed. Using the "RB_" prefix avoids future conflicts with Flink built-ins, aligns with existing rbm32/64 naming in Fluss, and ensures compatibility with pg_roaringbitmap conventions. All functions have been renamed. 3. On expanding the function set: Thank you for the references. I have added rb_build, rb_contains, and rb_to_array to Phase 1, and rb_xor_agg, rb_xor, and rb_andnot to Phase 2. I have also deferred rb_build_agg_64 due to the lack of standardized 64-bit serialization formats, consistent with the scope of FLIP-556. 3. On deferring pushdown: Agreed. The concerns regarding full table scans on the TabletServer and potential memory pressure are valid. I have deferred Layer 3 to future work and documented these architectural concerns. FIP-37 is now a focused single-layer proposal implementing 10 RoaringBitmap SQL functions registered via FlussCatalog on BYTES columns. The updated document here: https://cwiki.apache.org/confluence/display/FLUSS/FIP-37%3A+Native+RoaringBitmap+Integration+for+Apache+Fluss Best regards, Prajwal Banakar On Sun, 19 Apr 2026 at 21:23, Jark Wu <[email protected]> wrote: > Add the missing links in my previous email > > [1] https://github.com/flink-extended/flink-roaringbitmap > [2] https://github.com/ChenHuajun/pg_roaringbitmap > > On Sun, 19 Apr 2026 at 23:51, Jark Wu <[email protected]> wrote: > > > Hi Prajwal, > > > > Thanks for the proposal! RoaringBitmap SQL functions are a valuable > > addition to Fluss. I have a few suggestions: > > > > *1. Defer BITMAP Type Introduction* > > > > I suggest excluding the BITMAP type (Layer 1) from this FIP and keeping > > bitmap columns as BYTES. > > > > The primary benefit of a native BITMAP type — as proposed in Flink's > > FLIP-556 — is that functions can pass Bitmap Java objects in-memory > between > > operators without serialization/deserialization at each step. However, > > since FLIP-556 is still under discussion and not yet available in Flink, > > the Flink UDFs we register can only operate on byte[] (BYTES). On the > Fluss > > server side, the merge engine aggregators always deserialize from byte[], > > perform the OR, and serialize back — regardless of whether the logical > type > > is BYTES or BITMAP. > > > > Introducing the BITMAP type offers limited immediate benefits, while > > ensuring compatibility is a complex issue that warrants separate > > discussion. This includes addressing how type conversions between Fluss > and > > various engines bridge to BITMAP without breaking existing compatibility, > > which is not currently covered in this FIP. Therefore, I propose > deferring > > the introduction of the BITMAP type to simplify and accelerate the > approval > > of the current FIP. > > > > The FIP can focus on what matters most: SQL functions operating on BYTES > > columns that is roaringbitmap binary format underlying. We can revisit > the > > BITMAP type once FLIP-556 has been released in Flink. > > > > *2. Function Naming: Use "RB_" Prefix Instead of "BITMAP_"* > > > > I suggest using the "RB_" prefix rather than "BITMAP_". Reasons: > > > > (1) Intentional differentiation from Flink built-ins. When FLIP-556 > > eventually lands, Flink will have its own BITMAP_* built-in functions > > operating on a native BITMAP type. Our functions operate on BYTES and are > > registered via FlussCatalog. Using RB_ avoids ambiguity and accidental > > invocation of the wrong function at runtime. > > (2) Consistent with Fluss conventions. Fluss already uses rbm32/rbm64 as > > aggregation function identifiers in the merge engine. RB_ aligns > naturally > > with this existing naming. > > (3) PostgreSQL protocol compatibility. Fluss has future plans for PG > > protocol compatibility. The de facto standard for RoaringBitmap in > > PostgreSQL (pg_roaringbitmap) uses rb_ prefix. Adopting this now avoids a > > rename later. > > > > *3. Expand the Function Set* > > > > Referencing the function sets from flink-roaringbitmap[1] and > > pg_roaringbitmap[2], I suggest the following additions: > > > > Phase 1 (add these — they are very commonly used): > > > > * rb_build(ARRAY<INT>) → BYTES — construct a bitmap from an array without > > aggregation > > * rb_contains(BYTES, INT) → BOOLEAN — check if a bitmap contains a value > > * rb_to_array(BYTES) → ARRAY<INT> — convert bitmap back to array > > (debugging, integration) > > > > Phase 2 (add these to complete set algebra): > > > > * rb_xor_agg(BYTES) → BYTES — XOR aggregation > > * rb_xor(BYTES, BYTES) → BYTES — XOR of two bitmaps > > * rb_andnot(BYTES, BYTES) → BYTES — difference (AND NOT) of two bitmaps > > > > Also, consider deferring "bitmap_build_agg_64" — the 64-bit RoaringBitmap > > serialization format is not standardized, which limits cross-engine > > interoperability. FLIP-556 explicitly scoped to 32-bit only for this > reason. > > > > *4. Pushdown Optimization Needs Careful Scoping* > > > > The bitmap OR aggregation pushdown is the riskiest part of this proposal. > > Unlike count(*) (metadata-only), bitmap aggregation pushdown requires the > > TabletServer to scan all rows, deserialize each bitmap, and perform > > OR-merge. Key concerns: > > > > * Full table scan on server: Could starve normal read/write operations on > > the same tablet. > > * Memory pressure: Large bitmaps + concurrent queries could cause OOM. No > > memory budget or backpressure is described. > > * Server is not a query engine: Currently, TabletServer is designed for > > low-latency read/write, not OLAP-style full scans. > > > > Reading all the bytes and computing in the Flink/StarRocks may be more > > efficient and stable for now. I suggest deferring pushdown to future > work. > > > > Overall, this is a solid proposal. Narrowing the scope (skip BITMAP type, > > use RB_ prefix, expand functions, carefully scope pushdown) would make it > > more incremental and lower-risk. > > > > Best regards, > > Jark Wu > > >
