Hi Prajwal, Thanks for the proposal! RoaringBitmap SQL functions are a valuable addition to Fluss. I have a few suggestions:
*1. Defer BITMAP Type Introduction* I suggest excluding the BITMAP type (Layer 1) from this FIP and keeping bitmap columns as BYTES. The primary benefit of a native BITMAP type — as proposed in Flink's FLIP-556 — is that functions can pass Bitmap Java objects in-memory between operators without serialization/deserialization at each step. However, since FLIP-556 is still under discussion and not yet available in Flink, the Flink UDFs we register can only operate on byte[] (BYTES). On the Fluss server side, the merge engine aggregators always deserialize from byte[], perform the OR, and serialize back — regardless of whether the logical type is BYTES or BITMAP. Introducing the BITMAP type offers limited immediate benefits, while ensuring compatibility is a complex issue that warrants separate discussion. This includes addressing how type conversions between Fluss and various engines bridge to BITMAP without breaking existing compatibility, which is not currently covered in this FIP. Therefore, I propose deferring the introduction of the BITMAP type to simplify and accelerate the approval of the current FIP. The FIP can focus on what matters most: SQL functions operating on BYTES columns that is roaringbitmap binary format underlying. We can revisit the BITMAP type once FLIP-556 has been released in Flink. *2. Function Naming: Use "RB_" Prefix Instead of "BITMAP_"* I suggest using the "RB_" prefix rather than "BITMAP_". Reasons: (1) Intentional differentiation from Flink built-ins. When FLIP-556 eventually lands, Flink will have its own BITMAP_* built-in functions operating on a native BITMAP type. Our functions operate on BYTES and are registered via FlussCatalog. Using RB_ avoids ambiguity and accidental invocation of the wrong function at runtime. (2) Consistent with Fluss conventions. Fluss already uses rbm32/rbm64 as aggregation function identifiers in the merge engine. RB_ aligns naturally with this existing naming. (3) PostgreSQL protocol compatibility. Fluss has future plans for PG protocol compatibility. The de facto standard for RoaringBitmap in PostgreSQL (pg_roaringbitmap) uses rb_ prefix. Adopting this now avoids a rename later. *3. Expand the Function Set* Referencing the function sets from flink-roaringbitmap[1] and pg_roaringbitmap[2], I suggest the following additions: Phase 1 (add these — they are very commonly used): * rb_build(ARRAY<INT>) → BYTES — construct a bitmap from an array without aggregation * rb_contains(BYTES, INT) → BOOLEAN — check if a bitmap contains a value * rb_to_array(BYTES) → ARRAY<INT> — convert bitmap back to array (debugging, integration) Phase 2 (add these to complete set algebra): * rb_xor_agg(BYTES) → BYTES — XOR aggregation * rb_xor(BYTES, BYTES) → BYTES — XOR of two bitmaps * rb_andnot(BYTES, BYTES) → BYTES — difference (AND NOT) of two bitmaps Also, consider deferring "bitmap_build_agg_64" — the 64-bit RoaringBitmap serialization format is not standardized, which limits cross-engine interoperability. FLIP-556 explicitly scoped to 32-bit only for this reason. *4. Pushdown Optimization Needs Careful Scoping* The bitmap OR aggregation pushdown is the riskiest part of this proposal. Unlike count(*) (metadata-only), bitmap aggregation pushdown requires the TabletServer to scan all rows, deserialize each bitmap, and perform OR-merge. Key concerns: * Full table scan on server: Could starve normal read/write operations on the same tablet. * Memory pressure: Large bitmaps + concurrent queries could cause OOM. No memory budget or backpressure is described. * Server is not a query engine: Currently, TabletServer is designed for low-latency read/write, not OLAP-style full scans. Reading all the bytes and computing in the Flink/StarRocks may be more efficient and stable for now. I suggest deferring pushdown to future work. Overall, this is a solid proposal. Narrowing the scope (skip BITMAP type, use RB_ prefix, expand functions, carefully scope pushdown) would make it more incremental and lower-risk. Best regards, Jark Wu
