Hi all, With 0.1.0-incubating shipped, thanks to Yunhong(RM), Yuxia, Keith and everyone who contributed, I would like to open discussion on priorities for the next fluss-rust release.
I propose discussing 0.2.0-incubating roadmap, with the goal of making fluss-rust production-viable for analytical consumers such as DataFusion, Polars, DuckDB, and CDC pipelines, while also closing a known correctness gaps, types support. Rather than a flat list, I grouped the items into four themes, plus a parallel Elixir track. I also proposed an initial tiering of must-have versus stretch goals. Please push back where you disagree. THEME 1: CORRECTNESS: SCHEMA-AWARE DECODING ON KV LOOKUP [MUST-HAVE] KV rows in Fluss carry a 2-byte schema ID prefix. The KV batch reader already uses this ID to select the correct decoder via read_context.get_row_decoder. The lookup path does not. Today, both Lookuper::lookup and PrefixKeyLookuper::lookup decode every row using the table's latest row_type, so rows written under an earlier schema version can be returned with incorrect field values. Scope: Schema-ID-aware decoding in Lookuper and PrefixKeyLookuper Client-side schema cache (fetch schema by ID, LRU). The metadata path may need minor work depending on what is already cached in TableInfo Tests covering multi-schema-version rows through both lookup paths Out of scope for this theme: The KvRecordBatch decode is already schema-ID-aware - confined to Lookup only. Log tables are fine as it is. Tracking: https://github.com/apache/fluss-rust/issues/508 THEME 2: UNLOCK ANALYTICAL INTEGRATIONS [MUST-HAVE] Two features that together would make Fluss a first-class source for the analytical ecosystem. (a) Limit scan, with full client-side support Proto and RPC plumbing already landed (PR #472 / issue #315). The remaining work is exposing the limit through the public scanner API so that limit pushdown reaches DataFusion integrations. Issue #311 was filed explicitly to support fluss-datafusion and is also useful for other analytical consumers. https://github.com/apache/fluss-rust/issues/311 (b) Changelog subscription for KV tables This would let downstream consumers stream change events out of primary-key tables, which is the natural companion to upsert writes. https://github.com/apache/fluss-rust/issues/380 THEME 3: TYPE AND BINDING COMPLETENESS [MUST-HAVE] Close the remaining gaps so that all bindings reach parity on primitive and composite types. Map type in Rust core (depends on array support already landed) https://github.com/apache/fluss-rust/issues/387 Array support in C++ bindings https://github.com/apache/fluss-rust/issues/468 Cross-binding integration tests for arrays https://github.com/apache/fluss-rust/issues/441 Prefix lookup exposed in Python / C++ / Elixir Rust core just landed via PR #500 / issue #499. I will file tracking issues for binding exposure this week. Already landed and out of scope here: Array support in Rust (#386) Array support in Python (#469) THEME 4: FOUNDATIONS FOR THE NEXT CYCLE [STRETCH] Benchmark harness producing reproducible Rust / Python / C++ numbers, ideally suitable for nightly regression tracking in CI. A separate FIP with full design will follow. I already have a raw prototype that I use to verify regressions locally. https://github.com/apache/fluss-rust/issues/489 ArrowWriter pooling on the write path, mirroring Java's ArrowWriterPool. This complements the jemalloc / pre-size work (#429) and null-append optimization (#470) already landed. It would be best to land this on top of benchmark coverage so we can quantify the gain. https://github.com/apache/fluss-rust/issues/444 PARALLEL TRACK: ELIXIR BINDINGS 0.1 FEATURE PARITY The Elixir bindings (initial PR #452) need a broader set of features to reach parity with Rust, Python and C++. Open issues: #457, #458, #459, #460, #461, #462, #463, #464, #465, #466, #467, #493, #498 I am happy for this to slip to its own release if the core team agrees. GOVERNANCE TRACKS: SEPARATE FIPS Flagging these so the roadmap accounts for them, but each would get its own DISCUSS / VOTE thread: FIP: Benchmark harness design (supports Theme 4) FIP: Consolidate apache/fluss-rust into apache/fluss OPEN QUESTIONS - Is the proposed tiering reasonable, with Themes 1-3 as must-have and Theme 4 as stretch? Is anything missing or mis-prioritized? - Should we prioritize fluss-rust repo merging first? - Go/Typescript bindings to consider? - FIP-32 Gateway builds on fluss-rust (projection pushdown already supported). Should we add filter pushdown to 0.2.0 to unblock it, or defer? - Are there other items the community has been asking for that should be considered here, such as observability/metrics(we have https://github.com/apache/fluss-rust/issues/390 to start work there), TLS/mTLS, or auth beyond SASL/PLAIN? Looking forward to feedback. Best, — Anton
