Hi all, With the previous roadmap wrapping up - complex types (Array/Row/Map/nesting), limit and prefix scan, schema-aware KV decoding, the metrics framework, and write optimisations, thanks to everyone who contributed and reviewed, I'd like to open discussion on the roadmap.
I propose framing around a single goal: make fluss-rust a first-class analytical query surface - a Rust-native SQL/DataFrame path over Fluss (DataFusion, and through it Polars/DuckDB/gateways), reading the lake tier at scale, built on the analytical primitives we just landed. As before, I grouped the items into themes with an initial must-have / stretch positioning. Please push back where you disagree. THEME 1: DATAFUSION INTEGRATION ADAPTER [MUST-HAVE] A standalone fluss-datafusion crate exposing TableProvider + Catalog over the existing client, so SELECT ... FROM . works from any DataFusion-based engine. It should be framed as an integration adapter over the Rust core, not a bundled engine. A gateway (FIP-32) or any analytical consumer can use it or not. Scope: TableProvider + CatalogProvider/SchemaProvider backed by the metadata path. Map DataFusion's projection/filter/limit pushdown onto the client's existing access paths: a full primary-key equality becomes a lookup, a bucket-key prefix becomes a prefix lookup, LIMIT becomes a bounded scan, otherwise a log scan. Filters an access path fully satisfies are reported exact, the rest are left for DataFusion to apply (server-side filter pushdown is Theme 3). THEME 2: LAKE + LOG UNION READ (PAIMON-FIRST) [MUST-HAVE] A client-side read that presents a lake-enabled table as one table. Its history is tiered to the lake (Paimon) and recent writes are still in the Fluss log, the lake snapshot records the log offset it covers, so we read the lake up to that offset and the Fluss log past it, then combine. Scope: - Per bucket: the lake snapshot plus the Fluss log past the tiered offset. - Log tables: concatenate the two. PK tables: apply the newer log records to get current state. - Paimon first, reusing paimon-rust, Iceberg/Lance later the same way. The DataFusion adapter reads lake-enabled tables through this path. - Bounded batch first, streaming later Reusing paimon-rust means wrapping an existing reader, so it's rather feasible and straightforward. THEME 3: FOUNDATIONS [STRETCH] Server-side value-predicate (filter) pushdown. The proto field exists, but the client currently sends none. This prunes lake/log scans and unblocks the gateway's filter pushdown (the open question I raised for 0.2.0), cross-checked against the Java PredicateConverter semantics. THEME 4: GATEWAY [STRETCH] A thin HTTP frontend over the client we already have independent of Themes 1–3. It's how non-Rust/non-SQL clients (TS/JS, microservices, curl) write to Fluss and do simple key access. - Write: upsert/append, delete, create/drop tables, metadata. The mature client write path essentially. - Read: point lookup, prefix lookup, bounded log / CDC poll - one endpoint per existing primitive, bounded and stateless. The tradeoff: no joins, filters, or aggregations - that's the SQL surface, and it needs the adapter and probably will be an ongoing effort to optimise later. - If Themes 1–2 ship, we unlock SQL frontends on top - PostgreSQL via datafusion-postgres, as one example In short: REST is the no-regret floor, SQL is the ceiling deferred until Themes 1–2. PARALLEL TRACK: ELIXIR + BINDING EXPOSURE Continue Elixir parity and expose newly-landed primitives (prefix lookup, etc.) across Python / C++ / Elixir. GOVERNANCE TRACK FIP-40: Consolidate apache/fluss-rust into apache/fluss - PR is ready, looks pretty good, waiting for a good moment to rebase/merge. OPEN QUESTIONS - Themes 1–2 must-have, the rest stretch. Reasonable? Anything missing or mis-ordered? - Paimon first, Iceberg later? - Pin one arrow/DataFusion version across core, union-read/lake, and DF? - Separate crates: lake kernel, DataFusion adapter, gateway? Looking forward to feedback. -- Anton
