Thanks Anton for kicking this off — funnily enough, I'd been arriving at very similar conclusions while prototyping on my side, so let me share a bit of what came out of those experiments.
> THEME 1: DATAFUSION INTEGRATION ADAPTER [MUST-HAVE] Agree. Having the DataFusion integration live in fluss-rust would make it reusable across more downstream projects. One thing we might want to keep in mind: it would probably be worth aligning the DataFusion and Arrow versions with pg-datafusion so the whole stack stays on a consistent set of versions. > THEME 2: LAKE + LOG UNION READ (PAIMON-FIRST) [MUST-HAVE] Agree. I'd lean toward putting this in a separate fluss-lake module that exposes a client capable of doing the union read over the lake and the Fluss log. That way the lake-side dependencies stay isolated in their own module and don't bleed into the core fluss-rust code. One thing that might be worth flagging here: the lake side already supports predicate pushdown, while Fluss doesn't yet — so we'd probably think about how the union read handles that asymmetry in the interim. THEME 4: GATEWAY [STRETCH] — A thin HTTP frontend over the client I wonder if we could expand on this one a bit. My suggestion would be to split the SQL REST surface and the plain REST API into two separate FIPs. A read/write REST API on its own — somewhat in the spirit of kafka-rest, covering reads, writes, and metadata management — would already be a quick win for broadening Fluss's data ingestion/access reach, and it doesn't depend on anything else. The SQL REST surface, on the other hand, has to wait on Themes 1–2, so it might make sense to split them and ship the plain REST API first. Also, it might make sense to have the gateway as a separate crate inside the Fluss project itself — especially since fluss-rust will be moving into the Fluss repo anyway. On a personal note, this is something I'm really excited about and would love to help push forward. I'll share a write-up of what came out of my demo experiments with the community over the next few weeks. Best regards, Junbo Wang > 2026年6月23日 20:55，Yuxia Luo <[email protected]> 写道： > > Thanks for putting this together. +1 on the overall framing. My answers to > the open questions, plus one addition: > > Open questions: > - Themes 1-2 as must-have, rest stretch: agree, the ordering makes sense to > me. > - Paimon first, Iceberg later: +1. Reusing paimon-rust is the pragmatic path, > and the union-read design generalizes to Iceberg the same way later. > - Pin one arrow/DataFusion version across core, union-read/lake, and DF: yes, > we should pin a single version. Letting these drift will cause painful Arrow > type/ABI mismatches across the crate boundary, so a shared pinned version > (bumped deliberately, not per-crate) is worth the discipline. > - Separate crates (lake kernel, DataFusion adapter, gateway): +1 on separate > crates. It keeps the dependency surface clean - the adapter and gateway are > optional consumers, and the lake kernel shouldn't drag DataFusion into anyone > who only needs the core read path. > > One addition I'd like to propose for the scope: > PK-table CHANGELOG read, and expose it across the bindings - specifically C++ > and Python - so non-Rust CDC consumers can subscribe. > > On 2026/06/22 14:49:52 Anton Borisov wrote: >> Hi all, >> >> With the previous roadmap wrapping up - complex types >> (Array/Row/Map/nesting), limit and prefix scan, schema-aware KV >> decoding, the metrics framework, and write optimisations, thanks to >> everyone who contributed and reviewed, I'd like to open discussion on >> the roadmap. >> >> I propose framing around a single goal: make fluss-rust a first-class >> analytical query surface - a Rust-native SQL/DataFrame path over Fluss >> (DataFusion, and through it Polars/DuckDB/gateways), reading the lake >> tier at scale, built on the analytical primitives we just landed. >> >> As before, I grouped the items into themes with an initial must-have / >> stretch positioning. Please push back where you disagree. >> >> THEME 1: DATAFUSION INTEGRATION ADAPTER [MUST-HAVE] >> A standalone fluss-datafusion crate exposing TableProvider + Catalog >> over the existing client, so SELECT ... FROM . works from any >> DataFusion-based engine. It should be framed as an integration adapter >> over the Rust core, not a bundled engine. A gateway (FIP-32) or any >> analytical consumer can use it or not. >> >> Scope: TableProvider + CatalogProvider/SchemaProvider backed by the >> metadata path. >> >> Map DataFusion's projection/filter/limit pushdown onto the client's >> existing access paths: a full primary-key equality becomes a lookup, a >> bucket-key prefix becomes a prefix lookup, LIMIT becomes a bounded >> scan, otherwise a log scan. Filters an access path fully satisfies are >> reported exact, the rest are left for DataFusion to apply (server-side >> filter pushdown is Theme 3). >> >> THEME 2: LAKE + LOG UNION READ (PAIMON-FIRST) [MUST-HAVE] >> A client-side read that presents a lake-enabled table as one table. >> Its history is tiered to the lake (Paimon) and recent writes are still >> in the Fluss log, the lake snapshot records the log offset it covers, >> so we read the lake up to that offset and the Fluss log past it, then >> combine. >> Scope: >> - Per bucket: the lake snapshot plus the Fluss log past the tiered offset. >> - Log tables: concatenate the two. PK tables: apply the newer log >> records to get current state. >> - Paimon first, reusing paimon-rust, Iceberg/Lance later the same >> way. The DataFusion adapter reads lake-enabled tables through this >> path. >> - Bounded batch first, streaming later >> >> Reusing paimon-rust means wrapping an existing reader, so it's rather >> feasible and straightforward. >> >> THEME 3: FOUNDATIONS [STRETCH] >> Server-side value-predicate (filter) pushdown. The proto field exists, >> but the client currently sends none. This prunes lake/log scans and >> unblocks the gateway's filter pushdown (the open question I raised for >> 0.2.0), cross-checked against the Java PredicateConverter semantics. >> >> THEME 4: GATEWAY [STRETCH] >> A thin HTTP frontend over the client we already have independent of >> Themes 1–3. It's how non-Rust/non-SQL clients (TS/JS, microservices, >> curl) write to Fluss and do simple key access. >> - Write: upsert/append, delete, create/drop tables, metadata. The >> mature client write path essentially. >> - Read: point lookup, prefix lookup, bounded log / CDC poll - one >> endpoint per existing primitive, bounded and stateless. >> The tradeoff: no joins, filters, or aggregations - that's the SQL >> surface, and it needs the adapter and probably will be an ongoing >> effort to optimise later. >> - If Themes 1–2 ship, we unlock SQL frontends on top - PostgreSQL >> via datafusion-postgres, as one example >> In short: REST is the no-regret floor, SQL is the ceiling deferred >> until Themes 1–2. >> >> PARALLEL TRACK: ELIXIR + BINDING EXPOSURE >> Continue Elixir parity and expose newly-landed primitives (prefix >> lookup, etc.) across Python / C++ / Elixir. >> >> GOVERNANCE TRACK >> FIP-40: Consolidate apache/fluss-rust into apache/fluss - PR is ready, >> looks pretty good, waiting for a good moment to rebase/merge. >> >> OPEN QUESTIONS >> - Themes 1–2 must-have, the rest stretch. Reasonable? Anything >> missing or mis-ordered? >> - Paimon first, Iceberg later? >> - Pin one arrow/DataFusion version across core, union-read/lake, and DF? >> - Separate crates: lake kernel, DataFusion adapter, gateway? >> >> Looking forward to feedback. >> >> -- Anton >>
