Hi devs, At first, the Fluss Rust client still lacks several key capabilities — including Paimon data reading, schema evolution support, and client-side fault recovery (e.g., retry logic and metadata refresh during rolling upgrades). Overall, there is still a gap before it becomes production-ready.
Secondly, I've reviewed the community's Fluss Gateway design and noticed that authentication and authorization are not addressed. When using DataFusion as the query engine, this is not a concern due to its lightweight, embedded nature. However, as we move toward making Fluss Gateway production-ready, authentication and authorization become unavoidable. I'd like to raise a few design questions for discussion. Key Questions: 1. Authentication between Users and Fluss Gateway How should users authenticate with Fluss Gateway? Should this authentication mechanism be unified with Fluss's native authentication? >From what I've seen, most comparable products keep these two layers decoupled — the gateway has its own authentication, but ACLs are shared/integrated with the backend cluster. I'd prefer a similar approach: gateway-side authentication can be independent, but ACL enforcement should be unified with Fluss. 2. Gateway-to-Server Authentication: Single vs. Multi-User Does the connection between Fluss Gateway and Fluss Server need to support multiple user identities? Or is a single service account sufficient? 3. ACL Authorization: Gateway-side or Server-side? Should we perform per-user ACL checks at the Gateway layer, or delegate authorization to Fluss Server? This decision directly impacts connection management — whether the gateway maintains a single connection (single identity) or manages per-user connections to the backend. Comparison of Similar Products: I've surveyed several gateway/proxy implementations for reference: | Product | Authentication (User ↔ Gateway) | Authorization & Connection Management (Gateway ↔ Server) | |---------|-------------------------------|----------------------------------------------------------| | Apache Shenyu | Sign / JWT / OAuth2 | Single super-user; shared singleton connection | | Kafka REST Proxy | OSS: not supported; Confluent: HTTP Basic Auth, mTLS | Single super-user; shared singleton connection | | RocketMQ Proxy Server | AK/SK | ACL enforced at proxy side; no further authorization needed on server, thus singleton connection | | HBase REST Gateway | SPNEGO / Kerberos, HTTPS/TLS | Super-user delegates user identity; ACL enforced at server side; requires multi-user connection management | My Preference: I'm leaning toward the HBase Gateway approach — the gateway authenticates users independently, then proxies requests to the server using delegated user identities, with ACL enforcement happening on the Fluss Server side. This allows the gateway to manage connections for multiple users and keeps authorization logic centralized in the server, which is more maintainable and consistent. I'll follow up with a more detailed write-up of each product's auth/authz implementation. Looking forward to your thoughts. Best, Hongshun On Thu, Jul 2, 2026 at 11:48 AM Forward Xu <[email protected]> wrote: > Hi Anton, > > Thanks for putting this together, and thanks to everyone who landed the > previous roadmap. Framing the next phase around "fluss-rust as a > first-class analytical query surface" makes a lot of sense to me — it > builds naturally on the analytical primitives we just shipped. A few > thoughts, roughly following your structure. > > *Theme 1 — DataFusion integration adapter [MUST-HAVE]* +1. I strongly agree > it should be an *adapter over the Rust core*, not a bundled engine. Keeping > fluss-datafusion as a standalone crate exposing TableProvider + > CatalogProvider/SchemaProvider keeps the dependency direction clean and > lets any DataFusion-based consumer (or the gateway) opt in. Mapping > projection/filter/limit pushdown onto the existing access paths (PK > equality → lookup, bucket-key prefix → prefix lookup, LIMIT → bounded scan, > else log scan) is the right approach. One thing worth nailing down early is > the exactness contract we report back to DataFusion ( > TableProviderFilterPushDown::Exact vs Inexact), since getting that wrong > silently drops or double-applies filters. > > *Theme 2 — Lake + log union read (Paimon-first) [MUST-HAVE]* +1, and > Paimon-first is the right call — reusing paimon-rust by wrapping an > existing reader is low-risk, and the lake-snapshot-records-log-offset > design gives us a clean stitch point. Starting bounded-batch and deferring > streaming also de-risks correctness of the log-offset boundary before we > add continuous reads. The main correctness edge I'd like to see covered is > the boundary semantics for PK tables when applying newer log records over > the lake snapshot (dedup/ordering at the tiered offset). > > *Theme 3 — Foundations / server-side filter pushdown [STRETCH]* Agree this > is a stretch, but it's the item with the highest leverage since it prunes > both lake and log scans and unblocks the gateway's filter pushdown. > Cross-checking against the Java PredicateConverter semantics is important > so both clients behave identically. If capacity allows, I'd lean toward > pulling at least the simple value-predicate cases forward, since Themes 1 > and 4 both benefit. > > *Theme 4 — Gateway [STRETCH]* The "REST is the no-regret floor, SQL is the > ceiling deferred until Themes 1–2" framing is a good way to think about it. > Keeping it a thin, stateless HTTP frontend over the existing client > primitives (one endpoint per primitive) sounds right and keeps scope > contained. > > *Open questions* > > - *Themes 1–2 must-have, rest stretch — reasonable?* Yes, ordering looks > right to me. Nothing obviously missing. > - *Paimon first, Iceberg later?* Agree, provided the union-read path is > written against a lake abstraction so Iceberg/Lance slot in the same way > rather than requiring a rewrite. > - *Pin one arrow/DataFusion version across core, union-read/lake, and > DF?* Strong +1 — we should pin a single Arrow/DataFusion version > workspace-wide; version skew between these crates is a common and > painful > source of breakage. > - *Separate crates: lake kernel, DataFusion adapter, gateway?* +1 to > separate crates. It matches the "adapter, not bundled engine" principle > and > keeps the DataFusion dependency out of the core for consumers that don't > need it. > > One meta-question: do we have a rough sense of the target release (0.3.0?) > these must-haves land in, so we can size the Theme 3/4 stretch work > accordingly? > > Thanks again — happy to help on the DataFusion adapter side. > > Best, > > ForwardXu > > Junbo Wang <[email protected]> 于2026年7月1日周三 22:41写道： > > > > I'd also like us to focus on closing the gap with the Java client on > > predicate pushdown. > > Agreed — predicate pushdown is a great optimization, and the read path > > will benefit significantly from it. > > > > > > > > One thing that was missing in upstream paimon-rust was an > > object-store-safe existence check in the filesystem catalog. It checked > > whether database/table directories “exist”, which works on local > > filesystems but can fail on S3/OSS because those directories are just > > prefixes, not real objects. I forked it to fix that, so lake reads can > > reliably open Paimon tables from object storage. > > > > Thanks again, Anton, for putting this roadmap together! I think we could > > share it at the July 3rd community meeting. I've also been working on > some > > designs around lake kernel read and the Fluss Gateway REST API — happy to > > discuss those as well if time allows. > > > > > > Best regards, > > Junbo Wang > > > > > 2026年6月25日 07:23，Anton Borisov <[email protected]> 写道： > > > > > > Hi Yuxia and Junbo, > > > > > > Yuxia, on PK changelog reads across the bindings: I have already > > > prepared record-mode CDC for Rust, Python and C++. It's small, and > > > the PK union read in Theme 2 needs it anyway. > > > > > > Junbo, I looked through your prototype branch. It is clear that a lot > > > of Themes 1, 2 and 4 already exist there as working code and it does > > > seem to answer two of the open questions: the three-way crate split > > > (lake kernel / adapter / gateway), and pinning a single > Arrow/DataFusion > > > version at the workspace level. +1 to both. > > > > > > Two separate FIPs with detailed design for REST/SQL gateway parts - > > makes sense. > > > > > > One thing I noticed and want to discuss: > > > The "lake read" currently depends on a fork of paimon-rust, can you > share > > > what was missing in paimon-rust? > > > > > > I'd also like us to focus on closing the gap with the Java client on > > > predicate pushdown. > > > In the interim we can push predicates into the lake (Paimon already > > > supports it) and apply them as a filter pass over the real-time log > > > tail. > > > > > > Good to hear you want to drive this forward, given the prototype, that > > > makes sense to me. I am happy to review the design and following PRs. > > > > > > Looking forward to the write-up with your findings. > > > > > > -- Anton > > > > > > вт, 23 июн. 2026 г. в 15:10, Junbo Wang <[email protected]>: > > >> > > >> Thanks Anton for kicking this off — funnily enough, I'd been arriving > > at very similar conclusions while prototyping on my side, so let me > share a > > bit of what came out of those experiments. > > >> > > >>> THEME 1: DATAFUSION INTEGRATION ADAPTER [MUST-HAVE] > > >> > > >> Agree. Having the DataFusion integration live in fluss-rust would make > > it reusable across more downstream projects. One thing we might want to > > keep in mind: it would probably be worth aligning the DataFusion and > Arrow > > versions with pg-datafusion so the whole stack stays on a consistent set > of > > versions. > > >> > > >>> THEME 2: LAKE + LOG UNION READ (PAIMON-FIRST) [MUST-HAVE] > > >> > > >> Agree. I'd lean toward putting this in a separate fluss-lake module > > that exposes a client capable of doing the union read over the lake and > the > > Fluss log. That way the lake-side dependencies stay isolated in their own > > module and don't bleed into the core fluss-rust code. One thing that > might > > be worth flagging here: the lake side already supports predicate > pushdown, > > while Fluss doesn't yet — so we'd probably think about how the union read > > handles that asymmetry in the interim. > > >> > > >> THEME 4: GATEWAY [STRETCH] — A thin HTTP frontend over the client > > >> > > >> I wonder if we could expand on this one a bit. My suggestion would be > > to split the SQL REST surface and the plain REST API into two separate > > FIPs. A read/write REST API on its own — somewhat in the spirit of > > kafka-rest, covering reads, writes, and metadata management — would > already > > be a quick win for broadening Fluss's data ingestion/access reach, and it > > doesn't depend on anything else. The SQL REST surface, on the other hand, > > has to wait on Themes 1–2, so it might make sense to split them and ship > > the plain REST API first. > > >> > > >> > > >> > > >> Also, it might make sense to have the gateway as a separate crate > > inside the Fluss project itself — especially since fluss-rust will be > > moving into the Fluss repo anyway. > > >> > > >> On a personal note, this is something I'm really excited about and > > would love to help push forward. I'll share a write-up of what came out > of > > my demo experiments with the community over the next few weeks. > > >> > > >> > > >> Best regards, > > >> Junbo Wang > > >> > > >>> 2026年6月23日 20:55，Yuxia Luo <[email protected]> 写道： > > >>> > > >>> Thanks for putting this together. +1 on the overall framing. My > > answers to the open questions, plus one addition: > > >>> > > >>> Open questions: > > >>> - Themes 1-2 as must-have, rest stretch: agree, the ordering makes > > sense to me. > > >>> - Paimon first, Iceberg later: +1. Reusing paimon-rust is the > > pragmatic path, and the union-read design generalizes to Iceberg the same > > way later. > > >>> - Pin one arrow/DataFusion version across core, union-read/lake, and > > DF: yes, we should pin a single version. Letting these drift will cause > > painful Arrow type/ABI mismatches across the crate boundary, so a shared > > pinned version (bumped deliberately, not per-crate) is worth the > discipline. > > >>> - Separate crates (lake kernel, DataFusion adapter, gateway): +1 on > > separate crates. It keeps the dependency surface clean - the adapter and > > gateway are optional consumers, and the lake kernel shouldn't drag > > DataFusion into anyone who only needs the core read path. > > >>> > > >>> One addition I'd like to propose for the scope: > > >>> PK-table CHANGELOG read, and expose it across the bindings - > > specifically C++ and Python - so non-Rust CDC consumers can subscribe. > > >>> > > >>> On 2026/06/22 14:49:52 Anton Borisov wrote: > > >>>> Hi all, > > >>>> > > >>>> With the previous roadmap wrapping up - complex types > > >>>> (Array/Row/Map/nesting), limit and prefix scan, schema-aware KV > > >>>> decoding, the metrics framework, and write optimisations, thanks to > > >>>> everyone who contributed and reviewed, I'd like to open discussion > on > > >>>> the roadmap. > > >>>> > > >>>> I propose framing around a single goal: make fluss-rust a > first-class > > >>>> analytical query surface - a Rust-native SQL/DataFrame path over > Fluss > > >>>> (DataFusion, and through it Polars/DuckDB/gateways), reading the > lake > > >>>> tier at scale, built on the analytical primitives we just landed. > > >>>> > > >>>> As before, I grouped the items into themes with an initial > must-have / > > >>>> stretch positioning. Please push back where you disagree. > > >>>> > > >>>> THEME 1: DATAFUSION INTEGRATION ADAPTER [MUST-HAVE] > > >>>> A standalone fluss-datafusion crate exposing TableProvider + Catalog > > >>>> over the existing client, so SELECT ... FROM . works from any > > >>>> DataFusion-based engine. It should be framed as an integration > adapter > > >>>> over the Rust core, not a bundled engine. A gateway (FIP-32) or any > > >>>> analytical consumer can use it or not. > > >>>> > > >>>> Scope: TableProvider + CatalogProvider/SchemaProvider backed by the > > >>>> metadata path. > > >>>> > > >>>> Map DataFusion's projection/filter/limit pushdown onto the client's > > >>>> existing access paths: a full primary-key equality becomes a > lookup, a > > >>>> bucket-key prefix becomes a prefix lookup, LIMIT becomes a bounded > > >>>> scan, otherwise a log scan. Filters an access path fully satisfies > are > > >>>> reported exact, the rest are left for DataFusion to apply > (server-side > > >>>> filter pushdown is Theme 3). > > >>>> > > >>>> THEME 2: LAKE + LOG UNION READ (PAIMON-FIRST) [MUST-HAVE] > > >>>> A client-side read that presents a lake-enabled table as one table. > > >>>> Its history is tiered to the lake (Paimon) and recent writes are > still > > >>>> in the Fluss log, the lake snapshot records the log offset it > covers, > > >>>> so we read the lake up to that offset and the Fluss log past it, > then > > >>>> combine. > > >>>> Scope: > > >>>> - Per bucket: the lake snapshot plus the Fluss log past the tiered > > offset. > > >>>> - Log tables: concatenate the two. PK tables: apply the newer log > > >>>> records to get current state. > > >>>> - Paimon first, reusing paimon-rust, Iceberg/Lance later the same > > >>>> way. The DataFusion adapter reads lake-enabled tables through this > > >>>> path. > > >>>> - Bounded batch first, streaming later > > >>>> > > >>>> Reusing paimon-rust means wrapping an existing reader, so it's > rather > > >>>> feasible and straightforward. > > >>>> > > >>>> THEME 3: FOUNDATIONS [STRETCH] > > >>>> Server-side value-predicate (filter) pushdown. The proto field > exists, > > >>>> but the client currently sends none. This prunes lake/log scans and > > >>>> unblocks the gateway's filter pushdown (the open question I raised > for > > >>>> 0.2.0), cross-checked against the Java PredicateConverter semantics. > > >>>> > > >>>> THEME 4: GATEWAY [STRETCH] > > >>>> A thin HTTP frontend over the client we already have independent of > > >>>> Themes 1–3. It's how non-Rust/non-SQL clients (TS/JS, microservices, > > >>>> curl) write to Fluss and do simple key access. > > >>>> - Write: upsert/append, delete, create/drop tables, metadata. The > > >>>> mature client write path essentially. > > >>>> - Read: point lookup, prefix lookup, bounded log / CDC poll - one > > >>>> endpoint per existing primitive, bounded and stateless. > > >>>> The tradeoff: no joins, filters, or aggregations - that's the SQL > > >>>> surface, and it needs the adapter and probably will be an ongoing > > >>>> effort to optimise later. > > >>>> - If Themes 1–2 ship, we unlock SQL frontends on top - PostgreSQL > > >>>> via datafusion-postgres, as one example > > >>>> In short: REST is the no-regret floor, SQL is the ceiling deferred > > >>>> until Themes 1–2. > > >>>> > > >>>> PARALLEL TRACK: ELIXIR + BINDING EXPOSURE > > >>>> Continue Elixir parity and expose newly-landed primitives (prefix > > >>>> lookup, etc.) across Python / C++ / Elixir. > > >>>> > > >>>> GOVERNANCE TRACK > > >>>> FIP-40: Consolidate apache/fluss-rust into apache/fluss - PR is > ready, > > >>>> looks pretty good, waiting for a good moment to rebase/merge. > > >>>> > > >>>> OPEN QUESTIONS > > >>>> - Themes 1–2 must-have, the rest stretch. Reasonable? Anything > > >>>> missing or mis-ordered? > > >>>> - Paimon first, Iceberg later? > > >>>> - Pin one arrow/DataFusion version across core, union-read/lake, and > > DF? > > >>>> - Separate crates: lake kernel, DataFusion adapter, gateway? > > >>>> > > >>>> Looking forward to feedback. > > >>>> > > >>>> -- Anton > > >>>> > > >> > > > > >
