Hello, Reviving this discussion thread as we discussed including this for release 1.0 on community sync earlier.
David, George, have you been able to address the comments provided by Junbo and Anton? Leaving my comments on the FIP: 1. Let's explicitly call out that authentication and authorisation as out of scope for the FIP similar to Flink SQL gateway that has no built-in user authentication and delegated to proxy. 2. Thinking along the line of limiting scope, should we cut the number of supported protocol to ArrowFlight and REST? If I gather correctly, DataFusion support only requires Arrow Flight implementation. PostgreSQL wire and gRPC support can be added in later FIPs. Cutting scope allows for earlier delivery, iteration and bug fixes. 3. I am unfortunately unable to view the draft implementation anymore. Can you provide an updated link? Best regards Keith On Thu, May 7, 2026 at 2:20 PM Junbo Wang <[email protected]> wrote: > Thanks for driving this FIP — really excited about the Gateway direction. > Having standardized protocol endpoints will make it much easier for BI > tools (Tableau, Superset, Grafana) and AI Agents to interact with Fluss > without dealing with low-level client SDKs. This is a great step forward > for the ecosystem.A few thoughts as I went through the design: > > 1. Metadata Caching: Since the Gateway is a long-running service and > fluss-rust metadata uses error-driven invalidation, it might be helpful to > introduce a TTL-based cache layer at the Gateway level (e.g., 60s for table > schema). This would help ensure newly created tables and DDL changes remain > visible without relying solely on error-triggered refreshes. > > 2. Session Context: PostgreSQL Wire Protocol is stateful per connection > (search_path, prepared statements, SET variables), so a single shared > SessionContext might cause interference between clients. A layered approach > — shared Catalog with per-connection session isolation for PG Wire — could > address this. Flight SQL only needs lightweight handle management, while > gRPC/REST are fully stateless and don't require a SessionContext at all. > > 3. Protocol Priority & Architecture: Wondering if it makes sense to > prioritize Arrow Flight SQL + PostgreSQL Wire Protocol + REST for the first > phase, deferring gRPC given its overlap with REST. Also, the direct > NativeFlussBackend path for simple read/write (bypassing DataFusion) seems > described in the text but not yet reflected in the architecture diagram — > might be worth aligning those. > > Would love to discuss further and contribute to this effort. Thanks again! > > Best regards, > Junbo Wang > > > > 2026年3月12日 11:10，Hongshun Wang <[email protected]> 写道： > > > > Hi David, > > > > The current scope of the FIP is mainly focused on the query path, so > Query > > Gateway is intentionally positioned around query-related capabilities > > rather than as a full proxy server like the Pulsar broker. > > > > That said, whether it could evolve to support more proxy-like > > functionalities in the future depends on the long-term architecture > goals. > > If the intention is only to provide a unified query entry point, then > using > > DataFusion seems reasonable. However, if there is an expectation that it > > may later handle broader responsibilities, such as data reads/writes or > > more general request routing, then it would be worth considering whether > > the current design leaves enough room for extensibility. > > > > Using DataFusion itself is not necessarily a blocker, but it does mean > the > > architecture may become more query-engine-centric. If future requirements > > go significantly beyond query execution, additional abstraction layers or > > architectural adjustments might be needed. > > > > Best, > > Hongshun > > > > On Thu, Mar 12, 2026 at 4:38 AM David Reger <[email protected]> > > wrote: > > > >> Yes I think it‘s good for visibility and also motivation to contribute > to > >> the „official“ main repo. Other projects also include cross language > stuff > >> (e.g. Arrow). > >> > >> I cannot validate the extension of native Fluss APIs and write a more > >> native gateway with bypassing Datafusion as the work was done > completely by > >> George Stamatakis. > >> > >> But I think it’s a reasonable point for the full API support and also > for > >> high performance use cases. > >> > >> Kind regards > >> David > >> > >>> On 9. Mar 2026, at 20:40, Keith Lee <[email protected]> > wrote: > >>> > >>> Agree strongly on the points that have been made on building gateway > on > >>> fluss-rust as well as benefits of monorepo. > >>> > >>> I’m very excited to see the possibility of bringing fluss-rust under > >> fluss. > >>> The collaborations we have on fluss-rust side and the language itself > has > >>> been a delight. > >>> > >>> Best regards > >>> Keith Lee > >>> > >>> > >>>> On Mon, 9 Mar 2026 at 15:42, Jark Wu <[email protected]> wrote: > >>>> > >>>> Beyond this, I believe a monorepo would be more friendly to coding > >>>> agents by providing them with a more complete code context. This > >>>> approach would significantly improve efficiency in cross-language > >>>> implementations, such as porting features from Java to Rust. I view > >>>> this transition as a proactive and necessary step for the Fluss > >>>> project to move fast in the era of AI. > >>>> > >>>> Additionally, if the community considers merging repositories a viable > >>>> option, this initiative warrants a dedicated DISCUSS and vote thread. > >>>> Such a change requires at least three binding votes from PPMC members, > >>>> whereas a FIP only requires three binding votes from committers. > >>>> > >>>> Best, > >>>> Jark > >>>> > >>>>> On Mon, 9 Mar 2026 at 21:33, yuxia <[email protected]> > >> wrote: > >>>>> > >>>>> Thanks David for start the discussion. > >>>>> > >>>>> I'm big +1 to use Rust to build the gateway for > >>>>> - Performance: Rust avoids GC pauses, which is valuable for high-QPS > >>>> gateway workloads > >>>>> - Ecosystem: Rust integrates naturally with Arrow/DataFusion/Flight > >> SQL, > >>>> and has mature PostgreSQL protocol libraries. > >>>>> - Industry momentum: More infrastructure projects are adopting Rust > for > >>>> gateway layers (e.g., sglang-gateway, a Rust-based AI gateway) > >>>>> > >>>>> Also, as one the core maintainers of fluss-rust, I'm also +1 to move > >>>> fluss-rust to fluss main repo after the first release of fluss-rust > for > >>>>> - Faster feature delivery: It removes multi-repo release chaining and > >>>> shortens time-to-user for fixes and new features. > >>>>> - More contributors: A unified repo can attract broader contributor > >>>> participation across Java/Rust/gateway areas. > >>>>> - Better developer workflow: Developers can change Java + Rust + > >> gateway > >>>> code in one PR, reducing coordination overhead and improving review > >>>> efficiency. > >>>>> > >>>>> Best regards, > >>>>> Yuxia > >>>>> > >>>>> ----- 原始邮件 ----- > >>>>> 发件人: "Jark Wu" <[email protected]> > >>>>> 收件人: "dev" <[email protected]> > >>>>> 发送时间: 星期一, 2026年 3 月 09日 下午 8:50:08 > >>>>> 主题: Re: [DISCUSS] FIP-32 Multiprotocol Query Gateway for Apache Fluss > >>>>> > >>>>> Hi David, > >>>>> > >>>>> I really appreciate this proposal. It represents a significant step > >>>>> forward and opens up new possibilities for the Fluss project. > >>>>> > >>>>> I have uploaded the FIP markdown to the Apache cwiki to improve > >>>>> readability and granted you edit permissions. You can access and > >>>>> modify the document here: > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/FLUSS/FIP-32%3A+Multiprotocol+Query+Gateway+for+Apache+Fluss > >>>>> > >>>>> Overall, I think the proposal is well structured. Below are some > >>>>> detailed comments for further consideration: > >>>>> > >>>>> **1. Repository Strategy** > >>>>> I am a strong advocate for Rust and fully support leveraging > >>>>> DataFusion and fluss-rust to build the gateway. Given that the > gateway > >>>>> will become a critical component of Fluss, handling lightweight query > >>>>> execution tasks, adopting Rust and DataFusion from the outset > provides > >>>>> a solid foundation for performance and reliability. > >>>>> > >>>>> However, we need to carefully consider project dependency management. > >>>>> The current dependency chain Fluss gateway depends on fluss-rust, > >>>>> which in turn depends on fluss repo means that releasing a new > feature > >>>>> would require three separate repository releases. This could delay > >>>>> feature availability to users by nearly a year, which is inefficient. > >>>>> I suggest consolidating fluss-rust and fluss-gateway into the Fluss > >>>>> main repository (we can do this after the first release of > >>>>> fluss-rust). This approach would not only streamline the release > >>>>> process but also encourage more contributors to participate in > >>>>> Rust-related development and accelerate community iteration across > the > >>>>> Rust, CLI, and gateway ecosystems. > >>>>> > >>>>> **2. Separate into multiple FIPs > >>>>> I recommend splitting this proposal into multiple FIPs. The primary > >>>>> FIP should focus on the Gateway architecture, covering design > >>>>> decisions related to dependencies, language choice, repository > >>>>> structure, and essential features such as ACL support. Within this > >>>>> Gateway FIP, we can reference REST, ADBC, and PostgreSQL protocols in > >>>>> a high-level overview without subjecting them to the formal voting > >>>>> process. Each protocol should then have its own dedicated FIP to > >>>>> address protocol-specific complexities, including supported APIs, > >>>>> implementation details, and versioning strategies. This modular > >>>>> approach will make the review process more focused and manageable. > >>>>> > >>>>> **3. Configuration File Format** > >>>>> Fluss currently uses YAML for configuration files, which ensures > >>>>> consistency across different environments such as Flink SQL, shell > >>>>> commands, and Docker Compose. To maintain this uniformity, I suggest > >>>>> starting with YAML for the gateway configuration as well. This will > >>>>> provide a consistent external API for users. We can consider > >>>>> supporting additional formats like TOML or JSON in the future based > on > >>>>> community feedback. > >>>>> > >>>>> **4. Connection Parameter** > >>>>> We should adopt the standard `bootstrap.servers` parameter for the > >>>>> gateway to connect to the Fluss cluster, rather than using > >>>>> `coordinator_address`. The coordinator address may change due to > >>>>> failover or scaling events, whereas bootstrap servers provide a > stable > >>>>> entry point for client discovery and connection. > >>>>> > >>>>> **5. DataFusion as an Optional Component** > >>>>> While DataFusion is a powerful query engine, it may not be the > optimal > >>>>> choice for all gateway use cases. Some scenarios, such as > >>>>> high-frequency, lightweight lookup_kv or produce_log operations, > could > >>>>> experience performance bottlenecks due to DataFusion's planning and > >>>>> conversion overhead. Furthermore, certain Fluss-specific APIs are not > >>>>> supported by DataFusion. I propose making DataFusion an optional > >>>>> library that gateway implementations can choose to integrate. This > >>>>> design would also allow bypassing DataFusion entirely and calling the > >>>>> fluss-rust client directly when appropriate, providing greater > >>>>> flexibility and performance optimization opportunities. > >>>>> > >>>>> Please let me know your thoughts on these suggestions. I am happy to > >>>>> discuss further and help refine the proposal. > >>>>> > >>>>> Best regards, > >>>>> Jark > >>>>> > >>>>> On Sat, 7 Mar 2026 at 12:08, ForwardXu <[email protected]> wrote: > >>>>>> > >>>>>> Hi David, > >>>>>> > >>>>>> > >>>>>> Thanks a lot for sharing the FIP-32 draft and George’s prototype > work > >>>> on the multiprotocol query gateway! > >>>>>> > >>>>>> > >>>>>> Supporting multiple protocols (especially Flight SQL) via Apache > >>>> DataFusion is a great direction for Fluss, and I fully support this > >>>> initiative. > >>>>>> > >>>>>> > >>>>>> I’ve checked the repo and have one quick question: will this gateway > >>>> be designed to be pluggable, so that we can easily add new protocols > in > >> the > >>>> future? > >>>>>> > >>>>>> > >>>>>> Looking forward to further discussions, and I’m happy to help with > any > >>>> reviews or testing. > >>>>>> > >>>>>> > >>>>>> Best, > >>>>>> Forward > >>>>>> > >>>>>> 原始邮件 > >>>>>> > >>>>>> > >>>>>> 发件人：David Reger <[email protected]> > >>>>>> 发件时间：2026年3月7日 07:39 > >>>>>> 收件人：dev <[email protected]> > >>>>>> 主题：[DISCUSS] FIP-32 Multiprotocol Query Gateway for Apache Fluss > >>>>>> > >>>>>> > >>>>>> > >>>>>> Hey together, > >>>>>> > >>>>>> > >>>> > >> > Thanks to George we have already a nice draft of a query gateway using Apache Datafusion du support multiple protocols especially Flight SQL. > >>>>>> > >>>>>> His Repo is here > >>>> https://github.com/gstamatakis95/fluss/tree/feature/query-gateway. > >>>>>> > >>>>>> > >>>> > >> > Based on his work I created a FIP to further discuss how the query gateway fits into the project and how we want to integrate and extend it further. > >>>>>> > >>>>>> You can find the FIP Draft here > >>>> > >> > https://drive.google.com/file/d/1CC_uSQjtQmlMNGZyyHXr9K162BpKZvZO/view?usp=drivesdk > >>>>>> > >>>>>> > >>>> > Thank you and I appreciate every feedback. > >>>>>> > >>>>>> Kind regards > >>>>>> David > >>>> > >> > >
