Hi David, The current scope of the FIP is mainly focused on the query path, so Query Gateway is intentionally positioned around query-related capabilities rather than as a full proxy server like the Pulsar broker.
That said, whether it could evolve to support more proxy-like functionalities in the future depends on the long-term architecture goals. If the intention is only to provide a unified query entry point, then using DataFusion seems reasonable. However, if there is an expectation that it may later handle broader responsibilities, such as data reads/writes or more general request routing, then it would be worth considering whether the current design leaves enough room for extensibility. Using DataFusion itself is not necessarily a blocker, but it does mean the architecture may become more query-engine-centric. If future requirements go significantly beyond query execution, additional abstraction layers or architectural adjustments might be needed. Best, Hongshun On Thu, Mar 12, 2026 at 4:38 AM David Reger <[email protected]> wrote: > Yes I think it‘s good for visibility and also motivation to contribute to > the „official“ main repo. Other projects also include cross language stuff > (e.g. Arrow). > > I cannot validate the extension of native Fluss APIs and write a more > native gateway with bypassing Datafusion as the work was done completely by > George Stamatakis. > > But I think it’s a reasonable point for the full API support and also for > high performance use cases. > > Kind regards > David > > > On 9. Mar 2026, at 20:40, Keith Lee <[email protected]> wrote: > > > > Agree strongly on the points that have been made on building gateway on > > fluss-rust as well as benefits of monorepo. > > > > I’m very excited to see the possibility of bringing fluss-rust under > fluss. > > The collaborations we have on fluss-rust side and the language itself has > > been a delight. > > > > Best regards > > Keith Lee > > > > > >> On Mon, 9 Mar 2026 at 15:42, Jark Wu <[email protected]> wrote: > >> > >> Beyond this, I believe a monorepo would be more friendly to coding > >> agents by providing them with a more complete code context. This > >> approach would significantly improve efficiency in cross-language > >> implementations, such as porting features from Java to Rust. I view > >> this transition as a proactive and necessary step for the Fluss > >> project to move fast in the era of AI. > >> > >> Additionally, if the community considers merging repositories a viable > >> option, this initiative warrants a dedicated DISCUSS and vote thread. > >> Such a change requires at least three binding votes from PPMC members, > >> whereas a FIP only requires three binding votes from committers. > >> > >> Best, > >> Jark > >> > >>> On Mon, 9 Mar 2026 at 21:33, yuxia <[email protected]> > wrote: > >>> > >>> Thanks David for start the discussion. > >>> > >>> I'm big +1 to use Rust to build the gateway for > >>> - Performance: Rust avoids GC pauses, which is valuable for high-QPS > >> gateway workloads > >>> - Ecosystem: Rust integrates naturally with Arrow/DataFusion/Flight > SQL, > >> and has mature PostgreSQL protocol libraries. > >>> - Industry momentum: More infrastructure projects are adopting Rust for > >> gateway layers (e.g., sglang-gateway, a Rust-based AI gateway) > >>> > >>> Also, as one the core maintainers of fluss-rust, I'm also +1 to move > >> fluss-rust to fluss main repo after the first release of fluss-rust for > >>> - Faster feature delivery: It removes multi-repo release chaining and > >> shortens time-to-user for fixes and new features. > >>> - More contributors: A unified repo can attract broader contributor > >> participation across Java/Rust/gateway areas. > >>> - Better developer workflow: Developers can change Java + Rust + > gateway > >> code in one PR, reducing coordination overhead and improving review > >> efficiency. > >>> > >>> Best regards, > >>> Yuxia > >>> > >>> ----- 原始邮件 ----- > >>> 发件人: "Jark Wu" <[email protected]> > >>> 收件人: "dev" <[email protected]> > >>> 发送时间: 星期一, 2026年 3 月 09日 下午 8:50:08 > >>> 主题: Re: [DISCUSS] FIP-32 Multiprotocol Query Gateway for Apache Fluss > >>> > >>> Hi David, > >>> > >>> I really appreciate this proposal. It represents a significant step > >>> forward and opens up new possibilities for the Fluss project. > >>> > >>> I have uploaded the FIP markdown to the Apache cwiki to improve > >>> readability and granted you edit permissions. You can access and > >>> modify the document here: > >>> > >> > https://cwiki.apache.org/confluence/display/FLUSS/FIP-32%3A+Multiprotocol+Query+Gateway+for+Apache+Fluss > >>> > >>> Overall, I think the proposal is well structured. Below are some > >>> detailed comments for further consideration: > >>> > >>> **1. Repository Strategy** > >>> I am a strong advocate for Rust and fully support leveraging > >>> DataFusion and fluss-rust to build the gateway. Given that the gateway > >>> will become a critical component of Fluss, handling lightweight query > >>> execution tasks, adopting Rust and DataFusion from the outset provides > >>> a solid foundation for performance and reliability. > >>> > >>> However, we need to carefully consider project dependency management. > >>> The current dependency chain Fluss gateway depends on fluss-rust, > >>> which in turn depends on fluss repo means that releasing a new feature > >>> would require three separate repository releases. This could delay > >>> feature availability to users by nearly a year, which is inefficient. > >>> I suggest consolidating fluss-rust and fluss-gateway into the Fluss > >>> main repository (we can do this after the first release of > >>> fluss-rust). This approach would not only streamline the release > >>> process but also encourage more contributors to participate in > >>> Rust-related development and accelerate community iteration across the > >>> Rust, CLI, and gateway ecosystems. > >>> > >>> **2. Separate into multiple FIPs > >>> I recommend splitting this proposal into multiple FIPs. The primary > >>> FIP should focus on the Gateway architecture, covering design > >>> decisions related to dependencies, language choice, repository > >>> structure, and essential features such as ACL support. Within this > >>> Gateway FIP, we can reference REST, ADBC, and PostgreSQL protocols in > >>> a high-level overview without subjecting them to the formal voting > >>> process. Each protocol should then have its own dedicated FIP to > >>> address protocol-specific complexities, including supported APIs, > >>> implementation details, and versioning strategies. This modular > >>> approach will make the review process more focused and manageable. > >>> > >>> **3. Configuration File Format** > >>> Fluss currently uses YAML for configuration files, which ensures > >>> consistency across different environments such as Flink SQL, shell > >>> commands, and Docker Compose. To maintain this uniformity, I suggest > >>> starting with YAML for the gateway configuration as well. This will > >>> provide a consistent external API for users. We can consider > >>> supporting additional formats like TOML or JSON in the future based on > >>> community feedback. > >>> > >>> **4. Connection Parameter** > >>> We should adopt the standard `bootstrap.servers` parameter for the > >>> gateway to connect to the Fluss cluster, rather than using > >>> `coordinator_address`. The coordinator address may change due to > >>> failover or scaling events, whereas bootstrap servers provide a stable > >>> entry point for client discovery and connection. > >>> > >>> **5. DataFusion as an Optional Component** > >>> While DataFusion is a powerful query engine, it may not be the optimal > >>> choice for all gateway use cases. Some scenarios, such as > >>> high-frequency, lightweight lookup_kv or produce_log operations, could > >>> experience performance bottlenecks due to DataFusion's planning and > >>> conversion overhead. Furthermore, certain Fluss-specific APIs are not > >>> supported by DataFusion. I propose making DataFusion an optional > >>> library that gateway implementations can choose to integrate. This > >>> design would also allow bypassing DataFusion entirely and calling the > >>> fluss-rust client directly when appropriate, providing greater > >>> flexibility and performance optimization opportunities. > >>> > >>> Please let me know your thoughts on these suggestions. I am happy to > >>> discuss further and help refine the proposal. > >>> > >>> Best regards, > >>> Jark > >>> > >>> On Sat, 7 Mar 2026 at 12:08, ForwardXu <[email protected]> wrote: > >>>> > >>>> Hi David, > >>>> > >>>> > >>>> Thanks a lot for sharing the FIP-32 draft and George’s prototype work > >> on the multiprotocol query gateway! > >>>> > >>>> > >>>> Supporting multiple protocols (especially Flight SQL) via Apache > >> DataFusion is a great direction for Fluss, and I fully support this > >> initiative. > >>>> > >>>> > >>>> I’ve checked the repo and have one quick question: will this gateway > >> be designed to be pluggable, so that we can easily add new protocols in > the > >> future? > >>>> > >>>> > >>>> Looking forward to further discussions, and I’m happy to help with any > >> reviews or testing. > >>>> > >>>> > >>>> Best, > >>>> Forward > >>>> > >>>> 原始邮件 > >>>> > >>>> > >>>> 发件人：David Reger <[email protected]> > >>>> 发件时间：2026年3月7日 07:39 > >>>> 收件人：dev <[email protected]> > >>>> 主题：[DISCUSS] FIP-32 Multiprotocol Query Gateway for Apache Fluss > >>>> > >>>> > >>>> > >>>> Hey together, > >>>> > >>>> > >> > Thanks to George we have already a nice draft of a query gateway using Apache Datafusion du support multiple protocols especially Flight SQL. > >>>> > >>>> His Repo is here > >> https://github.com/gstamatakis95/fluss/tree/feature/query-gateway. > >>>> > >>>> > >> > Based on his work I created a FIP to further discuss how the query gateway fits into the project and how we want to integrate and extend it further. > >>>> > >>>> You can find the FIP Draft here > >> > https://drive.google.com/file/d/1CC_uSQjtQmlMNGZyyHXr9K162BpKZvZO/view?usp=drivesdk > >>>> > >>>> > >> Thank you and I appreciate every feedback. > >>>> > >>>> Kind regards > >>>> David > >> >
