Thanks David for start the discussion. I'm big +1 to use Rust to build the gateway for - Performance: Rust avoids GC pauses, which is valuable for high-QPS gateway workloads - Ecosystem: Rust integrates naturally with Arrow/DataFusion/Flight SQL, and has mature PostgreSQL protocol libraries. - Industry momentum: More infrastructure projects are adopting Rust for gateway layers (e.g., sglang-gateway, a Rust-based AI gateway)
Also, as one the core maintainers of fluss-rust, I'm also +1 to move fluss-rust to fluss main repo after the first release of fluss-rust for - Faster feature delivery: It removes multi-repo release chaining and shortens time-to-user for fixes and new features. - More contributors: A unified repo can attract broader contributor participation across Java/Rust/gateway areas. - Better developer workflow: Developers can change Java + Rust + gateway code in one PR, reducing coordination overhead and improving review efficiency. Best regards, Yuxia ----- 原始邮件 ----- 发件人: "Jark Wu" <[email protected]> 收件人: "dev" <[email protected]> 发送时间: 星期一, 2026年 3 月 09日 下午 8:50:08 主题: Re: [DISCUSS] FIP-32 Multiprotocol Query Gateway for Apache Fluss Hi David, I really appreciate this proposal. It represents a significant step forward and opens up new possibilities for the Fluss project. I have uploaded the FIP markdown to the Apache cwiki to improve readability and granted you edit permissions. You can access and modify the document here: https://cwiki.apache.org/confluence/display/FLUSS/FIP-32%3A+Multiprotocol+Query+Gateway+for+Apache+Fluss Overall, I think the proposal is well structured. Below are some detailed comments for further consideration: **1. Repository Strategy** I am a strong advocate for Rust and fully support leveraging DataFusion and fluss-rust to build the gateway. Given that the gateway will become a critical component of Fluss, handling lightweight query execution tasks, adopting Rust and DataFusion from the outset provides a solid foundation for performance and reliability. However, we need to carefully consider project dependency management. The current dependency chain Fluss gateway depends on fluss-rust, which in turn depends on fluss repo means that releasing a new feature would require three separate repository releases. This could delay feature availability to users by nearly a year, which is inefficient. I suggest consolidating fluss-rust and fluss-gateway into the Fluss main repository (we can do this after the first release of fluss-rust). This approach would not only streamline the release process but also encourage more contributors to participate in Rust-related development and accelerate community iteration across the Rust, CLI, and gateway ecosystems. **2. Separate into multiple FIPs I recommend splitting this proposal into multiple FIPs. The primary FIP should focus on the Gateway architecture, covering design decisions related to dependencies, language choice, repository structure, and essential features such as ACL support. Within this Gateway FIP, we can reference REST, ADBC, and PostgreSQL protocols in a high-level overview without subjecting them to the formal voting process. Each protocol should then have its own dedicated FIP to address protocol-specific complexities, including supported APIs, implementation details, and versioning strategies. This modular approach will make the review process more focused and manageable. **3. Configuration File Format** Fluss currently uses YAML for configuration files, which ensures consistency across different environments such as Flink SQL, shell commands, and Docker Compose. To maintain this uniformity, I suggest starting with YAML for the gateway configuration as well. This will provide a consistent external API for users. We can consider supporting additional formats like TOML or JSON in the future based on community feedback. **4. Connection Parameter** We should adopt the standard `bootstrap.servers` parameter for the gateway to connect to the Fluss cluster, rather than using `coordinator_address`. The coordinator address may change due to failover or scaling events, whereas bootstrap servers provide a stable entry point for client discovery and connection. **5. DataFusion as an Optional Component** While DataFusion is a powerful query engine, it may not be the optimal choice for all gateway use cases. Some scenarios, such as high-frequency, lightweight lookup_kv or produce_log operations, could experience performance bottlenecks due to DataFusion's planning and conversion overhead. Furthermore, certain Fluss-specific APIs are not supported by DataFusion. I propose making DataFusion an optional library that gateway implementations can choose to integrate. This design would also allow bypassing DataFusion entirely and calling the fluss-rust client directly when appropriate, providing greater flexibility and performance optimization opportunities. Please let me know your thoughts on these suggestions. I am happy to discuss further and help refine the proposal. Best regards, Jark On Sat, 7 Mar 2026 at 12:08, ForwardXu <[email protected]> wrote: > > Hi David, > > > Thanks a lot for sharing the FIP-32 draft and George’s prototype work on the > multiprotocol query gateway! > > > Supporting multiple protocols (especially Flight SQL) via Apache DataFusion > is a great direction for Fluss, and I fully support this initiative. > > > I’ve checked the repo and have one quick question: will this gateway be > designed to be pluggable, so that we can easily add new protocols in the > future? > > > Looking forward to further discussions, and I’m happy to help with any > reviews or testing. > > > Best, > Forward > > 原始邮件 > > > 发件人：David Reger <[email protected]> > 发件时间：2026年3月7日 07:39 > 收件人：dev <[email protected]> > 主题：[DISCUSS] FIP-32 Multiprotocol Query Gateway for Apache Fluss > > > > Hey together, > > Thanks to George we have already a nice draft of a query gateway using Apache Datafusion du support multiple protocols especially Flight SQL. > > His Repo is here https://github.com/gstamatakis95/fluss/tree/feature/query-gateway. > > Based on his work I created a FIP to further discuss how the query gateway fits into the project and how we want to integrate and extend it further. > > You can find the FIP Draft here https://drive.google.com/file/d/1CC_uSQjtQmlMNGZyyHXr9K162BpKZvZO/view?usp=drivesdk > > Thank you and I appreciate every feedback. > > Kind regards > David
