GitHub user aicam created a discussion: Replace Ingress and Envoy with Envoy-Gateway
# Proposal: Unify Proxy Architecture with Envoy Gateway ## Summary We are proposing a migration from our current dual-proxy setup (Ingress Nginx + Envoy) to a unified **Envoy Gateway** architecture. This change aims to simplify our infrastructure, adopt the modern Kubernetes Gateway API, and natively support the dynamic routing requirements of our ephemeral Computing Units. ## Motivation & Current Limitations Our system currently relies on **Ingress Nginx** for static routing and a separate **Envoy** instance for dynamic routing. While Ingress Nginx is a standard solution, it presents significant architectural limitations for our specific workload, particularly regarding the **Computing Units** in the `texera-workflow-computing-unit-pool` namespace. ### The Problems with the Current Stack 1. **Lack of Dynamic Routing:** Ingress Nginx operates on a configuration reload model. Every time an upstream service changes, Nginx must reload its configuration. This is inefficient for our Computing Units, which are highly transient (dynamically created and terminated). 2. **Dual-Proxy Complexity:** To bypass Nginx's limitations, we introduced a secondary Envoy proxy to handle WebSocket and HTTP connections to these dynamic units. This resulted in a "double-hop" architecture (User -> Ingress -> Envoy -> Compute), increasing latency and operational maintenance. 3. **Legacy Architecture:** Nginx's process-based architecture is less suited for high-churn service discovery compared to Envoy's modern, threaded architecture which utilizes the xDS protocol for seamless, hot-restart-free configuration updates. ## Proposed Solution: Envoy Gateway We propose replacing both the Ingress Controller and the standalone Envoy proxy with **Envoy Gateway**. Envoy Gateway is a Kubernetes-native implementation of the **Gateway API**. It manages Envoy proxies as the data plane, allowing us to handle both static system routes (Web App, Config Service) and dynamic ephemeral routes (Computing Units) in a single, unified layer. ### Key Benefits * **Unified Architecture:** Eliminates the maintenance overhead of managing two different proxy technologies. * **Native Dynamic Routing:** Envoy natively supports service discovery via xDS, allowing it to route to ephemeral pods in the `texera-workflow-computing-unit-pool` without the need for constant configuration reloads. * **Modern Standard:** Adopting the [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/) future-proofs our networking stack with standard resources (`Gateway`, `HTTPRoute`) rather than vendor-specific annotations. * **Protocol Support:** Seamless support for both HTTP/2 and WebSockets, which are critical for the interactive nature of our workflow system. ## Architecture Comparison ### Current Architecture *Traffic flows through Ingress for static routes, but requires a secondary Envoy hop for dynamic Compute Units.* Current architecture: <img width="1119" height="509" alt="Screenshot from 2026-01-29 13-16-45" src="https://github.com/user-attachments/assets/b73d1403-f973-4153-804e-b330c2824c19"; /> ### Target Architecture *A single Envoy Gateway layer handles all ingress traffic, routing directly to services and dynamically discovering Compute Units.* <img width="1253" height="559" alt="Screenshot from 2026-01-29 13-21-07" src="https://github.com/user-attachments/assets/1006d958-3c16-4acd-a902-30e4501e6361"; /> GitHub link: https://github.com/apache/texera/discussions/4182 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
