viirya commented on PR #56933: URL: https://github.com/apache/spark/pull/56933#issuecomment-5268035730
Following up on the routing options from the earlier discussion: I ran option C (`:authority`-based routing) end-to-end on a real kind cluster with ingress-nginx v1.15.1, since it's the shape the gRPC maintainers recommended in grpc/grpc#14900. It works, and notably it lets the gateway be reached without mounting it at a URL subpath. The mechanism: each service is addressed by its own Ingress `host:` value rather than by a path under a shared host. In K8s, different-`host` Ingresses are isolated, so multiple services can each mount at `path: /` on the same ingress-nginx without colliding. I verified this directly — two Ingresses on the same controller, both at `path: /`, `host: sparkconnect` (the gateway) and `host: othersvc` (a dummy) — then sent two gRPC calls to the **exact same connection target** (same host:port, SNI, cert, method path), varying only `default_authority`: | connection target | `:authority` | routed to | |---|---|---| | same endpoint | `sparkconnect` | the gateway (real Spark Connect response) | | same endpoint | `othersvc` | the other service | The ingress access log confirms the same `/spark.connect.SparkConnectService/Config` request goes to different upstreams purely by `:authority`. So clients share one entry point (one hostname:port), each service is distinguished by its `:authority` tag, the gRPC `:path` is untouched, and nothing needs a rewrite. The full picture I verified (current PySpark client, no code change — just channel options): - **Plaintext (h2c):** `default_authority=sparkconnect` → routed to gateway; a different authority → 404. Routing keys on `:authority`, not SNI (confirmed: real SNI + a nonexistent `:authority` → 404, not a TLS error). - **TLS:** works too, with one caveat worth stating. gRPC uses the authority as the TLS cert-verification name by default, so if your cert is for the real gateway hostname and you set `default_authority=sparkconnect`, the handshake fails (the tag isn't in the cert SAN). The fix is to keep the two names separate — `grpc.ssl_target_name_override` = the real cert hostname, `grpc.default_authority` = the routing tag. I confirmed both the failure and the fix. **A correction to my earlier snippet:** I wrote `default_authority` as `"sparkConnect"` (camel case). That's fine for bare nginx `server_name`, but a K8s Ingress `host:` must be a lowercase RFC 1123 name — `host: sparkConnect` is rejected by the API server outright. So the routing tag has to be lowercase (`sparkconnect`) on both sides. One more operational note: once a `tls:` section is present for that host, ingress-nginx `308`-redirects plaintext `:80` to HTTPS, so clients must connect over TLS. (I tested the Python client; I haven't checked grpc-java's authority/TLS behavior.) Net, this doesn't change my earlier reservation about option A — I'd still rather not grow the client toward putting a prefix on the method `:path`. But I wanted to confirm on a real ingress-nginx that the `:authority` route works, so multiple gRPC services can share one entry point without any of them needing a URL subpath or a `:path` rewrite. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
