Kaicheng Zhou created SPARK-61773:
-------------------------------------
Summary: Connect through the HTTP proxy of the environment, as
grpc-python does
Key: SPARK-61773
URL: https://issues.apache.org/jira/browse/SPARK-61773
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Kaicheng Zhou
Fix For: connect-rust-4.3.0
The Rust client (pyspark-client-rust and the native Rust API) always dials the
Spark Connect server directly, so it cannot connect from behind an egress HTTP
proxy. PySpark's client can: grpcio reads the proxy of the environment for
every channel and opens an HTTP/1.1 CONNECT tunnel. A user whose PySpark
connects through a proxy today only gets retries from the drop-in.
Honor the proxy of the environment with grpcio's rules:
- the proxy is the first of {{grpc_proxy}}, {{https_proxy}} and {{http_proxy}}
that is set (an empty value means no proxy); only the lowercase names are read
- {{no_grpc_proxy}}, or else {{no_proxy}}, lists the hosts that are not
proxied: the host or a parent domain, or an IP in a CIDR range
- only an {{http://}} proxy is supported; credentials in the URI are sent as
Proxy-Authorization: Basic
- TLS is negotiated with the Spark server inside the tunnel, so the proxy sees
neither the token nor the data
Two deliberate differences from grpcio: a loopback host is never proxied,
because a token is sent to localhost without TLS (grpcio proxies it, which
shows the token to the proxy in clear), and a proxy address with an invalid
host or port is an error instead of a channel that never connects.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]