Kaicheng Zhou created SPARK-61773:
-------------------------------------

             Summary: Connect through the HTTP proxy of the environment, as 
grpc-python does
                 Key: SPARK-61773
                 URL: https://issues.apache.org/jira/browse/SPARK-61773
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Kaicheng Zhou
             Fix For: connect-rust-4.3.0


The Rust client (pyspark-client-rust and the native Rust API) always dials the 
Spark Connect server directly, so it cannot connect from behind an egress HTTP 
proxy. PySpark's client can: grpcio reads the proxy of the environment for 
every channel and opens an HTTP/1.1 CONNECT tunnel. A user whose PySpark 
connects through a proxy today only gets retries from the drop-in.

Honor the proxy of the environment with grpcio's rules:
- the proxy is the first of {{grpc_proxy}}, {{https_proxy}} and {{http_proxy}} 
that is set (an empty value means no proxy); only the lowercase names are read
- {{no_grpc_proxy}}, or else {{no_proxy}}, lists the hosts that are not 
proxied: the host or a parent domain, or an IP in a CIDR range
- only an {{http://}} proxy is supported; credentials in the URI are sent as 
Proxy-Authorization: Basic
- TLS is negotiated with the Spark server inside the tunnel, so the proxy sees 
neither the token nor the data

Two deliberate differences from grpcio: a loopback host is never proxied, 
because a token is sent to localhost without TLS (grpcio proxies it, which 
shows the token to the proxy in clear), and a proxy address with an invalid 
host or port is an error instead of a channel that never connects.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to