Tim Meehan created SPARK-61412:
----------------------------------

             Summary: Send the user agent as client_type on the execute, 
analyze, config, interrupt, release and error-detail requests, as PySpark does
                 Key: SPARK-61412
                 URL: https://issues.apache.org/jira/browse/SPARK-61412
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Tim Meehan


The Rust client sends no {{client_type}} on ExecutePlan, AnalyzePlan, Config, 
Interrupt, ReleaseSession and ReleaseExecute, and the public 
fetch_error_details sends none unless the caller set it. PySpark sets 
{{client_type}} to the user agent on all of these (and on ReattachExecute and 
ReleaseExecute, which copy it from the ExecutePlan request they belong to). A 
server, proxy or audit log that reads {{client_type}} sees PySpark's requests 
with a client type and the Python client of spark-connect-rust's without one. 
An ExecutePlan hook also sees the field unset.

{code:python}
# proxy is a loopback gRPC proxy to the server that prints client_type of each 
request it forwards.
spark = 
SparkSession.builder.remote(f"sc://127.0.0.1:{proxy_port}").getOrCreate()
spark.range(3).collect()
spark.range(3).schema
spark.conf.set("k", "v")
spark.interruptAll()
spark.stop()
{code}

The client_type of each request that the proxy saw, against an OSS Spark 4.3.0 
Connect server on loopback:
* pyspark-client 4.3.0: ExecutePlan, AnalyzePlan, Config, Interrupt, 
ReleaseExecute and ReleaseSession each carry {{_SPARK_CONNECT_PYTHON 
spark/4.3.0 os/linux python/3.11.13}}.
* The Python client at spark-connect-rust master 125654b: the same six requests 
carry no {{client_type}} (the field is absent on the wire).

>From source, at 125654b, crates/spark-connect-core/src/client.rs: execute_plan 
>(970), execute_plan_reattachable (1172), analyze_plan (1104), config (1359), 
>interrupt (1385), conf_while_raising (1327, the Config read while an error is 
>raised) and release_session (1742) never set it, and the public 
>fetch_error_details (1249) does not fill it. ReattachExecute and 
>ReleaseExecute copy the field from an ExecutePlan request that has none 
>(crates/spark-connect-core/src/reattach.rs:113, 163).

Known differences that are out of scope here:
* The user agent itself differs: the Rust client ends it with {{python/rust}} 
(crates/spark-connect-core/src/channel.rs:294), PySpark with 
{{python/<interpreter version>}}.
* CloneSession: PySpark sends {{client_type="python"}}; the Rust client sends 
the user agent (client.rs:1704).
* AddArtifacts and ArtifactStatuses: PySpark sends no {{client_type}}; the Rust 
client sends the user agent (client.rs:1932, 1975).
* GetStatus is not implemented in the Rust client.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to