[ 
https://issues.apache.org/jira/browse/SPARK-61412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Meehan resolved SPARK-61412.
--------------------------------
    Resolution: Duplicate

> Send the user agent as client_type on the execute, analyze, config, 
> interrupt, release and error-detail requests, as PySpark does
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61412
>                 URL: https://issues.apache.org/jira/browse/SPARK-61412
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Tim Meehan
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> The Rust client sends no {{client_type}} on ExecutePlan, AnalyzePlan, Config, 
> Interrupt, ReleaseSession and ReleaseExecute, and the public 
> fetch_error_details sends none unless the caller set it. PySpark sets 
> {{client_type}} to the user agent on all of these (and on ReattachExecute and 
> ReleaseExecute, which copy it from the ExecutePlan request they belong to). A 
> server, proxy or audit log that reads {{client_type}} sees PySpark's requests 
> with a client type and the Python client of spark-connect-rust's without one. 
> An ExecutePlan hook also sees the field unset.
> {code:python}
> # proxy is a loopback gRPC proxy to the server that prints client_type of 
> each request it forwards.
> spark = 
> SparkSession.builder.remote(f"sc://127.0.0.1:{proxy_port}").getOrCreate()
> spark.range(3).collect()
> spark.range(3).schema
> spark.conf.set("k", "v")
> spark.interruptAll()
> spark.stop()
> {code}
> The client_type of each request that the proxy saw, against an OSS Spark 
> 4.3.0 Connect server on loopback:
> * pyspark-client 4.3.0: ExecutePlan, AnalyzePlan, Config, Interrupt, 
> ReleaseExecute and ReleaseSession each carry {{_SPARK_CONNECT_PYTHON 
> spark/4.3.0 os/linux python/3.11.13}}.
> * The Python client at spark-connect-rust master 125654b: the same six 
> requests carry no {{client_type}} (the field is absent on the wire).
> From source, at 125654b, crates/spark-connect-core/src/client.rs: 
> execute_plan (970), execute_plan_reattachable (1172), analyze_plan (1104), 
> config (1359), interrupt (1385), conf_while_raising (1327, the Config read 
> while an error is raised) and release_session (1742) never set it, and the 
> public fetch_error_details (1249) does not fill it. ReattachExecute and 
> ReleaseExecute copy the field from an ExecutePlan request that has none 
> (crates/spark-connect-core/src/reattach.rs:113, 163).
> Known differences that are out of scope here:
> * The user agent itself differs: the Rust client ends it with {{python/rust}} 
> (crates/spark-connect-core/src/channel.rs:294), PySpark with 
> {{python/<interpreter version>}}.
> * CloneSession: PySpark sends {{client_type="python"}}; the Rust client sends 
> the user agent (client.rs:1704).
> * AddArtifacts and ArtifactStatuses: PySpark sends no {{client_type}}; the 
> Rust client sends the user agent (client.rs:1932, 1975).
> * GetStatus is not implemented in the Rust client.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to