[
https://issues.apache.org/jira/browse/SPARK-61120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61120:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Add DataFrame._session, SparkSession._client and the results of
> execute_command that vendored modules read, in the Python client as PySpark
> does
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61120
> URL: https://issues.apache.org/jira/browse/SPARK-61120
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Modules vendored from Apache Spark v4.3.0 read private attributes of PySpark
> Connect's DataFrame, session and client that the Python client lacks:
> * {{DataFrame._session}}, the session of a DataFrame, which these read:
> ** {{pyspark.ml.util._cache_spark_dataset}}, which {{CrossValidator}} and
> {{TrainValidationSplit}} use with {{SPARKML_TEMP_DFS_PATH}}
> ** {{pyspark.sql.plot}}, for the maximum rows of a plot
> ** {{select_dtypes}} of {{pyspark.pandas}}
> On the client, {{df._session}} is a Column of that name, so with
> {{SPARKML_TEMP_DFS_PATH}} set, {{CrossValidator(...).fit(df)}} raises
> {{TypeError: 'Column' object is not callable}}.
> * {{SparkSession._client}} and {{SparkConnectClient._resources()}}, of which
> {{TorchDistributor}} reads the resources of the cluster on Spark Connect. The
> client has neither.
> * PySpark's {{SparkConnectClient.execute_command}} returns the result of a
> {{get_resources_command}} as a dict of {{ResourceInformation}}, of a
> {{create_resource_profile_command}} as its profile ID, and of a
> {{checkpoint_command}} as a DataFrame of its cached relation, and it also
> returns the {{pipeline_event_result}} of a response. The client returns the
> first as its message, and none of the others.
> The fix is to add them, as PySpark has them.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
