[
https://issues.apache.org/jira/browse/SPARK-61041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61041:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Raise SESSION_NOT_SAME for the DataFrames of other sessions in the Python
> client as PySpark does
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-61041
> URL: https://issues.apache.org/jira/browse/SPARK-61041
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, the DataFrame methods that take another DataFrame,
> {{join}}, {{crossJoin}}, {{lateralJoin}}, {{union}}, {{unionAll}},
> {{unionByName}}, {{intersect}}, {{intersectAll}}, {{subtract}},
> {{exceptAll}}, {{zip}}, {{nearestByJoin}}, {{sameSemantics}}, as well as
> {{GroupedData.cogroup}} and the {{initialState}} of {{transformWithState}}
> and {{transformWithStateInPandas}}, check that it is of the same session
> ({{_check_same_session}}), and raise {{SessionNotSameException}}
> ({{SESSION_NOT_SAME}}) for a DataFrame of another session, and for anything
> that is not a DataFrame.
> The Rust-backed Python client takes a DataFrame of another session, and sends
> its plan to the server of this session, where the relations of the other
> session, such as its temporary views, are not found. It raises a TypeError
> for what is not a DataFrame. Check the session as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
