Hyukjin Kwon created SPARK-61041:
------------------------------------
Summary: Raise SESSION_NOT_SAME for the DataFrames of other
sessions in the Python client as PySpark does
Key: SPARK-61041
URL: https://issues.apache.org/jira/browse/SPARK-61041
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, the DataFrame methods that take another DataFrame,
{{join}}, {{crossJoin}}, {{lateralJoin}}, {{union}}, {{unionAll}},
{{unionByName}}, {{intersect}}, {{intersectAll}}, {{subtract}}, {{exceptAll}},
{{zip}}, {{nearestByJoin}}, {{sameSemantics}}, as well as
{{GroupedData.cogroup}} and the {{initialState}} of {{transformWithState}} and
{{transformWithStateInPandas}}, check that it is of the same session
({{_check_same_session}}), and raise {{SessionNotSameException}}
({{SESSION_NOT_SAME}}) for a DataFrame of another session, and for anything
that is not a DataFrame.
The Rust-backed Python client takes a DataFrame of another session, and sends
its plan to the server of this session, where the relations of the other
session, such as its temporary views, are not found. It raises a TypeError for
what is not a DataFrame. Check the session as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
