Amogh Desai created SPARK-62068:
-----------------------------------
Summary: Support SparkSession.getOrCreateDataFrame in Spark Connect
Key: SPARK-62068
URL: https://issues.apache.org/jira/browse/SPARK-62068
Project: Spark
Issue Type: New Feature
Components: Connect
Affects Versions: 4.4.0
Reporter: Amogh Desai
Fix For: 4.5.0
https://issues.apache.org/jira/browse/SPARK-60775 adds
SparkSession.getOrCreateDataFrame(path, create) for Spark Classic.
It returns the DataFrame stored at path if one is there; otherwise, it calls
create,
writes the result as parquet, and returns it. On Spark Connect, it currently
throws
UNSUPPORTED_CONNECT_FEATURE.SESSION_GET_OR_CREATE_DATAFRAME (Scala) and
NOT_IMPLEMENTED (Python).
The goal is to extend support in connect too.
The approach I have in mind is:
- Expose the server-side check (isDataFrameMaterialized) to the client, forĀ
example through a new RPC (or a command?)
- The client evaluates the create function locally, the same way classic
pyspark does, then writes the result and the marker on the server.
Context: https://github.com/apache/spark/pull/60000#discussion_r4590551003
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]