Ruifeng Zheng created SPARK-60374:
-------------------------------------
Summary: Support client-generated checkpoint relation IDs
Key: SPARK-60374
URL: https://issues.apache.org/jira/browse/SPARK-60374
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 5.0.0
Reporter: Ruifeng Zheng
Spark Connect currently generates the IDs for checkpointed DataFrames on the
server. Libraries such as GraphFrames need to choose these handles on the
client so they can reference and release the corresponding cached DataFrames.
Add an optional relation_id to CheckpointCommand. The server should use a
supplied non-empty ID and continue generating a UUID when the field is absent.
IDs remain scoped to the Connect session and use the existing cached-relation
removal mechanism.
The Python Connect DataFrame.checkpoint() and DataFrame.localCheckpoint()
methods should generate UUIDs, pass them in the request, validate that the
server returns the requested ID, and construct CachedRemoteRelation from the
original ID and session. Public method signatures remain unchanged. Empty IDs
should receive a structured error, and duplicate IDs should retain the existing
error.
Python checkpoint methods will require a server that supports the field because
older servers return a different, server-generated ID. Other protocol clients
that omit the field retain the existing behavior.
Implementation PR: https://github.com/apache/spark/pull/59540
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
