[
https://issues.apache.org/jira/browse/SPARK-60374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng resolved SPARK-60374.
-----------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59540
[https://github.com/apache/spark/pull/59540]
> Support client-generated checkpoint relation IDs
> ------------------------------------------------
>
> Key: SPARK-60374
> URL: https://issues.apache.org/jira/browse/SPARK-60374
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Ruifeng Zheng
> Assignee: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> Spark Connect currently generates the IDs for checkpointed DataFrames on the
> server. Libraries such as GraphFrames need to choose these handles on the
> client so they can reference and release the corresponding cached DataFrames.
> Add an optional relation_id to CheckpointCommand. The server should use a
> supplied non-empty ID and continue generating a UUID when the field is
> absent. IDs remain scoped to the Connect session and use the existing
> cached-relation removal mechanism.
> The Python Connect DataFrame.checkpoint() and DataFrame.localCheckpoint()
> methods should generate UUIDs, pass them in the request, validate that the
> server returns the requested ID, and construct CachedRemoteRelation from the
> original ID and session. Public method signatures remain unchanged. Empty IDs
> should receive a structured error, and duplicate IDs should retain the
> existing error.
> Python checkpoint methods will require a server that supports the field
> because older servers return a different, server-generated ID. Other protocol
> clients that omit the field retain the existing behavior.
> Implementation PR: https://github.com/apache/spark/pull/59540
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
