zhengruifeng opened a new pull request, #59540: URL: https://github.com/apache/spark/pull/59540
### What changes were proposed in this pull request? Add an optional `relation_id` to `CheckpointCommand`. The server uses a supplied non-empty ID and generates a UUID when the field is absent. Both checkpoint variants continue returning the actual ID through `CheckpointCommandResult`. Reject empty IDs and duplicate IDs with structured errors. Duplicate registration preserves the existing DataFrame. IDs remain scoped to the Connect session and use the existing cached-relation removal mechanism. Extend the internal Python checkpoint command builder and regenerate the Python protobuf bindings. The public `DataFrame.checkpoint()` and `DataFrame.localCheckpoint()` signatures are unchanged. ### Why are the changes needed? Libraries such as GraphFrames need to choose checkpoint handles on the client so they can reference and release the corresponding server-side DataFrames. Currently, the Connect server always chooses the handle. Making the request field optional preserves the existing behavior for clients that omit it. A client relying on its requested ID must wait for successful registration and inspect the returned ID. Older servers ignore the new field and return a server-generated ID. ### Does this PR introduce _any_ user-facing change? Yes. Connect protocol clients can optionally supply a checkpoint relation ID. Existing requests retain server-generated IDs. Supplied IDs must be non-empty and unique among cached DataFrames in the session. ### How was this patch tested? - All 85 tests in `pyspark.sql.tests.connect.test_connect_plan` passed, including new coverage for optional relation ID serialization. - Added 11 Scala cases covering checkpoint/localCheckpoint, eager/lazy execution, supplied/generated IDs, cached-relation resolution and removal, empty and duplicate IDs, and session isolation. - Generated Java protocol classes and the Scala test suite/session cache compiled against cached dependencies. Scala syntax checks passed for all changed Scala files. - Protobuf breaking-change checks against `upstream/master` passed. Protobuf lint produced no new findings compared with the base branch. - Python lint/format checks, error catalog validation, and `git diff --check` passed. The Scala suite could not be executed locally: Maven hosts cannot be resolved in this environment, and the offline dependency cache is missing `scala-xml_2.13:2.5.0`. Full Scala validation remains pending CI. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
