This is an automated email from the ASF dual-hosted git repository.

zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 6043db3896b8 [SPARK-60374][CONNECT] Support client-generated 
checkpoint relation IDs
6043db3896b8 is described below

commit 6043db3896b802b508cf84577406e1eaa3db2f69
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Mon Oct 26 10:46:10 2026 +0800

    [SPARK-60374][CONNECT] Support client-generated checkpoint relation IDs
    
    ### What changes were proposed in this pull request?
    
    Add an optional `relation_id` to `CheckpointCommand`. The server uses a 
supplied non-empty ID and generates a UUID when the field is absent. Both 
checkpoint variants return the actual ID through `CheckpointCommandResult`.
    
    Python Connect `DataFrame.checkpoint()` and `DataFrame.localCheckpoint()` 
generate a UUID and send it through the internal command builder, whose ID 
remains optional. Response decoding returns the server-provided relation ID, 
and the DataFrame uses that returned ID and its originating session. Older 
servers that ignore the new field remain supported. Remove 
`proto_to_remote_cached_dataframe` and regenerate the Python protobuf bindings.
    
    Reject empty client-supplied IDs with a structured error. Checkpoint IDs 
are cached with an atomic `putIfAbsent`; an ID already in the cache produces 
`CONNECT_INVALID_PLAN.DATAFRAME_ALREADY_EXISTS` without replacing the existing 
entry. This applies to both client-generated and server-generated checkpoint 
IDs. ForeachBatch retains its existing collision handling. IDs remain scoped to 
the Connect session and use the existing cached-relation removal mechanism.
    
    ### Why are the changes needed?
    
    Libraries such as GraphFrames need to choose checkpoint handles on the 
client so they can reference and release the corresponding server-side 
DataFrames. Previously, the Connect server always chose the handle.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Protocol clients can optionally supply a checkpoint relation ID. 
Requests without one keep server-generated IDs. Python checkpoint methods 
request a UUID and use the actual ID returned by the server, preserving 
compatibility with servers that generate their own IDs. Public method 
signatures are unchanged.
    
    ### How was this patch tested?
    
    Added a session-holder regression test for duplicate cache entries. 
`SessionHolder.scala`, `SparkConnectPlannerSuite.scala`, 
`test_connect_basic.py`, and `test_client.py` are unchanged by this PR.
    
    - The updated planner and existing planner suite compiled locally against 
the PR's CI build; all 45 existing planner tests passed.
    - All 44 existing Python client tests, four DataFrame cache tests, and four 
existing Python GC tests passed in earlier local validation.
    - Scala formatting and `git diff --check` passed.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6)
    
    Closes #59540 from zhengruifeng/connect-client-checkpoint-relation-id.
    
    Authored-by: Ruifeng Zheng <[email protected]>
    Signed-off-by: Ruifeng Zheng <[email protected]>
---
 .../src/main/resources/error/error-conditions.json | 10 ++++++
 python/pyspark/sql/connect/client/core.py          |  5 +--
 python/pyspark/sql/connect/conversion.py           | 19 ------------
 python/pyspark/sql/connect/dataframe.py            | 36 ++++++++++++++++------
 python/pyspark/sql/connect/plan.py                 |  4 +++
 python/pyspark/sql/connect/proto/commands_pb2.py   | 16 +++++-----
 python/pyspark/sql/connect/proto/commands_pb2.pyi  | 20 ++++++++++++
 python/pyspark/sql/connect/proto/relations_pb2.pyi |  2 +-
 .../src/main/protobuf/spark/connect/commands.proto |  5 +++
 .../main/protobuf/spark/connect/relations.proto    |  2 +-
 .../sql/connect/planner/SparkConnectPlanner.scala  | 17 ++++++++--
 .../service/SparkConnectSessionHolderSuite.scala   | 19 +++++++++++-
 12 files changed, 108 insertions(+), 47 deletions(-)

diff --git a/common/utils/src/main/resources/error/error-conditions.json 
b/common/utils/src/main/resources/error/error-conditions.json
index b6bb675bac17..828aa3ac2b61 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -1426,11 +1426,21 @@
           "To disable plan compression, set 
'spark.connect.session.planCompression.threshold' to -1."
         ]
       },
+      "CHECKPOINT_RELATION_ID_EMPTY" : {
+        "message" : [
+          "The checkpoint relation ID must not be empty."
+        ]
+      },
       "CHUNKED_CACHED_LOCAL_RELATION_WITHOUT_DATA" : {
         "message" : [
           "ChunkedCachedLocalRelation should contain data."
         ]
       },
+      "DATAFRAME_ALREADY_EXISTS" : {
+        "message" : [
+          "A DataFrame with id <dfId> already exists in the session 
<sessionId>."
+        ]
+      },
       "DATAFRAME_NOT_FOUND" : {
         "message" : [
           "No DataFrame with id <dfId> is found in the session <sessionId>"
diff --git a/python/pyspark/sql/connect/client/core.py 
b/python/pyspark/sql/connect/client/core.py
index a88ce09b3c55..531528590ace 100644
--- a/python/pyspark/sql/connect/client/core.py
+++ b/python/pyspark/sql/connect/client/core.py
@@ -87,7 +87,6 @@ from pyspark.sql.connect.client.retries import (
     RetryPolicy,
 )
 from pyspark.sql.connect.conversion import (
-    proto_to_remote_cached_dataframe,
     proto_to_storage_level,
     storage_level_to_proto,
 )
@@ -1922,9 +1921,7 @@ class SparkConnectClient(object):
                 yield {"create_resource_profile_command_result": profile_id}
             if b.HasField("checkpoint_command_result"):
                 yield {
-                    "checkpoint_command_result": 
proto_to_remote_cached_dataframe(
-                        b.checkpoint_command_result.relation
-                    )
+                    "checkpoint_command_result": 
(b.checkpoint_command_result.relation.relation_id)
                 }
             if b.HasField("ml_command_result"):
                 yield {"ml_command_result": b.ml_command_result}
diff --git a/python/pyspark/sql/connect/conversion.py 
b/python/pyspark/sql/connect/conversion.py
index 220ac36a8a00..a9d8e6810921 100644
--- a/python/pyspark/sql/connect/conversion.py
+++ b/python/pyspark/sql/connect/conversion.py
@@ -14,14 +14,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-from typing import TYPE_CHECKING
-
 import pyspark.sql.connect.proto as pb2
 from pyspark.storagelevel import StorageLevel
 
-if TYPE_CHECKING:
-    from pyspark.sql.connect.dataframe import DataFrame
-
 
 def storage_level_to_proto(storage_level: StorageLevel) -> pb2.StorageLevel:
     assert storage_level is not None and isinstance(storage_level, 
StorageLevel)
@@ -43,17 +38,3 @@ def proto_to_storage_level(storage_level: pb2.StorageLevel) 
-> StorageLevel:
         deserialized=storage_level.deserialized,
         replication=storage_level.replication,
     )
-
-
-def proto_to_remote_cached_dataframe(relation: pb2.CachedRemoteRelation) -> 
"DataFrame":
-    assert relation is not None and isinstance(relation, 
pb2.CachedRemoteRelation)
-
-    import pyspark.sql.connect.plan as plan
-    from pyspark.sql.connect.dataframe import DataFrame
-    from pyspark.sql.connect.session import SparkSession
-
-    session = SparkSession.active()
-    return DataFrame(
-        plan=plan.CachedRemoteRelation(relation.relation_id, session),
-        session=session,
-    )
diff --git a/python/pyspark/sql/connect/dataframe.py 
b/python/pyspark/sql/connect/dataframe.py
index cb25a25a3e62..8d76ba09e6d7 100644
--- a/python/pyspark/sql/connect/dataframe.py
+++ b/python/pyspark/sql/connect/dataframe.py
@@ -22,6 +22,7 @@ import json
 import os
 import random
 import sys
+import uuid
 import warnings
 from collections.abc import Iterable
 from typing import (
@@ -2483,26 +2484,41 @@ class DataFrame(ParentDataFrame):
         return DataFrame(plan.Offset(child=self._plan, offset=num), 
session=self._session)
 
     def checkpoint(self, eager: bool = True) -> ParentDataFrame:
-        cmd = plan.Checkpoint(child=self._plan, local=False, eager=eager)
+        relation_id = str(uuid.uuid4())
+        cmd = plan.Checkpoint(
+            child=self._plan,
+            local=False,
+            eager=eager,
+            relation_id=relation_id,
+        )
         _, properties, self._execution_info = 
self._session.client.execute_command(
             cmd.command(self._session.client)
         )
-        assert "checkpoint_command_result" in properties
-        checkpointed = properties["checkpoint_command_result"]
-        assert isinstance(checkpointed._plan, plan.CachedRemoteRelation)
-        return checkpointed
+        returned_relation_id = properties["checkpoint_command_result"]
+        return DataFrame(
+            plan=plan.CachedRemoteRelation(returned_relation_id, 
self._session),
+            session=self._session,
+        )
 
     def localCheckpoint(
         self, eager: bool = True, storageLevel: Optional[StorageLevel] = None
     ) -> ParentDataFrame:
-        cmd = plan.Checkpoint(child=self._plan, local=True, eager=eager, 
storage_level=storageLevel)
+        relation_id = str(uuid.uuid4())
+        cmd = plan.Checkpoint(
+            child=self._plan,
+            local=True,
+            eager=eager,
+            storage_level=storageLevel,
+            relation_id=relation_id,
+        )
         _, properties, self._execution_info = 
self._session.client.execute_command(
             cmd.command(self._session.client)
         )
-        assert "checkpoint_command_result" in properties
-        checkpointed = properties["checkpoint_command_result"]
-        assert isinstance(checkpointed._plan, plan.CachedRemoteRelation)
-        return checkpointed
+        returned_relation_id = properties["checkpoint_command_result"]
+        return DataFrame(
+            plan=plan.CachedRemoteRelation(returned_relation_id, 
self._session),
+            session=self._session,
+        )
 
     def toJSON(self) -> ParentDataFrame:
         return self.select(F.to_json(F.struct(F.col("*"))).alias("value"))
diff --git a/python/pyspark/sql/connect/plan.py 
b/python/pyspark/sql/connect/plan.py
index 351f1be4f716..491178d2c92e 100644
--- a/python/pyspark/sql/connect/plan.py
+++ b/python/pyspark/sql/connect/plan.py
@@ -2384,11 +2384,13 @@ class Checkpoint(LogicalPlan):
         local: bool,
         eager: bool,
         storage_level: Optional[StorageLevel] = None,
+        relation_id: Optional[str] = None,
     ) -> None:
         super().__init__(child)
         self._local = local
         self._eager = eager
         self._storage_level = storage_level
+        self._relation_id = relation_id
 
     def command(self, session: "SparkConnectClient") -> proto.Command:
         cmd = proto.Command()
@@ -2400,6 +2402,8 @@ class Checkpoint(LogicalPlan):
         )
         if self._storage_level is not None:
             
checkpoint_command.storage_level.CopyFrom(storage_level_to_proto(self._storage_level))
+        if self._relation_id is not None:
+            checkpoint_command.relation_id = self._relation_id
         cmd.checkpoint_command.CopyFrom(checkpoint_command)
         return cmd
 
diff --git a/python/pyspark/sql/connect/proto/commands_pb2.py 
b/python/pyspark/sql/connect/proto/commands_pb2.py
index 78af0ddbfe31..c022c8c19d3d 100644
--- a/python/pyspark/sql/connect/proto/commands_pb2.py
+++ b/python/pyspark/sql/connect/proto/commands_pb2.py
@@ -44,7 +44,7 @@ from pyspark.sql.connect.proto import pipelines_pb2 as 
spark_dot_connect_dot_pip
 
 
 DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
-    
b'\n\x1cspark/connect/commands.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x1aspark/connect/common.proto\x1a\x1fspark/connect/expressions.proto\x1a\x1dspark/connect/relations.proto\x1a\x16spark/connect/ml.proto\x1a\x1dspark/connect/pipelines.proto"\xfb\x0e\n\x07\x43ommand\x12]\n\x11register_function\x18\x01
 
\x01(\x0b\x32..spark.connect.CommonInlineUserDefinedFunctionH\x00R\x10registerFunction\x12H\n\x0fwrite_operation\x18\x02
 \x01(\x0b\x32\x1d.spark.connect.WriteOpe [...]
+    
b'\n\x1cspark/connect/commands.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x1aspark/connect/common.proto\x1a\x1fspark/connect/expressions.proto\x1a\x1dspark/connect/relations.proto\x1a\x16spark/connect/ml.proto\x1a\x1dspark/connect/pipelines.proto"\xfb\x0e\n\x07\x43ommand\x12]\n\x11register_function\x18\x01
 
\x01(\x0b\x32..spark.connect.CommonInlineUserDefinedFunctionH\x00R\x10registerFunction\x12H\n\x0fwrite_operation\x18\x02
 \x01(\x0b\x32\x1d.spark.connect.WriteOpe [...]
 )
 
 _globals = globals()
@@ -83,8 +83,8 @@ if not _descriptor._USE_C_DESCRIPTORS:
     _globals["_GETRESOURCESCOMMANDRESULT_RESOURCESENTRY"]._serialized_options 
= b"8\001"
     _globals["_EXECUTEEXTERNALCOMMAND_OPTIONSENTRY"]._loaded_options = None
     _globals["_EXECUTEEXTERNALCOMMAND_OPTIONSENTRY"]._serialized_options = 
b"8\001"
-    _globals["_STREAMINGQUERYEVENTTYPE"]._serialized_start = 11920
-    _globals["_STREAMINGQUERYEVENTTYPE"]._serialized_end = 12053
+    _globals["_STREAMINGQUERYEVENTTYPE"]._serialized_start = 11974
+    _globals["_STREAMINGQUERYEVENTTYPE"]._serialized_end = 12107
     _globals["_COMMAND"]._serialized_start = 222
     _globals["_COMMAND"]._serialized_end = 2137
     _globals["_SQLCOMMAND"]._serialized_start = 2140
@@ -190,11 +190,11 @@ if not _descriptor._USE_C_DESCRIPTORS:
     _globals["_REMOVECACHEDREMOTERELATIONCOMMAND"]._serialized_start = 10903
     _globals["_REMOVECACHEDREMOTERELATIONCOMMAND"]._serialized_end = 11003
     _globals["_CHECKPOINTCOMMAND"]._serialized_start = 11006
-    _globals["_CHECKPOINTCOMMAND"]._serialized_end = 11211
-    _globals["_MERGEINTOTABLECOMMAND"]._serialized_start = 11214
-    _globals["_MERGEINTOTABLECOMMAND"]._serialized_end = 11702
-    _globals["_EXECUTEEXTERNALCOMMAND"]._serialized_start = 11705
-    _globals["_EXECUTEEXTERNALCOMMAND"]._serialized_end = 11917
+    _globals["_CHECKPOINTCOMMAND"]._serialized_end = 11265
+    _globals["_MERGEINTOTABLECOMMAND"]._serialized_start = 11268
+    _globals["_MERGEINTOTABLECOMMAND"]._serialized_end = 11756
+    _globals["_EXECUTEEXTERNALCOMMAND"]._serialized_start = 11759
+    _globals["_EXECUTEEXTERNALCOMMAND"]._serialized_end = 11971
     _globals["_EXECUTEEXTERNALCOMMAND_OPTIONSENTRY"]._serialized_start = 3424
     _globals["_EXECUTEEXTERNALCOMMAND_OPTIONSENTRY"]._serialized_end = 3482
 # @@protoc_insertion_point(module_scope)
diff --git a/python/pyspark/sql/connect/proto/commands_pb2.pyi 
b/python/pyspark/sql/connect/proto/commands_pb2.pyi
index cc1330f11f8a..9b7fccac6478 100644
--- a/python/pyspark/sql/connect/proto/commands_pb2.pyi
+++ b/python/pyspark/sql/connect/proto/commands_pb2.pyi
@@ -2244,6 +2244,7 @@ class CheckpointCommand(google.protobuf.message.Message):
     LOCAL_FIELD_NUMBER: builtins.int
     EAGER_FIELD_NUMBER: builtins.int
     STORAGE_LEVEL_FIELD_NUMBER: builtins.int
+    RELATION_ID_FIELD_NUMBER: builtins.int
     @property
     def relation(self) -> pyspark.sql.connect.proto.relations_pb2.Relation:
         """(Required) The logical plan to checkpoint."""
@@ -2256,6 +2257,11 @@ class CheckpointCommand(google.protobuf.message.Message):
     @property
     def storage_level(self) -> 
pyspark.sql.connect.proto.common_pb2.StorageLevel:
         """(Optional) For local checkpoint, the storage level to use."""
+    relation_id: builtins.str
+    """(Optional) ID to assign to the checkpointed DataFrame in the current 
session.
+    If omitted, the server generates an ID. If provided, it must be non-empty 
and
+    must not already identify a cached DataFrame in this session.
+    """
     def __init__(
         self,
         *,
@@ -2263,14 +2269,19 @@ class 
CheckpointCommand(google.protobuf.message.Message):
         local: builtins.bool = ...,
         eager: builtins.bool = ...,
         storage_level: pyspark.sql.connect.proto.common_pb2.StorageLevel | 
None = ...,
+        relation_id: builtins.str | None = ...,
     ) -> None: ...
     def HasField(
         self,
         field_name: typing_extensions.Literal[
+            "_relation_id",
+            b"_relation_id",
             "_storage_level",
             b"_storage_level",
             "relation",
             b"relation",
+            "relation_id",
+            b"relation_id",
             "storage_level",
             b"storage_level",
         ],
@@ -2278,6 +2289,8 @@ class CheckpointCommand(google.protobuf.message.Message):
     def ClearField(
         self,
         field_name: typing_extensions.Literal[
+            "_relation_id",
+            b"_relation_id",
             "_storage_level",
             b"_storage_level",
             "eager",
@@ -2286,10 +2299,17 @@ class 
CheckpointCommand(google.protobuf.message.Message):
             b"local",
             "relation",
             b"relation",
+            "relation_id",
+            b"relation_id",
             "storage_level",
             b"storage_level",
         ],
     ) -> None: ...
+    @typing.overload
+    def WhichOneof(
+        self, oneof_group: typing_extensions.Literal["_relation_id", 
b"_relation_id"]
+    ) -> typing_extensions.Literal["relation_id"] | None: ...
+    @typing.overload
     def WhichOneof(
         self, oneof_group: typing_extensions.Literal["_storage_level", 
b"_storage_level"]
     ) -> typing_extensions.Literal["storage_level"] | None: ...
diff --git a/python/pyspark/sql/connect/proto/relations_pb2.pyi 
b/python/pyspark/sql/connect/proto/relations_pb2.pyi
index a286e560c0dc..82fd1a39e18d 100644
--- a/python/pyspark/sql/connect/proto/relations_pb2.pyi
+++ b/python/pyspark/sql/connect/proto/relations_pb2.pyi
@@ -2281,7 +2281,7 @@ class 
CachedRemoteRelation(google.protobuf.message.Message):
 
     RELATION_ID_FIELD_NUMBER: builtins.int
     relation_id: builtins.str
-    """(Required) ID of the remote related (assigned by the service)."""
+    """(Required) ID of the remote relation in the current session."""
     def __init__(
         self,
         *,
diff --git a/sql/connect/common/src/main/protobuf/spark/connect/commands.proto 
b/sql/connect/common/src/main/protobuf/spark/connect/commands.proto
index dcf5aff2366f..2fb391892a81 100644
--- a/sql/connect/common/src/main/protobuf/spark/connect/commands.proto
+++ b/sql/connect/common/src/main/protobuf/spark/connect/commands.proto
@@ -522,6 +522,11 @@ message CheckpointCommand {
 
   // (Optional) For local checkpoint, the storage level to use.
   optional StorageLevel storage_level = 4;
+
+  // (Optional) ID to assign to the checkpointed DataFrame in the current 
session.
+  // If omitted, the server generates an ID. If provided, it must be non-empty 
and
+  // must not already identify a cached DataFrame in this session.
+  optional string relation_id = 5;
 }
 
 message MergeIntoTableCommand {
diff --git a/sql/connect/common/src/main/protobuf/spark/connect/relations.proto 
b/sql/connect/common/src/main/protobuf/spark/connect/relations.proto
index ac6f9b917894..d64a47e92d37 100644
--- a/sql/connect/common/src/main/protobuf/spark/connect/relations.proto
+++ b/sql/connect/common/src/main/protobuf/spark/connect/relations.proto
@@ -556,7 +556,7 @@ message ChunkedCachedLocalRelation {
 
 // Represents a remote relation that has been cached on server.
 message CachedRemoteRelation {
-  // (Required) ID of the remote related (assigned by the service).
+  // (Required) ID of the remote relation in the current session.
   string relation_id = 1;
 }
 
diff --git 
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
 
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
index 28efd70d44be..cbc6d40386a5 100644
--- 
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
+++ 
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
@@ -60,7 +60,7 @@ import 
org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, CharVarcharUtils}
 import org.apache.spark.sql.classic.{Catalog, DataFrameWriter, Dataset, 
MergeIntoWriter, RelationalGroupedDataset, SparkSession, TypedAggUtils, 
UserDefinedFunctionUtils}
 import org.apache.spark.sql.classic.ClassicConversions._
 import org.apache.spark.sql.connect.client.arrow.ArrowSerializer
-import org.apache.spark.sql.connect.common.{DataTypeProtoConverter, 
ForeachWriterPacket, LiteralValueProtoConverter, StorageLevelProtoConverter, 
StreamingListenerPacket, UdfPacket, UdfSerialization}
+import org.apache.spark.sql.connect.common.{DataTypeProtoConverter, 
ForeachWriterPacket, InvalidPlanInput, LiteralValueProtoConverter, 
StorageLevelProtoConverter, StreamingListenerPacket, UdfPacket, 
UdfSerialization}
 import 
org.apache.spark.sql.connect.config.Connect.CONNECT_GRPC_ARROW_MAX_BATCH_SIZE
 import org.apache.spark.sql.connect.ml.MLHandler
 import org.apache.spark.sql.connect.pipelines.PipelinesHandler
@@ -4187,6 +4187,14 @@ class SparkConnectPlanner(
   private def handleCheckpointCommand(
       checkpointCommand: CheckpointCommand,
       responseObserver: StreamObserver[proto.ExecutePlanResponse]): Unit = {
+    val dfId = if (checkpointCommand.hasRelationId) {
+      if (checkpointCommand.getRelationId.isEmpty) {
+        throw 
InvalidPlanInput("CONNECT_INVALID_PLAN.CHECKPOINT_RELATION_ID_EMPTY", Map.empty)
+      }
+      checkpointCommand.getRelationId
+    } else {
+      UUID.randomUUID().toString
+    }
     val target = Dataset
       .ofRows(session, transformRelation(checkpointCommand.getRelation))
     val checkpointed = if (checkpointCommand.getLocal) {
@@ -4202,9 +4210,12 @@ class SparkConnectPlanner(
       target.checkpoint(eager = checkpointCommand.getEager)
     }
 
-    val dfId = UUID.randomUUID().toString
     logInfo(log"Caching DataFrame with id ${MDC(DATAFRAME_ID, dfId)}")
-    sessionHolder.cacheDataFrameById(dfId, checkpointed)
+    if (sessionHolder.dataFrameCache.putIfAbsent(dfId, checkpointed) != null) {
+      throw InvalidPlanInput(
+        "CONNECT_INVALID_PLAN.DATAFRAME_ALREADY_EXISTS",
+        Map("dfId" -> dfId, "sessionId" -> sessionId))
+    }
 
     executeHolder.eventsManager.postFinished()
     responseObserver.onNext(
diff --git 
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionHolderSuite.scala
 
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionHolderSuite.scala
index f34f46101136..3369e25c78c3 100644
--- 
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionHolderSuite.scala
+++ 
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionHolderSuite.scala
@@ -30,7 +30,7 @@ import scala.util.control.NonFatal
 import com.google.common.collect.Lists
 import org.scalatest.time.SpanSugar._
 
-import org.apache.spark.SparkEnv
+import org.apache.spark.{SparkEnv, SparkException}
 import org.apache.spark.api.python.SimplePythonFunction
 import org.apache.spark.connect.proto
 import org.apache.spark.sql.IntegratedUDFTestUtils
@@ -69,6 +69,23 @@ class SparkConnectSessionHolderSuite extends 
SharedSparkSession {
     assert(expectedDf2 == df2)
   }
 
+  test("DataFrame cache: Duplicate IDs are not overwritten") {
+    val sessionHolder = SparkConnectTestUtils.createDummySessionHolder(spark)
+    val df1 = spark.range(1).toDF()
+    val df2 = spark.range(2).toDF()
+    val dfId = "duplicate-id"
+
+    sessionHolder.cacheDataFrameById(dfId, df1)
+    val error = intercept[SparkException] {
+      sessionHolder.cacheDataFrameById(dfId, df2)
+    }
+    checkError(
+      exception = error,
+      condition = "INTERNAL_ERROR",
+      parameters = Map("message" -> s"A dataframe is already associated with 
id $dfId"))
+    assert(sessionHolder.getDataFrameOrThrow(dfId) eq df1)
+  }
+
   test("DataFrame cache: Should throw when dataframe is not found") {
     val sessionHolder = SparkConnectTestUtils.createDummySessionHolder(spark)
     import sessionHolder.session.implicits._


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to