This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new b4bc0a6c1dce [SPARK-60374][CONNECT] Support client-generated
checkpoint relation IDs
b4bc0a6c1dce is described below
commit b4bc0a6c1dce7f5354dc617110eee59945974a4f
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Mon Oct 26 10:46:10 2026 +0800
[SPARK-60374][CONNECT] Support client-generated checkpoint relation IDs
### What changes were proposed in this pull request?
Add an optional `relation_id` to `CheckpointCommand`. The server uses a
supplied non-empty ID and generates a UUID when the field is absent. Both
checkpoint variants return the actual ID through `CheckpointCommandResult`.
Python Connect `DataFrame.checkpoint()` and `DataFrame.localCheckpoint()`
generate a UUID and send it through the internal command builder, whose ID
remains optional. Response decoding returns the server-provided relation ID,
and the DataFrame uses that returned ID and its originating session. Older
servers that ignore the new field remain supported. Remove
`proto_to_remote_cached_dataframe` and regenerate the Python protobuf bindings.
Reject empty client-supplied IDs with a structured error. Checkpoint IDs
are cached with an atomic `putIfAbsent`; an ID already in the cache produces
`CONNECT_INVALID_PLAN.DATAFRAME_ALREADY_EXISTS` without replacing the existing
entry. This applies to both client-generated and server-generated checkpoint
IDs. ForeachBatch retains its existing collision handling. IDs remain scoped to
the Connect session and use the existing cached-relation removal mechanism.
### Why are the changes needed?
Libraries such as GraphFrames need to choose checkpoint handles on the
client so they can reference and release the corresponding server-side
DataFrames. Previously, the Connect server always chose the handle.
### Does this PR introduce _any_ user-facing change?
Yes. Protocol clients can optionally supply a checkpoint relation ID.
Requests without one keep server-generated IDs. Python checkpoint methods
request a UUID and use the actual ID returned by the server, preserving
compatibility with servers that generate their own IDs. Public method
signatures are unchanged.
### How was this patch tested?
Added a session-holder regression test for duplicate cache entries.
`SessionHolder.scala`, `SparkConnectPlannerSuite.scala`,
`test_connect_basic.py`, and `test_client.py` are unchanged by this PR.
- The updated planner and existing planner suite compiled locally against
the PR's CI build; all 45 existing planner tests passed.
- All 44 existing Python client tests, four DataFrame cache tests, and four
existing Python GC tests passed in earlier local validation.
- Scala formatting and `git diff --check` passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
Closes #59540 from zhengruifeng/connect-client-checkpoint-relation-id.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 6043db3896b802b508cf84577406e1eaa3db2f69)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 10 ++++++
python/pyspark/sql/connect/client/core.py | 5 +--
python/pyspark/sql/connect/conversion.py | 19 ------------
python/pyspark/sql/connect/dataframe.py | 36 ++++++++++++++++------
python/pyspark/sql/connect/plan.py | 4 +++
python/pyspark/sql/connect/proto/commands_pb2.py | 16 +++++-----
python/pyspark/sql/connect/proto/commands_pb2.pyi | 20 ++++++++++++
python/pyspark/sql/connect/proto/relations_pb2.pyi | 2 +-
.../src/main/protobuf/spark/connect/commands.proto | 5 +++
.../main/protobuf/spark/connect/relations.proto | 2 +-
.../sql/connect/planner/SparkConnectPlanner.scala | 17 ++++++++--
.../service/SparkConnectSessionHolderSuite.scala | 19 +++++++++++-
12 files changed, 108 insertions(+), 47 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index 71ef75a4b712..a0ee5facb3be 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -1426,11 +1426,21 @@
"To disable plan compression, set
'spark.connect.session.planCompression.threshold' to -1."
]
},
+ "CHECKPOINT_RELATION_ID_EMPTY" : {
+ "message" : [
+ "The checkpoint relation ID must not be empty."
+ ]
+ },
"CHUNKED_CACHED_LOCAL_RELATION_WITHOUT_DATA" : {
"message" : [
"ChunkedCachedLocalRelation should contain data."
]
},
+ "DATAFRAME_ALREADY_EXISTS" : {
+ "message" : [
+ "A DataFrame with id <dfId> already exists in the session
<sessionId>."
+ ]
+ },
"DATAFRAME_NOT_FOUND" : {
"message" : [
"No DataFrame with id <dfId> is found in the session <sessionId>"
diff --git a/python/pyspark/sql/connect/client/core.py
b/python/pyspark/sql/connect/client/core.py
index a88ce09b3c55..531528590ace 100644
--- a/python/pyspark/sql/connect/client/core.py
+++ b/python/pyspark/sql/connect/client/core.py
@@ -87,7 +87,6 @@ from pyspark.sql.connect.client.retries import (
RetryPolicy,
)
from pyspark.sql.connect.conversion import (
- proto_to_remote_cached_dataframe,
proto_to_storage_level,
storage_level_to_proto,
)
@@ -1922,9 +1921,7 @@ class SparkConnectClient(object):
yield {"create_resource_profile_command_result": profile_id}
if b.HasField("checkpoint_command_result"):
yield {
- "checkpoint_command_result":
proto_to_remote_cached_dataframe(
- b.checkpoint_command_result.relation
- )
+ "checkpoint_command_result":
(b.checkpoint_command_result.relation.relation_id)
}
if b.HasField("ml_command_result"):
yield {"ml_command_result": b.ml_command_result}
diff --git a/python/pyspark/sql/connect/conversion.py
b/python/pyspark/sql/connect/conversion.py
index 220ac36a8a00..a9d8e6810921 100644
--- a/python/pyspark/sql/connect/conversion.py
+++ b/python/pyspark/sql/connect/conversion.py
@@ -14,14 +14,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-from typing import TYPE_CHECKING
-
import pyspark.sql.connect.proto as pb2
from pyspark.storagelevel import StorageLevel
-if TYPE_CHECKING:
- from pyspark.sql.connect.dataframe import DataFrame
-
def storage_level_to_proto(storage_level: StorageLevel) -> pb2.StorageLevel:
assert storage_level is not None and isinstance(storage_level,
StorageLevel)
@@ -43,17 +38,3 @@ def proto_to_storage_level(storage_level: pb2.StorageLevel)
-> StorageLevel:
deserialized=storage_level.deserialized,
replication=storage_level.replication,
)
-
-
-def proto_to_remote_cached_dataframe(relation: pb2.CachedRemoteRelation) ->
"DataFrame":
- assert relation is not None and isinstance(relation,
pb2.CachedRemoteRelation)
-
- import pyspark.sql.connect.plan as plan
- from pyspark.sql.connect.dataframe import DataFrame
- from pyspark.sql.connect.session import SparkSession
-
- session = SparkSession.active()
- return DataFrame(
- plan=plan.CachedRemoteRelation(relation.relation_id, session),
- session=session,
- )
diff --git a/python/pyspark/sql/connect/dataframe.py
b/python/pyspark/sql/connect/dataframe.py
index cb25a25a3e62..8d76ba09e6d7 100644
--- a/python/pyspark/sql/connect/dataframe.py
+++ b/python/pyspark/sql/connect/dataframe.py
@@ -22,6 +22,7 @@ import json
import os
import random
import sys
+import uuid
import warnings
from collections.abc import Iterable
from typing import (
@@ -2483,26 +2484,41 @@ class DataFrame(ParentDataFrame):
return DataFrame(plan.Offset(child=self._plan, offset=num),
session=self._session)
def checkpoint(self, eager: bool = True) -> ParentDataFrame:
- cmd = plan.Checkpoint(child=self._plan, local=False, eager=eager)
+ relation_id = str(uuid.uuid4())
+ cmd = plan.Checkpoint(
+ child=self._plan,
+ local=False,
+ eager=eager,
+ relation_id=relation_id,
+ )
_, properties, self._execution_info =
self._session.client.execute_command(
cmd.command(self._session.client)
)
- assert "checkpoint_command_result" in properties
- checkpointed = properties["checkpoint_command_result"]
- assert isinstance(checkpointed._plan, plan.CachedRemoteRelation)
- return checkpointed
+ returned_relation_id = properties["checkpoint_command_result"]
+ return DataFrame(
+ plan=plan.CachedRemoteRelation(returned_relation_id,
self._session),
+ session=self._session,
+ )
def localCheckpoint(
self, eager: bool = True, storageLevel: Optional[StorageLevel] = None
) -> ParentDataFrame:
- cmd = plan.Checkpoint(child=self._plan, local=True, eager=eager,
storage_level=storageLevel)
+ relation_id = str(uuid.uuid4())
+ cmd = plan.Checkpoint(
+ child=self._plan,
+ local=True,
+ eager=eager,
+ storage_level=storageLevel,
+ relation_id=relation_id,
+ )
_, properties, self._execution_info =
self._session.client.execute_command(
cmd.command(self._session.client)
)
- assert "checkpoint_command_result" in properties
- checkpointed = properties["checkpoint_command_result"]
- assert isinstance(checkpointed._plan, plan.CachedRemoteRelation)
- return checkpointed
+ returned_relation_id = properties["checkpoint_command_result"]
+ return DataFrame(
+ plan=plan.CachedRemoteRelation(returned_relation_id,
self._session),
+ session=self._session,
+ )
def toJSON(self) -> ParentDataFrame:
return self.select(F.to_json(F.struct(F.col("*"))).alias("value"))
diff --git a/python/pyspark/sql/connect/plan.py
b/python/pyspark/sql/connect/plan.py
index 351f1be4f716..491178d2c92e 100644
--- a/python/pyspark/sql/connect/plan.py
+++ b/python/pyspark/sql/connect/plan.py
@@ -2384,11 +2384,13 @@ class Checkpoint(LogicalPlan):
local: bool,
eager: bool,
storage_level: Optional[StorageLevel] = None,
+ relation_id: Optional[str] = None,
) -> None:
super().__init__(child)
self._local = local
self._eager = eager
self._storage_level = storage_level
+ self._relation_id = relation_id
def command(self, session: "SparkConnectClient") -> proto.Command:
cmd = proto.Command()
@@ -2400,6 +2402,8 @@ class Checkpoint(LogicalPlan):
)
if self._storage_level is not None:
checkpoint_command.storage_level.CopyFrom(storage_level_to_proto(self._storage_level))
+ if self._relation_id is not None:
+ checkpoint_command.relation_id = self._relation_id
cmd.checkpoint_command.CopyFrom(checkpoint_command)
return cmd
diff --git a/python/pyspark/sql/connect/proto/commands_pb2.py
b/python/pyspark/sql/connect/proto/commands_pb2.py
index 78af0ddbfe31..c022c8c19d3d 100644
--- a/python/pyspark/sql/connect/proto/commands_pb2.py
+++ b/python/pyspark/sql/connect/proto/commands_pb2.py
@@ -44,7 +44,7 @@ from pyspark.sql.connect.proto import pipelines_pb2 as
spark_dot_connect_dot_pip
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
-
b'\n\x1cspark/connect/commands.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x1aspark/connect/common.proto\x1a\x1fspark/connect/expressions.proto\x1a\x1dspark/connect/relations.proto\x1a\x16spark/connect/ml.proto\x1a\x1dspark/connect/pipelines.proto"\xfb\x0e\n\x07\x43ommand\x12]\n\x11register_function\x18\x01
\x01(\x0b\x32..spark.connect.CommonInlineUserDefinedFunctionH\x00R\x10registerFunction\x12H\n\x0fwrite_operation\x18\x02
\x01(\x0b\x32\x1d.spark.connect.WriteOpe [...]
+
b'\n\x1cspark/connect/commands.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x1aspark/connect/common.proto\x1a\x1fspark/connect/expressions.proto\x1a\x1dspark/connect/relations.proto\x1a\x16spark/connect/ml.proto\x1a\x1dspark/connect/pipelines.proto"\xfb\x0e\n\x07\x43ommand\x12]\n\x11register_function\x18\x01
\x01(\x0b\x32..spark.connect.CommonInlineUserDefinedFunctionH\x00R\x10registerFunction\x12H\n\x0fwrite_operation\x18\x02
\x01(\x0b\x32\x1d.spark.connect.WriteOpe [...]
)
_globals = globals()
@@ -83,8 +83,8 @@ if not _descriptor._USE_C_DESCRIPTORS:
_globals["_GETRESOURCESCOMMANDRESULT_RESOURCESENTRY"]._serialized_options
= b"8\001"
_globals["_EXECUTEEXTERNALCOMMAND_OPTIONSENTRY"]._loaded_options = None
_globals["_EXECUTEEXTERNALCOMMAND_OPTIONSENTRY"]._serialized_options =
b"8\001"
- _globals["_STREAMINGQUERYEVENTTYPE"]._serialized_start = 11920
- _globals["_STREAMINGQUERYEVENTTYPE"]._serialized_end = 12053
+ _globals["_STREAMINGQUERYEVENTTYPE"]._serialized_start = 11974
+ _globals["_STREAMINGQUERYEVENTTYPE"]._serialized_end = 12107
_globals["_COMMAND"]._serialized_start = 222
_globals["_COMMAND"]._serialized_end = 2137
_globals["_SQLCOMMAND"]._serialized_start = 2140
@@ -190,11 +190,11 @@ if not _descriptor._USE_C_DESCRIPTORS:
_globals["_REMOVECACHEDREMOTERELATIONCOMMAND"]._serialized_start = 10903
_globals["_REMOVECACHEDREMOTERELATIONCOMMAND"]._serialized_end = 11003
_globals["_CHECKPOINTCOMMAND"]._serialized_start = 11006
- _globals["_CHECKPOINTCOMMAND"]._serialized_end = 11211
- _globals["_MERGEINTOTABLECOMMAND"]._serialized_start = 11214
- _globals["_MERGEINTOTABLECOMMAND"]._serialized_end = 11702
- _globals["_EXECUTEEXTERNALCOMMAND"]._serialized_start = 11705
- _globals["_EXECUTEEXTERNALCOMMAND"]._serialized_end = 11917
+ _globals["_CHECKPOINTCOMMAND"]._serialized_end = 11265
+ _globals["_MERGEINTOTABLECOMMAND"]._serialized_start = 11268
+ _globals["_MERGEINTOTABLECOMMAND"]._serialized_end = 11756
+ _globals["_EXECUTEEXTERNALCOMMAND"]._serialized_start = 11759
+ _globals["_EXECUTEEXTERNALCOMMAND"]._serialized_end = 11971
_globals["_EXECUTEEXTERNALCOMMAND_OPTIONSENTRY"]._serialized_start = 3424
_globals["_EXECUTEEXTERNALCOMMAND_OPTIONSENTRY"]._serialized_end = 3482
# @@protoc_insertion_point(module_scope)
diff --git a/python/pyspark/sql/connect/proto/commands_pb2.pyi
b/python/pyspark/sql/connect/proto/commands_pb2.pyi
index cc1330f11f8a..9b7fccac6478 100644
--- a/python/pyspark/sql/connect/proto/commands_pb2.pyi
+++ b/python/pyspark/sql/connect/proto/commands_pb2.pyi
@@ -2244,6 +2244,7 @@ class CheckpointCommand(google.protobuf.message.Message):
LOCAL_FIELD_NUMBER: builtins.int
EAGER_FIELD_NUMBER: builtins.int
STORAGE_LEVEL_FIELD_NUMBER: builtins.int
+ RELATION_ID_FIELD_NUMBER: builtins.int
@property
def relation(self) -> pyspark.sql.connect.proto.relations_pb2.Relation:
"""(Required) The logical plan to checkpoint."""
@@ -2256,6 +2257,11 @@ class CheckpointCommand(google.protobuf.message.Message):
@property
def storage_level(self) ->
pyspark.sql.connect.proto.common_pb2.StorageLevel:
"""(Optional) For local checkpoint, the storage level to use."""
+ relation_id: builtins.str
+ """(Optional) ID to assign to the checkpointed DataFrame in the current
session.
+ If omitted, the server generates an ID. If provided, it must be non-empty
and
+ must not already identify a cached DataFrame in this session.
+ """
def __init__(
self,
*,
@@ -2263,14 +2269,19 @@ class
CheckpointCommand(google.protobuf.message.Message):
local: builtins.bool = ...,
eager: builtins.bool = ...,
storage_level: pyspark.sql.connect.proto.common_pb2.StorageLevel |
None = ...,
+ relation_id: builtins.str | None = ...,
) -> None: ...
def HasField(
self,
field_name: typing_extensions.Literal[
+ "_relation_id",
+ b"_relation_id",
"_storage_level",
b"_storage_level",
"relation",
b"relation",
+ "relation_id",
+ b"relation_id",
"storage_level",
b"storage_level",
],
@@ -2278,6 +2289,8 @@ class CheckpointCommand(google.protobuf.message.Message):
def ClearField(
self,
field_name: typing_extensions.Literal[
+ "_relation_id",
+ b"_relation_id",
"_storage_level",
b"_storage_level",
"eager",
@@ -2286,10 +2299,17 @@ class
CheckpointCommand(google.protobuf.message.Message):
b"local",
"relation",
b"relation",
+ "relation_id",
+ b"relation_id",
"storage_level",
b"storage_level",
],
) -> None: ...
+ @typing.overload
+ def WhichOneof(
+ self, oneof_group: typing_extensions.Literal["_relation_id",
b"_relation_id"]
+ ) -> typing_extensions.Literal["relation_id"] | None: ...
+ @typing.overload
def WhichOneof(
self, oneof_group: typing_extensions.Literal["_storage_level",
b"_storage_level"]
) -> typing_extensions.Literal["storage_level"] | None: ...
diff --git a/python/pyspark/sql/connect/proto/relations_pb2.pyi
b/python/pyspark/sql/connect/proto/relations_pb2.pyi
index a286e560c0dc..82fd1a39e18d 100644
--- a/python/pyspark/sql/connect/proto/relations_pb2.pyi
+++ b/python/pyspark/sql/connect/proto/relations_pb2.pyi
@@ -2281,7 +2281,7 @@ class
CachedRemoteRelation(google.protobuf.message.Message):
RELATION_ID_FIELD_NUMBER: builtins.int
relation_id: builtins.str
- """(Required) ID of the remote related (assigned by the service)."""
+ """(Required) ID of the remote relation in the current session."""
def __init__(
self,
*,
diff --git a/sql/connect/common/src/main/protobuf/spark/connect/commands.proto
b/sql/connect/common/src/main/protobuf/spark/connect/commands.proto
index dcf5aff2366f..2fb391892a81 100644
--- a/sql/connect/common/src/main/protobuf/spark/connect/commands.proto
+++ b/sql/connect/common/src/main/protobuf/spark/connect/commands.proto
@@ -522,6 +522,11 @@ message CheckpointCommand {
// (Optional) For local checkpoint, the storage level to use.
optional StorageLevel storage_level = 4;
+
+ // (Optional) ID to assign to the checkpointed DataFrame in the current
session.
+ // If omitted, the server generates an ID. If provided, it must be non-empty
and
+ // must not already identify a cached DataFrame in this session.
+ optional string relation_id = 5;
}
message MergeIntoTableCommand {
diff --git a/sql/connect/common/src/main/protobuf/spark/connect/relations.proto
b/sql/connect/common/src/main/protobuf/spark/connect/relations.proto
index ac6f9b917894..d64a47e92d37 100644
--- a/sql/connect/common/src/main/protobuf/spark/connect/relations.proto
+++ b/sql/connect/common/src/main/protobuf/spark/connect/relations.proto
@@ -556,7 +556,7 @@ message ChunkedCachedLocalRelation {
// Represents a remote relation that has been cached on server.
message CachedRemoteRelation {
- // (Required) ID of the remote related (assigned by the service).
+ // (Required) ID of the remote relation in the current session.
string relation_id = 1;
}
diff --git
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
index 28efd70d44be..cbc6d40386a5 100644
---
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
+++
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
@@ -60,7 +60,7 @@ import
org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, CharVarcharUtils}
import org.apache.spark.sql.classic.{Catalog, DataFrameWriter, Dataset,
MergeIntoWriter, RelationalGroupedDataset, SparkSession, TypedAggUtils,
UserDefinedFunctionUtils}
import org.apache.spark.sql.classic.ClassicConversions._
import org.apache.spark.sql.connect.client.arrow.ArrowSerializer
-import org.apache.spark.sql.connect.common.{DataTypeProtoConverter,
ForeachWriterPacket, LiteralValueProtoConverter, StorageLevelProtoConverter,
StreamingListenerPacket, UdfPacket, UdfSerialization}
+import org.apache.spark.sql.connect.common.{DataTypeProtoConverter,
ForeachWriterPacket, InvalidPlanInput, LiteralValueProtoConverter,
StorageLevelProtoConverter, StreamingListenerPacket, UdfPacket,
UdfSerialization}
import
org.apache.spark.sql.connect.config.Connect.CONNECT_GRPC_ARROW_MAX_BATCH_SIZE
import org.apache.spark.sql.connect.ml.MLHandler
import org.apache.spark.sql.connect.pipelines.PipelinesHandler
@@ -4187,6 +4187,14 @@ class SparkConnectPlanner(
private def handleCheckpointCommand(
checkpointCommand: CheckpointCommand,
responseObserver: StreamObserver[proto.ExecutePlanResponse]): Unit = {
+ val dfId = if (checkpointCommand.hasRelationId) {
+ if (checkpointCommand.getRelationId.isEmpty) {
+ throw
InvalidPlanInput("CONNECT_INVALID_PLAN.CHECKPOINT_RELATION_ID_EMPTY", Map.empty)
+ }
+ checkpointCommand.getRelationId
+ } else {
+ UUID.randomUUID().toString
+ }
val target = Dataset
.ofRows(session, transformRelation(checkpointCommand.getRelation))
val checkpointed = if (checkpointCommand.getLocal) {
@@ -4202,9 +4210,12 @@ class SparkConnectPlanner(
target.checkpoint(eager = checkpointCommand.getEager)
}
- val dfId = UUID.randomUUID().toString
logInfo(log"Caching DataFrame with id ${MDC(DATAFRAME_ID, dfId)}")
- sessionHolder.cacheDataFrameById(dfId, checkpointed)
+ if (sessionHolder.dataFrameCache.putIfAbsent(dfId, checkpointed) != null) {
+ throw InvalidPlanInput(
+ "CONNECT_INVALID_PLAN.DATAFRAME_ALREADY_EXISTS",
+ Map("dfId" -> dfId, "sessionId" -> sessionId))
+ }
executeHolder.eventsManager.postFinished()
responseObserver.onNext(
diff --git
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionHolderSuite.scala
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionHolderSuite.scala
index f34f46101136..3369e25c78c3 100644
---
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionHolderSuite.scala
+++
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionHolderSuite.scala
@@ -30,7 +30,7 @@ import scala.util.control.NonFatal
import com.google.common.collect.Lists
import org.scalatest.time.SpanSugar._
-import org.apache.spark.SparkEnv
+import org.apache.spark.{SparkEnv, SparkException}
import org.apache.spark.api.python.SimplePythonFunction
import org.apache.spark.connect.proto
import org.apache.spark.sql.IntegratedUDFTestUtils
@@ -69,6 +69,23 @@ class SparkConnectSessionHolderSuite extends
SharedSparkSession {
assert(expectedDf2 == df2)
}
+ test("DataFrame cache: Duplicate IDs are not overwritten") {
+ val sessionHolder = SparkConnectTestUtils.createDummySessionHolder(spark)
+ val df1 = spark.range(1).toDF()
+ val df2 = spark.range(2).toDF()
+ val dfId = "duplicate-id"
+
+ sessionHolder.cacheDataFrameById(dfId, df1)
+ val error = intercept[SparkException] {
+ sessionHolder.cacheDataFrameById(dfId, df2)
+ }
+ checkError(
+ exception = error,
+ condition = "INTERNAL_ERROR",
+ parameters = Map("message" -> s"A dataframe is already associated with
id $dfId"))
+ assert(sessionHolder.getDataFrameOrThrow(dfId) eq df1)
+ }
+
test("DataFrame cache: Should throw when dataframe is not found") {
val sessionHolder = SparkConnectTestUtils.createDummySessionHolder(spark)
import sessionHolder.session.implicits._
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]