rangadi commented on code in PR #41580:
URL: https://github.com/apache/spark/pull/41580#discussion_r1228842072
##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -788,6 +790,13 @@ class SparkConnectPlanner(val session: SparkSession)
extends Logging {
.logicalPlan
}
+ private def transformCachedRemoteRelation(session: SparkSession, rel:
proto.CachedRemoteRelation)
+ : LogicalPlan = {
+ SparkConnectService.cachedDataFrameManager
+ .get(session, rel.getRelationId)
+ .logicalPlan
Review Comment:
[Continuation of the comment
[here](https://github.com/apache/spark/pull/41146/files#r1228238923)]
@grundprinzip SessionHolder is not currently passed down and I wasn't sure
if we should add more state to it yet.
For now, keeping a separate cache. I will add a comment.
##########
connector/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -395,6 +396,12 @@ message CachedLocalRelation {
string hash = 3;
}
+// Represents a remote relation that has been cached on server.
+message CachedRemoteRelation {
+ // (Required) ID of the remote related (assigned by the service).
+ string relation_id = 3;
Review Comment:
[continuation of the comment
[here](https://github.com/apache/spark/pull/41146/files#r1228237503)]
@grundprinzip removed _user_id_ & 'session_id'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
