SCHJonathan commented on code in PR #52154:
URL: https://github.com/apache/spark/pull/52154#discussion_r2388960688
##########
sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto:
##########
@@ -97,11 +99,21 @@ message PipelineCommand {
// Name of the dataset this flow writes to. Can be partially or fully
qualified.
optional string target_dataset_name = 3;
- // An unresolved relation that defines the dataset's flow.
+ // An unresolved relation that defines the dataset's flow. Empty if the
query function
+ // that defines the flow cannot be analyzed at the time of flow definition.
Review Comment:
Instead the server would bookkeeping the unresolved dependency from the
previously triggered `AnalyzePlan` RPC. SC client can just sent `None` as the
`relation` when defining the flow, and the server would use the tracked
unresolved dependencies to register the Flow in the DataflowGraph.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
