sryza commented on code in PR #52685:
URL: https://github.com/apache/spark/pull/52685#discussion_r2452446304
##########
sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto:
##########
@@ -280,3 +280,9 @@ message SourceCodeLocation {
message PipelineQueryFunctionExecutionSignal {
repeated string flow_names = 1;
}
+
+// Provides context for pipeline analysis during Spark Connect query execution.
+message PipelineAnalysisContext {
+ // Defines the path assigned at runtime when the pipeline initializes
+ optional string definition_path = 1;
Review Comment:
Is my understanding correct that the purpose of this is that, if it's
included with a `spark.sql("USE DATABASE")` command, that database will apply
only to datasets defined within the same definitions file? I don't see an
equivalent `definition_path` to link to elsewhere (e.g. on
`SourceCodeLocation`). Should there be?
cc @SCHJonathan
##########
sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto:
##########
@@ -280,3 +280,9 @@ message SourceCodeLocation {
message PipelineQueryFunctionExecutionSignal {
repeated string flow_names = 1;
}
+
+// Provides context for pipeline analysis during Spark Connect query execution.
+message PipelineAnalysisContext {
+ // Defines the path assigned at runtime when the pipeline initializes
+ optional string definition_path = 1;
+}
Review Comment:
Seems useful to include an extension field just to be safe.
##########
sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto:
##########
@@ -280,3 +280,9 @@ message SourceCodeLocation {
message PipelineQueryFunctionExecutionSignal {
repeated string flow_names = 1;
}
+
+// Provides context for pipeline analysis during Spark Connect query execution.
+message PipelineAnalysisContext {
Review Comment:
This should include a dataflow graph ID
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
