sryza commented on code in PR #52685:
URL: https://github.com/apache/spark/pull/52685#discussion_r2453468624
##########
sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto:
##########
@@ -280,3 +280,15 @@ message SourceCodeLocation {
message PipelineQueryFunctionExecutionSignal {
repeated string flow_names = 1;
}
+
+// Contextual metadata used during pipeline analysis within Spark Connect
queries.
+message PipelineAnalysisContext {
+ // Unique identifier of the dataflow graph associated with this pipeline.
+ optional string dataflow_graph_id = 1;
+
+ // Defines the path assigned at runtime when the pipeline initializes
+ optional string definition_path = 2;
Review Comment:
We determined that we need a corresponding field on `SourceCodeLocation`,
right? Also, perhaps we should give this consistent naming with the `file_name`
field there? Perhaps `outer_file_name`?
##########
sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto:
##########
@@ -280,3 +280,15 @@ message SourceCodeLocation {
message PipelineQueryFunctionExecutionSignal {
repeated string flow_names = 1;
}
+
+// Contextual metadata used during pipeline analysis within Spark Connect
queries.
+message PipelineAnalysisContext {
+ // Unique identifier of the dataflow graph associated with this pipeline.
+ optional string dataflow_graph_id = 1;
+
+ // Defines the path assigned at runtime when the pipeline initializes
+ optional string definition_path = 2;
+
+ // Reserved field for protocol extensions.
+ google.protobuf.Any extension = 999;
Review Comment:
extension fields should be repeated
##########
sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto:
##########
@@ -280,3 +280,15 @@ message SourceCodeLocation {
message PipelineQueryFunctionExecutionSignal {
repeated string flow_names = 1;
}
+
+// Contextual metadata used during pipeline analysis within Spark Connect
queries.
+message PipelineAnalysisContext {
+ // Unique identifier of the dataflow graph associated with this pipeline.
+ optional string dataflow_graph_id = 1;
+
+ // Defines the path assigned at runtime when the pipeline initializes
Review Comment:
Extreme nitpick: include period at end
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
