amaliujia commented on code in PR #40586:
URL: https://github.com/apache/spark/pull/40586#discussion_r1152388067
##########
connector/connect/common/src/main/protobuf/spark/connect/commands.proto:
##########
@@ -177,3 +179,97 @@ message WriteOperationV2 {
// (Optional) A condition for overwrite saving mode
Expression overwrite_condition = 8;
}
+
+message WriteStreamOperation {
+
+ // (Required) The output of the `input` streaming relation will be written.
+ Relation input = 1;
+
+ string format = 2;
+ map<string, string> options = 3;
+ repeated string partitioning_column_names = 4;
+
+ oneof trigger {
+ string processing_time_interval = 5;
+ bool available_now = 6;
+ bool one_time = 7;
+ string continuous_checkpoint_interval = 8;
+ }
+
+ string output_mode = 9;
+ string query_name = 10;
+
+ oneof sink_destination {
+ string path = 11;
+ string table_name = 12;
+ }
+}
+
+message StreamingQueryStartResult {
+
+ string name = 1;
+
+ // (Required)
+ string id = 2;
+
+ // (Required)
+ string run_id = 3;
+
+ // TODO: How do we indicate errors?
+ // TODO: Consider adding StreamingQueryStatusResult here.
+}
+
+message StreamingQueryCommand {
+
+ // (Required)
+ string id = 1;
+
+ oneof command_type {
+ StatusCommand status = 2;
+ bool stop = 3;
+ bool process_all_available = 4;
+ ExplainCommand explain = 5;
+
+ // TODO(SPARK-42960) Add more commands: await_termination(), exception()
etc.
+ }
+
+ message StatusCommand {
+ int32 recent_progress_limit = 1;
+ }
+
+ message ExplainCommand {
Review Comment:
Can we re-use the existing explain command?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
