jingz-db commented on code in PR #49560:
URL: https://github.com/apache/spark/pull/49560#discussion_r1962978619
##########
sql/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -1031,6 +1031,26 @@ message GroupMap {
// (Optional) The schema for the grouped state.
optional DataType state_schema = 10;
+
+ // Below fields are used by TransformWithState and TransformWithStateInPandas
+ // (Optional) TransformWithState related parameters.
+ optional TransformWithStateInfo transform_with_state_info = 11;
+}
+
+// Additional input parameters used for TransformWithState operator.
+message TransformWithStateInfo {
+ // (Required) Time mode string for transformWithState.
+ string time_mode = 1;
+
+ // (Required) Output mode string for transformWithState.
+ string output_mode = 2;
Review Comment:
> What is different between this output mode and that of the GroupMap
message?
TransformWithState has its own dedicated Scala class of
[TimeMode](https://github.com/apache/spark/blob/master/sql/api/src/main/java/org/apache/spark/sql/streaming/TimeMode.java#L26)
which is different from the time mode class used by GroupMap timemode so we
will need a separate field wrapped under `TransformWithStateInfo`. But the
output mode is the same class. I will remove the output mode entry here.
> Should this be an enum?
For Python, time mode and output mode are user input strings and they will
only be restored on JVM into Scala class
[here](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/classic/RelationalGroupedDataset.scala#L491-L492).
So I think it is probably better to reuse the existing methods for restoring
user input string into Scala class and keep the string format here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
