grundprinzip commented on code in PR #39585:
URL: https://github.com/apache/spark/pull/39585#discussion_r1072064530
##########
connector/connect/common/src/main/protobuf/spark/connect/expressions.proto:
##########
@@ -217,6 +219,19 @@ message Expression {
bool is_user_defined_function = 4;
}
+ message PythonFunction {
+ bytes command = 1;
+ }
+
+ message PythonUDF {
+ string function_name = 1;
+ PythonFunction function = 2;
+ string output_type = 3;
+ repeated Expression arguments = 4;
+ int32 eval_type = 5;
+ bool deterministic = 6;
+ }
+
Review Comment:
My suggestion would be to change the message types to something more like
the below. This will allow us to add a Scala UDF to the `oneof` and simplify
the PythonUDF message. In there we inline the bytes directly. I don't think
there is a need to keep it in a separate message.
```suggestion
message ScalarInlineUserDefinedFunction {
string function_name = 1;
bool deterministic = 2;
repeated Expression arguments = 3;
oneof function {
PythonUDF python_udf = 5;
}
}
message PythonUDF {
string output_type = 1;
int32 eval_type = 2;
bytes command = 3;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
