[GitHub] [spark] zhengruifeng commented on a diff in pull request #38915: [SPARK-41382][CONNECT][PYTHON] Implement `product` function
zhengruifeng commented on code in PR #38915:
URL: https://github.com/apache/spark/pull/38915#discussion_r1040759564
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##
@@ -552,6 +554,27 @@ class SparkConnectPlanner(session: SparkSession) {
}
}
+ /**
+ * For some reason, not all functions are registered in 'FunctionRegistry'.
For a unregistered
+ * function, we can still wrap it under the proto 'UnresolvedFunction', and
then resolve it in
+ * this method.
+ */
+ private def transformUnregisteredFunction(
+ fun: proto.Expression.UnresolvedFunction): Option[Expression] = {
+fun.getFunctionName match {
+ case "product" =>
+if (fun.getArgumentsCount != 1) {
+ throw InvalidPlanInput("Product requires single child expression")
Review Comment:
I'm not sure about the exception type here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #38915: [SPARK-41382][CONNECT][PYTHON] Implement `product` function
zhengruifeng commented on code in PR #38915:
URL: https://github.com/apache/spark/pull/38915#discussion_r1040331473
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##
@@ -539,6 +539,15 @@ class SparkConnectPlanner(session: SparkSession) {
* @return
*/
private def transformScalarFunction(fun:
proto.Expression.UnresolvedFunction): Expression = {
+if (fun.getFunctionName == "product") {
Review Comment:
good point, I think we will have more similar cases, let me add a new
function for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #38915: [SPARK-41382][CONNECT][PYTHON] Implement `product` function
zhengruifeng commented on code in PR #38915:
URL: https://github.com/apache/spark/pull/38915#discussion_r1040330791
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##
@@ -539,6 +539,15 @@ class SparkConnectPlanner(session: SparkSession) {
* @return
*/
private def transformScalarFunction(fun:
proto.Expression.UnresolvedFunction): Expression = {
+if (fun.getFunctionName == "product") {
Review Comment:
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #38915: [SPARK-41382][CONNECT][PYTHON] Implement `product` function
zhengruifeng commented on code in PR #38915:
URL: https://github.com/apache/spark/pull/38915#discussion_r1039400753
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##
@@ -539,6 +539,15 @@ class SparkConnectPlanner(session: SparkSession) {
* @return
*/
private def transformScalarFunction(fun:
proto.Expression.UnresolvedFunction): Expression = {
+if (fun.getFunctionName == "product") {
Review Comment:
Had a discussion with @cloud-fan , we don't want to expose all expressions
in SQL syntax, and `product` is such an example.
I think we will also deal with dedicated expressions for Pandas-on-Spark in
this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
