[GitHub] [spark] beliefer commented on a diff in pull request #40467: [SPARK-42584][CONNECT] Improve output of `Column.explain`
beliefer commented on code in PR #40467: URL: https://github.com/apache/spark/pull/40467#discussion_r1144208941 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Column.scala: ## @@ -1211,13 +1211,11 @@ class Column private[sql] (private[sql] val expr: proto.Expression) extends Logg * @group df_ops * @since 3.4.0 */ - def explain(extended: Boolean): Unit = { + def explain(extended: Boolean)(implicit spark: SparkSession): Unit = { Review Comment: I understand what you mean roughly. We don't need follow SQL, but the output reflects the hierarchy of the tree. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #40467: [SPARK-42584][CONNECT] Improve output of `Column.explain`
beliefer commented on code in PR #40467: URL: https://github.com/apache/spark/pull/40467#discussion_r1144182236 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Column.scala: ## @@ -1211,13 +1211,11 @@ class Column private[sql] (private[sql] val expr: proto.Expression) extends Logg * @group df_ops * @since 3.4.0 */ - def explain(extended: Boolean): Unit = { + def explain(extended: Boolean)(implicit spark: SparkSession): Unit = { Review Comment: @hvanhovell Do you mean to construct local catalyst expression tree at connect client side ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #40467: [SPARK-42584][CONNECT] Improve output of `Column.explain`
beliefer commented on code in PR #40467: URL: https://github.com/apache/spark/pull/40467#discussion_r1144175741 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Column.scala: ## @@ -1211,13 +1211,11 @@ class Column private[sql] (private[sql] val expr: proto.Expression) extends Logg * @group df_ops * @since 3.4.0 */ - def explain(extended: Boolean): Unit = { + def explain(extended: Boolean)(implicit spark: SparkSession): Unit = { Review Comment: @hvanhovell If so, the output of `Column.explain` can't keep consistent with Spark SQL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #40467: [SPARK-42584][CONNECT] Improve output of `Column.explain`
beliefer commented on code in PR #40467: URL: https://github.com/apache/spark/pull/40467#discussion_r1142827066 ## connector/connect/common/src/main/protobuf/spark/connect/base.proto: ## @@ -163,6 +164,14 @@ message AnalyzePlanRequest { // (Required) The logical plan to get a hashCode. Plan plan = 1; } + + // Explains the expression based on extended is true or not. + message ExplainExpression { +// (Required) The expression to be analyzed. +Expression expr = 1; + +bool extended = 2; Review Comment: Thank you! I forgot it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #40467: [SPARK-42584][CONNECT] Improve output of `Column.explain`
beliefer commented on code in PR #40467: URL: https://github.com/apache/spark/pull/40467#discussion_r1142826734 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Column.scala: ## @@ -1211,13 +1211,11 @@ class Column private[sql] (private[sql] val expr: proto.Expression) extends Logg * @group df_ops * @since 3.4.0 */ - def explain(extended: Boolean): Unit = { + def explain(extended: Boolean)(implicit spark: SparkSession): Unit = { Review Comment: Good question. It seems we must send the msg to server side. So we require `SparkSession`. But it's a question how to get `SparkSession` in connect's `Column` ? or we define the SparkSession in `Column` directly? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org