[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/20803 Hi @jerryshao @cloud-fan @dongjoon-hyun, I would like to close this PR and open another one https://github.com/apache/spark/pull/20876, would you please move to that? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/20803  --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/20803 I have decoupled the sqlText with sql execution. In current implementation, when user invoke spark.sql(xx), it will create a new SparkListenerSQLTextCaptured event to listenerbus. Then in SQLAppStatusListener, the information will be stored and all the sql sentences will display in AllExecutionPage in order with submission time, instead of in each ExecutionPage. I will upload the commit after testing. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
@cloud-fan, please review.
Now the test result is:
val df = spark.sql("x")
spark.range(10).count() // noting shows in UI
df.collect() // shows sql text "x"
df.count() // noting show in UI
df.show() // shows sql text "x"
df.filter(...).collect() // shows sql text "x"
df.filter(...).count() // noting shows
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20803 Sorry I didn't clarify it clearly enough. I was not suggesting to show sql text for all of these cases, but tried to raise a discussion about when we should show sql text. e.g. for `df.count()` and `df.filter(...).count` seems we should not show. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
@cloud-fan @jerryshao In the last commit, seems I faced a Scala bug. :-(
> [error]
/Users/lajin/git/my/spark/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:63:
in object Dataset, multiple overloaded alternatives of define default
arguments
[error] Error occurred in an application involving default arguments.
[error] private[sql] object Dataset {
[error] ^
https://stackoverflow.com/questions/24991209/scala-2-11-complains-with-multiple-overloaded-alternatives-of-method
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
@cloud-fan, please review.
The test result is:
val df = spark.sql("x")
spark.range(10).count() // noting show in UI
df.collect() // show sql text "x" on the UI
df.count() // show sql text "x" on the UI
df.show() // show sql text "x" on the UI
df.filter(...).collect() // show sql text "x" on the UI
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/20803 Hi @wangyum, the problem about variable substitution now is resolved. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/20803 Thanks a lot, @cloud-fan . The problems like missing the `--hivevar` also exist in current implementation (display sql text in jobs pages). I will try to fix it in my ticket. Probably accurately, this patch not only moves the sql text from jobs page to sql query page, but also resolves the problem that sql text cannot be captured from `bin/spark-submit` or `bin/spark-shell`. You know `bin/spark-sql` (client deploy mode) is mostly used in ad-hoc scenario. Besides that, lots of Spark SQL scenarios like daily job in warehouse, ETL job, and others which need to be submitted to cluster, [SPARK-4871](https://issues.apache.org/jira/browse/SPARK-4871) didn't cover it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20803 So this patch duplicates the SQL text info on the jobs page to the SQL query page. I think it's good and more user-friendly, but we need to make sure the underlying implementation reuse the code, to avoid problems like missing the `--hivevar`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
@wangyum Good point. Unfortunately it is `select ${a}, ${b}`. Let me fix it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/20803
```bash
cat < test.sql
select '\${a}', '\${b}';
EOF
spark-sql --hiveconf a=avalue --hivevar b=bvalue -f test.sql
```
SQL text is `select ${a}, ${b}` or `select avalue, bvalue`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/20803 > What if this SQL statement contains --hiveconf or --hivevar? What's meaning? Can you give an example? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/20803 @cloud-fan one SQL execution only has one sql statement whatever how many jobs it triggered. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/20803 1. Double click this SQL statement can show full SQL statement: https://github.com/apache/spark/pull/6646 2. What if this SQL statement contains `--hiveconf` or `--hivevar`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20803 what if an SQL execution triggers multiple jobs? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20803 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/20803 @gatorsmile @cloud-fan Could you add some comments? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20803 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
