Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
Hi @jerryshao @cloud-fan @dongjoon-hyun, I would like to close this PR and
open another one https://github.com/apache/spark/pull/20876, would you please
move to that?
---
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
---
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
I have decoupled the sqlText with sql execution. In current implementation,
when user invoke spark.sql(xx), it will create a new
SparkListenerSQLTextCaptured event to listenerbus. Then in
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
@cloud-fan, please review.
Now the test result is:
val df = spark.sql("x")
spark.range(10).count() // noting shows in UI
df.collect() // shows sql text "x"
df.count() //
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20803
Sorry I didn't clarify it clearly enough. I was not suggesting to show sql
text for all of these cases, but tried to raise a discussion about when we
should show sql text. e.g. for `df.count()`
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
@cloud-fan @jerryshao In the last commit, seems I faced a Scala bug. :-(
> [error]
/Users/lajin/git/my/spark/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:63:
in object
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
@cloud-fan, please review.
The test result is:
val df = spark.sql("x")
spark.range(10).count() // noting show in UI
df.collect() // show sql text "x" on the UI
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
Hi @wangyum, the problem about variable substitution now is resolved.
---
-
To unsubscribe, e-mail:
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
Thanks a lot, @cloud-fan . The problems like missing the `--hivevar` also
exist in current implementation (display sql text in jobs pages). I will try to
fix it in my ticket. Probably accurately,
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20803
So this patch duplicates the SQL text info on the jobs page to the SQL
query page. I think it's good and more user-friendly, but we need to make sure
the underlying implementation reuse the code,
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
@wangyum Good point. Unfortunately it is `select ${a}, ${b}`. Let me fix it.
---
-
To unsubscribe, e-mail:
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/20803
```bash
cat < test.sql
select '\${a}', '\${b}';
EOF
spark-sql --hiveconf a=avalue --hivevar b=bvalue -f test.sql
```
SQL text is `select ${a}, ${b}` or `select avalue,
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
> What if this SQL statement contains --hiveconf or --hivevar?
What's meaning? Can you give an example?
---
-
To
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
@cloud-fan one SQL execution only has one sql statement whatever how many
jobs it triggered.
---
-
To unsubscribe, e-mail:
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/20803
1. Double click this SQL statement can show full SQL statement:
https://github.com/apache/spark/pull/6646
2. What if this SQL statement contains `--hiveconf` or `--hivevar`?
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20803
what if an SQL execution triggers multiple jobs?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20803
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20803
@gatorsmile @cloud-fan Could you add some comments?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20803
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
19 matches
Mail list logo
