[GitHub] spark pull request #23046: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.en...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/23046 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23046: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.en...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/23046#discussion_r234088968
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -280,7 +280,7 @@ object ShuffleExchangeExec {
}
// The comparator for comparing row hashcode, which should
always be Integer.
val prefixComparator = PrefixComparators.LONG
- val canUseRadixSort =
SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)
+ val canUseRadixSort = SQLConf.get.enableRadixSort
--- End diff --
It's a small bug fix, so no need to backport to all the branches. I think
2.4 is good enough
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23046: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.en...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/23046#discussion_r234073703
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -280,7 +280,7 @@ object ShuffleExchangeExec {
}
// The comparator for comparing row hashcode, which should
always be Integer.
val prefixComparator = PrefixComparators.LONG
- val canUseRadixSort =
SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)
+ val canUseRadixSort = SQLConf.get.enableRadixSort
--- End diff --
Yes .. I don't mind it but was just thinking that we don't necessarily
backport to all the branches if there's any concern. I will leave it to you
guys as well.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23046: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.en...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23046#discussion_r234073072
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -280,7 +280,7 @@ object ShuffleExchangeExec {
}
// The comparator for comparing row hashcode, which should
always be Integer.
val prefixComparator = PrefixComparators.LONG
- val canUseRadixSort =
SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)
+ val canUseRadixSort = SQLConf.get.enableRadixSort
--- End diff --
Ah, yes, to be exact, if users specified the config to `SparkConf` before
Spark ran, it could be read.
I'd leave which branch we should backport to to you and other reviewers.
@jiangxb1987 @cloud-fan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23046: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.en...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/23046#discussion_r234063905
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -280,7 +280,7 @@ object ShuffleExchangeExec {
}
// The comparator for comparing row hashcode, which should
always be Integer.
val prefixComparator = PrefixComparators.LONG
- val canUseRadixSort =
SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)
+ val canUseRadixSort = SQLConf.get.enableRadixSort
--- End diff --
@ueshin, BTW, for clarification, it does read the configuration but does
not respect when the configuration is given to the session, right? I think we
don't need to backport this through all other branches.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23046: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.en...
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/23046 [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.enableRadixSort` instead of `SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)`. ## What changes were proposed in this pull request? This is a follow-up of #20393. We should read the conf `"spark.sql.sort.enableRadixSort"` from `SQLConf` instead of `SparkConf`, i.e., use `SQLConf.get.enableRadixSort` instead of `SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)`, otherwise the config is never read. ## How was this patch tested? Existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ueshin/apache-spark issues/SPARK-23207/conf Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23046.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23046 commit d8db5f3759b998e3ef5720f56c1a048c1494a1c1 Author: Takuya UESHIN Date: 2018-11-15T11:30:13Z Use `SQLConf.get.enableRadixSort` instead of `SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
