This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 033784c [KYUUBI #1018] Set
spark.sql.execution.topKSortFallbackThreshold to 10000
033784c is described below
commit 033784c691880083d798c4be1653cb40fc28da87
Author: Brian Yue <[email protected]>
AuthorDate: Thu Sep 9 13:36:34 2021 +0800
[KYUUBI #1018] Set spark.sql.execution.topKSortFallbackThreshold to 10000
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in
https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your
PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
To avoid performance issues in the topK scenario and close #1018 .
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run
test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests)
locally before make a pull request
Closes #1054 from byyue/feature-1018.
Closes #1018
21de1fcb [Brian Yue] Set spark.sql.execution.topKSortFallbackThreshold to
10000
Authored-by: Brian Yue <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
---
.../src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala | 1 +
1 file changed, 1 insertion(+)
diff --git
a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
index 6968e07..9c2832c 100644
---
a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
+++
b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
@@ -96,6 +96,7 @@ object SparkSQLEngine extends Logging {
def createSpark(): SparkSession = {
val sparkConf = new SparkConf()
+ sparkConf.setIfMissing("spark.sql.execution.topKSortFallbackThreshold",
"10000")
sparkConf.setIfMissing("spark.sql.legacy.castComplexTypesToString.enabled",
"true")
sparkConf.setIfMissing("spark.master", "local")
sparkConf.setIfMissing("spark.ui.port", "0")
