This is an automated email from the ASF dual-hosted git repository.
bowenliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new c8a138f98 [KYUUBI #4933] [DOCS] [MINOR] Mark
`spark.sql.optimizer.insertRepartitionNum` config for Spark 3.1 only
c8a138f98 is described below
commit c8a138f9861986b325fcf00d4795dfe75d19c9f7
Author: liangbowen <[email protected]>
AuthorDate: Fri Jun 9 08:30:23 2023 +0800
[KYUUBI #4933] [DOCS] [MINOR] Mark
`spark.sql.optimizer.insertRepartitionNum` config for Spark 3.1 only
### _Why are the changes needed?_
- Update doc to mark the spark plugin's config
`spark.sql.optimizer.insertRepartitionNum` used for Spark 3.1 only
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run
test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests)
locally before make a pull request
Closes #4933 from bowenliang123/insert-num.
Closes #4933
5ed6e2867 [liangbowen] comment and style
280a6af03 [liangbowen] spark.sql.optimizer.insertRepartitionNum only
available for Spark 3.1.x
7f01cf3b6 [liangbowen] spark.sql.optimizer.insertRepartitionNum only
available for Spark 3.1.x
Authored-by: liangbowen <[email protected]>
Signed-off-by: liangbowen <[email protected]>
---
docs/extensions/engines/spark/rules.md | 2 +-
.../src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/docs/extensions/engines/spark/rules.md
b/docs/extensions/engines/spark/rules.md
index 50bf087a8..4614f5244 100644
--- a/docs/extensions/engines/spark/rules.md
+++ b/docs/extensions/engines/spark/rules.md
@@ -66,7 +66,7 @@ Kyuubi provides some configs to make these feature easy to
use.
| Name |
Default Value |
Description
| Since |
|---------------------------------------------------------------------|----------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------|
| spark.sql.optimizer.insertRepartitionBeforeWrite.enabled | true
| Add repartition node at the top of query
plan. An approach of merging small files.
| 1.2.0 |
-| spark.sql.optimizer.insertRepartitionNum | none
| The partition number if
`spark.sql.optimizer.insertRepartitionBeforeWrite.enabled` is enabled. If AQE
is disabled, the default value is `spark.sql.shuffle.partitions`. If AQE is
enabled, the default value is none that means depend on AQE.
| 1.2.0 |
+| spark.sql.optimizer.insertRepartitionNum | none
| The partition number if
`spark.sql.optimizer.insertRepartitionBeforeWrite.enabled` is enabled. If AQE
is disabled, the default value is `spark.sql.shuffle.partitions`. If AQE is
enabled, the default value is none that means depend on AQE. This config is
used for Spark 3.1 only.
| 1.2.0 |
| spark.sql.optimizer.dynamicPartitionInsertionRepartitionNum | 100
| The partition number of each dynamic
partition if `spark.sql.optimizer.insertRepartitionBeforeWrite.enabled` is
enabled. We will repartition by dynamic partition columns to reduce the small
file but that can cause data skew. This config is to extend the partition of
dynamic partition column to avoid skew but may generate some small files. |
1.2.0 |
| spark.sql.optimizer.forceShuffleBeforeJoin.enabled | false
| Ensure shuffle node exists before shuffled
join (shj and smj) to make AQE `OptimizeSkewedJoin` works (complex scenario
join, multi table join).
| 1.2.0 |
| spark.sql.optimizer.finalStageConfigIsolation.enabled | false
| If true, the final stage support use
different config with previous stage. The prefix of final stage config key
should be `spark.sql.finalStage.`. For example, the raw spark config:
`spark.sql.adaptive.advisoryPartitionSizeInBytes`, then the final stage config
should be: `spark.sql.finalStage.adaptive.advisoryPartitionSizeInBytes`.
| 1.2.0 |
diff --git
a/extensions/spark/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala
b/extensions/spark/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala
index cb2f1130e..fa118a3e2 100644
---
a/extensions/spark/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala
+++
b/extensions/spark/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala
@@ -34,7 +34,8 @@ object KyuubiSQLConf {
buildConf("spark.sql.optimizer.insertRepartitionNum")
.doc(s"The partition number if ${INSERT_REPARTITION_BEFORE_WRITE.key} is
enabled. " +
s"If AQE is disabled, the default value is
${SQLConf.SHUFFLE_PARTITIONS.key}. " +
- "If AQE is enabled, the default value is none that means depend on
AQE.")
+ "If AQE is enabled, the default value is none that means depend on
AQE. " +
+ "This config is used for Spark 3.1 only.")
.version("1.2.0")
.intConf
.createOptional
