ulysses-you opened a new pull request, #7581:
URL: https://github.com/apache/kyuubi/pull/7581
### Why are the changes needed?
When both
`spark.sql.adaptive.rebalancePartitionsAdvisoryPartitionSizeInBytes`
and `spark.sql.finalStage.adaptive.advisoryPartitionSizeInBytes` are set,
`KyuubiSQLConf.getAdvisoryPartitionSize` previously let the rebalance config
unconditionally take precedence. This could pick a smaller advisory partition
size than the finalStage config even when the finalStage value is larger.
This changes the resolution so the larger of the two values is used when both
are set, which better reflects the intent of both knobs. The doc for the
rebalance config is updated accordingly. The change is applied to the
spark-3.5, 4.0 and 4.1 extension modules.
### How was this patch tested?
Updated `RebalanceBeforeWritingSuite` in all three modules to cover:
- both set, rebalance larger -> rebalance wins
- both set, finalStage larger -> finalStage wins
- both set to the same value
Ran the suite locally:
```
build/mvn test -pl extensions/spark/kyuubi-extension-spark-3-5 -am
-Pspark-3.5 \
-Dtest=none
-DwildcardSuites=org.apache.spark.sql.RebalanceBeforeWritingSuite # 14 passed
build/mvn test -pl extensions/spark/kyuubi-extension-spark-4-0 -am
-Pspark-4.0 -Pscala-2.13 \
-Dtest=none
-DwildcardSuites=org.apache.spark.sql.RebalanceBeforeWritingSuite # 12 passed
build/mvn test -pl extensions/spark/kyuubi-extension-spark-4-1 -am
-Pspark-4.1 -Pscala-2.13 \
-Dtest=none
-DwildcardSuites=org.apache.spark.sql.RebalanceBeforeWritingSuite # 12 passed
```
### Was this patch authored or co-authored using generative AI tooling?
Yes.
Assisted-by: Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
