This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 3638992 [SPARK-31810][TEST] Fix AlterTableRecoverPartitions test
using incorrect api to modify RDD_PARALLEL_LISTING_THRESHOLD
3638992 is described below
commit 363899267d6f8afd5314f694a71fb034a7a2d409
Author: Prakhar Jain <[email protected]>
AuthorDate: Tue May 26 14:13:02 2020 +0900
[SPARK-31810][TEST] Fix AlterTableRecoverPartitions test using incorrect
api to modify RDD_PARALLEL_LISTING_THRESHOLD
### What changes were proposed in this pull request?
Use the correct API in AlterTableRecoverPartition tests to modify the
`RDD_PARALLEL_LISTING_THRESHOLD` conf.
### Why are the changes needed?
The existing AlterTableRecoverPartitions test modify the
RDD_PARALLEL_LISTING_THRESHOLD as a SQLConf using the withSQLConf API. But
since, this is not a SQLConf, it is not overridden and so the test doesn't end
up testing the required behaviour.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
This is UT Fix. UTs are still passing after the fix.
Closes #28634 from prakharjain09/SPARK-31810-fix-recover-partitions.
Authored-by: Prakhar Jain <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
(cherry picked from commit 452594f5a43bc5d98fb42bf0927638d139f6d17c)
Signed-off-by: HyukjinKwon <[email protected]>
---
.../org/apache/spark/sql/execution/command/DDLSuite.scala | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
index 10ad8ac..e4709e4 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
@@ -1203,14 +1203,24 @@ abstract class DDLSuite extends QueryTest with
SQLTestUtils {
}
test("alter table: recover partitions (sequential)") {
- withSQLConf(RDD_PARALLEL_LISTING_THRESHOLD.key -> "10") {
+ val oldRddParallelListingThreshold = spark.sparkContext.conf.get(
+ RDD_PARALLEL_LISTING_THRESHOLD)
+ try {
+ spark.sparkContext.conf.set(RDD_PARALLEL_LISTING_THRESHOLD.key, "10")
testRecoverPartitions()
+ } finally {
+ spark.sparkContext.conf.set(RDD_PARALLEL_LISTING_THRESHOLD,
oldRddParallelListingThreshold)
}
}
test("alter table: recover partition (parallel)") {
- withSQLConf(RDD_PARALLEL_LISTING_THRESHOLD.key -> "0") {
+ val oldRddParallelListingThreshold = spark.sparkContext.conf.get(
+ RDD_PARALLEL_LISTING_THRESHOLD)
+ try {
+ spark.sparkContext.conf.set(RDD_PARALLEL_LISTING_THRESHOLD.key, "0")
testRecoverPartitions()
+ } finally {
+ spark.sparkContext.conf.set(RDD_PARALLEL_LISTING_THRESHOLD,
oldRddParallelListingThreshold)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
