Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22233#discussion_r212804252
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -1131,7 +1135,7 @@ abstract class DDLSuite extends QueryTest with
SQLTestUtils {
}
test("alter table: recover partition (parallel)") {
- withSQLConf("spark.rdd.parallelListingThreshold" -> "1") {
+ withSQLConf("spark.rdd.parallelListingThreshold" -> "0") {
--- End diff --
On the recursive calls this condition
https://github.com/apache/spark/blob/131ca146ed390cd0109cd6e8c95b61e418507080/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala#L685-L686
is false because `statuses.length` is `1` and `threshold` is `1`. So, it leads
to sequential listening of files. I just enforce parallel scanning even for 1
sub-folder.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
