This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 1b2ccc7c5cf2 [SPARK-57695][CORE] Make TestUtils.recursiveList
null-safe by reusing Utils.recursiveList
1b2ccc7c5cf2 is described below
commit 1b2ccc7c5cf27682c4d58c9fd559c4e5819eb0d1
Author: YangJie <[email protected]>
AuthorDate: Wed Jul 1 00:48:07 2026 +0200
[SPARK-57695][CORE] Make TestUtils.recursiveList null-safe by reusing
Utils.recursiveList
### What changes were proposed in this pull request?
`TestUtils.recursiveList` duplicated the recursive directory walk already
provided by `Utils.recursiveList` (i.e. `SparkFileUtils.recursiveList`). This
PR removes the duplicated copy and delegates to `Utils.recursiveList`:
```scala
// before
def recursiveList(f: File): Array[File] = {
require(f.isDirectory)
val current = f.listFiles
current ++ current.filter(_.isDirectory).flatMap(recursiveList)
}
// after
def recursiveList(f: File): Array[File] = Utils.recursiveList(f)
```
### Why are the changes needed?
The duplicated implementation carried the same two issues that were just
fixed in `SparkFileUtils.recursiveList` under
[SPARK-57530](https://issues.apache.org/jira/browse/SPARK-57530):
1. It called `File.listFiles` without a null check, so an IO error (or the
directory being removed mid-walk) would throw an NPE.
2. The `current ++ ... .flatMap(...)` form had no linear-time guarantee.
By delegating to `Utils.recursiveList`, `TestUtils.recursiveList`
automatically picks up the null-safety (a directory that cannot be listed is
skipped with a warning instead of throwing) and the O(n) traversal from
SPARK-57530, and the duplicated logic is removed.
This is a follow-up to SPARK-57530, which is already merged to master.
### Does this PR introduce _any_ user-facing change?
No. `TestUtils` is a test-only `private[spark]` helper; this is an internal
refactor with no behavior change for any successful directory walk.
### How was this patch tested?
Existing tests that use `TestUtils.recursiveList` continue to exercise it.
The behavioral contract for a readable directory tree is unchanged (same set of
files returned); the only difference is that an unreadable directory is now
skipped with a warning rather than throwing, matching `Utils.recursiveList`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
Closes #56901 from LuciferYang/SPARK-57695-testutils-recursivelist.
Authored-by: YangJie <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit a4ce62b8bff4c8963d473a6228a3f01247f8e7e7)
Signed-off-by: Max Gekk <[email protected]>
---
core/src/main/scala/org/apache/spark/TestUtils.scala | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/TestUtils.scala
b/core/src/main/scala/org/apache/spark/TestUtils.scala
index 1bfc6c3d0fad..46f20825e55b 100644
--- a/core/src/main/scala/org/apache/spark/TestUtils.scala
+++ b/core/src/main/scala/org/apache/spark/TestUtils.scala
@@ -387,11 +387,7 @@ private[spark] object TestUtils extends SparkTestUtils {
/**
* Lists files recursively.
*/
- def recursiveList(f: File): Array[File] = {
- require(f.isDirectory)
- val current = f.listFiles
- current ++ current.filter(_.isDirectory).flatMap(recursiveList)
- }
+ def recursiveList(f: File): Array[File] = Utils.recursiveList(f)
/**
* Returns the list of files at 'path' recursively. This skips files that
are ignored normally
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
