LuciferYang opened a new pull request, #56901:
URL: https://github.com/apache/spark/pull/56901
### What changes were proposed in this pull request?
`TestUtils.recursiveList` duplicated the recursive directory walk already
provided by `Utils.recursiveList` (i.e. `SparkFileUtils.recursiveList`). This
PR removes the duplicated copy and delegates to `Utils.recursiveList`:
```scala
// before
def recursiveList(f: File): Array[File] = {
require(f.isDirectory)
val current = f.listFiles
current ++ current.filter(_.isDirectory).flatMap(recursiveList)
}
// after
def recursiveList(f: File): Array[File] = Utils.recursiveList(f)
```
### Why are the changes needed?
The duplicated implementation carried the same two issues that were just
fixed in `SparkFileUtils.recursiveList` under
[SPARK-57530](https://issues.apache.org/jira/browse/SPARK-57530):
1. It called `File.listFiles` without a null check, so an IO error (or the
directory being removed mid-walk) would throw an NPE.
2. The `current ++ ... .flatMap(...)` form had no linear-time guarantee.
By delegating to `Utils.recursiveList`, `TestUtils.recursiveList`
automatically picks up the null-safety (a directory that cannot be listed is
skipped with a warning instead of throwing) and the O(n) traversal from
SPARK-57530, and the duplicated logic is removed.
This is a follow-up to SPARK-57530, which is already merged to master.
### Does this PR introduce _any_ user-facing change?
No. `TestUtils` is a test-only `private[spark]` helper; this is an internal
refactor with no behavior change for any successful directory walk.
### How was this patch tested?
Existing tests that use `TestUtils.recursiveList` continue to exercise it.
The behavioral contract for a readable directory tree is unchanged (same set of
files returned); the only difference is that an unreadable directory is now
skipped with a warning rather than throwing, matching `Utils.recursiveList`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
