Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19184#discussion_r137988200
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java
---
@@ -104,6 +124,10 @@ public void loadNext() throws IOException {
if (taskContext != null) {
taskContext.killTaskIfInterrupted();
}
+ if (this.din == null) {
+ // Good time to init (if all files are opened, we can get Too Many
files exception)
+ initStreams();
+ }
--- End diff --
The valid fix should be to import a new config to control the concurrent
number of opened spill files, it also means you should use some data structure
to keep and track the request of open spill files.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
