Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19184#discussion_r137971517
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java
---
@@ -104,6 +124,10 @@ public void loadNext() throws IOException {
if (taskContext != null) {
taskContext.killTaskIfInterrupted();
}
+ if (this.din == null) {
+ // Good time to init (if all files are opened, we can get Too Many
files exception)
+ initStreams();
+ }
--- End diff --
Can this solve the too many file open issue? When we do merging the
readers, it is possibly that all the readers in priority queue still have
records and are asked for records (so their files open). You still can
encounter too many file open issue.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
