Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19184#discussion_r137981905
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java
---
@@ -104,6 +124,10 @@ public void loadNext() throws IOException {
if (taskContext != null) {
taskContext.killTaskIfInterrupted();
}
+ if (this.din == null) {
+ // Good time to init (if all files are opened, we can get Too Many
files exception)
--- End diff --
This comment looks confusing. Maybe `It is the time to initialize and hold
the input stream of the spill file for loading records. Keeps the input stream
open too early will very possibly encounter too many file open issue.`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
