Github user tsdeng commented on the pull request:
https://github.com/apache/spark/pull/3302#issuecomment-63347621
I guess, the original intention of this variable is to make sure there is
at least 1K records before each spilling. We saw a too many files open
exception due to this variable is not being updated correctly. Of course , this
is not the root cause of the issue, I currently have another working branch
trying to tackle the deeper cause of this, as mentioned in
https://issues.apache.org/jira/browse/SPARK-4452. But at the same time, I'm
sending this PR to fix the updating the elementsRead to alleviate the problem
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
