Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/1485#issuecomment-49526386
@fireflyc Spark should not be scheduling more than N concurrent tasks on an
Executor. It appears that the tasks may be returning "success" but then don't
actually return the thread to the thread pool.
This is itself a bug -- could you run "jstack" on your Executor process to
see where the threads are stuck?
Perhaps new tasks are just starting before the old threads finish cleaning
up, and thus this solution is the right one, but I'd like to find out exactly
why.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
