Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/4051#issuecomment-70606552
when we set initial number to min, there is a delay to request more
executors. because in ExecutorAllocationManager's addExecutors
numExecutorsToAdd starts from 1, and next minute is double. so when we request
more executors, there is very slow and it spends several minutes.so maybe we
need to change this scaling mechanism?
another situation is:
when stage has skew task and individual tasks need to run for a long time,
number of executor will be to a small value, and then next stage need many
executors to run tasks, so ExecutorAllocationManager need to request many
executors. at this time it also need to spend a few minutes for reaching number
of pending tasks. @andrewor14
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
