GitHub user Astralidea opened a pull request:
https://github.com/apache/spark/pull/15588
[SPARK-18039][Scheduler] fix bug maxRegisteredWaitingTime does not work
The synchronous mode of driver and executor is through dummy job is only
ensure 1 executor connect to driver.
In my cluster I need to ensure each executor have one receiver.
Thinking about following example:
If spark.cores.max=4 and spark.executor.cores=2 therefore, it will launch 2
executor instance.
The spark first job is running dummy job is always 70 tasks. it takes about
4 seconds.
case 1:
if in this 4 seconds only one executor (E1) connect to driver and
another(E2) not
executor 1 will start 2 receiver and did not working tasks. because it had
used 2 core.
executor 2 will only do tasks not running receiver .because I write code
set 2 receiver stream.
therefore the batch running slowly and it have network data
transmission.(about 3s)
case 2:
in this 4 seconds 2 eecutor connected to driver
executor 1 start 1 receiver used 1 core and could do task
executor 2 start 1 receiver used 1 core and could do task
it is balanced scheduler and running fast (about 0.1s)
So I hope I could set maxRegisteredWaiting to make sure if I have a slowly
executor to startup and have a better receiver policy like every executor have
one receiver.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Astralidea/spark SPARK-18039
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15588.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15588
----
commit 41fc65e827420710784bf8208bf065e3f0fd9d16
Author: xueyan.li <[email protected]>
Date: 2016-10-21T11:21:17Z
Merge pull request #1 from apache/master
update
commit 84d533f6607a29120cd4cb66b081628f4383139f
Author: astralidea <[email protected]>
Date: 2016-10-21T16:23:01Z
add maxRegisteredWaitingTime conf for receiver
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
