GitHub user Astralidea opened a pull request: https://github.com/apache/spark/pull/15588
[SPARK-18039][Scheduler] fix bug maxRegisteredWaitingTime does not work The synchronous mode of driver and executor is through dummy job is only ensure 1 executor connect to driver. In my cluster I need to ensure each executor have one receiver. Thinking about following example: If spark.cores.max=4 and spark.executor.cores=2 therefore, it will launch 2 executor instance. The spark first job is running dummy job is always 70 tasks. it takes about 4 seconds. case 1: if in this 4 seconds only one executor (E1) connect to driver and another(E2) not executor 1 will start 2 receiver and did not working tasks. because it had used 2 core. executor 2 will only do tasks not running receiver .because I write code set 2 receiver stream. therefore the batch running slowly and it have network data transmission.(about 3s) case 2: in this 4 seconds 2 eecutor connected to driver executor 1 start 1 receiver used 1 core and could do task executor 2 start 1 receiver used 1 core and could do task it is balanced scheduler and running fast (about 0.1s) So I hope I could set maxRegisteredWaiting to make sure if I have a slowly executor to startup and have a better receiver policy like every executor have one receiver. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Astralidea/spark SPARK-18039 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15588.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15588 ---- commit 41fc65e827420710784bf8208bf065e3f0fd9d16 Author: xueyan.li <meetup...@gmail.com> Date: 2016-10-21T11:21:17Z Merge pull request #1 from apache/master update commit 84d533f6607a29120cd4cb66b081628f4383139f Author: astralidea <astrali...@163.com> Date: 2016-10-21T16:23:01Z add maxRegisteredWaitingTime conf for receiver ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org