[
https://issues.apache.org/jira/browse/SPARK-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangTaoTheTonic updated SPARK-3411:
-----------------------------------
Description:
If the waiting driver array is too big, the drivers in it will be dispatched to
the first worker we get(if it has enough resources), with or without the
Randomization.
We should do randomization every time we dispatch a driver, in order to better
balance drivers.
Update(2014/9/6):Doing shuffle is much slower, so we use round robin to avoid
it.
was:
If the waiting driver array is too big, the drivers in it will be dispatched to
the first worker we get(if it has enough resources), with or without the
Randomization.
We should do randomization every time we dispatch a driver, in order to better
balance drivers.
> Improve load-balancing of concurrently-submitted drivers across workers
> -----------------------------------------------------------------------
>
> Key: SPARK-3411
> URL: https://issues.apache.org/jira/browse/SPARK-3411
> Project: Spark
> Issue Type: Improvement
> Components: Deploy
> Reporter: WangTaoTheTonic
> Priority: Minor
>
> If the waiting driver array is too big, the drivers in it will be dispatched
> to the first worker we get(if it has enough resources), with or without the
> Randomization.
> We should do randomization every time we dispatch a driver, in order to
> better balance drivers.
> Update(2014/9/6):Doing shuffle is much slower, so we use round robin to avoid
> it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
