Shivaram Venkataraman created SPARK-19485:
---------------------------------------------
Summary: Launch tasks async i.e. dont wait for the network
Key: SPARK-19485
URL: https://issues.apache.org/jira/browse/SPARK-19485
Project: Spark
Issue Type: Improvement
Components: Scheduler
Affects Versions: 2.1.0
Reporter: Shivaram Venkataraman
Currently the scheduling thread in CoarseGrainedSchedulerBackend is used to
both walk through the list of offers and to serialize, create RPCs and send
messages over the network.
For stages with large number of tasks we can avoid blocking on RPCs /
serialization by moving that to a separate thread in CGSB. As a part of this
JIRA we can first investigate the potential benefits of doing this for
different kinds of jobs (one large stage, many independent small stages etc.)
and then propose a code change.
cc [~kayousterhout]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
