Shivaram Venkataraman created SPARK-19486:
---------------------------------------------
Summary: Investigate using multiple threads for task serialization
Key: SPARK-19486
URL: https://issues.apache.org/jira/browse/SPARK-19486
Project: Spark
Issue Type: Improvement
Components: Scheduler
Affects Versions: 2.1.0
Reporter: Shivaram Venkataraman
This is related to SPARK-18890, where all the serialization logic is moved into
the Scheduler backend thread. As a follow on to this we can investigate using a
thread pool to serialize a number of tasks together instead of using a single
thread to serialize all of them.
Note that this may not yield sufficient benefits unless the driver has enough
cores and we don't run into contention across threads. We can first investigate
potential benefits and if there are sufficient benefits we can create a PR for
this.
cc [~kayousterhout]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
