Guoqiang Li created SPARK-17931:
-----------------------------------
Summary: taskScheduler has some unneeded serialization
Key: SPARK-17931
URL: https://issues.apache.org/jira/browse/SPARK-17931
Project: Spark
Issue Type: Improvement
Components: Spark Core
Reporter: Guoqiang Li
When taskScheduler instantiates TaskDescription, it calls
`Task.serializeWithDependencies(task, sched.sc.addedFiles, sched.sc.addedJars,
ser)`. It serializes task and its dependency.
But after SPARK-2521 has been merged into the master, the ResultTask class and
ShuffleMapTask class no longer contain rdd and closure objects.
TaskDescription class can be changed as below:
{noformat}
class TaskDescription[T](
val taskId: Long,
val attemptNumber: Int,
val executorId: String,
val name: String,
val index: Int,
val task: Task[T]) extends Serializable
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
