Guoqiang Li created SPARK-17931:
-----------------------------------

             Summary: taskScheduler has some unneeded serialization
                 Key: SPARK-17931
                 URL: https://issues.apache.org/jira/browse/SPARK-17931
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
            Reporter: Guoqiang Li


When taskScheduler instantiates TaskDescription, it calls 
`Task.serializeWithDependencies(task, sched.sc.addedFiles, sched.sc.addedJars, 
ser)`.  It serializes task and its dependency. 

But after SPARK-2521 has been merged into the master, the ResultTask class and 
ShuffleMapTask  class no longer contain rdd and closure objects. 
TaskDescription class can be changed as below:

{noformat}
class TaskDescription[T](
val taskId: Long,
val attemptNumber: Int,
val executorId: String,
val name: String,
val index: Int, 
val task: Task[T]) extends Serializable
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to