Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r98540621
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -51,8 +54,39 @@ private[spark] class TaskDescription(
val index: Int, // Index within this task's TaskSet
val addedFiles: Map[String, Long],
val addedJars: Map[String, Long],
- val properties: Properties,
- val serializedTask: ByteBuffer) {
+ val properties: Properties) extends Logging {
+
+ def this(
+ taskId: Long,
+ attemptNumber: Int,
+ executorId: String,
+ name: String,
+ index: Int, // Index within this task's TaskSet
+ addedFiles: Map[String, Long],
+ addedJars: Map[String, Long],
+ properties: Properties,
+ task: Task[_]) {
+ this(taskId, attemptNumber, executorId, name, index,
+ addedFiles, addedJars, properties)
+ task_ = task
--- End diff --
I find this second constructor and the use of task_ to be confusing,
because it's not easy for a reader to reason about which variable are defined
(and used) where. I'd prefer to eliminate this constructor, and add an
argument to the normal constructor:
`@transient private val task: Task[_]`
with a comment about it that says something like "This is only used on the
driver to pass the Task object between various scheduler components.". The
transient annotation doesn't actually have any impact since we manually
serialize the task description, but I think is helpful to clearly signal that
the task variable won't be serialized to be sent to the worker.
Then, I think the decode function should explicitly pass in "task = null"
when creating the TaskDescription.
@squito made an earlier comment that seemed potentially in conflict with
this suggestion -- Imran what do you think about this approach?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
