mridulm commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r816434674
##########
File path:
core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
##########
@@ -599,7 +599,8 @@ class EventLoggingListenerSuite extends SparkFunSuite with
LocalSparkContext wit
stageId: Int,
taskType: String,
executorMetrics: ExecutorMetrics): SparkListenerTaskEnd = {
- val taskInfo = new TaskInfo(taskId, taskIndex, 0, 1553291556000L,
executorId, "executor",
+ val taskInfo = new TaskInfo(
+ taskId, taskIndex, 0, taskIndex, 1553291556000L, executorId, "executor",
Review comment:
nit: We should rename `taskIndex` as `partitionId` in this method -
taskIndex happens to be partition id here.
##########
File path:
core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
##########
@@ -1860,15 +1860,16 @@ abstract class AppStatusListenerSuite extends
SparkFunSuite with BeforeAndAfter
private def newAttempt(orig: TaskInfo, nextId: Long): TaskInfo = {
// Task reattempts have a different ID, but the same index as the original.
- new TaskInfo(nextId, orig.index, orig.attemptNumber + 1, time,
orig.executorId,
+ new TaskInfo(
+ nextId, orig.index, orig.attemptNumber + 1, orig.partitionId, time,
orig.executorId,
s"${orig.executorId}.example.com", TaskLocality.PROCESS_LOCAL,
orig.speculative)
}
private def createTasks(count: Int, execs: Array[String]): Seq[TaskInfo] = {
(1 to count).map { id =>
val exec = execs(id.toInt % execs.length)
val taskId = nextTaskId()
- new TaskInfo(taskId, taskId.toInt, 1, time, exec, s"$exec.example.com",
+ new TaskInfo(taskId, taskId.toInt, 1, taskId.toInt, time, exec,
s"$exec.example.com",
TaskLocality.PROCESS_LOCAL, id % 2 == 0)
Review comment:
Looks like this method is incorrect to begin with.
task index should be < number of tasks in stage and partition id should be <
number of partitions.
We should change both to `id` instead.
Thoughts @stczwd, @Ngone51, @cloud-fan ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
