SaurabhChawla100 commented on a change in pull request #32381:
URL: https://github.com/apache/spark/pull/32381#discussion_r623284990
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
##########
@@ -66,7 +70,9 @@ private[ui] class AllJobsPage(parent: JobsTab, store:
AppStatusStore) extends We
private def makeJobEvent(jobs: Seq[v1.JobData]): Seq[String] = {
jobs.filter { job =>
job.status != JobExecutionStatus.UNKNOWN && job.submissionTime.isDefined
- }.map { job =>
+ }.sortBy { j =>
+ -math.max(j.submissionTime.get.getTime,
j.completionTime.map(_.getTime).getOrElse(-1L))
Review comment:
We are here considering the latest submitted or latest completed jobs
based on the submissionTime and completionTime
So If we take this example
```
scala> case class Test(startTime: Long, endTime: Long)
defined class Test
val addSeq: Seq[Test] = Seq(Test(2,7), Test(1,5), Test(4,5), Test(2,5),
Test(3,4), Test(1,-1))
scala> addSeq
res135: Seq[Test] = List(Test(1,5), Test(2,7), Test(3,4), Test(4,5),
Test(2,5), Test(1,-1))
```
If we take latest startTime and latest endTime and take only 2 items from
the seq than it would be
List(Test(2,7), Test(4,5))
where as per the above code by taking the max for startTime and endTime
```
scala> addSeq.sortBy { e => -math.max( e.startTime, e.endTime) }
res136: Seq[Test] = List(Test(2,7), Test(1,5), Test(4,5), Test(2,5),
Test(3,4), Test(1,-1))
scala> addSeq.sortBy { e => -math.max( e.startTime, e.endTime) }.take(2)
res137: Seq[Test] = List(Test(2,7), Test(1,5))
```
if we take both the tuples and compare the result we can get the latest
start and latest end value
```
scala> addSeq.sortWith{ (t1,t2) => !(t1.startTime < t2.startTime ||
t1.endTime < t2.endTime)}
res138: Seq[Test] = List(Test(2,7), Test(4,5), Test(2,5), Test(1,5),
Test(3,4), Test(1,-1))
scala> addSeq.sortWith{ (t1,t2) => !(t1.startTime < t2.startTime ||
t1.endTime < t2.endTime)}.take(2)
res139: Seq[Test] = List(Test(2,7), Test(4,5))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
