Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/16527
I use following code log trim stages/jobs time consuming:
```:scala
/** If stages is too large, remove and garbage collect old stages */
private def trimStagesIfNecessary(stages: ListBuffer[StageInfo]) =
synchronized {
if (stages.size > retainedStages) {
val start = System.currentTimeMillis()
val toRemove = (stages.size - retainedStages)
stages.take(toRemove).foreach { s =>
stageIdToData.remove((s.stageId, s.attemptId))
stageIdToInfo.remove(s.stageId)
}
stages.trimStart(toRemove)
logInfo(s"Trim stages time consuming: ${System.currentTimeMillis() -
start}")
}
}
/** If jobs is too large, remove and garbage collect old jobs */
private def trimJobsIfNecessary(jobs: ListBuffer[JobUIData]) =
synchronized {
if (jobs.size > retainedJobs) {
val start = System.currentTimeMillis()
val toRemove = (jobs.size - retainedJobs)
jobs.take(toRemove).foreach { job =>
// Remove the job's UI data, if it exists
jobIdToData.remove(job.jobId).foreach { removedJob =>
// A null jobGroupId is used for jobs that are run without a job
group
val jobGroupId = removedJob.jobGroup.orNull
// Remove the job group -> job mapping entry, if it exists
jobGroupToJobIds.get(jobGroupId).foreach { jobsInGroup =>
jobsInGroup.remove(job.jobId)
// If this was the last job in this job group, remove the map
entry for the job group
if (jobsInGroup.isEmpty) {
jobGroupToJobIds.remove(jobGroupId)
}
}
}
}
jobs.trimStart(toRemove)
logInfo(s"Trim jobs time consuming: ${System.currentTimeMillis() -
start}")
}
}
```
and the result is:
```
tail -f test-time-consuming.log | grep time
17/01/13 10:03:39 INFO JobProgressListener: Trim stages time consuming: 3
17/01/13 10:03:39 INFO JobProgressListener: Trim jobs time consuming: 4
17/01/13 10:03:39 INFO JobProgressListener: Trim stages time consuming: 0
17/01/13 10:03:47 INFO JobProgressListener: Trim stages time consuming: 0
17/01/13 10:03:47 INFO JobProgressListener: Trim jobs time consuming: 0
17/01/13 10:03:47 INFO JobProgressListener: Trim stages time consuming: 0
17/01/13 10:03:56 INFO JobProgressListener: Trim stages time consuming: 1
17/01/13 10:03:56 INFO JobProgressListener: Trim jobs time consuming: 0
17/01/13 10:03:56 INFO JobProgressListener: Trim stages time consuming: 0
17/01/13 10:04:04 INFO JobProgressListener: Trim stages time consuming: 0
17/01/13 10:04:04 INFO JobProgressListener: Trim jobs time consuming: 0
17/01/13 10:04:04 INFO JobProgressListener: Trim stages time consuming: 0
```
It may be fine just change `retainedTasks`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
