Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15743#discussion_r86406738
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1089,7 +1089,7 @@ class DAGScheduler(
// To avoid UI cruft, ignore cases where value wasn't updated
if (acc.name.isDefined && !updates.isZero) {
stage.latestInfo.accumulables(id) = acc.toInfo(None,
Some(acc.value))
- event.taskInfo.accumulables += acc.toInfo(Some(updates.value),
Some(acc.value))
+ event.taskInfo.accumulables ::= acc.toInfo(Some(updates.value),
Some(acc.value))
--- End diff --
Something I was a bit unsure of but want to revisit now:
It looks like the loop
```
for (accumulableInfo <- info.accumulables) {
stageData.accumulables(accumulableInfo.id) = accumulableInfo
}
```
in `JobProgressListener.onTaskEnd()` may assume that accumulator updates
are appended to the list so that the newest update for a particular accumulator
appears last, but _prepending_ here will change that. I don't think that this
behavior is covered by any existing unit tests, though. Instead of using a list
here, I might want to use an immutable `Queue`. Let me go ahead and make this
change now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
