Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/7770#issuecomment-127328436
@squito we won't remove `TaskMetrics` before 2.0 because it breaks
compatibility, as you mentioned. Also, in my mind the semantics of accumulators
during retries are fairly clear actually; if a task fails, its accumulator
values are not counted, whereas with `TaskMetrics` you have to check the task
end status in all the downstream listeners, thus requiring code duplication for
correctness.
I agree that accumulators are not perfect. The fact that there is a
singleton `Accumulator` object per JVM is a little strange, for instance. For a
clean redesign in 2.0 ideally we can clean up a lot of the remaining sources of
confusion with accumulators and unify it with `TaskMetrics` somehow so we get
the best of both worlds (e.g. provide type safety). Until then, I personally
find accumulators easier to work with.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
