Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8090#issuecomment-130093251
Hi @andrewor14 , no that is not quite what I was saying. I was basically
just repeating my comments from the earlier PR, and how I was confused on what
we the goal is here, and that we can probably do something much simpler:
https://github.com/apache/spark/pull/7770#discussion_r36189276. Admittedly I
never even considered this interaction w/ skipped stages, but I was in general
wary of making a change that seemed more complicated than it needed to be.
In fact, the questions you are asking are *exactly* my questions about the
initial approach. I don't understand when & why we want to reset those values
-- there are so many seemingly similar cases which result in different
behavior, I can't figure out what the desired effect for the end user really
is. The global value of the accumulator is going to be a mix of different
stages no matter what, since you can have multiple stages executing
simultaneously, and even multiple attempts for one stage. It seems like you
just want the value you see in the SparkListener `StageInfo`s to have the value
for just the stage. But that is possible with the much simpler implementation
of just changing the internal accumulators in `Stage` to:
```scala
private var _internalAccumulators: Seq[Accumulator[Long]] =
InternalAccumulator.create()
```
That also passes all your tests, so I'm not sure why we want something more
complex. It'll just keep incrementing the value across multiple attempts, but
unless we clearly we define the desired behavior (across multiple attempts,
partial recomputations, full recomputations, full recomputations but that get
broken out into 2 attempts, recomputations via a skipped stage, etc.) I don't
understand doing anything else. At least this would be doing something that is
well defined.
"safe" is an interesting word to use ... the safe thing to do is to not
touch the DAGScheduler (as it seems we do even when we know its broken) and let
the metric be incorrect in cases we don't properly understand.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
