Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5550#issuecomment-98827840
Thanks @JoshRosen that makes sense. In that case we're back to @XuTingjun's
original patch, which simply doesn't subtract the number of skipped stages from
the total (denominator).
Can a stage ever be skipped without first completing once? This could be a
dumb question. That is, is it possible that a stage is then counted in the
total (denominator) but not in completed stages (numerator) because it was
skipped and was never seen to complete? Then we have the problem that it will
never reach m/m complete.
If that can't happen, then I agree with the original change actually.
Then I think we need _both_ the current state of this change, which defends
against a slightly different problem (stage completes, retries, but then
_fails_, but is always counted complete), _and_ the original change, which just
removed count of skipped from the total.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
