Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18070
thanks for the udpates. I was testing this out by running large job with
speculative tasks and I am still seeing the stage summary show failed tasks.
It looks like its due to this code:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala#L396
Where it doesn't differentiate the commit denied message so i think we need
to handle it there so the stats show up properly.
It would also be good to add a unit test for that where you can look at the
stageData to make sure the numFailedTasks is what you expect.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
