Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/3009#issuecomment-63426293
There are several alternative approaches, and they largely differentiate on
who is the intended consumer of the progress bar information. For example, a
system administrator type who is knowledgeable about how Spark works would
likely be happier with progress bars that indicate as much as possible about
the number of stages and tasks and their completion, even if that means that
those numbers won't be fixed, might mover backwards, etc. On the other hand,
if you're dealing with a less knowledgeable consumer who only wants to know
whether the job is making some kind of progress, then you can play all sorts of
tricks, such as not displaying absolute numbers of tasks to be completed but
only the completed percentage of whatever the scheduler currently knows about,
not moving the progress bar backwards when tasks or stages need to be re-run,
moving to 100% complete when the job is done, etc.
What is most appropriate for Spark's own webui is matter still open for
debate, but the progress API should at least make the rawer numbers available
so that those crafting their own progress indicator outside of Spark's webui
can make independent decisions about what is appropriate to display for their
particular intended users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
