Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4841#issuecomment-77227608
Let's summarize the original goals and the current state of SPARK-5771.
From there we can decide how to move forward with this issue:
- Before #4567, a completed application that called `sc.stop()` will show
"0 cores" on the UI, and one that did not call this will show however many
cores it was using the instant before it was terminated.
- #4567 is an attempt to make this consistent, such that the number of
cores displayed on the UI in both cases is non-zero. However, this shows the
number of cores requested by the application, which may not make sense for
applications that did not explicitly request any cores.
- #4800 hides the default number of cores (Int.MaxValue) behind `*`. This
is not important to our discussion.
It seems from all of this that the deeper issue is that we use the number
of cores requested by the application, which doesn't make sense for all
applications. Ideally we want something that represents "the number of cores
used by an application before it terminates", but the existing code in `Master`
doesn't allow us to do that easily.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
