Github user liyezhang556520 commented on the pull request:
https://github.com/apache/spark/pull/7753#issuecomment-142980989
>1. I don't think there is any need to separate out the memory used by the
client and server portions. These are internal details that the end-user
doesn't care about -- in fact you're already correctly simplifying by the time
it gets to the web UI, but you could really combine them immediately in
TransportMetrics. That would help simplify the code I think.
This make sense, I'll merge them in the Metrics itself.
> 2. We shouldn't say "Net" memory used, users might think that means total
memory used. I guess we should say "Network"
Oh, I forgot thet "net" has another meaning.... Sorry for that.
> 4.Could the stage table include the max memory per executor per stage as
well? That would be great to help users quickly identify the stages which
require the most memory
Yes, User can see the max memory per executor per stage, there is a
`StageMemoryPage` attached to `MemoryTab`, which will show the executors'
status (max memory) for each finished stage.
> 5. How many additional events are getting logged? With the current
architecture, there is some pressure to not log too much -- both to keep log
sizes small for later processing, and also to make sure the driver doesn't get
too busy just logging (which eventually leads to it dropping events).
additional events are only logged when there is stages complete or
executors removed. Please see the [design
doc](https://issues.apache.org/jira/secure/attachment/12762171/Tracking%20Spark%20Memory%20Usage%20-%20Phase%201.pdf)
but the code is not aligned with the doc yet, I'll update the code first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
