Github user squito commented on the issue:
https://github.com/apache/spark/pull/14617
hi @jerryshao good points. First, we should probably move this discussion
to jira so its more visible -- feel free to open two issues for these if you
want, or first discuss on dev@. (Sorry its my fault for starting a discussion
here in the PR comments after the original change was merged ...)
On your first point, I like the idea of exposing more information on memory
usage, but is there anything meaningful to report on execution memory in the
rest api? It doesn't really seem like there is. Maybe we should rename the
fields, but keep backwards compatibility in mind.
For the second point about storage memory limits -- its a good question
about what it should report with the Unified Memory Manager. I thought we'd
just expose the portion of memory immune to eviction,
`spark.memory.storageFraction`. But perhaps I misunderstand what is going on
now.
Again, probably better to have the design discussion on jiras.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
