Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14617
@squito , by revising this code, I found there're some places which are
misleading and could be improved:
* All the memory usage referred here about on-heap memory and off-heap
memory (used, remaining, max) are only about storage memory. But in our REST
API and metrics source we don't specify this, this may mislead the user. I
think this problem existed after we incorporated execution memory usage, we
neglected that part of memory usage and only count in storage memory usage.
Either we should rename the field in REST API and others to specifically
saying that is storage memory, or we should also count in execution memory
usage.
* With `UnifiedMemoryManager` incorporated, the maximum storage memory
(on-heap and off-heap) will be changed in the runtime according to the
execution memory, but here in the BlockManager's `StorageStatus` and other
places we assume this is static. It is not accurate to reflect the current
Storage Memory size. We should also handle this.
What is your opinion about it, would be grateful to hear your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
