Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/7753#issuecomment-126534206
Hi @squito , IIUC here the code always set the max memory usage:
```
@volatile private var _nettyReadOnheapSize: Long = _
def nettyReadOnheapSize: Long = _nettyReadOnheapSize
private[spark] def setNettyReadOnheapSize(value: Long, time: SysTime):
Unit = {
_nettyReadOnheapSize = value
if (value > _maxNettyReadOnheapSizeTime._1) {
_maxNettyReadOnheapSizeTime = (value, time)
}
}
```
If current memory usage is smaller than the previous one, simple will
discard it.
From my understanding it is better to transfer the raw metrics to the
driver, and further processed by each listener as they wanted, that will be
more flexible. If people want the minimum memory usage for example, they could
process it by their logic in listener. Here only get the max memory usage will
restrict the flexibility for different user.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
