Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18935
@squito , here is the list of full metrics I dumped from running
application. By default, if we don't enable verbose metric, only 2 metrics
("usedHeapMemory" and "usedDirectMemory") will be tracked. If we enable verbose
metrics, then the details of each arena will be tracked, the number of detailed
metrics is 16, so the total verbose metric number will be "16 * arena number *
2", here arena number is based on the number of cores, and "2" represents
"direct arena" and "heap arena".
I agree this might be too verbose for a user to track netty memory usage,
so by default I will not enable verbose metric. Unless if developers wants to
know the detailed behaviors of network memory usage, then they could enable
this configuration, this might be useful for tuning and performance analysis.
What do you think?
```
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveBytes
value = 16777216
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveHugeAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveNormalAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveSmallAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveTinyAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numAllocations
value = 7
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numDeallocations
value = 7
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numHugeAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numHugeDeallocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numNormalAllocations
value = 2
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numNormalDeallocations
value = 2
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numSmallAllocations
value = 3
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numSmallDeallocations
value = 3
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numTinyAllocations
value = 2
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numTinyDeallocations
value = 2
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveBytes
value = 16777216
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveHugeAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveNormalAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveSmallAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveTinyAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numAllocations
value = 4
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numDeallocations
value = 4
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numHugeAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numHugeDeallocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numNormalAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numNormalDeallocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numSmallAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numSmallDeallocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numTinyAllocations
value = 4
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numTinyDeallocations
value = 4
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.usedDirectMemory
value = 16777216
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.usedHeapMemory
value = 16777216
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
