Gera Shegalov created YARN-1529:
-----------------------------------
Summary: Add Localization overhead metrics to NM
Key: YARN-1529
URL: https://issues.apache.org/jira/browse/YARN-1529
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Users are often unaware of localization cost that their jobs incur. To measure
effectiveness of localization caches it is necessary to expose the overhead in
the form of metrics.
We propose addition of the following metrics to NodeManagerMetrics.
When a container is about to launch, its set of LocalResources has to be
fetched from a central location, typically on HDFS, that results in a number of
download requests for the files missing in caches.
LocalizedFilesMissed: total files (requests) downloaded from DFS. Cache misses.
LocalizedFilesCached: total localization requests that were served from local
caches. Cache hits.
LocalizedBytesMissed: total bytes downloaded from DFS due to cache misses.
LocalizedBytesCached: total bytes satisfied from local caches.
Localized(Files|Bytes)CachedRatio: percentage of localized (files|bytes) that
were served out of cache: ratio = 100 * caches / (caches + misses)
LocalizationDownloadNanos: total elapsed time in nanoseconds for a container to
go from ResourceRequestTransition to LocalizedTransition
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
