[
https://issues.apache.org/jira/browse/HADOOP-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13031:
-----------------------------------
Description:
[HADOOP-13065] added a new interface for retrieving FS and FC Statistics. This
jira is to refactor the code that maintains rack-aware read metrics to use the
newly added StorageStatistics. Specially,
# Rack-aware read bytes metrics is mostly specific to HDFS. For example, local
file system doesn't need that. We consider to move it from base
FileSystemStorageStatistics to a dedicated HDFS specific StorageStatistics
sub-class.
# We would have to develop an optimized thread-local mechanism to do this, to
avoid causing a performance regression in HDFS stream performance.
Optionally, it would be better to simply move this to HDFS's existing
per-stream {{ReadStatistics}} for now. As [HDFS-9579] states, ReadStatistics
metrics are only accessible via {{DFSClient}} or {{DFSInputStream}}. Not
something that application framework such as MR and Tez can get to.
was:
According to discussion in [HDFS-10175], using a composite (e.g. enum map,
array) data structure to manage the distance->bytesRead mapping will probably
make the code simpler.
# {{StatisticsData}} will be a bit shorter by delegating the operations to the
composite data structure.
# The {{incrementBytesReadByDistance(int distance, long newBytes)}} and
{{getBytesReadByDistance(int distance)}} which switch-case all hard-code
variables, may be simplified as we can set/get the {{bytesRead}} by distance
directly from map/array.
This jira is to track the discussion and effort of refactoring the code that
maintains rack-aware counters.
> Refactor the code that maintains rack-aware counters in FileSystem$Statistics
> -----------------------------------------------------------------------------
>
> Key: HADOOP-13031
> URL: https://issues.apache.org/jira/browse/HADOOP-13031
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Mingliang Liu
>
> [HADOOP-13065] added a new interface for retrieving FS and FC Statistics.
> This jira is to refactor the code that maintains rack-aware read metrics to
> use the newly added StorageStatistics. Specially,
> # Rack-aware read bytes metrics is mostly specific to HDFS. For example,
> local file system doesn't need that. We consider to move it from base
> FileSystemStorageStatistics to a dedicated HDFS specific StorageStatistics
> sub-class.
> # We would have to develop an optimized thread-local mechanism to do this, to
> avoid causing a performance regression in HDFS stream performance.
> Optionally, it would be better to simply move this to HDFS's existing
> per-stream {{ReadStatistics}} for now. As [HDFS-9579] states, ReadStatistics
> metrics are only accessible via {{DFSClient}} or {{DFSInputStream}}. Not
> something that application framework such as MR and Tez can get to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
