Steve Loughran created HADOOP-18374:
---------------------------------------
Summary: DistCP: Aggregate IOStatistics Counters in MapReduce
Counters
Key: HADOOP-18374
URL: https://issues.apache.org/jira/browse/HADOOP-18374
Project: Hadoop Common
Issue Type: Sub-task
Components: tools/distcp
Affects Versions: 3.3.9
Reporter: Steve Loughran
Assignee: Mehakmeet Singh
Distcp can collect IOStatisticsContext counter values and report them to the
console. it can't do the timings in min/mean/max though, as there's no way to
aggregate them properly.
# Publish statistics to MapReduce counters in the tasks within
CopyMapper.copyFileWithRetry().
# The counters will be automatically logged in Job.monitorAndPrintJob() when
DistCp is executed with the -verbose option; no need for changes there.
# We could also publish the iOStatistic means by publishing sample count and
total sum as two separate counters
# In AbstractContractDistCpTest, add an override point for subclasses to list
which metrics they will issue; assert that values are generated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
