HeartSaVioR opened a new pull request #2624:
URL: https://github.com/apache/hadoop/pull/2624
This PR proposes to add latency information in
FileOutputCommitter.mergePaths, so that we can trace how much latency specific
directory takes to merge.
This information would provide some value on investigation when the commit
in FileOutputCommitter takes huge time than expected. This class logged the
call with from/to params in debug level which looks insufficient to trace the
latency of specific directory due to recursive call.
No test added as there's nothing to test actually. Manual test done via
adding below in log4j.properties
```
log4j.logger.org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter=DEBUG
```
and ran tests in TestFileOutputCommitter.
```
2021-01-18 16:14:03,475 DEBUG [main] output.FileOutputCommitter
(FileOutputCommitter.java:mergePaths(461)) - Merging data from
DeprecatedRawLocalFileStatus{path=file:/Users/jlim/WorkArea/JavaProjects/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/test-dir/org.apache.hadoop.mapreduce.lib.output.TestFileOutputCommitter/_temporary/0/task_200707121733_0001_m_000000;
isDirectory=true; modification_time=1610954043000; access_time=1610954043000;
owner=; group=; permission=rwxrwxrwx; isSymlink=false; hasAcl=false;
isEncrypted=false; isErasureCoded=false} to
file:/Users/jlim/WorkArea/JavaProjects/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/test-dir/org.apache.hadoop.mapreduce.lib.output.TestFileOutputCommitter
...
2021-01-18 16:14:03,476 DEBUG [main] output.FileOutputCommitter
(FileOutputCommitter.java:mergePaths(502)) - Merged data from
file:/.../hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/test-dir/org.apache.hadoop.mapreduce.lib.output.TestFileOutputCommitter/_temporary/0/task_200707121733_0001_m_000000
to
file:/.../hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/test-dir/org.apache.hadoop.mapreduce.lib.output.TestFileOutputCommitter
in 1 ms
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
