[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055829#comment-15055829
]
Wellington Chevreuil commented on HDFS-9521:
--------------------------------------------
Thanks [~liuml07]. The original idea on measuring download time and individual
disk fsync time separately and log it as INFO is to help troubleshooting issues
once it occurs. Leaving it as DEBUG would require a restart of Namenodes and
hope for the incident to re occur. Would you think leave this as INFO would add
too much overhead on this feature?
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> -------------------------------------------------------------------------------------------------------
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Minor
> Attachments: HDFS-9521.patch
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
> ((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
