[
https://issues.apache.org/jira/browse/HDFS-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16716085#comment-16716085
]
Pranay Singh commented on HDFS-14084:
-------------------------------------
[~xkrogen] I changed the logging from DEBUG to INFO in my test setup and did a
listing, copy of file from local filesystem to HDFS and this is what shows up
....
2018-12-10 14:39:54,391 INFO ipc.ProtobufRpcEngine: RPC Client stats: {}
AddBlockNumOps = 1
AddBlockAvgTime = 21.0
CreateNumOps = 1
CreateAvgTime = 39.0
GetFileInfoNumOps = 4
GetFileInfoAvgTime = 1.0
GetServerDefaultsNumOps = 1
GetServerDefaultsAvgTime = 6.0
2018-12-10 14:39:54,471 INFO ipc.ProtobufRpcEngine: RPC Client stats: {}
AddBlockNumOps = 1
AddBlockAvgTime = 21.0
CreateNumOps = 1
CreateAvgTime = 39.0
CompleteNumOps = 1
CompleteAvgTime = 4.0
GetFileInfoNumOps = 4
GetFileInfoAvgTime = 1.0
GetServerDefaultsNumOps = 1
GetServerDefaultsAvgTime = 6.0
2018-12-10 14:39:54,880 INFO ipc.ProtobufRpcEngine: RPC Client stats: {}
AddBlockNumOps = 1
AddBlockAvgTime = 21.0
CreateNumOps = 1
CreateAvgTime = 39.0
CompleteNumOps = 2
CompleteAvgTime = 7.0
GetFileInfoNumOps = 4
GetFileInfoAvgTime = 1.0
GetServerDefaultsNumOps = 1
GetServerDefaultsAvgTime = 6.0
2018-12-10 14:39:54,896 INFO ipc.ProtobufRpcEngine: RPC Client stats: {}
RenameNumOps = 1
RenameAvgTime = 12.0
AddBlockNumOps = 1
AddBlockAvgTime = 21.0
CreateNumOps = 1
CreateAvgTime = 39.0
CompleteNumOps = 2
CompleteAvgTime = 7.0
GetFileInfoNumOps = 4
GetFileInfoAvgTime = 1.0
GetServerDefaultsNumOps = 1
GetServerDefaultsAvgTime = 6.0
> Need for more stats in DFSClient
> --------------------------------
>
> Key: HDFS-14084
> URL: https://issues.apache.org/jira/browse/HDFS-14084
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Pranay Singh
> Assignee: Pranay Singh
> Priority: Minor
> Attachments: HDFS-14084.001.patch, HDFS-14084.002.patch
>
>
> The usage of HDFS has changed from being used as a map-reduce filesystem, now
> it's becoming more of like a general purpose filesystem. In most of the cases
> there are issues with the Namenode so we have metrics to know the workload or
> stress on Namenode.
> However, there is a need to have more statistics collected for different
> operations/RPCs in DFSClient to know which RPC operations are taking longer
> time or to know what is the frequency of the operation.These statistics can
> be exposed to the users of DFS Client and they can periodically log or do
> some sort of flow control if the response is slow. This will also help to
> isolate HDFS issue in a mixed environment where on a node say we have Spark,
> HBase and Impala running together. We can check the throughput of different
> operation across client and isolate the problem caused because of noisy
> neighbor or network congestion or shared JVM.
> We have dealt with several problems from the field for which there is no
> conclusive evidence as to what caused the problem. If we had metrics or stats
> in DFSClient we would be better equipped to solve such complex problems.
> List of jiras for reference:
> -------------------------
> HADOOP-15538 HADOOP-15530 ( client side deadlock)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
