Xiaoyu Yao commented on HDFS-10930:

Thanks [~daryn] for the feedback. I've put my answers to the questions inline. 
Feel free to share your thoughts of other approaches. 

bq. do we really need to move non-redundant and non-general use methods into a 
new class just to add a log line for duration?

Refactor and adding trace is just the first step. It helps us add 
instrumentation and per volume I/O metrics in other follow up JIRAs. 

bq. I'm not a huge fan of kitchen sink classes of static methods because it's 
counter to polymorphism and it usually makes tests, particularly mocks, wicked 
hard to write.

DatanodeFaultInjection and mocks exist in datanode unit tests. The majority of 
the extracted static methods in DatanodeIO class wraps low level OS calls so 
that we can easily add datanode I/O instrumentation and collect finer grain 
metrics without churning datanode code everywhere. The test/mock can still be 
added at higher level from the callers' perspective with the verification the 
metrics update. 

> Refactor: Wrap Datanode IO related operations
> ---------------------------------------------
>                 Key: HDFS-10930
>                 URL: https://issues.apache.org/jira/browse/HDFS-10930
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>            Reporter: Xiaoyu Yao
>            Assignee: Xiaoyu Yao
>         Attachments: HDFS-10930.01.patch, HDFS-10930.02.patch
> Datanode IO (Disk/Network) related operations and instrumentations are 
> currently spilled in many classes such as DataNode.java, BlockReceiver.java, 
> BlockSender.java, FsDatasetImpl.java, FsVolumeImpl.java, 
> DirectoryScanner.java, BlockScanner.java, FsDatasetAsyncDiskService.java, 
> LocalReplica.java, LocalReplicaPipeline.java, Storage.java, etc. 
> This ticket is opened to consolidate IO related operations for easy 
> instrumentation, metrics collection, logging and trouble shooting. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to