[
https://issues.apache.org/jira/browse/HDFS-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129925#comment-13129925
]
Uma Maheswara Rao G commented on HDFS-1447:
-------------------------------------------
@Suresh:
Thanks for taking a look. Yes, i know the max files per dir is 64 by default.
Please check other JIRAs in this umberilla for further optimizations. Wanted to
use the same scan for recoverTmpUnlinkedFiles with other JIRA.
With the less file we may not get good improvements and with this single JIRA.
Above figures are to show the relative improvement in optimizing the loops.
@Todd, Yes, the scanning logic was good in HDFS-2384. But in FSDataSet case, we
need to maintain the tree as well rite.
Also in C code i have seen there is a genstamps array with blkids. Long array
will consume more memory right?
> Make getGenerationStampFromFile() more efficient, so it doesn't reprocess
> full directory listing for every block
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1447
> URL: https://issues.apache.org/jira/browse/HDFS-1447
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Attachments: HDFS-1447.patch, Test_HDFS_1447_NotForCommitt.java.patch
>
>
> Make getGenerationStampFromFile() more efficient. Currently this routine is
> called by addToReplicasMap() for every blockfile in the directory tree, and
> it walks each file's containing directory on every call. There is a simple
> refactoring that should make it more efficient.
> This work item is one of four sub-tasks for HDFS-1443, Improve Datanode
> startup time.
> The fix will probably be folded into sibling task HDFS-1446, which is already
> refactoring the method that calls getGenerationStampFromFile().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
