[
https://issues.apache.org/jira/browse/HADOOP-6467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837012#action_12837012
]
Tsz Wo (Nicholas), SZE commented on HADOOP-6467:
------------------------------------------------
I ran wordcount with the v2 patch twice. Both took 16 mins. Not sure why it
ran slower than the previous patch.
{noformat}
-bash-3.1$ date; time $H ${WC_CMD} ${HAR_FULL}/${DIR} ${TT_WC}3 Mon Feb 22
23:23:03 UTC 2010
10/02/22 23:23:19 INFO input.FileInputFormat: Total input paths to process :
100000
10/02/22 23:33:53 INFO mapred.JobClient: Running job: job_201002042035_75937
10/02/22 23:33:54 INFO mapred.JobClient: map 0% reduce 0% ...
10/02/22 23:39:08 INFO mapred.JobClient: map 100% reduce 100% ...
10/02/22 23:39:14 INFO mapred.JobClient: Reduce input records=17729
real 16m11.171s
user 2m24.217s
sys 0m39.462s
{noformat}
Anyway, below are some improvement suggestions:
- use a variable to store harPath.depth() so that the same value won't be
computed again and again.
- Unfortunately, Path is quite expensive (see HADOOP-6532). It is better to
check child.startsWith(parentString) before creating thisPath. i.e. replace
{code}
+ String child = lineFeed.substring(0, lineFeed.indexOf(" "));
+ Path thisPath = new Path(child);
+ if ((child.startsWith(parentString)) && (thisPath.depth() ==
harPath.depth() + 1)) {
+ ...
{code}
with
{code}
+ String child = lineFeed.substring(0, lineFeed.indexOf(" "));
+ if (child.startsWith(parentString)) {
+ Path thisPath = new Path(child);
+ if (thisPath.depth() == harPath.depth() + 1) {
+ ...
{code}
- Also, there are lines more than 80 characters.
> Performance improvement for liststatus on directories in hadoop archives.
> -------------------------------------------------------------------------
>
> Key: HADOOP-6467
> URL: https://issues.apache.org/jira/browse/HADOOP-6467
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: Archives_performance.docx, Archives_performance.docx,
> HADOOP-6467-v2.patch, HADOOP-6467-y.0.20-branch-v2.patch,
> HADOOP-6467-y0.20-branch.patch, HADOOP-6467.patch, HADOOP-6467.patch,
> HADOOP-6467.patch
>
>
> A liststatus call on a directory in hadoop archives leads to ( 2* number of
> files in directory) open calls to the namenode. This is very sub optimal and
> needs to be fixed to make it performant enough to be used on a daily basis.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
