[
https://issues.apache.org/jira/browse/CRUNCH-683?focusedWorklogId=231949&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-231949
]
ASF GitHub Bot logged work on CRUNCH-683:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Apr/19 09:47
Start Date: 24/Apr/19 09:47
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #23: CRUNCH-683
Avoid unnecessary listStatus calls from getPathSize computation
URL: https://github.com/apache/crunch/pull/23#discussion_r278042702
##########
File path:
crunch-core/src/main/java/org/apache/crunch/io/SourceTargetHelper.java
##########
@@ -41,17 +41,23 @@ public static long getPathSize(FileSystem fs, Path path)
throws IOException {
}
long size = 0;
for (FileStatus status : stati) {
- if (status.isDir()) {
- for (FileStatus st : fs.listStatus(status.getPath())) {
- size += getPathSize(fs, st.getPath());
- }
- } else {
- size += status.getLen();
- }
+ size += getPathSize(fs, status);
}
return size;
}
-
+
+ private static long getPathSize(final FileSystem fs, final FileStatus
status) throws IOException {
Review comment:
This is still doing a recursive treewalk. IF you do
`FileSystem.listFiles(path, true)` you get a deep listing of files only from
the store. For an object store with an optimised implementation (as s3a does),
only one HTTP request is made per 1000 objects, irrespective of the depth of
the tree. For HDFS &c there's still efficiencies, especially when you have a
directory with many millions of files in: in listStatus() all the results have
to get serialized and marshalled over as one, rather than paged over to the
client
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 231949)
Time Spent: 20m (was: 10m)
> Avoid unnecessary listStatus calls from getSize computation
> -----------------------------------------------------------
>
> Key: CRUNCH-683
> URL: https://issues.apache.org/jira/browse/CRUNCH-683
> Project: Crunch
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.14.0
> Reporter: Jon Hemphill
> Assignee: Josh Wills
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The getPathSize computation in SourceTargetHelper currently makes unnecessary
> listStatus calls when recursing over a directory, which can cause performance
> issues when the filesystem is an object store such as S3. The performance can
> be improved with the addition of a private method to use for the getPathSize
> recursion that takes a known FIleStatus object as a parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
