[
https://issues.apache.org/jira/browse/HADOOP-13704?focusedWorklogId=728944&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728944
]
ASF GitHub Bot logged work on HADOOP-13704:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 17/Feb/22 13:52
Start Date: 17/Feb/22 13:52
Worklog Time Spent: 10m
Work Description: ahmarsuhail commented on a change in pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#discussion_r809069339
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/GetContentSummaryOperation.java
##########
@@ -204,5 +233,6 @@ S3AFileStatus probePathStatus(Path path,
RemoteIterator<S3AFileStatus> listStatusIterator(Path path)
Review comment:
yup, have removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 728944)
Time Spent: 1h 10m (was: 1h)
> S3A getContentSummary() to move to listFiles(recursive) to count children;
> instrument use
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-13704
> URL: https://issues.apache.org/jira/browse/HADOOP-13704
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0
> Reporter: Steve Loughran
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Hive and a bit of Spark use {{getContentSummary()}} to get some summary stats
> of a filesystem. This is very expensive on S3A (and any other object store),
> especially as the base implementation does the recursive tree walk.
> Because of HADOOP-13208, we have a full enumeration of files under a path
> without directory costs...S3A can/should switch to this to speed up those
> places where the operation is called.
> Also
> * API call needs FS spec and contract tests
> * S3A could instrument invocation, so as to enable real-world popularity to
> be measured
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
