Mustafa Iman created HADOOP-16801:

             Summary: S3Guard queries S3 with recursive file listings
                 Key: HADOOP-16801
             Project: Hadoop Common
          Issue Type: Bug
          Components: tools
            Reporter: Mustafa Iman
         Attachments: HADOOP-aws-no-prefetch.prelim.patch

S3AFileSystem#listFiles with recursive option, queries S3 even when directory 
listing is authoritative. FileStatusListingIterator is created with given 
entries from metadata store 
 . However, FileStatusListingIterator has an ObjectListingIterator that 
prefetches from s3 regardless of authoritative listing. We observed this 
behavior when using DynamDBMetadataStore.

I suppressed the unnecessary S3 calls by providing a dumb listing iterator to 
listFiles call in the provided patch. Obviously this is not a solution. Just 
demonstrating the source of the problem.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to