[ https://issues.apache.org/jira/browse/HADOOP-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16155261#comment-16155261 ]
Steve Loughran commented on HADOOP-13421: ----------------------------------------- {code} fs.s3a.use.list.v1 {code} better {code} fs.s3a.list.version="1" {code} line us up for a v3 algorithm in some time in the future. This will propagate into the s3a FS code too. * Can just call the class {{ListRequest}}, or, if you really want an s3 prefix. One thing to consider: what if someone who only has the v1 API wants to run the tests? I know Thomas and Ewan are happy, but don't know about others. Maybe: if the version API is explicitly set to v1 for a bucket then skip the V2 tests? Looking at the code though, I think that's a moot issue. If someone forces the list API to v1 in their settings, that will be implicitly picked yp by everything except the v1 test, which just becomes a duplicate of the superclass in terms of test coverage...its not going to fail. > Switch to v2 of the S3 List Objects API in S3A > ---------------------------------------------- > > Key: HADOOP-13421 > URL: https://issues.apache.org/jira/browse/HADOOP-13421 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 2.8.0 > Reporter: Steven K. Wong > Assignee: Aaron Fabbri > Priority: Minor > Attachments: HADOOP-13421.002.patch, > HADOOP-13421-HADOOP-13345.001.patch > > > Unlike [version > 1|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html] of the > S3 List Objects API, [version > 2|http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html] by > default does not fetch object owner information, which S3A doesn't need > anyway. By switching to v2, there will be less data to transfer/process. > Also, it should be more robust when listing a versioned bucket with "a large > number of delete markers" ([according to > AWS|https://aws.amazon.com/releasenotes/Java/0735652458007581]). > Methods in S3AFileSystem that use this API include: > * getFileStatus(Path) > * innerDelete(Path, boolean) > * innerListStatus(Path) > * innerRename(Path, Path) > Requires AWS SDK 1.10.75 or later. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org