[
https://issues.apache.org/jira/browse/HADOOP-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HADOOP-12444:
--------------------------------------
Attachment: HADOOP-12444.3.patch
Thanks [~thodemoor]. Attaching the revised patch. I will upload the test
report shortly.
There are the 2 tests which fail in both master and with-patch.
AWS tests without patch (“mvn clean package” from
hadoop/hadoop-tools/hadoop-aws):
======================================================
Results :
========
Failed tests:
TestS3Credentials.noSecretShouldThrow Expected exception:
java.lang.IllegalArgumentException
TestS3Credentials.noAccessIdShouldThrow Expected exception:
java.lang.IllegalArgumentException
Tests in error:
TestS3AContractRootDir>AbstractContractRootDirectoryTest.testListEmptyRootDirectory:134
» FileNotFound
TestS3AConfiguration.TestAutomaticProxyPortSelection:138 » AmazonS3 Forbidden
...
Tests run: 220, Failures: 2, Errors: 2, Skipped: 6
AWS tests with patch
================
Results :
========
Failed tests:
TestS3Credentials.noSecretShouldThrow Expected exception:
java.lang.IllegalArgumentException
TestS3Credentials.noAccessIdShouldThrow Expected exception:
java.lang.IllegalArgumentException
Tests in error:
TestS3AContractRootDir>AbstractContractRootDirectoryTest.testListEmptyRootDirectory:134
» FileNotFound
TestS3AConfiguration.TestAutomaticProxyPortSelection:138 » AmazonS3 Forbidden
...
Tests run: 220, Failures: 2, Errors: 2, Skipped: 6
{noformat}
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 8.75 sec <<<
FAILURE! - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir)
Time elapsed: 1.633 sec <<< ERROR!
java.io.FileNotFoundException: No such file or directory: /
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1000)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:738)
at
org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:134)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
TestAutomaticProxyPortSelection(org.apache.hadoop.fs.s3a.TestS3AConfiguration)
Time elapsed: 620.356 sec <<< ERROR!
com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon
S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: null)
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3738)
at
com.amazonaws.services.s3.AmazonS3Client.listMultipartUploads(AmazonS3Client.java:2796)
at
com.amazonaws.services.s3.transfer.TransferManager.abortMultipartUploads(TransferManager.java:1217)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:313)
at
org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:51)
at
org.apache.hadoop.fs.s3a.TestS3AConfiguration.TestAutomaticProxyPortSelection(TestS3AConfiguration.java:138)
{noformat}
> Consider implementing lazy seek in S3AInputStream
> -------------------------------------------------
>
> Key: HADOOP-12444
> URL: https://issues.apache.org/jira/browse/HADOOP-12444
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.7.1
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: HADOOP-12444.1.patch, HADOOP-12444.2.patch,
> HADOOP-12444.3.patch, HADOOP-12444.WIP.patch
>
>
> - Currently, "read(long position, byte[] buffer, int offset, int length)" is
> not implemented in S3AInputStream (unlike DFSInputStream). So,
> "readFully(long position, byte[] buffer, int offset, int length)" in
> S3AInputStream goes through the default implementation of seek(), read(),
> seek() in FSInputStream.
> - However, seek() in S3AInputStream involves re-opening of connection to S3
> everytime
> (https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L115).
>
> - It would be good to consider having a lazy seek implementation to reduce
> connection overheads to S3. (e.g Presto implements lazy seek.
> https://github.com/facebook/presto/blob/master/presto-hive/src/main/java/com/facebook/presto/hive/PrestoS3FileSystem.java#L623)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
