Zoran Rajic created HADOOP-12753:
------------------------------------
Summary: S3A jUnit tests failing if using HTTP proxy
Key: HADOOP-12753
URL: https://issues.apache.org/jira/browse/HADOOP-12753
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 2.7.2
Environment: Test hardware:
* issue identified while running jUnit tests on MacBook Pro
PREREQUISITE:
* have a functional HTTP proxy running (ie. SQUID HTTP proxy @
http://127.0.0.1:3128)
* S3A test configuration (relevant options):
<!-- NOTE: by not specifying endpoint, we'll use the default AWS S3 -->
<property>
<name>fs.s3a.endpoint</name>
<value></value>
</property>
<!-- NOTE: please configure functional HTTP proxy with
fs.s3a.proxy.{host,port} -->
<property>
<name>fs.s3a.proxy.host</name>
<value>127.0.0.1</value>
</property>
<property>
<name>fs.s3a.proxy.port</name>
<value>3128</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<value>_REAL_AWS_S3_ACCESS_KEY</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<value>_REAL_AWS_S3_SECRET_KEY</value>
</property>
Reporter: Zoran Rajic
Priority: Minor
Some companies have restricted firewalls and don't allow the "direct access" to
Web-pages or Internet, so their employees have to use the SOCKS or the HTTP
proxy.
This also means that the engineers who develop and test the Hadoop code and
s3a:// -filesystem inside the company firewalls, need to access the "external"
Amazon S3 service via the HTTP proxy.
Unfortunately, there are S3A jUnit test failures when run with the HTTP proxy
(see Environment field above).
--
Details:
* steps to reproduce:
cd hadoop_git/hadoop-tools/hadoop-aws
mvn clean test -Dtest=TestS3AConfiguration
[...]
Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
Tests run: 5, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 2.737 sec
<<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
TestAutomaticProxyPortSelection(org.apache.hadoop.fs.s3a.TestS3AConfiguration)
Time elapsed: 0.737 sec <<< FAILURE!
java.lang.AssertionError: Expected a connection error for proxy server
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.fs.s3a.TestS3AConfiguration.TestAutomaticProxyPortSelection(TestS3AConfiguration.java:130)
TestProxyPortWithoutHost(org.apache.hadoop.fs.s3a.TestS3AConfiguration)
Time elapsed: 0.624 sec <<< ERROR!
com.amazonaws.AmazonClientException: Unable to execute HTTP request:
Connection to http://127.0.0.1:1 refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:384)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
at
com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
at
com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
at
org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:51)
at
org.apache.hadoop.fs.s3a.TestS3AConfiguration.TestProxyPortWithoutHost(TestS3AConfiguration.java:111)
Results :
Failed tests:
TestS3AConfiguration.TestAutomaticProxyPortSelection:130 Expected a
connection error for proxy server
Tests in error:
TestS3AConfiguration.TestProxyPortWithoutHost:111 ยป AmazonClient Unable to
exe...
Tests run: 5, Failures: 1, Errors: 1, Skipped: 0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
