[
https://issues.apache.org/jira/browse/HADOOP-17337?focusedWorklogId=532505&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532505
]
ASF GitHub Bot logged work on HADOOP-17337:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 07/Jan/21 14:02
Start Date: 07/Jan/21 14:02
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2599:
URL: https://github.com/apache/hadoop/pull/2599#issuecomment-756134591
if it's not found you don't get any attempts to patch in the wildfly/openssl
library. HTTPS connections may be slower.
You will stay with the java https factory. I think you may retain the
inefficient algorithm which came with java8; there's a bit of fixup in the
hadoop-common lib to strip out one of them.
Other than that, if you have wildfly and openssl on the CP, you won't see
any speedup, because you won't get that native library binding
that wildfly/openssl binding has been a PITA; very brittle. I don't want to
go near it if I can help it, and doing the fixup for unshaded JARs will add a
codepath which will not get tested during normal builds. We have enough
problems that the JAR and various openssl releases have been incompatible,
making it brittle on some linux images. The fact that an azure SDK kept the old
widfly classes hidden unshaded in its JAR made this very hard to eliminate as
source of stack traces.
Given you want to use the unshaded JARs for lighter images, I think you'll
have to accept that a price of that is "no speedup from native openssl". That's
all. What is important is: you won't get linkage errors any more
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 532505)
Time Spent: 50m (was: 40m)
> NetworkBinding has a runtime class dependency on a third-party shaded class
> ---------------------------------------------------------------------------
>
> Key: HADOOP-17337
> URL: https://issues.apache.org/jira/browse/HADOOP-17337
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Chris Wensel
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The hadoop-aws library has a dependency on
> 'com.amazonaws':aws-java-sdk-bundle' which in turn is a fat jar of all AWS
> SDK libraries and shaded dependencies.
>
> This dependency is 181MB.
>
> Some applications using the S3AFilesystem may be sensitive to having a large
> footprint. For example, building an application using Parquet and bundled
> with Docker.
>
> Typically, in prior Hadoop versions, the bundle was replaced by the specific
> AWS SDK dependencies, dropping the overall footprint.
>
> In 3.3 (and maybe prior versions) this strategy does not work because of the
> following exception:
> {{java.lang.NoClassDefFoundError:
> com/amazonaws/thirdparty/apache/http/conn/socket/ConnectionSocketFactory}}
> {{ at
> org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1335)}}
> {{ at
> org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1290)}}
> {{ at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1247)}}
> {{ at
> org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:61)}}
> {{ at
> org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:644)}}
> {{ at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:390)}}
> {{ at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)}}
> {{ at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)}}
> {{ at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)}}
> {{ at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)}}
> {{ at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
