[
https://issues.apache.org/jira/browse/HADOOP-17337?focusedWorklogId=532657&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532657
]
ASF GitHub Bot logged work on HADOOP-17337:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 07/Jan/21 18:27
Start Date: 07/Jan/21 18:27
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2599:
URL: https://github.com/apache/hadoop/pull/2599#issuecomment-756296069
> is it not possible, if not reasonable, to have both by adding a new class
that couples to the unshaded class in tandem to the current proposed class that
couples to the shaded? this seems like a reasonable tax on the choice to use a
shaded library and coupling to its internals.
yes, if someone plans to do it. They'd need to do 100% reflection binding
(this code compiles against the shaded one) and presumably there'd be e a
binding mechanism which switches between them.
I'm not going to go near this. What this patch tries to do is make your
deployments work.
As to why we use the shaded library, you have to look at the dependencies of
aws sdk, version problems we've had with json parsers, httpclient libs and the
like to understand we've got good reason to say "use the shaded SDK". There's
an implicit "all issues which can be fixed by "move to shaded AWS SDK" will be
closed as wontfix".
We've chosen to use the shaded SDK to make those version problems go away.
It's heavy, but makes versions manageable. I don't want to force people onto
going onto the unshaded JARs if they really want to (hence this patch), but I'm
not personally worried about the fact there may be a performance impact. Sorry.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 532657)
Time Spent: 1h 10m (was: 1h)
> NetworkBinding has a runtime class dependency on a third-party shaded class
> ---------------------------------------------------------------------------
>
> Key: HADOOP-17337
> URL: https://issues.apache.org/jira/browse/HADOOP-17337
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Chris Wensel
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The hadoop-aws library has a dependency on
> 'com.amazonaws':aws-java-sdk-bundle' which in turn is a fat jar of all AWS
> SDK libraries and shaded dependencies.
>
> This dependency is 181MB.
>
> Some applications using the S3AFilesystem may be sensitive to having a large
> footprint. For example, building an application using Parquet and bundled
> with Docker.
>
> Typically, in prior Hadoop versions, the bundle was replaced by the specific
> AWS SDK dependencies, dropping the overall footprint.
>
> In 3.3 (and maybe prior versions) this strategy does not work because of the
> following exception:
> {{java.lang.NoClassDefFoundError:
> com/amazonaws/thirdparty/apache/http/conn/socket/ConnectionSocketFactory}}
> {{ at
> org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1335)}}
> {{ at
> org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1290)}}
> {{ at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1247)}}
> {{ at
> org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:61)}}
> {{ at
> org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:644)}}
> {{ at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:390)}}
> {{ at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)}}
> {{ at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)}}
> {{ at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)}}
> {{ at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)}}
> {{ at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
