[
https://issues.apache.org/jira/browse/HADOOP-19083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17955674#comment-17955674
]
Steve Loughran commented on HADOOP-19083:
-----------------------------------------
bq. since all the 3rd party libraries AWS includes in the bundle are shaded
without being relocated.
Not sure what you mean here. All 100% of external dependencies should be under
software.amazon.awssdk.thirdparty. And looking at 2.29.52, all I can see is a
org.reactivestreams package in the wrong place
that's the main reason we use bundle.jar. Yes its huge. Yes it hurts download
and docker launch times. Yes there's lots of stuff we don't need. But it
completely eliminates all classpath nightmares. And we do try to make sure that
if you strip your own deployments down then there's nothing in the production
code which fails to link.
Now, there are past jiras related to somehow doing a lean dependency JAR, but
nothing has progressed. You can look for them `-but really this is complex
enough to start discussing on the hadoop common dev list.
> provide hadoop binary tarball without aws v2 sdk
> ------------------------------------------------
>
> Key: HADOOP-19083
> URL: https://issues.apache.org/jira/browse/HADOOP-19083
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.1
>
>
> Have the default hadoop binary .tar.gz exclude the aws v2 sdk by default.
> This SDK brings the total size of the distribution to about 1 GB.
> Proposed
> * add a profile to include the aws sdk in the dist module
> * document it for local building
> * for release builds, we modify our release ant builds to generate modified
> x86 and arm64 releases without the file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
