[
https://issues.apache.org/jira/browse/HADOOP-19696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18039692#comment-18039692
]
ASF GitHub Bot commented on HADOOP-19696:
-----------------------------------------
steveloughran commented on PR #8094:
URL: https://github.com/apache/hadoop/pull/8094#issuecomment-3559789024
@pan3793 I understand your concerns but this is actually a packaging
improvement
The current build
* contains a tools/lib/bundle.jar for aws, which is too big for distribution
* doesn't put anyone elses cloud connector on the classpath. People doing
their own distros end up moving them or doing other classpath stuff.
* forces a release process where we have to take the full tar, untar it,
remove that bundle.tar, rebuild and sign again:
https://github.com/apache/hadoop-release-support/blob/main/build.xml#L1315
moving the hadoop-* cloud libs into common-lib but leaving out dependencies
means the stuff is in the right place, provided users manually add the
dependencies (which I'm not going to build with except for hadoop-azure as
that's the httpclient and wildfly libs we use elsewhere.
This makes releasing _easier_, and it makes adding the dependencies easier
as the current setup requires a user to add the specific bundle-jar version the
release was built with, same for the other components.
> hadoop binary distribution to move cloud connectors to hadoop common/lib
> ------------------------------------------------------------------------
>
> Key: HADOOP-19696
> URL: https://issues.apache.org/jira/browse/HADOOP-19696
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, fs/gcs, fs/huawei, fs/s3
> Affects Versions: 3.4.2
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Place all the cloud connector hadoop-* artifacts and dependencies into
> hadoop/common/lib so that the stores can be directly accessed.
> * filesystem operations against abfs, s3a, gcs, etc don't need any effort
> setting things up.
> * Releases without the aws bundle.jar can be trivially updated by adding any
> version of the sdk libraries to the common/lib dir.
> This adds a lot more stuff into the distribution, so I'm doing the following
> design
> * all hadoop-* modules in common/lib
> * minimal dependencies for hadoop-azure and hadoop-gcs (once we get those
> right!)
> * hadoop-aws: everything except bundle.jar
> * other connectors: only included with explicit profiles.
> ASF releases will support azure out the box, the others once you add the
> dependencies. And anyone can build their own release with everything
> One concern here, we make hadoop-cloud-storage artifact incomplete at pulling
> in things when depended on. We may need a separate module for the distro
> setup.
> Noticed during this that the hadoop-tos component is shaded and includes
> stuff (httpclient5) that we need under control. Filed HADOOP-19708 and
> incorporating here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
