[
https://issues.apache.org/jira/browse/HADOOP-19696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-19696:
------------------------------------
Description:
Place all the cloud connector hadoop-* artifacts and dependencies into
hadoop/common/lib so that the stores can be directly accessed.
* filesystem operations against abfs, s3a, gcs, etc don't need any effort
setting things up.
* Releases without the aws bundle.jar can be trivially updated by adding any
version of the sdk libraries to the common/lib dir.
This adds a lot more stuff into the distribution, so I'm doing the following
design
* all hadoop-* modules in common/lib
* minimal dependencies for hadoop-azure and hadoop-gcs (once we get those
right!)
* hadoop-aws: everything except bundle.jar
* other connectors: only included with explicit profiles.
ASF releases will support azure, gcs out the box, the others once you add the
dependencies. And anyone can build their own release with everything
One concern here, we make hadoop-cloud-storage artifact incomplete at pulling
in things when depended on. We may need a separate module for the distro setup.
Noticed during this that the hadoop-tos component is shaded and includes stuff
(httpclient5) that we need under control. Filed HADOOP-19708 and incorporating
here.
was:
Place all the cloud connector hadoop-* artifacts and dependencies into
hadoop/common/lib so that the stores can be directly accessed.
* filesystem operations against abfs, s3a, gcs, etc don't need any effort
setting things up.
* Releases without the aws bundle.jar can be trivially updated by adding any
version of the sdk libraries to the common/lib dir.
> hadoop binary distribution to move cloud connectors to hadoop common/lib
> ------------------------------------------------------------------------
>
> Key: HADOOP-19696
> URL: https://issues.apache.org/jira/browse/HADOOP-19696
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, fs/gcs, fs/huawei, fs/s3
> Affects Versions: 3.4.2
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Place all the cloud connector hadoop-* artifacts and dependencies into
> hadoop/common/lib so that the stores can be directly accessed.
> * filesystem operations against abfs, s3a, gcs, etc don't need any effort
> setting things up.
> * Releases without the aws bundle.jar can be trivially updated by adding any
> version of the sdk libraries to the common/lib dir.
> This adds a lot more stuff into the distribution, so I'm doing the following
> design
> * all hadoop-* modules in common/lib
> * minimal dependencies for hadoop-azure and hadoop-gcs (once we get those
> right!)
> * hadoop-aws: everything except bundle.jar
> * other connectors: only included with explicit profiles.
> ASF releases will support azure, gcs out the box, the others once you add the
> dependencies. And anyone can build their own release with everything
> One concern here, we make hadoop-cloud-storage artifact incomplete at pulling
> in things when depended on. We may need a separate module for the distro
> setup.
> Noticed during this that the hadoop-tos component is shaded and includes
> stuff (httpclient5) that we need under control. Filed HADOOP-19708 and
> incorporating here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
