[
https://issues.apache.org/jira/browse/HADOOP-19762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18047724#comment-18047724
]
ASF GitHub Bot commented on HADOOP-19762:
-----------------------------------------
cnauroth opened a new pull request, #8147:
URL: https://github.com/apache/hadoop/pull/8147
### Description of PR
When we introduced hadoop-gcp, we discussed whether this belongs under
hadoop-cloud-storage-project or hadoop-tools in a comment chain starting here:
https://issues.apache.org/jira/browse/HADOOP-19343?focusedCommentId=17941734&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17941734
We chose hadoop-tools, but in retrospect, hadoop-cloud-storage-project is
really the right choice. hadoop-aws and hadoop-azure remain under hadoop-tools
mostly for legacy reasons. Moving them under hadoop-cloud-storage-project would
cause a lot of maintenance burden while backporting patches to prior versions.
For a new, unreleased cloud `FileSystem`, we have the opportunity to ship it in
hadoop-cloud-storage-project from the start.
1. `git mv hadoop-tools/hadoop-gcp hadoop-cloud-storage-project`
2. Remove hadoop-gcp module from hadoop-tools/pom.xml and add to
hadoop-cloud-storage-project/pom.xml.
3. Small modification in hadoop-gcp/pom.xml to the way
`maven-dependency-plugin:list` is called for shell profile generation.
4. Reorder the module list in the top-level pom.xml so that hadoop-dist is
always last. Otherwise, hadoop-dist runs the dist-layout-stitching script
before hadoop-gcp has run its shading in the package phase, so it accidentally
picks up the unshaded jar.
### How was this patch tested?
1. Create a distro including hadoop-gcp:
```
mvn clean package -Pdist -Dtar -Dhadoop-gcp-package -DskipTests
```
2. Extract:
```
tar xf ~/hadoop/hadoop-dist/target/hadoop-3.5.0-SNAPSHOT.tar.gz
```
3. Confirm inclusion of hadoop-gcp.jar:
```
ls -lrth
hadoop-3.5.0-SNAPSHOT/share/hadoop/common/lib/hadoop-gcp-3.5.0-SNAPSHOT.jar
-rw-r--r
> Move hadoop-gcp from hadoop-tools to hadoop-cloud-storage-project
> -----------------------------------------------------------------
>
> Key: HADOOP-19762
> URL: https://issues.apache.org/jira/browse/HADOOP-19762
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/gcs
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Priority: Major
>
> When we introduced hadoop-gcp, we discussed whether this belongs under
> hadoop-cloud-storage-project or hadoop-tools in a comment chain starting here:
> https://issues.apache.org/jira/browse/HADOOP-19343?focusedCommentId=17941734&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17941734
> We chose hadoop-tools, but in retrospect, hadoop-cloud-storage-project is
> really the right choice. hadoop-aws and hadoop-azure remain under
> hadoop-tools mostly for legacy reasons. Moving them under
> hadoop-cloud-storage-project would cause a lot of maintenance burden while
> backporting patches to prior versions. For a new, unreleased cloud
> {{FileSystem}}, we have the opportunity to ship it in
> hadoop-cloud-storage-project from the start.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
