cnauroth opened a new pull request, #8147: URL: https://github.com/apache/hadoop/pull/8147
### Description of PR When we introduced hadoop-gcp, we discussed whether this belongs under hadoop-cloud-storage-project or hadoop-tools in a comment chain starting here: https://issues.apache.org/jira/browse/HADOOP-19343?focusedCommentId=17941734&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17941734 We chose hadoop-tools, but in retrospect, hadoop-cloud-storage-project is really the right choice. hadoop-aws and hadoop-azure remain under hadoop-tools mostly for legacy reasons. Moving them under hadoop-cloud-storage-project would cause a lot of maintenance burden while backporting patches to prior versions. For a new, unreleased cloud `FileSystem`, we have the opportunity to ship it in hadoop-cloud-storage-project from the start. 1. `git mv hadoop-tools/hadoop-gcp hadoop-cloud-storage-project` 2. Remove hadoop-gcp module from hadoop-tools/pom.xml and add to hadoop-cloud-storage-project/pom.xml. 3. Small modification in hadoop-gcp/pom.xml to the way `maven-dependency-plugin:list` is called for shell profile generation. 4. Reorder the module list in the top-level pom.xml so that hadoop-dist is always last. Otherwise, hadoop-dist runs the dist-layout-stitching script before hadoop-gcp has run its shading in the package phase, so it accidentally picks up the unshaded jar. ### How was this patch tested? 1. Create a distro including hadoop-gcp: ``` mvn clean package -Pdist -Dtar -Dhadoop-gcp-package -DskipTests ``` 2. Extract: ``` tar xf ~/hadoop/hadoop-dist/target/hadoop-3.5.0-SNAPSHOT.tar.gz ``` 3. Confirm inclusion of hadoop-gcp.jar: ``` ls -lrth hadoop-3.5.0-SNAPSHOT/share/hadoop/common/lib/hadoop-gcp-3.5.0-SNAPSHOT.jar -rw-r--r-- 1 cnauroth cnauroth 34M Dec 26 23:09 hadoop-3.5.0-SNAPSHOT/share/hadoop/common/lib/hadoop-gcp-3.5.0-SNAPSHOT.jar ``` 4. Test a command accessing GCS: ``` bin/hadoop fs \ -D fs.gs.auth.type=SERVICE_ACCOUNT_JSON_KEYFILE \ -D fs.gs.auth.service.account.json.keyfile=/usr/local/google/home/cnauroth/cnauroth-test.json \ -ls gs://cnauroth-hive-metastore-proxy-dist ``` 5. Run hadoop-gcp integration tests: ``` mvn verify ``` ### For code changes: - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [X] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
