zhouaoe opened a new pull request, #7205: URL: https://github.com/apache/hadoop/pull/7205
### Description of PR AliyunOSS is about to adjust its security policy: only V4 signature requests will be supported in the public cloud. Therefore, support for V4 signatures is also required in Hadoop, and V4 signatures will be the default. Change 1: Upgraded the OSS SDK version to 3.18.1 to support V4 authentication, while also fixing some bugs inherent in the SDK. Change 2: Added configuration options to enable V4 authentication support in the hadoop-oss-connector. ### How was this patch tested? Integrated testing was conducted to test V4 authentication. A new test, ITAliyunOSSSignatureV4, is added, which verifies the authentication method using V4. Additionally, it confirms that V1 authentication is still used by default. ```shell mvn test -Dtest=org.apache.hadoop.fs.aliyun.oss.ITAliyunOSSSignatureV4 -pl hadoop-tools/hadoop-aliyun [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.apache.hadoop.fs.aliyun.oss.ITAliyunOSSSignatureV4 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.934 s - in org.apache.hadoop.fs.aliyun.oss.ITAliyunOSSSignatureV4 [INFO] [INFO] Results: [INFO] [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4.655 s [INFO] Finished at: 2024-12-03T09:01:00Z [INFO] ------------------------------------------------------------------------ ``` Other Aliyun oss integration tests are also executed. ```shell mvn test -pl hadoop-tools/hadoop-aliyun [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSBlockOutputStream [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.258 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSBlockOutputStream [INFO] Running org.apache.hadoop.fs.aliyun.oss.yarn.TestOSS [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.808 s - in org.apache.hadoop.fs.aliyun.oss.yarn.TestOSS [INFO] Running org.apache.hadoop.fs.aliyun.oss.yarn.TestOSSMiniYarnCluster [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.128 s - in org.apache.hadoop.fs.aliyun.oss.yarn.TestOSSMiniYarnCluster [INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [INFO] Tests run: 53, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.578 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemStore [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.816 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemStore [INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContext [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.262 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContext [INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextCreateMkdir [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.784 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextCreateMkdir [INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextMainOperations [WARNING] Tests run: 73, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 16.198 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextMainOperations [INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextStatistics [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.682 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextStatistics [INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextUtil [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.23 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextUtil [INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextURI [WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.146 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextURI [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDelete [WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.158 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDelete [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractOpen [WARNING] Tests run: 19, Failures: 0, Errors: 0, Skipped: 19, Time elapsed: 0.201 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractOpen [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractSeek [WARNING] Tests run: 19, Failures: 0, Errors: 0, Skipped: 19, Time elapsed: 0.204 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractSeek [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractGetFileStatus [WARNING] Tests run: 20, Failures: 0, Errors: 0, Skipped: 20, Time elapsed: 0.197 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractGetFileStatus [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDistCp [WARNING] Tests run: 13, Failures: 0, Errors: 0, Skipped: 13, Time elapsed: 0.179 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDistCp [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractMkdir [WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.164 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractMkdir [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRootDir [WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 9, Time elapsed: 0.164 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRootDir [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRename [WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 0.167 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRename [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractCreate [WARNING] Tests run: 16, Failures: 0, Errors: 0, Skipped: 16, Time elapsed: 0.182 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractCreate [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractGetFileStatusV1List [WARNING] Tests run: 20, Failures: 0, Errors: 0, Skipped: 20, Time elapsed: 0.199 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractGetFileStatusV1List [INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSInputStream [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.352 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSInputStream [INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunCredentials [WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.131 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunCredentials [INFO] [INFO] Results: [INFO] [WARNING] Tests run: 328, Failures: 0, Errors: 0, Skipped: 150 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:12 min [INFO] Finished at: 2024-12-03T08:15:52Z [INFO] ------------------------------------------------------------------------ ``` ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
