openinx opened a new pull request #2230: URL: https://github.com/apache/iceberg/pull/2230
Made this pull request to support aliyun OSS ([Object Storage Service](https://www.alibabacloud.com/help/doc-detail/31947.htm?spm=a2c63.p38356.b99.978.7155466aSTYesW)). Aliyun OSS is a very popular object storage service among public could services, it's necessary to support it (similar to aws s3) in apache iceberg official repo (In fact I saw many asia users are asking whether there's an official support for aliyun oss). When I prepare this patch, the biggest obstacle is: How to write unit tests for the OSS+iceberg integration work. For aws s3, adobe create a great tool named [s3mock](https://github.com/adobe/S3Mock), saying it could just setup a local mini cluster that support aws s3 protocal, then we could easily write unit tests based that mini s3 clusters. For aliyun OSS services, we don't have that great tool (the official aliyun oss sdk also don't provide the mini cluster jar). I don't expect to commit plenty of codes with integration tests only but without unit tests. So I implemented a similar local oss mini clusters, it's similar to adobe's work but it's more lightweight because we don't want every API to be implemented in apache iceberg. Implementing the basic OSS API in a local mini cluster is enough, it takes hundreds lines of code to accomplish this. Currently, I've implemented all the iceberg FileIO's interfaces by aliyun oss sdk, and make both the unit tests and integration tests work well. * How to run the aliyun oss unit tests ? ``` ./gradlew :iceberg-aliyun:test ``` * How to run the aliyun oss integration tests ? ``` export OSS_TEST_RULE_CLASS_IMPL=org.apache.iceberg.aliyun.oss.OSSIntegrationTestRule export OSS_TEST_ENDPOINT=<aliyun-oss-endpoint> export OSS_TEST_ACCESS_KEY=<your-aliyun-oss-access-key> export OSS_TEST_ACCESS_SECRET=<your-aliyun-oss-access-secret-key> export OSS_TEST_BUCKET=<your-oss-bucket-name> export OSS_TEST_KEY_PREFIX=<your-oss-object-key-prefix>. # optional. ./gradlew :iceberg-aliyun:test ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
