Hi, That's interesting to come across this discussion. Apache OpenDAL offers native support for COS (although we primarily utilize it for native IAM support and are not currently aware of any performance gaps). Perhaps we could consider adding a new file IO class based on OpenDAL to support this?
Additionally, OpenDAL provides native support for nearly all storage services, ranging from S3, Gcs to Azblob, which could be a valuable addition for us. For more information about OpenDAL: https://github.com/apache/opendal We also offer a native Java binding: https://github.com/apache/opendal/tree/main/bindings/java On Thu, Jan 9, 2025, at 11:05, Jingsong Li wrote: > Hi jinhui, > > Thanks for driving this, can you provide some information about "When > using the s3 protocol to access the fusion bucket, the traffic will > pass through the access layer twice", is there an official website? > > Best, > Jingsong > > On Thu, Jan 9, 2025 at 10:56 AM 刘金辉 <965147...@qq.com.invalid> wrote: >> >> Introduction >> Tencent COS is a famous object storage system provided by Tencent Corp. >> Hadoop-COS is a client that makes the upper computing systems based on HDFS >> be able to use the COS as its underlying storage system. tencent cloud is >> top 2 cloud vendors in China market and the object store COS >> （https://intl.cloud.tencent.com/product/cos） is widely used among China’s >> cloud users but now it is hard for hadoop user to access data laid on COS >> storage >> >> >> We can use S3 SDK and COSN SDK to access COSN，but >> 1. When using the s3 protocol to access the fusion bucket, the traffic will >> pass through the access layer twice, the traffic will be doubled, and the >> performance will be impaired. This problem will not occur when using cosn. >> 2. Compared with cos ordinary buckets, fusion buckets have performance >> advantages in list, rename, truncate and other operations. >> 3. If you must use the s3 protocol to access cos, we do not recommend using >> fusion buckets, then the s3 protocol + cos ordinary bucket will encounter >> some of the above performance issues. >> 4. Other data lake frameworks also support access to cosn, oss, S3, etc. >> >> >> >> >> Therefore, in order to allow more users to use paimon with high performance, >> it is necessary to support cosn access to paimon. >> >> >> >> >> >> >> Please give your opinions. thanks >> >> >> >> >> >> 刘金辉 >> 965147...@qq.com >> >> >> >> -- Xuanwo https://xuanwo.io/
