Hi xuanwo, thanks for the suggestion. We are talking about getting rid of the middle layer, and use COSN directly.
So if OpenDAL has no difference to COSN, it is better to use it directly. Best, Jingsong 刘金辉 <965147...@qq.com.invalid>于2025年1月15日 周三19:02写道： > hi，Jingsong teacher. > Here is some information，，please check thanks > Optimization methods of cosn: > 1. Performance optimization: > a. Combined with the architecture of COS fusion bucket, separate metadata > flow and data flow to achieve high-performance file semantic access > b. Based on COS distributed metadata engine, provide high-performance > metadata access > c. Provide high-concurrency data flow through pre-reading strategy, block > processing and other means > 2. Integrate Tencent Cloud service role authentication capabilities to > improve access security > 3. Integrate Tencent Cloud log reporting capabilities, support task-level > log reporting, and improve observability > > > > > > S3 Architecture: > > > > > > > > > > > Cosn Architecture > > liujinhui > > > > > 刘金辉 > 965147...@qq.com > > > > > > > > > ------------------ 原始邮件 ------------------ > 发件人: > "dev" > < > xua...@apache.org>; > 发送时间: 2025年1月9日(星期四) 下午4:20 > 收件人: "dev"<dev@paimon.apache.org>; > > 主题: Re: [DISCUSS]paimon support tencent cosn > > > > Hi, > > That's interesting to come across this discussion. Apache OpenDAL offers > native support for COS (although we primarily utilize it for native IAM > support and are not currently aware of any performance gaps). Perhaps we > could consider adding a new file IO class based on OpenDAL to support this? > > Additionally, OpenDAL provides native support for nearly all storage > services, ranging from S3, Gcs to Azblob, which could be a valuable > addition for us. > > For more information about OpenDAL: https://github.com/apache/opendal > We also offer a native Java binding: > https://github.com/apache/opendal/tree/main/bindings/java > > On Thu, Jan 9, 2025, at 11:05, Jingsong Li wrote: > > Hi jinhui, > > > > Thanks for driving this, can you provide some information about "When > > using the s3 protocol to access the fusion bucket, the traffic will > > pass through the access layer twice", is there an official website? > > > > Best, > > Jingsong > > > > On Thu, Jan 9, 2025 at 10:56 AM 刘金辉 <965147...@qq.com.invalid> > wrote: > >> > >> Introduction > >> Tencent COS is a famous object storage system provided by Tencent > Corp. Hadoop-COS is a client that makes the upper computing systems based > on HDFS be able to use the COS as its underlying storage system. tencent > cloud is top 2 cloud vendors in China market and the object store COS （ > https://intl.cloud.tencent.com/product/cos） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage&nbsp; > >> > >> > >> We can use S3 SDK and COSN SDK to access COSN，but > >> 1. When using the s3 protocol to access the fusion bucket, the > traffic will pass through the access layer twice, the traffic will be > doubled, and the performance will be impaired. This problem will not occur > when using cosn. > >> 2. Compared with cos ordinary buckets, fusion buckets have > performance advantages in list, rename, truncate and other operations. > >> 3. If you must use the s3 protocol to access cos, we do not > recommend using fusion buckets, then the s3 protocol + cos ordinary bucket > will encounter some of the above performance issues. > >> 4. Other data lake frameworks also support access to cosn, oss, > S3, etc. > >> > >> > >> > >> > >> Therefore, in order to allow more users to use paimon with high > performance, it is necessary to support cosn access to paimon. > >> > >> > >> > >> > >> > >> > >> Please give your opinions. thanks > >> > >> > >> > >> > >> > >> 刘金辉 > >> 965147...@qq.com > >> > >> > >> > >> &nbsp; > > -- > Xuanwo > > https://xuanwo.io/
