Hi, jinhui. Very thanks for explaining the difference between s3 and cosn. I asked the Tencent Cloud and it is indeed. So just go head for it.
Best, tanjialiang. Jingsong Li <jingsongl...@gmail.com> 于2025年1月15日周三 22:27写道： > Hi, jinhui, > > Thanks for your information. > > I understood the difference. > > It is time to do vote if nobody has more concerns. > > Best, > Jingsong > > Jingsong Li <jingsongl...@gmail.com>于2025年1月15日 周三22:23写道： > > > Hi xuanwo, thanks for the suggestion. > > > > We are talking about getting rid of the middle layer, and use COSN > > directly. > > > > So if OpenDAL has no difference to COSN, it is better to use it directly. > > > > Best, > > Jingsong > > > > 刘金辉 <965147...@qq.com.invalid>于2025年1月15日 周三19:02写道： > > > >> hi，Jingsong teacher. > >> Here is some information，，please check thanks > >> Optimization methods of cosn: > >> 1. Performance optimization: > >> a. Combined with the architecture of COS fusion bucket, separate > metadata > >> flow and data flow to achieve high-performance file semantic access > >> b. Based on COS distributed metadata engine, provide high-performance > >> metadata access > >> c. Provide high-concurrency data flow through pre-reading strategy, > block > >> processing and other means > >> 2. Integrate Tencent Cloud service role authentication capabilities to > >> improve access security > >> 3. Integrate Tencent Cloud log reporting capabilities, support > task-level > >> log reporting, and improve observability > >> > >> > >> > >> > >> > >> S3 Architecture: > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> Cosn Architecture > >> > >> liujinhui > >> > >> > >> > >> > >> 刘金辉 > >> 965147...@qq.com > >> > >> > >> > >> > >> > >> > >> > >> > >> ------------------ 原始邮件 ------------------ > >> 发件人: > >> "dev" > >> < > >> xua...@apache.org>; > >> 发送时间: 2025年1月9日(星期四) 下午4:20 > >> 收件人: "dev"<dev@paimon.apache.org>; > >> > >> 主题: Re: [DISCUSS]paimon support tencent cosn > >> > >> > >> > >> Hi, > >> > >> That's interesting to come across this discussion. Apache OpenDAL offers > >> native support for COS (although we primarily utilize it for native IAM > >> support and are not currently aware of any performance gaps). Perhaps we > >> could consider adding a new file IO class based on OpenDAL to support > this? > >> > >> Additionally, OpenDAL provides native support for nearly all storage > >> services, ranging from S3, Gcs to Azblob, which could be a valuable > >> addition for us. > >> > >> For more information about OpenDAL: https://github.com/apache/opendal > >> We also offer a native Java binding: > >> https://github.com/apache/opendal/tree/main/bindings/java > >> > >> On Thu, Jan 9, 2025, at 11:05, Jingsong Li wrote: > >> > Hi jinhui, > >> > > >> > Thanks for driving this, can you provide some information about > "When > >> > using the s3 protocol to access the fusion bucket, the traffic will > >> > pass through the access layer twice", is there an official website? > >> > > >> > Best, > >> > Jingsong > >> > > >> > On Thu, Jan 9, 2025 at 10:56 AM 刘金辉 <965147...@qq.com.invalid> > >> wrote: > >> >> > >> >> Introduction > >> >> Tencent COS is a famous object storage system provided by > >> Tencent Corp. Hadoop-COS is a client that makes the upper computing > systems > >> based on HDFS be able to use the COS as its underlying storage system. > >> tencent cloud is top 2 cloud vendors in China market and the object > store > >> COS （https://intl.cloud.tencent.com/product/cos） is widely used among > >> China’s cloud users but now it is hard for hadoop user to access data > laid > >> on COS storage&nbsp; > >> >> > >> >> > >> >> We can use S3 SDK and COSN SDK to access COSN，but > >> >> 1. When using the s3 protocol to access the fusion bucket, the > >> traffic will pass through the access layer twice, the traffic will be > >> doubled, and the performance will be impaired. This problem will not > occur > >> when using cosn. > >> >> 2. Compared with cos ordinary buckets, fusion buckets have > >> performance advantages in list, rename, truncate and other operations. > >> >> 3. If you must use the s3 protocol to access cos, we do not > >> recommend using fusion buckets, then the s3 protocol + cos ordinary > bucket > >> will encounter some of the above performance issues. > >> >> 4. Other data lake frameworks also support access to cosn, oss, > >> S3, etc. > >> >> > >> >> > >> >> > >> >> > >> >> Therefore, in order to allow more users to use paimon with high > >> performance, it is necessary to support cosn access to paimon. > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> Please give your opinions. thanks > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> 刘金辉 > >> >> 965147...@qq.com > >> >> > >> >> > >> >> > >> >> &nbsp; > >> > >> -- > >> Xuanwo > >> > >> https://xuanwo.io/ > > > > >
