hi.all?? Happy New Year to all the friends in the paimon community??start a vote on Paimon to support Tencent Cloud Cosn??
Introduction Tencent COS is a famous object storage system provided by Tencent Corp. Hadoop-COS is a client that makes the upper computing systems based on HDFS be able to use the COS as its underlying storage system. tencent cloud is top 2 cloud vendors in China market and the object store COS ??https://intl.cloud.tencent.com/product/cos?? is widely used among China??s cloud users but now it is hard for hadoop user to access data laid on COS storage We can use S3 SDK and COSN SDK to access COSN??but 1. When using the s3 protocol to access the fusion bucket, the traffic will pass through the access layer twice, the traffic will be doubled, and the performance will be impaired. This problem will not occur when using cosn. 2. Compared with cos ordinary buckets, fusion buckets have performance advantages in list, rename, truncate and other operations. 3. If you must use the s3 protocol to access cos, we do not recommend using fusion buckets, then the s3 protocol + cos ordinary bucket will encounter some of the above performance issues. 4. Other data lake frameworks also support access to cosn, oss, S3, etc. Optimization methods of cosn: 1. Performance optimization: a. Combined with the architecture of COS fusion bucket, separate metadata flow and data flow to achieve high-performance file semantic access b. Based on COS distributed metadata engine, provide high-performance metadata access c. Provide high-concurrency data flow through pre-reading strategy, block processing and other means 2. Integrate Tencent Cloud service role authentication capabilities to improve access security 3. Integrate Tencent Cloud log reporting capabilities, support task-level log reporting, and improve observability S3 Architecture: Cosn Architecture best??liujinhui The vote will be open for at least 72 hours or until necessary numbers of votes are reached. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove with the reason ?????? 965147...@qq.com
