Archie-selfless commented on issue #6572:
URL: https://github.com/apache/hudi/issues/6572#issuecomment-1280713709
@yihua I'm really sorry for replying the current issue so late! With the
help of my colleagues, I completed the Hive-sync by accessing S3 protocol. I
think it's because the org.apache.hudi.hive.HiveSyncTool always fetch the
s3\s3a as protocol. My current core-site.xml is here:
```xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>cons://bigdata-1252394470</value>
<description>COS bucket name</description>
</property>
<property>
<name>fs.cosn.userinfo.secretId</name>
<value>XXXXXXXX</value>
<description>Tencent Cloud Secret Id</description>
</property>
<property>
<name>fs.cosn.userinfo.secretKey</name>
<value>XXXXXXXX</value>
<description>Tencent Cloud Secret Key</description>
</property>
<property>
<name>fs.cosn.bucket.region</name>
<value>ap-guangzhou</value>
<description>The region where the bucket is located.</description>
</property>
<property>
<name>fs.cosn.bucket.endpoint_suffix</name>
<value>cos.ap-guangzhou.myqcloud.com</value>
<description>
COS endpoint to connect to.
For public cloud users, it is recommended not to set this
option, and only the correct area field is required.
</description>
</property>
<property>
<name>fs.cosn.impl</name>
<value>org.apache.hadoop.fs.CosFileSystem</value>
<description>The implementation class of the CosN
Filesystem.</description>
</property>
<property>
<name>fs.AbstractFileSystem.cosn.impl</name>
<value>org.apache.hadoop.fs.CosN</value>
<description>The implementation class of the CosN
AbstractFileSystem.</description>
</property>
<!-- ############### If I add below params, everything goes well
############### -->
<property>
<name>fs.s3.impl</name>
<value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>
<property>
<name>fs.s3a.impl</name>
<value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>
<property>
<name>fs.s3n.impl</name>
<value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
</property>
<property>
<name>fs.s3.endpoint</name>
<value>cos.ap-guangzhou.myqcloud.com</value>
</property>
<property>
<name>fs.s3a.endpoint</name>
<value>cos.ap-guangzhou.myqcloud.com</value>
</property>
<property>
<name>fs.s3n.endpoint</name>
<value>cos.ap-guangzhou.myqcloud.com</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<value>XXXXXXXX</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<value>XXXXXXXX</value>
</property>
<property>
<name>fs.s3a.aws.credentials.provider</name>
<value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
</property>
</configuration>
```
**Main Summary**: org.apache.hudi.hive.HiveSyncTool always fetch the s3\s3a
as protocol.(Based Hudi-0.12.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
