cccs-mjs commented on issue #3546: URL: https://github.com/apache/iceberg/issues/3546#issuecomment-1059284873
> By the way, another issue with using Hadoop catalog with S3 is that S3 does not support atomic file write, but Hadoop catalog relies on the atomic write of a table version hint file to ensure transaction. So it is definitely not recommended for production usage. In the Iceberg docs for Spark Configuration (https://iceberg.apache.org/docs/latest/spark-configuration/#using-catalog-specific-hadoop-configuration-values) it shows using a Hadoop catalog with s3. Is that configuration also not recommended for production usage? > spark.sql.catalog.hadoop_prod.hadoop.fs.s3a.endpoint = http://aws-local:9000 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
