voonhous commented on code in PR #19423: URL: https://github.com/apache/hudi/pull/19423#discussion_r3877842134
########## hudi-notebooks/README.md: ########## @@ -79,6 +79,40 @@ This project provides a ready-to-use Docker Compose environment for running Apac - Spark, Hive, and Hudi configs are in `conf/` and automatically copied into containers. - S3 access keys and endpoints are set for MinIO and referenced in Spark/Hive configs. +### Using another S3-compatible object store + +Hudi reads and writes through the Hadoop S3A connector, so the same `fs.s3a.*` +settings work against any S3-compatible object store: Amazon S3, or a compatible +provider such as Backblaze B2, Cloudflare R2, or MinIO. This demo points +`fs.s3a.endpoint` at the bundled MinIO service; to target a different store, +edit `conf/hadoop/core-site.xml` (and the matching Spark/Hive configs) with that +provider's endpoint and credentials: + +```xml +<property> + <name>fs.s3a.endpoint</name> + <!-- Amazon S3: https://s3.<region>.amazonaws.com + any other provider: its S3 endpoint URL --> + <value>https://<s3-endpoint></value> +</property> Review Comment: Done in 6b4a280: the three values are plain placeholder text now (https://your-s3-endpoint.example.com, ACCESS_KEY, SECRET_KEY) and the block parses as XML. ########## hudi-notebooks/README.md: ########## @@ -79,6 +79,40 @@ This project provides a ready-to-use Docker Compose environment for running Apac - Spark, Hive, and Hudi configs are in `conf/` and automatically copied into containers. - S3 access keys and endpoints are set for MinIO and referenced in Spark/Hive configs. +### Using another S3-compatible object store + +Hudi reads and writes through the Hadoop S3A connector, so the same `fs.s3a.*` +settings work against any S3-compatible object store: Amazon S3, or a compatible +provider such as Backblaze B2, Cloudflare R2, or MinIO. This demo points +`fs.s3a.endpoint` at the bundled MinIO service; to target a different store, +edit `conf/hadoop/core-site.xml` (and the matching Spark/Hive configs) with that +provider's endpoint and credentials: + +```xml +<property> + <name>fs.s3a.endpoint</name> + <!-- Amazon S3: https://s3.<region>.amazonaws.com + any other provider: its S3 endpoint URL --> + <value>https://<s3-endpoint></value> +</property> +<property> + <name>fs.s3a.access.key</name> + <value><access-key></value> +</property> +<property> + <name>fs.s3a.secret.key</name> + <value><secret-key></value> +</property> +<property> + <name>fs.s3a.path.style.access</name> + <value>true</value> Review Comment: Agreed on the inconsistency: 6b4a280 adds a comment on that property saying MinIO and most other providers need it and Amazon S3 does not, mirroring the comment on the endpoint property. One correction: Amazon S3 still accepts path-style requests, so leaving it at true there is unnecessary rather than broken; the comment says to leave it out. ########## hudi-notebooks/README.md: ########## @@ -79,6 +79,40 @@ This project provides a ready-to-use Docker Compose environment for running Apac - Spark, Hive, and Hudi configs are in `conf/` and automatically copied into containers. - S3 access keys and endpoints are set for MinIO and referenced in Spark/Hive configs. +### Using another S3-compatible object store + +Hudi reads and writes through the Hadoop S3A connector, so the same `fs.s3a.*` +settings work against any S3-compatible object store: Amazon S3, or a compatible +provider such as Backblaze B2, Cloudflare R2, or MinIO. This demo points +`fs.s3a.endpoint` at the bundled MinIO service; to target a different store, +edit `conf/hadoop/core-site.xml` (and the matching Spark/Hive configs) with that +provider's endpoint and credentials: + +```xml +<property> + <name>fs.s3a.endpoint</name> + <!-- Amazon S3: https://s3.<region>.amazonaws.com + any other provider: its S3 endpoint URL --> + <value>https://<s3-endpoint></value> +</property> Review Comment: Runnable as written. The images ship Hadoop 3.3.4 (Dockerfile.spark), 3.4.1 (Dockerfile.spark4) and hadoop-aws 3.1.0 (Dockerfile.hive), and the bundled conf/hadoop/core-site.xml already uses a scheme-prefixed endpoint (http://minio:9000) against all of them. On SDK v1, DefaultS3ClientFactory.createEndpointConfiguration passes fs.s3a.endpoint through RuntimeHttpUtils.toUri, which takes a full URI and only prepends a scheme when there is none; SDK v2's getS3Endpoint does the same. No change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
