hudi-agent commented on code in PR #19423: URL: https://github.com/apache/hudi/pull/19423#discussion_r3716153827
########## hudi-notebooks/README.md: ########## @@ -79,6 +79,40 @@ This project provides a ready-to-use Docker Compose environment for running Apac - Spark, Hive, and Hudi configs are in `conf/` and automatically copied into containers. - S3 access keys and endpoints are set for MinIO and referenced in Spark/Hive configs. +### Using another S3-compatible object store + +Hudi reads and writes through the Hadoop S3A connector, so the same `fs.s3a.*` +settings work against any S3-compatible object store: Amazon S3, or a compatible +provider such as Backblaze B2, Cloudflare R2, or MinIO. This demo points +`fs.s3a.endpoint` at the bundled MinIO service; to target a different store, +edit `conf/hadoop/core-site.xml` (and the matching Spark/Hive configs) with that +provider's endpoint and credentials: + +```xml +<property> + <name>fs.s3a.endpoint</name> + <!-- Amazon S3: https://s3.<region>.amazonaws.com + any other provider: its S3 endpoint URL --> + <value>https://<s3-endpoint></value> +</property> Review Comment: 🤖 The example uses a scheme-prefixed endpoint (`https://<s3-endpoint>` / `https://s3.<region>.amazonaws.com`). Whether `fs.s3a.endpoint` should include the `https://` scheme depends on the Hadoop/AWS SDK version bundled here: with the older AWS SDK v1 (Hadoop 3.3.x and earlier) `fs.s3a.endpoint` expected a bare host (e.g. `s3.<region>.amazonaws.com`) with TLS controlled by `fs.s3a.connection.ssl.enabled`, whereas AWS SDK v2 (Hadoop 3.4+) accepts a full URI. Could you confirm which Hadoop version the notebooks image ships and adjust the endpoint format accordingly so the example is runnable as written? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## hudi-notebooks/README.md: ########## @@ -79,6 +79,40 @@ This project provides a ready-to-use Docker Compose environment for running Apac - Spark, Hive, and Hudi configs are in `conf/` and automatically copied into containers. - S3 access keys and endpoints are set for MinIO and referenced in Spark/Hive configs. +### Using another S3-compatible object store + +Hudi reads and writes through the Hadoop S3A connector, so the same `fs.s3a.*` +settings work against any S3-compatible object store: Amazon S3, or a compatible +provider such as Backblaze B2, Cloudflare R2, or MinIO. This demo points +`fs.s3a.endpoint` at the bundled MinIO service; to target a different store, +edit `conf/hadoop/core-site.xml` (and the matching Spark/Hive configs) with that +provider's endpoint and credentials: + +```xml +<property> + <name>fs.s3a.endpoint</name> + <!-- Amazon S3: https://s3.<region>.amazonaws.com + any other provider: its S3 endpoint URL --> + <value>https://<s3-endpoint></value> +</property> +<property> + <name>fs.s3a.access.key</name> + <value><access-key></value> +</property> +<property> + <name>fs.s3a.secret.key</name> + <value><secret-key></value> +</property> +<property> + <name>fs.s3a.path.style.access</name> + <value>true</value> Review Comment: 🤖 The snippet sets `fs.s3a.path.style.access=true` unconditionally, but the closing paragraph correctly says Amazon S3 does not need it. Path-style access can actually break access to Amazon S3 buckets (which default to virtual-hosted style), so a reader who copies this block wholesale for AWS may hit failures. It might help to note in the snippet's comment that `path.style.access=true` applies to MinIO/most other providers and should be omitted (or set to `false`) for Amazon S3. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
