rangareddy commented on issue #15198:
URL: https://github.com/apache/hudi/issues/15198#issuecomment-5140099175
Reviewed this as part of the JIRA-migrated backlog triage. The two details
in the report that look
contradictory — a CSV table writing fine to the same bucket, and the same
failure appearing **even when
MinIO is stopped** — turn out to be the key to it, so here is what is
actually happening.
**Hudi's Flink writer talks to S3 through Hadoop's `s3a` directly, not
through Flink's filesystem plugin.**
Your CSV table used Flink's `filesystem` connector, which goes through the
`flink-s3-fs-hadoop` **plugin**
configured with `s3.*` keys in `flink-conf.yaml`. Hudi instead calls
`S3AFileSystem` itself (visible in your
stack: `StreamerUtil.tableExists` → `S3AFileSystem.exists`), and it builds
its Hadoop `Configuration` from
only three sources — see `FlinkClientUtil#getHadoopConf` and
`HadoopConfigurations#getHadoopConf`:
1. `core-site.xml` found under `HADOOP_CONF_DIR`, `HADOOP_HOME`, or
`fs.hdfs.hadoopconf`
(first location that resolves wins);
2. `flink-conf.yaml` entries prefixed **`flink.hadoop.`**;
3. table options prefixed **`hadoop.`**.
Flink's plugin-style `s3.*` settings are not among them, which is why the
plugin-backed CSV write succeeded
while Hudi got `403 Forbidden` on `.hoodie` — and why the error was
identical with MinIO stopped: Hudi had
no endpoint or credentials, so it was never reaching your MinIO at all.
Also worth noting: `hoodie.aws.access.key` / `hoodie.aws.secret.key` do not
configure the filesystem.
`HoodieAWSConfig` scopes those to Amazon CloudWatch (metrics) and Amazon
Glue (metadata); nothing maps them
onto `fs.s3a.*`.
Any one of these three will work. As table options:
```sql
create table t1s3hudi(id int PRIMARY KEY, name varchar(50)) with (
'connector' = 'hudi',
'path' = 's3a://test123/t1s3hudi',
'table.type' = 'MERGE_ON_READ',
'hadoop.fs.s3a.access.key' = 'minioadmin',
'hadoop.fs.s3a.secret.key' = 'minioadmin',
'hadoop.fs.s3a.endpoint' = 'http://localhost:9000',
'hadoop.fs.s3a.path.style.access' = 'true'
);
```
Or in `flink-conf.yaml`, which applies to every Hudi table:
```yaml
flink.hadoop.fs.s3a.access.key: minioadmin
flink.hadoop.fs.s3a.secret.key: minioadmin
flink.hadoop.fs.s3a.endpoint: http://localhost:9000
flink.hadoop.fs.s3a.path.style.access: true
```
Or point `HADOOP_CONF_DIR` at a directory containing a `core-site.xml` with
those same `fs.s3a.*`
properties. The `endpoint` and `path.style.access` settings are required for
MinIO specifically; without the
endpoint, `s3a` talks to AWS rather than your local server.
Two further notes:
- The versions here are well out of support — Hudi 0.10.1 and Flink 1.13.6.
Current Hudi supports the Flink
1.17+ lines, so it is worth retesting on a recent bundle if you still have
this workload.
- The documentation deserves part of the blame: [AWS
S3](https://hudi.apache.org/docs/s3_hoodie) is written
entirely for Spark ("configure your `SparkSession` or `SparkContext`") and
says nothing about the Flink
configuration paths above. That gap is being addressed separately.
So this is a configuration issue rather than a defect in Hudi, and there is
no code change to make here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
