cbg-wx opened a new issue, #19191:
URL: https://github.com/apache/hudi/issues/19191
### Feature Description
**What the feature achieves:**
Add a new Flink option parameter hadoop.conf.dir to load Hadoop
configuration files, enabling Flink applications to read and write Hudi tables
stored on HDFS across different Hadoop clusters, while isolating Flink’s HDFS
rocksdb state storage from the Hadoop cluster configuration.
**Why this feature is needed:**
Because the Hadoop cluster used for Flink RocksDB state backend may be
different from the Hadoop cluster where Hudi tables are stored, and Flink jobs
may need to read and write Hudi tables across multiple Hadoop clusters, a new
Hudi data source parameter is required to specify the path of Hadoop
configuration files.
### User Experience
**How users will use this feature:**
- Configuration changes needed
Users can configure the hadoop.conf.dir property in the Hudi table
parameters of Flink read/write jobs, specifying the Hadoop configuration
directory for either the source or target Hadoop cluster.
- API changes
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/HadoopConfigurations.java
```java
public static org.apache.hadoop.conf.Configuration
getHadoopConf(Configuration conf) {
org.apache.hadoop.conf.Configuration hadoopConf =
FlinkClientUtil.getHadoopConf();
// If hadoop.conf.dir is explicitly configured, load from that
directory.
String hadoopConfDir =
conf.getString(FlinkOptions.HADOOP_CONF_DIR.key(), null);
if (hadoopConfDir != null) {
org.apache.hadoop.conf.Configuration dirConf =
loadHadoopConfFromDir(hadoopConfDir);
if (dirConf != null) {
hadoopConf = dirConf;
}
}
Map<String, String> hadoopOptions =
FlinkOptions.getPropertiesWithPrefix(conf.toMap(), HADOOP_PREFIX);
hadoopOptions.remove("conf.dir");
hadoopOptions.forEach(hadoopConf::set);
Map<String, String> ioOptions = OptionsResolver.getIOOptions(conf);
ioOptions.forEach(hadoopConf::set);
return hadoopConf;
}
```
- Usage examples
```sql
CREATE TABLE hudi_remote_table (...)
WITH (
'connector' = 'hudi',
'path' = 'hdfs://remote-cluster:8020/path/to/table',
'hadoop.conf.dir' = '/path/to/remote-cluster/hadoop/conf',
...
)
```
### Hudi RFC Requirements
**RFC PR link:** (if applicable)
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
yes
- Does this change storage format? (Yes/No)
no
- Justification:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
