[
https://issues.apache.org/jira/browse/FLINK-25413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qiunan updated FLINK-25413:
---------------------------
Description:
In FLINK-16005[flink-yarn] Support yarn and hadoop config override.
In flink-conf.yaml
flink.hadoop.dfs.namenode.rpc-address.nameservice1.nn1: bigdata1:8020
flink.hadoop.dfs.namenode.rpc-address.nameservice1.nn2: bigdata2:8020
flink.hadoop.dfs.client.failover.proxy.provider.nameservice1:
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
flink.hadoop.dfs.ha.namenodes.nameservice1: nn1,nn2
flink.hadoop.dfs.nameservices: nameservice1
code：
{code:java}
// Approach 4: Flink configuration
// add all configuration key with prefix 'flink.hadoop.' in flink conf to
hadoop conf
for (String key : flinkConfiguration.keySet()) {
for (String prefix : FLINK_CONFIG_PREFIXES) {
if (key.startsWith(prefix)) {
String newKey = key.substring(prefix.length());
String value = flinkConfiguration.getString(key, null);
result.set(newKey, value);
LOG.debug(
"Adding Flink config entry for {} as {}={} to Hadoop
config",
key,
newKey,
value);
foundHadoopConfiguration = true;
}
}
} {code}
If my HADOOP_CONF_DIR hdfs-site.xml have dfs.nameservices: nameservice2, see
the code logic this config will be override. I think this config should not be
override append will be better. if override we should add all config, but we
have many clusters in production, it is impossible to configure all
configurations in flink-conf.yaml.
was:
In [FLINK-16005][flink-yarn] Support yarn and hadoop config override.
However, I think we should append methods to better meet our production needs.
For example, the HDFS cluster in production has its own configuration. We only
need to add hdfs that cannot be written to. At the same time, there are many
clusters in production. We cannot cover all the configurations.
In flink-conf.yaml
flink.hadoop.dfs.nameservices: nameservice1
The problem now is, if flink.hadoop.dfs.nameservices: nameservice1 we can only
write to the cluster of nameservice1, all other clusters cannot be written, we
need to add the configuration of each cluster to write.
> Use append dfs.nameservices hadoop config to replace overwrite
> --------------------------------------------------------------
>
> Key: FLINK-25413
> URL: https://issues.apache.org/jira/browse/FLINK-25413
> Project: Flink
> Issue Type: Improvement
> Reporter: qiunan
> Priority: Major
>
> In FLINK-16005[flink-yarn] Support yarn and hadoop config override.
> In flink-conf.yaml
> flink.hadoop.dfs.namenode.rpc-address.nameservice1.nn1: bigdata1:8020
> flink.hadoop.dfs.namenode.rpc-address.nameservice1.nn2: bigdata2:8020
> flink.hadoop.dfs.client.failover.proxy.provider.nameservice1:
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> flink.hadoop.dfs.ha.namenodes.nameservice1: nn1,nn2
> flink.hadoop.dfs.nameservices: nameservice1
> code：
> {code:java}
> // Approach 4: Flink configuration
> // add all configuration key with prefix 'flink.hadoop.' in flink conf to
> hadoop conf
> for (String key : flinkConfiguration.keySet()) {
> for (String prefix : FLINK_CONFIG_PREFIXES) {
> if (key.startsWith(prefix)) {
> String newKey = key.substring(prefix.length());
> String value = flinkConfiguration.getString(key, null);
> result.set(newKey, value);
> LOG.debug(
> "Adding Flink config entry for {} as {}={} to Hadoop
> config",
> key,
> newKey,
> value);
> foundHadoopConfiguration = true;
> }
> }
> } {code}
> If my HADOOP_CONF_DIR hdfs-site.xml have dfs.nameservices: nameservice2, see
> the code logic this config will be override. I think this config should not
> be override append will be better. if override we should add all config, but
> we have many clusters in production, it is impossible to configure all
> configurations in flink-conf.yaml.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
