zhaojintaozhao opened a new issue #5175:
URL: https://github.com/apache/incubator-doris/issues/5175
In my system, hdfs system uses the HA mode. I need to set many configuration
items of the HA mode in the load sql, such as dfs.nameservices,
dfs.ha.namenodes.xxx..... Oh, that is too complex.
For example:
```
LOAD LABEL example_db.label
(
DATA INFILE("hdfs://hacluster/my_bucket/input/file")
INTO table `my_table`
)
WITH BROKER "broker_name"
(
"dfs.nameservices" = "my_ha",
"dfs.ha.namenodes.my_ha" = "my_namenode1, my_namenode2",
"dfs.namenode.rpc-address.my_ha.my_namenode1" = "nn1_host:rpc_port",
"dfs.namenode.rpc-address.my_ha.my_namenode2" = "nn2_host:rpc_port",
"dfs.client.failover.proxy.provider" =
"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"
)
```
These configuration items are saved in the hdfs-site.xml and core-site.xml
files of the hdfs system. These configurations are generally fixed.
I suggest to add the hdfs HA configuration item to the apache-hdfs-broker
configuration file. These configurations are point to hdfs-site.xml or
core-site.xml. In this way, we do not need to set many HDFS HA attributes in
each load SQL properties.
For example:
```
Add the following configuration to the apache-hdfs-broker configuration file:
hadoop_core_site_file_path=/home/core-site.xml
hadoop_hdfs_site_file_path=/home/hdfs-site.xml
kerberos_krb5_file_path=/home/krb5.conf
```
In this case, we do not need to set the HDFS HA mode configuration when load
sql.
```
LOAD LABEL Example_db.label6
(
DATA INFILE("bos://my_bucket/input/file")
INTO table `my_table`
)
WITH BROKER "broker_name"
()
```
I optimized this feature and my system works properly.
I want to integrate my optimizations into the community.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
