advancedxy commented on PR #1121:
URL:
https://github.com/apache/incubator-uniffle/pull/1121#issuecomment-1674428813
If we are going to refactor it rather than to patch it. I would go with
option 3 but with some modifications.
> Add new config, format is
rss.coordinator.remote.storage.conf.path.{cluster} file://xxxx
The remote storage configs are stored in file://xxxx.
I don't think we should store storage conf in another file, the `dynamic
client conf file` should be self contained.
For option 3, I believe we should define a new JavaClass to hold all the
related client conf. Currently, it should be something like:
```java
Class DynamicClientConf {
Map<String, String> clientConf;
List<String> availableRemoteClusters;
Map<String, Map<String, String>> clusterConfigs;
}
```
And a parser to parse the dynamic client conf file into the above JavaClass.
The parser could be a json parser, a toml parser, or a properties parser (java
has built-in support) or this current home-made one before this pr.
To be backward-compatibility, I would extract the current parse logic into a
`SimpleDynamicClientConfParser` that matches the original logic.
To solves the problem you mentioned , I would suggested add a new properties
parser(without introduce any new dependency) that with some additional parsing
rule.
```dynamic_client_conf.properties
k1 = v1
k2 : v2
rss.coordinator.remote.storage.path = hdfs://a,hdfs://b,hdfs://c
# the dynamic part, option 1: encode each cluster conf in a json string,
note the remote_storage_path is used to identify remote clusters.
rss.coordinator.remote.storage.cluster.conf.a = {"remote_storage_path":
"hdfs://a", "k1": "v1", "k2": "v2"}
# option 2: flatten each key value pair
rss.coordinator.remote.storage.cluster.a.remote_storage_path = hdfs://a
rss.coordinator.remote.storage.cluster.a.k1 = v1
rss.coordinator.remote.storage.cluster.a.k2 = v2
```
We can add other parsers if the above properties is not convenient to use,
such as toml parser.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
