[
https://issues.apache.org/jira/browse/HUDI-89?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17205751#comment-17205751
]
sivabalan narayanan commented on HUDI-89:
-----------------------------------------
sorry, I was busy for the last few weeks. Here is my understanding. I don't
have full context around moving configs to right classes. I need sometime to
look into CompactionConfig, StorageConfig, HoodieCleanConfig. But wrt revamping
Configs in general here is the idea.
As of now, config management is naive. Let's say we want to add a new config,
we add a key string to HoodieWriteConfig, and then add a default, expose getter
and setter with builder pattern. Call into setting up defaults for properties
not set. and then build the HoodieWriteConfig. We wish to introduce a class
called ConfigOption (source:
[[1]|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/ConfigOption.java
|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/ConfigOption.java)]],
[[2]|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/Configuration.java
|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/ConfigOption.java)]]
and
[[3]|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/ConfigOptions.java|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/ConfigOption.java)]]
[)|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/ConfigOption.java)]
We are not looking for a full fledged ConfigOption(which include fallback keys
and and stuff), but just key, value, defaultValue and description for now. We
can iteratively add more features. For eg: there was some prep work done
earlier on this regards [here|https://github.com/apache/hudi/pull/1094/files].
By this, we can bind a key, default value, description for every config value
together. With this, the default value is maintained along w/ the actual config
in ConfigOption and so get() should return the actual value if set. if not will
return the default value if set. Also, description will come in handy when we
want to generate release docs.
And btw, with this change, we also want to rename HoodieWriteConfig to
HoodieClientConfig.
[~vinoth]: I understand we don't want to do a complete overhaul which involves
changes to the way users set the properties. So, may I know how do we go about
populating ConfigOptions from a map of properties or from a property file. In
other words, how do we intercept the value type from the property. Or am I
missing something on what changes we need to make.
> Clean up placement, naming, defaults of HoodieWriteConfig
> ---------------------------------------------------------
>
> Key: HUDI-89
> URL: https://issues.apache.org/jira/browse/HUDI-89
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Code Cleanup, Usability, Writer Core
> Reporter: Vinoth Chandar
> Assignee: Vinoth Chandar
> Priority: Major
>
> # Rename HoodieWriteConfig to HoodieClientConfig
> # Move bunch of configs from CompactionConfig to StorageConfig
> # Introduce new HoodieCleanConfig
> # Should we consider lombok or something to automate the
> defaults/getters/setters
> # Consistent name of properties/defaults
> # Enforce bounds more strictly
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
