+1 from me. Long time coming. On Mon, Apr 19, 2021 at 12:02 PM Ding, Wenning <wenni...@amazon.com.invalid> wrote:
> Hi, > I planned to refactor the current Hudi configuration framework. lamberken< > https://github.com/lamberken> did similar things before: > https://github.com/apache/hudi/pull/1094 and I’d like to continue this > work and add more features in ConfigOption class. > > The motivation of this change is, as lamberken mentioned, “Currently, > config items and their default value are dispersed in the java class file. > It's could be confused when config items are defined more and more”. Having > this change would make Hudi developers easy to use and check these > configurations. > > Also, we can also bind configuration description within the ConfigOption > class. And for the next step, we could also do something similar to Flink > to automatically add/update property description on the Hudi website: > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/description/Description.java > . > Besides, we can also bind an inference function within the ConfigOption > class which can provide an inference mechanism for some of our > configurations based on rules. For example, we can infer the key generator > class based on the Hudi record fields & partition fields. For example, if > the record key field contains comma which indicate that there are multiple > record keys, then by default we should use ComplexKeyGenerator. If there’s > no partition column, we should use NonpartitionedKeyGenerator. Having this > inference mechanism can make Hudi be more intelligent so that users don’t > need to set so many parameters from their client side. > The disadvantage of this change is for users who are now using e.g. > HoodieBootstrapConfig.BOOTSTRAP_BASE_PATH_PROP in their client code, they > need to either replace it with > HoodieBootstrapConfig.BOOTSTRAP_BASE_PATH_PROP.key() or > hoodie.bootstrap.base.path. > I opened a demo for this: https://github.com/apache/hudi/pull/2833. Feel > free to discuss under this thread and provide any suggestion! > > Related JIRAs: > https://issues.apache.org/jira/browse/HUDI-89 > https://issues.apache.org/jira/projects/HUDI/issues/HUDI-375 > > Thanks, > Wenning > > > >
