[jira] [Commented] (HUDI-375) Refactor the configure framework of hudi project
[ https://issues.apache.org/jira/browse/HUDI-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991712#comment-16991712 ] lamber-ken commented on HUDI-375: - Hi, [~vinoth], after refactoring these, the code structure is concise and clear, each component just care about its own options, and developers will use these options easily. Let's imagine a scenario, as hudi project growth, more and more components will be introduced into project, their keys and default values are defined in a fat config file. If use a parameter in a wrong place uncarefully, it may have a great impact to us. > Refactor the configure framework of hudi project > > > Key: HUDI-375 > URL: https://issues.apache.org/jira/browse/HUDI-375 > Project: Apache Hudi (incubating) > Issue Type: Improvement >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > > Currently, config items and their default value are dispersed in the java > class file. It's easy to confuse when config items are defined more and more, > so it's necessary to refactor the configure framework. > May some things need to consider > # config item and default value may defined in a class > # provide a mechanism which can extract some config items for specific > component. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-375) Refactor the configure framework of hudi project
[ https://issues.apache.org/jira/browse/HUDI-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991140#comment-16991140 ] Vinoth Chandar commented on HUDI-375: - [~lamber-ken] I'd like to always start from the problems. I checked out your classes already. The Config class that holds a config key, value, default, and may be a doc as well makes sense to me. But, I remain skeptical on overall ROI if this would involve breaking existing clients. lets start a DISCUSS thread and take it from therE? > Refactor the configure framework of hudi project > > > Key: HUDI-375 > URL: https://issues.apache.org/jira/browse/HUDI-375 > Project: Apache Hudi (incubating) > Issue Type: Improvement >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > > Currently, config items and their default value are dispersed in the java > class file. It's easy to confuse when config items are defined more and more, > so it's necessary to refactor the configure framework. > May some things need to consider > # config item and default value may defined in a class > # provide a mechanism which can extract some config items for specific > component. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-375) Refactor the configure framework of hudi project
[ https://issues.apache.org/jira/browse/HUDI-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990067#comment-16990067 ] lamber-ken commented on HUDI-375: - Hi, [~vinoth] the main purpose of this issue is to solve the fat config class like HoodieWriteConfig and make the configuration framework easy to use for users and developers. First, for the first question, I'll bring up a discuss via email to talk about this issue with community later. As you said, this may affect current config class, the clients also needs to modify. Second, for the second question, you can clone my demo project which contains two modules, they are simplified models, you will understand the mechanism. > Refactor the configure framework of hudi project > > > Key: HUDI-375 > URL: https://issues.apache.org/jira/browse/HUDI-375 > Project: Apache Hudi (incubating) > Issue Type: Improvement >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > > Currently, config items and their default value are dispersed in the java > class file. It's easy to confuse when config items are defined more and more, > so it's necessary to refactor the configure framework. > May some things need to consider > # config item and default value may defined in a class > # provide a mechanism which can extract some config items for specific > component. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-375) Refactor the configure framework of hudi project
[ https://issues.apache.org/jira/browse/HUDI-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990036#comment-16990036 ] Vinoth Chandar commented on HUDI-375: - I am okay with introducing a class that binds the key, config value together.. But I would like to not move the existing config classes much. Clients may be configuring those using the HoodieWriteClient.Builder or as plain properties passed infile, movingthem or renaming properties, will cause breakages.. we cannot do that. What exactly is in scope for this work? > provide a mechanism which can extract some config items for specific > component. Whats the use-case for this? Could you please clarify? > Refactor the configure framework of hudi project > > > Key: HUDI-375 > URL: https://issues.apache.org/jira/browse/HUDI-375 > Project: Apache Hudi (incubating) > Issue Type: Improvement >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > > Currently, config items and their default value are dispersed in the java > class file. It's easy to confuse when config items are defined more and more, > so it's necessary to refactor the configure framework. > May some things need to consider > # config item and default value may defined in a class > # provide a mechanism which can extract some config items for specific > component. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-375) Refactor the configure framework of hudi project
[ https://issues.apache.org/jira/browse/HUDI-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986303#comment-16986303 ] lamber-ken commented on HUDI-375: - hi, [~vinoth] if you have good idea, you can comment here. :) > Refactor the configure framework of hudi project > > > Key: HUDI-375 > URL: https://issues.apache.org/jira/browse/HUDI-375 > Project: Apache Hudi (incubating) > Issue Type: Improvement >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > > Currently, config items and their default value are dispersed in the java > class file. It's easy to confuse when config items are defined more and more, > so it's necessary to refactor the configure framework. > May some things need to consider > # config item and default value may defined in a class > # provide a mechanism which can extract some config items for specific > component. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-375) Refactor the configure framework of hudi project
[ https://issues.apache.org/jira/browse/HUDI-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986018#comment-16986018 ] lamber-ken commented on HUDI-375: - I wrote a demo here, any suggestion are welcome, thanks. https://github.com/lamber-ken/hudi-config > Refactor the configure framework of hudi project > > > Key: HUDI-375 > URL: https://issues.apache.org/jira/browse/HUDI-375 > Project: Apache Hudi (incubating) > Issue Type: Improvement >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > > Currently, config items and their default value are dispersed in the java > class file. It's easy to confuse when config items are defined more and more, > so it's necessary to refactor the configure framework. > May some things need to consider > # config item and default value may defined in a class > # provide a mechanism which can extract some config items for specific > component. -- This message was sent by Atlassian Jira (v8.3.4#803005)
