hudi-bot opened a new issue, #15780: URL: https://github.com/apache/hudi/issues/15780
Take for example `hoodie.datasource.write.keygenerator.type` and its counterpart `hoodie.datasource.write.keygenerator.class`. We should make configs like keygenerator.type for configs currently like keygenerator.class. Take for example `hoodie.datasource.write.record.merger.impls`. If you want HoodieAvroRecordMerger you need to set the value to org.apache.hudi.common.model.HoodieAvroRecordMerger. However, if you want HoodieSparkRecordMerger, the full class path is org.apache.hudi.HoodieSparkRecordMerger. Differences like this are an easy cause for mistakes. ## JIRA info - Link: https://issues.apache.org/jira/browse/HUDI-5761 - Type: Improvement - Epic: https://issues.apache.org/jira/browse/HUDI-5738 - Fix version(s): - 1.1.0 --- ## Comments 03/Apr/23 13:35;zxcoccer;Hi, I am trying to complete this issue, but there is some confusion in the process of completing the issue, how to get the `*_type` value elegantly, the method I can think of now is to correspond to a method for each `_type` Value, like this {code:java} switch (keyGeneratorTypeEnum){ case SIMPLE: return new SimpleAvroKeyGenerator(props); case COMPLEX: return new ComplexAvroKeyGenerator(props); case TIMESTAMP: return new TimestampBasedAvroKeyGenerator(props); case NON_PARTITION: return new NonpartitionedAvroKeyGenerator(props); case GLOBAL_DELETE: return new GlobalAvroDeleteKeyGenerator(props); } {code} By the way, can you assign this ticket to me? I will try my best to complete it;;; --- 03/Apr/23 15:39;jonvex;Ok so keygenerator.type is now not such a good example because https://issues.apache.org/jira/browse/HUDI-5929 . So I will give a different example. In HoodieBootstrapConfig.java we have config "hoodie.bootstrap.mode.selector". We have selectors that are already created: org.apache.hudi.client.bootstrap.selector.BootstrapRegexModeSelector org.apache.hudi.client.bootstrap.selector.FullRecordBootstrapModeSelector org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector But you need to read the code to figure that out and it is long to type out. We want to create a new enum {code:java} public enum BootstrapSelectorType { REGEX, FULL_RECORD, METADATA_ONLY, CUSTOM } {code} And create a new config "hoodie.bootstrap.mode.type" that takes in those config values Then, in HoodieWriteConfig.java there is a method {code:java} public String getBootstrapModeSelectorClass() { return getString(HoodieBootstrapConfig.MODE_SELECTOR_CLASS_NAME); } {code} We want to replace it with something like {code:java} public String getBootstrapModeSelectorClass() { switch (getString(HoodieBootstrapConfig.MODE_SELECTOR_TYPE_NAME)) { case REGEX: return "org.apache.hudi.client.bootstrap.selector.BootstrapRegexModeSelector"; case FULL_RECORD: return "org.apache.hudi.client.bootstrap.selector.FullRecordBootstrapModeSelector"; case METADATA_ONLY: return " org.apache.hudi.client.bootstrap.selector.FullRecordBootstrapModeSelector"; case CUSTOM: return getString(HoodieBootstrapConfig.MODE_SELECTOR_CLASS_NAME); } } {code} Then repeat this with all other configs that take in class: hoodie.datasource.write.payload.class hoodie.datasource.write.record.merger.impls hoodie.writestatus.class hoodie.fileid.prefix.provider.class for example. There are probably more than that. To get assigned tickets, you need to email [[email protected]|mailto:[email protected]] and ask to be added as a contributor. ;;; --- 05/Apr/23 11:38;zxcoccer;Thank you very much for your reply. I will complete this task soon.:);;; --- 24/Apr/23 18:27;jonvex;Taking this over because there is a lot of documentation required;;; --- 24/Apr/23 18:40;jonvex;Additional classes to look at: h3. -hoodie.datasource.hive_sync.partition_extractor_class- h3. -hoodie.meta.sync.client.tool.class- h3. hoodie.precommit.validators h3. hoodie.filesystem.operation.retry.exceptions h3. hoodie.bulkinsert.user.defined.partitioner.class h3. hoodie.write.concurrency.early.conflict.detection.strategy h3. hoodie.compaction.strategy h3. hoodie.clustering.execution.strategy.class h3. hoodie.clustering.plan.strategy.class h3. hoodie.clustering.updates.strategy h3. hoodie.bootstrap.full.input.provider h3. hoodie.bootstrap.index.class h3. hoodie.bootstrap.partitionpath.translator.class h4. hoodie.write.commit.callback.class h4. hoodie.write.lock.conflict.resolution.strategy h4. hoodie.write.lock.provider h4. hoodie.index.hbase.qps.allocator.class h3. hoodie.meta.sync.datahub.dataset.identifier.class h3. hoodie.schemaprovider.class h3. hoodie.filesystem.view.remote.retry.exceptions[|https://hudi.apache.org/docs/configurations#hoodiefilesystemviewremoteretryexceptions] ;;; -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
