[
https://issues.apache.org/jira/browse/HUDI-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-3640:
----------------------------
Status: In Progress (was: Open)
> Use default key generator class if not provided by user in
> TwoToThreeUpgradeHandler
> -----------------------------------------------------------------------------------
>
> Key: HUDI-3640
> URL: https://issues.apache.org/jira/browse/HUDI-3640
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Ethan Guo
> Assignee: Ethan Guo
> Priority: Blocker
> Fix For: 0.11.0
>
>
> When upgrading from table version 2 to 3, the logic checks if the key
> generator class is provided by user config. We should still use default
> SimpleKeyGenerator if not provided, instead of throwing an error in the
> upgrade step.
> {code:java}
> ValidationUtils.checkState(keyGenClassName != null, String.format("Missing
> config: %s or %s",
> HoodieTableConfig.KEY_GENERATOR_CLASS_NAME,
> HoodieWriteConfig.KEYGENERATOR_CLASS_NAME)); {code}
>
>
> {code:java}
> 22/03/14 12:28:10 ERROR HoodieDeltaStreamer: Shutting down delta-sync due to
> exception
> java.lang.IllegalStateException: Missing config: Key:
> 'hoodie.table.keygenerator.class' , default: null description: Key Generator
> class property for the hoodie table since version: version is not defined
> deprecated after: version is not defined) or Key:
> 'hoodie.datasource.write.keygenerator.class' , default: null description: Key
> generator class, that implements `org.apache.hudi.keygen.KeyGenerator`
> extract a key out of incoming records. since version: version is not defined
> deprecated after: version is not defined)
> at
> org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:67)
> at
> org.apache.hudi.table.upgrade.TwoToThreeUpgradeHandler.upgrade(TwoToThreeUpgradeHandler.java:50)
> at
> org.apache.hudi.table.upgrade.UpgradeDowngrade.upgrade(UpgradeDowngrade.java:145)
> at
> org.apache.hudi.table.upgrade.UpgradeDowngrade.run(UpgradeDowngrade.java:118)
> at
> org.apache.hudi.client.BaseHoodieWriteClient.tryUpgrade(BaseHoodieWriteClient.java:1407)
> at
> org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1302)
> at
> org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:151)
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:577)
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:329)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:656)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 22/03/14 12:28:10 INFO HoodieDeltaStreamer: Delta Sync shutdown. Error ?true
> 22/03/14 12:28:10 WARN HoodieDeltaStreamer: Gracefully shutting down compactor
> 22/03/14 12:28:16 INFO AsyncCompactService: Compactor shutting down properly!!
> 22/03/14 12:28:16 INFO HoodieDeltaStreamer: DeltaSync shutdown. Closing write
> client. Error?true
> 22/03/14 12:28:16 ERROR HoodieAsyncService: Service shutdown with error
> java.util.concurrent.ExecutionException:
> org.apache.hudi.exception.HoodieException: Missing config: Key:
> 'hoodie.table.keygenerator.class' , default: null description: Key Generator
> class property for the hoodie table since version: version is not defined
> deprecated after: version is not defined) or Key:
> 'hoodie.datasource.write.keygenerator.class' , default: null description: Key
> generator class, that implements `org.apache.hudi.keygen.KeyGenerator`
> extract a key out of incoming records. since version: version is not defined
> deprecated after: version is not defined)
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:103)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$1(HoodieDeltaStreamer.java:182)
> at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:179)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:530)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
