Ethan Guo created HUDI-3640:
-------------------------------
Summary: Use default key generator class if not provided by user
in TwoToThreeUpgradeHandler
Key: HUDI-3640
URL: https://issues.apache.org/jira/browse/HUDI-3640
Project: Apache Hudi
Issue Type: Task
Reporter: Ethan Guo
Assignee: Ethan Guo
Fix For: 0.11.0
When upgrading from table version 2 to 3, the logic checks if the key generator
class is provided by user config. We should still use default
SimpleKeyGenerator if not provided, instead of throwing an error in the upgrade
step.
{code:java}
ValidationUtils.checkState(keyGenClassName != null, String.format("Missing
config: %s or %s",
HoodieTableConfig.KEY_GENERATOR_CLASS_NAME,
HoodieWriteConfig.KEYGENERATOR_CLASS_NAME)); {code}
{code:java}
22/03/14 12:28:10 ERROR HoodieDeltaStreamer: Shutting down delta-sync due to
exception
java.lang.IllegalStateException: Missing config: Key:
'hoodie.table.keygenerator.class' , default: null description: Key Generator
class property for the hoodie table since version: version is not defined
deprecated after: version is not defined) or Key:
'hoodie.datasource.write.keygenerator.class' , default: null description: Key
generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract
a key out of incoming records. since version: version is not defined deprecated
after: version is not defined)
at
org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:67)
at
org.apache.hudi.table.upgrade.TwoToThreeUpgradeHandler.upgrade(TwoToThreeUpgradeHandler.java:50)
at
org.apache.hudi.table.upgrade.UpgradeDowngrade.upgrade(UpgradeDowngrade.java:145)
at
org.apache.hudi.table.upgrade.UpgradeDowngrade.run(UpgradeDowngrade.java:118)
at
org.apache.hudi.client.BaseHoodieWriteClient.tryUpgrade(BaseHoodieWriteClient.java:1407)
at
org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1302)
at
org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:151)
at
org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:577)
at
org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:329)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:656)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
22/03/14 12:28:10 INFO HoodieDeltaStreamer: Delta Sync shutdown. Error ?true
22/03/14 12:28:10 WARN HoodieDeltaStreamer: Gracefully shutting down compactor
22/03/14 12:28:16 INFO AsyncCompactService: Compactor shutting down properly!!
22/03/14 12:28:16 INFO HoodieDeltaStreamer: DeltaSync shutdown. Closing write
client. Error?true
22/03/14 12:28:16 ERROR HoodieAsyncService: Service shutdown with error
java.util.concurrent.ExecutionException:
org.apache.hudi.exception.HoodieException: Missing config: Key:
'hoodie.table.keygenerator.class' , default: null description: Key Generator
class property for the hoodie table since version: version is not defined
deprecated after: version is not defined) or Key:
'hoodie.datasource.write.keygenerator.class' , default: null description: Key
generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract
a key out of incoming records. since version: version is not defined deprecated
after: version is not defined)
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at
org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:103)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$1(HoodieDeltaStreamer.java:182)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:179)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:530)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
