[
https://issues.apache.org/jira/browse/HUDI-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17903359#comment-17903359
]
Aditya Goenka commented on HUDI-8591:
-------------------------------------
Fixed via https://github.com/apache/hudi/pull/12371
> Deltastreamer failing with zookeeper lock configs even with single_writer
> -------------------------------------------------------------------------
>
> Key: HUDI-8591
> URL: https://issues.apache.org/jira/browse/HUDI-8591
> Project: Apache Hudi
> Issue Type: Bug
> Components: deltastreamer
> Reporter: Aditya Goenka
> Priority: Blocker
> Fix For: 1.0.0
>
>
> Even without setting any concurrency related configuration, deltastreamer
> code is failing with `Caused by: java.lang.IllegalArgumentException: Property
> hoodie.write.lock.zookeeper.lock_key not found`.
> After setting zookeeper configs it works fine.
> *Root Cause -*
> This issue is not present in 0.X releases as we dont need the lock for
> instant generation for single_writer scenarios. With 1.X releases, we will
> need lock provider for instant generation. EMR default hudi settings have
> Zookeeper lock provider but lock_key is not set.
> To fix this issue we need to set table name as default lock key if the
> concurrency mode is single writer
>
> *Full stack trace*
> ```
> Exception in thread "main" org.apache.hudi.exception.HoodieException: Unable
> to instantiate class
> org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
> at
> org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:75)
> at
> org.apache.hudi.common.table.timeline.TimeGeneratorBase.getLockProvider(TimeGeneratorBase.java:93)
> at
> org.apache.hudi.common.table.timeline.TimeGeneratorBase.unlock(TimeGeneratorBase.java:117)
> at
> org.apache.hudi.common.table.timeline.SkewAdjustingTimeGenerator.generateTime(SkewAdjustingTimeGenerator.java:51)
> at
> org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator.lambda$createNewInstantTime$0(HoodieInstantTimeGenerator.java:78)
> at
> java.util.concurrent.atomic.AtomicReference.updateAndGet(AtomicReference.java:179)
> at
> org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator.createNewInstantTime(HoodieInstantTimeGenerator.java:75)
> at
> org.apache.hudi.common.table.timeline.TimelineUtils.generateInstantTime(TimelineUtils.java:541)
> at
> org.apache.hudi.common.table.timeline.TimelineUtils.generateInstantTime(TimelineUtils.java:529)
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.createNewInstantTime(HoodieTableMetaClient.java:515)
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.createNewInstantTime(HoodieTableMetaClient.java:504)
> at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:470)
> at
> org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:902)
> at
> org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
> at org.apache.hudi.common.util.Option.ifPresent(Option.java:101)
> at
> org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:222)
> at
> org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:637)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:73)
> ... 28 more
> Caused by: java.lang.IllegalArgumentException: Property
> hoodie.write.lock.zookeeper.lock_key not found
> at
> org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys(ConfigUtils.java:358)
> at
> org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys(ConfigUtils.java:332)
> at
> org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider.getLockKey(ZookeeperBasedLockProvider.java:55)
> at
> org.apache.hudi.client.transaction.lock.BaseZookeeperBasedLockProvider.<init>(BaseZookeeperBasedLockProvider.java:70)
> at
> org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider.<init>(ZookeeperBasedLockProvider.java:44)
> ... 33 more
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
