[
https://issues.apache.org/jira/browse/HUDI-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17604867#comment-17604867
]
Volodymyr Burenin commented on HUDI-4845:
-----------------------------------------
Full stack trace.
{code:java}
22/09/14 16:48:05 ERROR HoodieDeltaStreamer: Got error running delta sync once.
Shutting down
org.apache.hudi.exception.HoodieException: Could not sync using the meta sync
class org.apache.hudi.hive.HiveSyncTool
at
org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:61)
at
org.apache.hudi.utilities.deltastreamer.DeltaSync.syncMeta(DeltaSync.java:711)
at
org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:630)
at
org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:333)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$2(HoodieDeltaStreamer.java:200)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:198)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:553)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception
when hive syncing fegs_features_v3
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:143)
at
org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:59)
... 19 more
Caused by: org.apache.hudi.exception.HoodieException: Error fetching partition
paths from metadata table
at org.apache.hudi.common.fs.FSUtils.getAllPartitionPaths(FSUtils.java:305)
at
org.apache.hudi.sync.common.AbstractSyncHoodieClient.getPartitionsWrittenToSince(AbstractSyncHoodieClient.java:189)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:228)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:152)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:140)
... 20 more
Caused by: org.apache.hudi.exception.HoodieException:
org.apache.hudi.exception.HoodieException: Error occurs when executing map
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at
java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600)
at
java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:678)
at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:737)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:919)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at
org.apache.hudi.common.engine.HoodieLocalEngineContext.map(HoodieLocalEngineContext.java:84)
at
org.apache.hudi.metadata.FileSystemBackedTableMetadata.getAllPartitionPaths(FileSystemBackedTableMetadata.java:86)
at org.apache.hudi.common.fs.FSUtils.getAllPartitionPaths(FSUtils.java:303)
... 24 more
Caused by: org.apache.hudi.exception.HoodieException: Error occurs when
executing map
at
org.apache.hudi.common.function.FunctionWrapper.lambda$throwingMapWrapper$0(FunctionWrapper.java:40)
at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at
java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:952)
at
java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:926)
at java.base/java.util.stream.AbstractTask.compute(AbstractTask.java:327)
at
java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at
java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at
java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at
java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.util.concurrent.RejectedExecutionException: Thread limit
exceeded replacing blocked worker
at
java.base/java.util.concurrent.ForkJoinPool.tryCompensate(ForkJoinPool.java:1575)
at
java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3115)
at
java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)
at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)
at org.apache.hadoop.util.functional.FutureIO.awaitFuture(FutureIO.java:73)
at
org.apache.hadoop.fs.impl.FutureIOSupport.awaitFuture(FutureIOSupport.java:65)
at
org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.next(Listing.java:821)
at
org.apache.hadoop.fs.s3a.Listing$FileStatusListingIterator.requestNextBatch(Listing.java:612)
at
org.apache.hadoop.fs.s3a.Listing$FileStatusListingIterator.<init>(Listing.java:536)
at
org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:173)
at
org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:148)
at
org.apache.hadoop.fs.s3a.Listing.getFileStatusesAssumingNonEmptyDir(Listing.java:414)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:2834)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$14(S3AFileSystem.java:2812)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:2810)
at
org.apache.hudi.metadata.FileSystemBackedTableMetadata.lambda$getAllPartitionPaths$4e4c112d$1(FileSystemBackedTableMetadata.java:88)
at
org.apache.hudi.common.function.FunctionWrapper.lambda$throwingMapWrapper$0(FunctionWrapper.java:38)
... 13 more
{code}
> HiveSync fails while scanning a table with a large number of partitions
> -----------------------------------------------------------------------
>
> Key: HUDI-4845
> URL: https://issues.apache.org/jira/browse/HUDI-4845
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Volodymyr Burenin
> Priority: Major
>
>
> When I try to recreate a table in metastore that has around 4k partitions, I
> get this exceptions during
> org.apache.hudi.common.fs.FSUtils.getAllPartitionPaths call.
> {code:java}
> Caused by: java.util.concurrent.RejectedExecutionException: Thread limit
> exceeded replacing blocked worker
> at
> java.base/java.util.concurrent.ForkJoinPool.tryCompensate(ForkJoinPool.java:1575)
> at
> java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3115)
> at
> java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)
> at
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)
> at org.apache.hadoop.util.functional.FutureIO.awaitFuture(FutureIO.java:73)
> at
> org.apache.hadoop.fs.impl.FutureIOSupport.awaitFuture(FutureIOSupport.java:65)
> at
> org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.next(Listing.java:821)
> at
> org.apache.hadoop.fs.s3a.Listing$FileStatusListingIterator.requestNextBatch(Listing.java:612)
> at
> org.apache.hadoop.fs.s3a.Listing$FileStatusListingIterator.<init>(Listing.java:536)
> at
> org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:173)
> at
> org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:148)
> at
> org.apache.hadoop.fs.s3a.Listing.getFileStatusesAssumingNonEmptyDir(Listing.java:414){code}
> I tracked down the problem down here:
> [https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/metadata/FileSystemBackedTableMetadata.java]
> Apparently this value is too high:
> private static final int DEFAULT_LISTING_PARALLELISM = 1500;
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
