xccui commented on issue #8325:
URL: https://github.com/apache/hudi/issues/8325#issuecomment-1496179790
> > I hit the same exception in a Flink writer job. It happened when the job
was trying to recover from a failure.
> > Hudi version: 0.13.0 Flink version: 1.16.1
>
> have you try to reproduce the error? does it happen when you restart the
job manually? If it does reproduce, can you share your configs and table fields
type so we can try to reproduce locally and do some debugging.
It happened when a job tried to recover from a failure. Here is the full
stack trace.
```
2023-04-03 18:25:00 [stream_write: response_insertion_ids (5/5)#4] ERROR
org.apache.hudi.io.HoodieAppendHandle [] - Error writing
record HoodieRecord{key=HoodieKey {
recordKey=platformId:120,requestId:cfuleqeijb4qrau0tl4g
partitionPath=dt=2023-02-28/hr=01}, currentLocation='HoodieRecordLocation
{instantTime=U, fileId=null}', newLocation='null'}
org.apache.hudi.exception.HoodieException: Unable to instantiate class
org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:163)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:114)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:113)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.table.view.HoodieTableFileSystemView.<init>(HoodieTableFileSystemView.java:107)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.table.view.FileSystemViewManager.createInMemoryFileSystemView(FileSystemViewManager.java:177)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.table.view.FileSystemViewManager.lambda$createViewManager$5fcdabfe$1(FileSystemViewManager.java:272)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.table.view.FileSystemViewManager.lambda$getFileSystemView$1(FileSystemViewManager.java:115)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown
Source) ~[?:?]
at
org.apache.hudi.common.table.view.FileSystemViewManager.getFileSystemView(FileSystemViewManager.java:114)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.table.HoodieTable.getSliceView(HoodieTable.java:316)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.io.HoodieAppendHandle.init(HoodieAppendHandle.java:162)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.io.HoodieAppendHandle.doWrite(HoodieAppendHandle.java:486)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.io.HoodieWriteHandle.write(HoodieWriteHandle.java:175)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.execution.ExplicitWriteHandler.consume(ExplicitWriteHandler.java:49)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.execution.ExplicitWriteHandler.consume(ExplicitWriteHandler.java:35)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.util.queue.SimpleExecutor.execute(SimpleExecutor.java:67)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.execution.FlinkLazyInsertIterable.computeNext(FlinkLazyInsertIterable.java:64)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.execution.FlinkLazyInsertIterable.computeNext(FlinkLazyInsertIterable.java:43)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at java.util.Iterator.forEachRemaining(Unknown Source) ~[?:?]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:113)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:69)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.table.action.commit.FlinkWriteHelper.write(FlinkWriteHelper.java:77)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.table.action.commit.delta.FlinkUpsertDeltaCommitActionExecutor.execute(FlinkUpsertDeltaCommitActionExecutor.java:51)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.table.HoodieFlinkMergeOnReadTable.upsert(HoodieFlinkMergeOnReadTable.java:69)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.client.HoodieFlinkWriteClient.upsert(HoodieFlinkWriteClient.java:147)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$1(StreamWriteFunction.java:191)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:478)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at java.util.LinkedHashMap$LinkedValues.forEach(Unknown Source) ~[?:?]
at
org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:468)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:138)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:167)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:87)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:222)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:173)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:345)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:228)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:213)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:192)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:726)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:363)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$13(StreamTask.java:1281)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1269)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1226)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:287)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:64)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:488)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.io.checkpointing.AlternatingWaitingForFirstBarrierUnaligned.barrierReceived(AlternatingWaitingForFirstBarrierUnaligned.java:78)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlternatingAlignedBarrierHandlerState.barrierReceived(AbstractAlternatingAlignedBarrierHandlerState.java:55)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.lambda$processBarrier$2(SingleCheckpointBarrierHandler.java:234)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.markCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:262)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:231)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:542)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
[flink-dist-1.16.1.jar:1.16.1]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.reflect.InvocationTargetException
at
jdk.internal.reflect.GeneratedConstructorAccessor265.newInstance(Unknown
Source) ~[?:?]
at
jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:77)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
... 69 more
Caused by: org.apache.hudi.exception.HoodieIOException: getFileStatus on
s3a://path-to-table/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile:
com.amazonaws.AbortedException:
at
org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<init>(HFileBootstrapIndex.java:110)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
jdk.internal.reflect.GeneratedConstructorAccessor265.newInstance(Unknown
Source) ~[?:?]
at
jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:77)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
... 69 more
Caused by: java.io.InterruptedIOException: getFileStatus on
s3a://path-to-table/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile:
com.amazonaws.AbortedException:
at
org.apache.hadoop.fs.s3a.S3AUtils.translateInterruptedException(S3AUtils.java:395)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:201)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:175)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3799)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$exists$34(S3AFileSystem.java:4703)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4701)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.exists(HoodieWrapperFileSystem.java:566)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<init>(HFileBootstrapIndex.java:108)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
jdk.internal.reflect.GeneratedConstructorAccessor265.newInstance(Unknown
Source) ~[?:?]
at
jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:77)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
... 69 more
Caused by: com.amazonaws.AbortedException:
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleInterruptedException(AmazonHttpClient.java:880)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:757)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$10(S3AFileSystem.java:2545)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:377)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2533)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2513)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3776)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$exists$34(S3AFileSystem.java:4703)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4701)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.exists(HoodieWrapperFileSystem.java:566)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<init>(HFileBootstrapIndex.java:108)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
jdk.internal.reflect.GeneratedConstructorAccessor265.newInstance(Unknown
Source) ~[?:?]
at
jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:77)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
... 69 more
Caused by: com.amazonaws.http.timers.client.SdkInterruptedException
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:935)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:921)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1115)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$10(S3AFileSystem.java:2545)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:377)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2533)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2513)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3776)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$exists$34(S3AFileSystem.java:4703)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4701)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.exists(HoodieWrapperFileSystem.java:566)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<init>(HFileBootstrapIndex.java:108)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at
jdk.internal.reflect.GeneratedConstructorAccessor265.newInstance(Unknown
Source) ~[?:?]
at
jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:77)
~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
... 69 more
```
It happened more often when I enabled the metadata table in the past.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
