peach12345 opened a new issue, #14388:
URL: https://github.com/apache/iceberg/issues/14388
### Apache Iceberg version
1.10.0 (latest release)
### Query engine
Flink
### Please describe the bug 🐞
Hi,
we encountered the following error while using the new table maintenance
feature from iceberg 1.10.
Rewriting data files and deleting orphaned files seems to work but deleting
the metadata seems not to work.
Could it be related to the following stack trace?
`2025-10-21 10:59:47,648 ERROR
org.apache.iceberg.flink.maintenance.operator.ListMetadataFiles [] - Exception
listing metadata files for rest_catalog.default.myevent t at 1761044386378
software.amazon.awssdk.services.s3.model.NoSuchKeyException: (Service: S3,
Status Code: 404, Request ID: NQD8R3KWQZBZ21ZB, Extended Request ID:
y0MrqMeEGZOEEOWiyvCTr+M0lI5/FE4LbjCOUMwTrivOTLng5/fjcHkWC7WZvRRIIQFLH8SnTdDEYXA9cAYJVHfD0wcVtXQMRxuwubMUYoU=)
(SDK Attempt Count: 1) (Service: S3, Status Code: 404, Request ID:
NQD8R3KWQZBZ21ZB, Extended Request ID:
y0MrqMeEGZOEEOWiyvCTr+M0lI5/FE4LbjCOUMwTrivOTLng5/fjcHkWC7WZvRRIIQFLH8SnTdDEYXA9cAYJVHfD0wcVtXQMRxuwubMUYoU=)
at
software.amazon.awssdk.services.s3.model.NoSuchKeyException$BuilderImpl.build(NoSuchKeyException.java:150)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.services.s3.model.NoSuchKeyException$BuilderImpl.build(NoSuchKeyException.java:98)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor.modifyException(ExceptionTranslationInterceptor.java:65)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyException(ExecutionInterceptorChain.java:181)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.runModifyException(ExceptionReportingUtils.java:54)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.reportFailureToInterceptors(ExceptionReportingUtils.java:38)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:39)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
software.amazon.awssdk.services.s3.DefaultS3Client.headObject(DefaultS3Client.java:7623)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.aws.s3.BaseS3File.getObjectMetadata(BaseS3File.java:96)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.aws.s3.S3InputFile.getLength(S3InputFile.java:156)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:102)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:77)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:37)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.relocated.com.google.common.collect.Iterables.addAll(Iterables.java:332)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.relocated.com.google.common.collect.Lists.newLinkedList(Lists.java:261)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at org.apache.iceberg.ManifestLists.read(ManifestLists.java:42)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.BaseSnapshot.cacheManifests(BaseSnapshot.java:185)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at org.apache.iceberg.BaseSnapshot.allManifests(BaseSnapshot.java:203)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.flink.maintenance.operator.ListMetadataFiles.lambda$processElement$0(ListMetadataFiles.java:83)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.relocated.com.google.common.collect.ImmutableList.forEach(ImmutableList.java:421)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.flink.maintenance.operator.ListMetadataFiles.processElement(ListMetadataFiles.java:71)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.flink.maintenance.operator.ListMetadataFiles.processElement(ListMetadataFiles.java:37)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:64)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.collectAndCheckIfChained(ChainingOutput.java:91)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.collectAndCheckIfChained(ChainingOutput.java:41)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:84)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:35)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:38)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.collectAndCheckIfChained(ChainingOutput.java:91)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.collectAndCheckIfChained(ChainingOutput.java:41)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:84)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:35)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.operators.TimestampsAndWatermarksOperator.processElement(TimestampsAndWatermarksOperator.java:140)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:53)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.iceberg.flink.maintenance.operator.TriggerManager.checkAndFire(TriggerManager.java:247)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.flink.maintenance.operator.TriggerManager.onTimer(TriggerManager.java:207)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.flink.streaming.api.operators.KeyedProcessOperator.invokeUserFunction(KeyedProcessOperator.java:95)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.api.operators.KeyedProcessOperator.onProcessingTime(KeyedProcessOperator.java:80)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:303)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1851)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$deferCallbackToMailbox$27(StreamTask.java:1842)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:118)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:415)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:384)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:369)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:980)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:963)
[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:942)
[flink-dist-2.1.0.jar:2.1.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756)
[flink-dist-2.1.0.jar:2.1.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:568)
[flink-dist-2.1.0.jar:2.1.0]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2025-10-21 10:59:47,650 WARN
org.apache.iceberg.flink.maintenance.operator.ListFileSystemFiles [] -
Exception listing files for s3://my-bucket/lakehouse/default/myevent at
1761044386378
java.io.UncheckedIOException:
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme
"s3"
at
org.apache.iceberg.util.FileSystemWalker.listDirRecursivelyWithHadoop(FileSystemWalker.java:150)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.flink.maintenance.operator.ListFileSystemFiles.processElement(ListFileSystemFiles.java:111)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.flink.maintenance.operator.ListFileSystemFiles.processElement(ListFileSystemFiles.java:47)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:64)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.collectAndCheckIfChained(ChainingOutput.java:91)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.collectAndCheckIfChained(ChainingOutput.java:41)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:84)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:35)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:38)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.collectAndCheckIfChained(ChainingOutput.java:91)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.ChainingOutput.collectAndCheckIfChained(ChainingOutput.java:41)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:84)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:35)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.operators.TimestampsAndWatermarksOperator.processElement(TimestampsAndWatermarksOperator.java:140)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:53)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.iceberg.flink.maintenance.operator.TriggerManager.checkAndFire(TriggerManager.java:247)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.flink.maintenance.operator.TriggerManager.onTimer(TriggerManager.java:207)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.flink.streaming.api.operators.KeyedProcessOperator.invokeUserFunction(KeyedProcessOperator.java:95)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.api.operators.KeyedProcessOperator.onProcessingTime(KeyedProcessOperator.java:80)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:303)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1851)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$deferCallbackToMailbox$27(StreamTask.java:1842)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:118)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:415)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:384)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:369)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:980)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917)
~[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:963)
[flink-dist-2.1.0.jar:2.1.0]
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:942)
[flink-dist-2.1.0.jar:2.1.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756)
[flink-dist-2.1.0.jar:2.1.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:568)
[flink-dist-2.1.0.jar:2.1.0]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
FileSystem for scheme "s3"
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3581)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3612)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:172)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3716)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3667)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:366)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
at
org.apache.iceberg.util.FileSystemWalker.listDirRecursivelyWithHadoop(FileSystemWalker.java:122)
~[blob_p-ef15c0f9ed3a94bf50410ee82ee6b54aef3e1cb9-25ceff8143517830512ab6bf201c518d:?]
... 41 more`
We are currently using Flink 2.0 with Iceberg 1.10, leveraging AWS S3 as our
storage layer and Snowflake Open Catalog as the catalog.
Job-Configuration:
scheduleOnInterval(10 minutes)
minInputFiles(2)
maxFileGroupSizeBytes(1GB)
For deleteOrphanFiles:
deleteBatchSize(100)
scheduleOnInterval(10 mins)
minAge(1 hour)
For ExpireSnapshots:
deleteBatchSize(100)
scheduleOnInterval(10 mins)
maxSnapshotAge(1 hour)
We’d appreciate any insights or suggestions — especially regarding the
ExpireSnapshots job, as it doesn’t seem to be cleaning up old snapshot metadata
files as expected.
Thanks in advance for any kind of help!
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
