Lucas Borges created FLINK-38324:
------------------------------------
Summary: Job fails to restore keyed state backend when using Forst
state backend on S3
Key: FLINK-38324
URL: https://issues.apache.org/jira/browse/FLINK-38324
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Affects Versions: 2.0.0, 2.1.0
Environment: Observed on both Flink 2.0 and 2.1,
running with Forst state backend and state stored on S3,
job deployed on kubernetes using the Flink apache kubernetes operator.
Reporter: Lucas Borges
Task manager fails with the following exception:
Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state
backend for AsyncStreamFlatMap_34849252e53e8aeadce2388c44ea80ff_(1/1) from any
of the 1 provided restore options.
Which seems to be caused by the following error:
Caused by: java.io.IOException: java.io.FileNotFoundException: No such file or
directory:
s3a://flink-state/flink-2-smoke-testing-job/checkpoints/f3436c2b6f059985605ce8a4f3cdc841/shared/op_AsyncStreamFlatMap_34849252e53e8aeadce2388c44ea80ff__1_1__attempt_0/db/a62f79a2-bca7-40ff-9723-01d9a0d98f4a
Full stack trace here:
{panel}
java.lang.Exception: Exception while creating StreamOperatorStateContext.
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:359)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:280)
at
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreStateAndGates(StreamTask.java:858)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$restoreInternal$5(StreamTask.java:812)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:812)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:771)
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:963)
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:932)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:568)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state
backend for AsyncStreamFlatMap_34849252e53e8aeadce2388c44ea80ff_(1/1) from any
of the 1 provided restore options.
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:165)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:486)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:195)
... 12 more
Caused by: org.apache.flink.runtime.state.BackendBuildingException: Caught
unexpected exception.
at
org.apache.flink.state.forst.ForStKeyedStateBackendBuilder.build(ForStKeyedStateBackendBuilder.java:319)
at
org.apache.flink.state.forst.ForStStateBackend.createAsyncKeyedStateBackend(ForStStateBackend.java:474)
at
org.apache.flink.state.forst.ForStStateBackend.createAsyncKeyedStateBackend(ForStStateBackend.java:98)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$3(StreamTaskStateInitializerImpl.java:475)
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:173)
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:137)
... 14 more
Caused by: java.io.IOException: java.io.FileNotFoundException: No such file or
directory:
s3a://flink-state/flink-2-smoke-testing-job/checkpoints/f3436c2b6f059985605ce8a4f3cdc841/shared/op_AsyncStreamFlatMap_34849252e53e8aeadce2388c44ea80ff__1_1__attempt_0/db/a62f79a2-bca7-40ff-9723-01d9a0d98f4a
at
org.apache.flink.state.forst.datatransfer.CopyDataTransferStrategy.copyFileFromCheckpoint(CopyDataTransferStrategy.java:282)
at
org.apache.flink.state.forst.datatransfer.CopyDataTransferStrategy.transferFromCheckpoint(CopyDataTransferStrategy.java:91)
at
org.apache.flink.state.forst.datatransfer.ForStStateDataTransfer.lambda$transferAllStateDataToDirectoryAsync$4(ForStStateDataTransfer.java:305)
at
org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:48)
at
java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
... 1 more
Caused by: java.io.FileNotFoundException: No such file or directory:
s3a://flink-state/flink-2-smoke-testing-job/checkpoints/f3436c2b6f059985605ce8a4f3cdc841/shared/op_AsyncStreamFlatMap_34849252e53e8aeadce2388c44ea80ff__1_1__attempt_0/db/a62f79a2-bca7-40ff-9723-01d9a0d98f4a
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3866)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.extractOrFetchSimpleFileStatus(S3AFileSystem.java:5401)
at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:1465)
at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:1441)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:976)
at
org.apache.flink.fs.s3hadoop.common.HadoopFileSystem.open(HadoopFileSystem.java:134)
at
org.apache.flink.fs.s3hadoop.common.HadoopFileSystem.open(HadoopFileSystem.java:38)
at
org.apache.flink.core.fs.PluginFileSystemFactory$ClassLoaderFixingFileSystem.open(PluginFileSystemFactory.java:128)
at
org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:77)
at
org.apache.flink.state.forst.datatransfer.CopyDataTransferStrategy.copyFileFromCheckpoint(CopyDataTransferStrategy.java:261)
... 7 more {panel}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
