Hi Xianwen, Looks like the State Processor API needs to be updated for the new state backend factory stack. For now, just use RocksDBStateBackend and it will work as intended.
I've opened a ticket: https://issues.apache.org/jira/browse/FLINK-23728 Seth On Wed, Aug 11, 2021 at 2:08 AM xianwen jin <xianwenji...@gmail.com> wrote: > Hi Yun, > > Here's the stack trace > > java.util.concurrent.ExecutionException: java.io.IOException: Size of the > state is larger than the maximum permitted memory-backed state. > Size=100091190 , maxSize=5242880 . Consider using a different state > backend, like the File System State backend. > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > ~[?:1.8.0_292] > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > ~[?:1.8.0_292] > at > org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:636) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>(OperatorSnapshotFinalizer.java:54) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.state.api.output.SnapshotUtils.snapshot(SnapshotUtils.java:67) > ~[?:?] > at > org.apache.flink.state.api.output.operators.KeyedStateBootstrapOperator.endInput(KeyedStateBootstrapOperator.java:90) > ~[?:?] > at > org.apache.flink.state.api.output.BoundedStreamTask.processInput(BoundedStreamTask.java:107) > ~[?:?] > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:681) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:636) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:620) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.state.api.output.BoundedOneInputStreamTaskRunner.mapPartition(BoundedOneInputStreamTaskRunner.java:80) > ~[?:?] > at > org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:113) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:519) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:360) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292] > Caused by: java.io.IOException: Size of the state is larger than the > maximum permitted memory-backed state. Size=100091190 , maxSize=5242880 . > Consider using a different state backend, like the File System State > backend. > at > org.apache.flink.runtime.state.memory.MemCheckpointStreamFactory.checkSize(MemCheckpointStreamFactory.java:61) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.state.memory.MemCheckpointStreamFactory$MemoryCheckpointOutputStream.closeAndGetBytes(MemCheckpointStreamFactory.java:141) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.state.memory.MemCheckpointStreamFactory$MemoryCheckpointOutputStream.closeAndGetHandle(MemCheckpointStreamFactory.java:121) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.state.CheckpointStreamWithResultProvider$PrimaryStreamOnly.closeAndFinalizeCheckpointStreamResult(CheckpointStreamWithResultProvider.java:75) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.state.FullSnapshotAsyncWriter.get(FullSnapshotAsyncWriter.java:87) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.state.SnapshotStrategyRunner$1.callInternal(SnapshotStrategyRunner.java:91) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.state.SnapshotStrategyRunner$1.callInternal(SnapshotStrategyRunner.java:88) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.state.AsyncSnapshotCallable.call(AsyncSnapshotCallable.java:78) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[?:1.8.0_292] > at > org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:633) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > ... 16 more > 2021-08-09 15:59:53,314 INFO > org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy > [] - Calculating tasks to restart to recover the failed task > 488bec663292a28f42ac2a625db66756_0. > 2021-08-09 15:59:53,315 INFO > org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy > [] - 25 tasks should be restarted to recover the failed task > 488bec663292a28f42ac2a625db66756_0. > 2021-08-09 15:59:53,364 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job > PipelineName (43e80a9e125b45e6be8ac263f1c8fd4e) switched from state RUNNING > to FAILING. > org.apache.flink.runtime.JobException: Recovery is suppressed by > NoRestartBackoffTimeStrategy > at > org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:207) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:197) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:188) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:677) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:435) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source) > ~[?:?] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_292] > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292] > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) > ~[flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at akka.actor.Actor.aroundReceive(Actor.scala:517) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at akka.actor.Actor.aroundReceive$(Actor.scala:515) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > at > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > [flink-dist_2.12-1.13.1-stream1.jar:1.13.1-stream1] > > Thanks. > > On 2021/08/11 02:30:41, "Yun Gao" <yungao...@aliyun.com> wrote: > > Hi Xianwen, > > > > Could you also attach the full stack of the exception~? > > > > Very thanks > > > > > > ------------------------------------------------------------------ > > Sender:xianwen jin<xianwenji...@gmail.com> > > Date:2021/08/10 21:03:44 > > Recipient:<user@flink.apache.org> > > Theme:State Processor API with EmbeddedRocksDBStateBackend > > > > Hi Flink Community, > > I have an issue regarding create a savepoint with > EmbeddedRocksDBStateBackend. > > Since I have to use dataset for state processor api, it seems that the > checkpoint storage is always set to jobmanager even though I pass: > > state.checkpoint-storage: filesystem > > state.checkpoints.dir: 's3://flink-bucket01/checkpoints/saldo-es/state' > > in the flink-conf.yaml. > > This causes an exception when the state to be written is more than 5 MB: > > java.io.IOException: Size of the state is larger than the maximum > permitted memory-backed state. Size=100091190 , maxSize=5242880 . Consider > using a different state backend, like the File System State backend. > > > > There's a way to set checkpoint storage to filesystem when I'm using > state processor api? > > > > Thanks for any advice or help > > >