Hi,
The cause of the error is clearly shown in the log. You need to check the
logic in 'com.flink.batch.lbs.running.hbase.udf.
BatchRunningTimesHBaseUdfRichSinkFunction#flush'
Best,
Weihua
On Fri, Feb 24, 2023 at 4:26 PM simple <1028108...@qq.com> wrote:
>
> 2023-02-24 16:05:26,104 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
> Custom Source (1/1)
> (9644f5bc58ed204c6d31bb133156fa95_bc764cd8ddf7a0cff126f51c16239658_0_0)
> switched from RUNNING to CANCELING.
> 2023-02-24 16:05:26,108 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Flat Map (
> 1/2)
> (9644f5bc58ed204c6d31bb133156fa95_0a448493b4782967b150582570326227_0_0)
> switched from RUNNING to CANCELING.
> 2023-02-24 16:05:26,108 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Flat Map (
> 2/2)
> (9644f5bc58ed204c6d31bb133156fa95_0a448493b4782967b150582570326227_1_0)
> switched from RUNNING to CANCELING.
> 2023-02-24 16:05:26,108 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
> KeyedProcess -> Sink: Unnamed (2/2)
> (9644f5bc58ed204c6d31bb133156fa95_e70bbd798b564e0a50e10e343f1ac56b_1_0)
> switched from RUNNING to CANCELING.
> 2023-02-24 16:05:26,137 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Flat Map (
> 1/2)
> (9644f5bc58ed204c6d31bb133156fa95_0a448493b4782967b150582570326227_0_0)
> switched from CANCELING to CANCELED.
> 2023-02-24 16:05:26,140 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Flat Map (
> 2/2)
> (9644f5bc58ed204c6d31bb133156fa95_0a448493b4782967b150582570326227_1_0)
> switched from CANCELING to CANCELED.
> 2023-02-24 16:05:26,141 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
> KeyedProcess -> Sink: Unnamed (2/2)
> (9644f5bc58ed204c6d31bb133156fa95_e70bbd798b564e0a50e10e343f1ac56b_1_0)
> switched from CANCELING to CANCELED.
> 2023-02-24 16:05:26,145 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> 299ff4a6ea193d7c110a19a3de33edab:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=1}]
> 2023-02-24 16:05:26,152 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
> Custom Source (1/1)
> (9644f5bc58ed204c6d31bb133156fa95_bc764cd8ddf7a0cff126f51c16239658_0_0)
> switched from CANCELING to CANCELED.
> 2023-02-24 16:05:26,155 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Clearing resource requirements of job 299ff4a6ea193d7c110a19a3de33edab
> 2023-02-24 16:05:26,154 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job [
> 10204-08]BatchRunningTimesLBSHBaseJobApp
> (299ff4a6ea193d7c110a19a3de33edab) switched from state FAILING to FAILED.
> org.apache.flink.runtime.JobException: Recovery is suppressed by
> NoRestartBackoffTimeStrategy
> at
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:139)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:83)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:256)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:247)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:240)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:738)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:715)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:78)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:477)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:?]
> at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:309)
> ~[flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
> ~[flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:307)
> ~[flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:222)
> ~[flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:84)
> ~[flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:168)
> ~[flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at akka.actor.Actor.aroundReceive(Actor.scala:537)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at akka.actor.Actor.aroundReceive$(Actor.scala:535)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at akka.actor.ActorCell.invoke(ActorCell.scala:548)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at akka.dispatch.Mailbox.run(Mailbox.scala:231)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
> [flink-rpc-akka_24401257-5903-4217-89f4-09e8c9e6f005.jar:1.16.0]
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
> at
> java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
> [?:?]
> at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
> [?:?]
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
> [?:?]
> Caused by: org.apache.flink.runtime.taskmanager.AsynchronousException:
> Caught exception while processing timer.
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$StreamTaskAsyncExceptionHandler.handleAsyncException(StreamTask.java:1568)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.handleAsyncException(StreamTask.java:1543)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1683)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1672)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at java.lang.Thread.run(Thread.java:834) ~[?:?]
> Caused by: org.apache.flink.streaming.runtime.tasks.TimerException:
> org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
> Could not forward element to next operator
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1683)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1672)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at java.lang.Thread.run(Thread.java:834) ~[?:?]
> Caused by:
> org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
> Could not forward element to next operator
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:99)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.transformation(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:87)
> ~[?:?]
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.resultOut(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:76)
> ~[?:?]
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.onTimerCallback(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:48)
> ~[?:?]
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.onTimer(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:43)
> ~[?:?]
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.invokeUserFunction(KeyedProcessOperator.java:91)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.onProcessingTime(KeyedProcessOperator.java:76)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:284)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1681)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1672)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at java.lang.Thread.run(Thread.java:834) ~[?:?]
> Caused by: java.lang.RuntimeException: exception
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.checkErrorAndRethrow(BatchRunningTimesHBaseUdfRichSinkFunction.java:264)
> ~[?:?]
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.flush(BatchRunningTimesHBaseUdfRichSinkFunction.java:257)
> ~[?:?]
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.invoke(BatchRunningTimesHBaseUdfRichSinkFunction.java:235)
> ~[?:?]
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.invoke(BatchRunningTimesHBaseUdfRichSinkFunction.java:37)
> ~[?:?]
> at
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.transformation(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:87)
> ~[?:?]
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.resultOut(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:76)
> ~[?:?]
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.onTimerCallback(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:48)
> ~[?:?]
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.onTimer(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:43)
> ~[?:?]
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.invokeUserFunction(KeyedProcessOperator.java:91)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.onProcessingTime(KeyedProcessOperator.java:76)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:284)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1681)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1672)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
> ~[flink-streaming-java-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
> ~[flink-runtime-1.16.0.jar:1.16.0]
> at java.lang.Thread.run(Thread.java:834) ~[?:?]
> Caused by: java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
> ~[?:?]
> at java.util.ArrayList$Itr.next(ArrayList.java:997) ~[?:?]
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:189)
> ~[hbase-client-2.2.2.jar:2.2.2]
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.flush(BatchRunningTimesHBaseUdfRichSinkFunction.java:244)
> ~[?:?]
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.lambda$open$0(BatchRunningTimesHBaseUdfRichSinkFunction.java:97)
> ~[?:?]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> ~[?:?]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> ~[?:?]
> at java.lang.Thread.run(Thread.java:834) ~[?:?]
> 2023-02-24 16:05:26,158 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Stopping
> checkpoint coordinator for job 299ff4a6ea193d7c110a19a3de33edab.
> 2023-02-24 16:05:26,178 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job
> 299ff4a6ea193d7c110a19a3de33edab reached terminal state FAILED.
> org.apache.flink.runtime.JobException: Recovery is suppressed by
> NoRestartBackoffTimeStrategy
> at
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:139)
> at
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:83)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:256)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:247)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:240)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:738)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:715)
> at
> org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:78)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:477)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62
> )
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43
> )
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:309)
> at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:307)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:222)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:84)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:168)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
> at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> at akka.actor.Actor.aroundReceive(Actor.scala:537)
> at akka.actor.Actor.aroundReceive$(Actor.scala:535)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
> at akka.actor.ActorCell.invoke(ActorCell.scala:548)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
> at akka.dispatch.Mailbox.run(Mailbox.scala:231)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
> at
> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
> at
> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020
> )
> at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656
> )
> at
> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594
> )
> at
> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183
> )
> Caused by: org.apache.flink.runtime.taskmanager.AsynchronousException:
> Caught exception while processing timer.
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$StreamTaskAsyncExceptionHandler.handleAsyncException(StreamTask.java:1568)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.handleAsyncException(StreamTask.java:1543)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1683)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1672)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by:
> TimerException{org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
> Could not forward element to next operator}
> ... 15 more
> Caused by:
> org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
> Could not forward element to next operator
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:99)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
> at
> org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.transformation(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:87)
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.resultOut(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:76)
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.onTimerCallback(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:48)
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.onTimer(BatchRunningTimesHBaseUdfKeyedProcessFunctionV2.java:43)
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.invokeUserFunction(KeyedProcessOperator.java:91)
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.onProcessingTime(KeyedProcessOperator.java:76)
> at
> org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:284)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1681)
> ... 14 more
> Caused by: java.lang.RuntimeException: exception
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.checkErrorAndRethrow(BatchRunningTimesHBaseUdfRichSinkFunction.java:264)
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.flush(BatchRunningTimesHBaseUdfRichSinkFunction.java:257)
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.invoke(BatchRunningTimesHBaseUdfRichSinkFunction.java:235)
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.invoke(BatchRunningTimesHBaseUdfRichSinkFunction.java:37)
> at
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
> ... 27 more
> Caused by: java.util.ConcurrentModificationException
> at
> java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043
> )
> at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:189)
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.flush(BatchRunningTimesHBaseUdfRichSinkFunction.java:244)
> at
> com.flink.batch.lbs.running.hbase.udf.BatchRunningTimesHBaseUdfRichSinkFunction.lambda$open$0(BatchRunningTimesHBaseUdfRichSinkFunction.java:97)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515
> )
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305
> )
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128
> )
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628
> )
> ... 1 more
> 2023-02-24 16:05:26,225 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job
> 299ff4a6ea193d7c110a19a3de33edab has been registered for cleanup in the
> JobResultStore after reaching a terminal state.
> 2023-02-24 16:05:26,231 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Stopping
> the JobMaster for job '[10204-08]BatchRunningTimesLBSHBaseJobApp'
> (299ff4a6ea193d7c110a19a3de33edab).
> 2023-02-24 16:05:26,237 INFO
> org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] -
> Shutting down
> 2023-02-24 16:05:26,239 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] -
> Disconnect TaskExecutor localhost:45013-1b5bd3 because: Stopping JobMaster
> for job '[10204-08]BatchRunningTimesLBSHBaseJobApp'
> (299ff4a6ea193d7c110a19a3de33edab).
> 2023-02-24 16:05:26,240 INFO
> org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool [] -
> Releasing slot [ce80357b0859df0b268566294786c8a9].
> 2023-02-24 16:05:26,242 INFO
> org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool [] -
> Releasing slot [7f78bde0f41f2cf0931a01a54db98bfb].
> 2023-02-24 16:05:26,243 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Close
> ResourceManager connection 4d0adaf4e63930c3693def906b4a3e7e: Stopping
> JobMaster for job '[10204-08]BatchRunningTimesLBSHBaseJobApp'
> (299ff4a6ea193d7c110a19a3de33edab).
> 2023-02-24 16:05:26,246 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
> Disconnect job manager 00000000000000000000000000000000
> @akka.tcp://flink@localhost:6123/user/rpc/jobmanager_2 for job
> 299ff4a6ea193d7c110a19a3de33edab from the resource manager.
>
> ------------------------------
> 冉庆超
> 1028108...@qq.com
>
> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=%E5%86%89%E5%BA%86%E8%B6%85&icon=http%3A%2F%2Fthirdqq.qlogo.cn%2Fg%3Fb%3Dsdk%26k%3DKiaSrhuQkdzkuLbXWejpF0g%26s%3D100%26t%3D1570761082%3Frand%3D1641817208&mail=1028108734%40qq.com&code=>
>
>
