[
https://issues.apache.org/jira/browse/FLINK-26109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17491896#comment-17491896
]
Chesnay Schepler commented on FLINK-26109:
------------------------------------------
hmm. For some reason the TM is being shut down which fails the job. There
doesn't seem to be any errors though that could cause it...my first thought was
that the shutdown of the cluster might be noticed that quickly that the JM
still has time to fail the job, but the shutdown happens after the log check.
> Avro Confluent Schema Registry nightly end-to-end test failed on azure
> ----------------------------------------------------------------------
>
> Key: FLINK-26109
> URL: https://issues.apache.org/jira/browse/FLINK-26109
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Affects Versions: 1.15.0
> Reporter: Yun Gao
> Priority: Major
> Labels: test-stability
>
> {code:java}
> Feb 12 07:55:02 Stopping job timeout watchdog (with pid=130662)
> Feb 12 07:55:03 Checking for errors...
> Feb 12 07:55:03 Found error in log files; printing first 500 lines; see full
> logs for details:
> ...
> az209-567.vil1xujjdrkuxjp2ihtao45w0e.ax.internal.cloudapp.net
> (dataPort=41161).
> org.apache.flink.util.FlinkException: The TaskExecutor is shutting down.
> at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.onStop(TaskExecutor.java:456)
> ~[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
> ~[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.lambda$terminate$0(AkkaRpcActor.java:568)
> ~[flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
> ~[flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:567)
> ~[flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:191)
> ~[flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
> ~[flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
> ~[flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> ~[flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> ~[flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
> ~[flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> [flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> [flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at akka.actor.Actor.aroundReceive(Actor.scala:537)
> [flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at akka.actor.Actor.aroundReceive$(Actor.scala:535)
> [flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
> [flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
> [flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at akka.actor.ActorCell.invoke(ActorCell.scala:548)
> [flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
> [flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at akka.dispatch.Mailbox.run(Mailbox.scala:231)
> [flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
> [flink-rpc-akka_7dcae025-2017-4b0f-828d-f89a7ceb9bf7.jar:1.15-SNAPSHOT]
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> [?:1.8.0_312]
> at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> [?:1.8.0_312]
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> [?:1.8.0_312]
> {code}
> Also in the TM's log
>
> {code:java}
> 2022-02-12 07:55:04,764 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - RECEIVED
> SIGNAL 15: SIGTERM. Shutting down as requested.
> 2022-02-12 07:55:04,765 INFO
> org.apache.flink.runtime.blob.PermanentBlobCache [] - Shutting
> down BLOB cache
> 2022-02-12 07:55:04,767 INFO
> org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager [] -
> Shutting down TaskExecutorLocalStateStoresManager.
> 2022-02-12 07:55:04,768 INFO
> org.apache.flink.runtime.io.disk.FileChannelManagerImpl [] -
> FileChannelManager removed spill file directory
> /tmp/flink-io-e1efe10a-812c-476b-b48a-e16f6908ada4
> 2022-02-12 07:55:04,769 INFO
> org.apache.flink.runtime.blob.TransientBlobCache [] - Shutting
> down BLOB cache
> 2022-02-12 07:55:04,771 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Stopping
> TaskExecutor akka.tcp://[email protected]:45841/user/rpc/taskmanager_0.
> 2022-02-12 07:55:04,771 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Close
> ResourceManager connection 36726faf0b67603703f3d376f7193b16.
> 2022-02-12 07:55:04,772 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Close
> JobManager connection for job 4a839a700a510dd3fab48d7c6f1621b8.
> 2022-02-12 07:55:04,772 INFO org.apache.flink.runtime.taskmanager.Task
> [] - Attempting to fail task externally Sink: Writer -> Sink:
> Committer (1/1)#0 (67db69a53ebf3d709632d204db3c12e7).
> 2022-02-12 07:55:04,773 WARN org.apache.flink.runtime.taskmanager.Task
> [] - Sink: Writer -> Sink: Committer (1/1)#0
> (67db69a53ebf3d709632d204db3c12e7) switched from RUNNING to FAILED with
> failure cause: org.apache.flink.util.FlinkException: Disconnect from
> JobManager responsible for 4a839a700a510dd3fab48d7c6f1621b8.
> at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.disconnectJobManagerConnection(TaskExecutor.java:1679)
> at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$closeJob$18(TaskExecutor.java:1660)
> at java.util.Optional.ifPresent(Optional.java:159)
> at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.closeJob(TaskExecutor.java:1658)
> at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.onStop(TaskExecutor.java:462)
> at
> org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.lambda$terminate$0(AkkaRpcActor.java:568)
> at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:567)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:191)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
> at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> at akka.actor.Actor.aroundReceive(Actor.scala:537)
> at akka.actor.Actor.aroundReceive$(Actor.scala:535)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
> at akka.actor.ActorCell.invoke(ActorCell.scala:548)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
> at akka.dispatch.Mailbox.run(Mailbox.scala:231)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Caused by: org.apache.flink.util.FlinkException: The TaskExecutor is shutting
> down.
> at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.onStop(TaskExecutor.java:456)
> ... 24 more {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=31305&view=logs&j=fb37c667-81b7-5c22-dd91-846535e99a97&t=39a035c3-c65e-573c-fb66-104c66c28912&l=3871
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
