[
https://issues.apache.org/jira/browse/FLINK-34489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821201#comment-17821201
]
Matthias Pohl commented on FLINK-34489:
---------------------------------------
The first case is caused by a TaskManager not coming up on time leaving the job
with an expected parallelism of 4 hanging on 3 TaskManagers/Slots. The failed
TM reveals the following error logs:
{code}
[...]
2024-02-21 07:30:28,915 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not
resolve ResourceManager address
pekko.ssl.tcp://flink@localhost:6123/user/rpc/resourcemanager_*, retrying in
10000 ms: Could not connect to rpc endpoint under address pekko.ssl
.tcp://flink@localhost:6123/user/rpc/resourcemanager_*.
2024-02-21 07:30:38,939 WARN org.apache.pekko.remote.EndpointReader
[] - Discarding inbound message to
[Actor[pekko://flink/temp/_user_rpc_resourcemanager_*$o]] in read-only
association to [pekko.ssl.tcp://flink@localhost:6123]. If this happens often
you may
consider using pekko.remote.use-passive-connections=off or use Artery TCP.
2024-02-21 07:30:48,954 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not
resolve ResourceManager address
pekko.ssl.tcp://flink@localhost:6123/user/rpc/resourcemanager_*, retrying in
10000 ms: Could not connect to rpc endpoint under address pekko.ssl
.tcp://flink@localhost:6123/user/rpc/resourcemanager_*.
2024-02-21 07:30:58,353 ERROR
org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Fatal error
occurred in TaskExecutor
pekko.ssl.tcp://flink@localhost:33341/user/rpc/taskmanager_0.
org.apache.flink.runtime.taskexecutor.exceptions.RegistrationTimeoutException:
Could not register at the ResourceManager within the specified maximum
registration duration PT5M. This indicates a problem with this instance.
Terminating now.
at
org.apache.flink.runtime.taskexecutor.TaskExecutor.registrationTimeout(TaskExecutor.java:1614)
~[flink-dist-1.20-SNAPSHOT.jar:1.20-SNAPSHOT]
at
org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$startRegistrationTimeout$18(TaskExecutor.java:1599)
~[flink-dist-1.20-SNAPSHOT.jar:1.20-SNAPSHOT]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRunAsync$4(PekkoRpcActor.java:451)
~[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
~[flink-dist-1.20-SNAPSHOT.jar:1.20-SNAPSHOT]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRunAsync(PekkoRpcActor.java:451)
~[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:218)
~[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:168)
~[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at
org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at
org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at
org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at
org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253)
[flink-rpc-akka1f9d7113-8709-4799-8cb2-0d9f15b6f908.jar:1.20-SNAPSHOT]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[?:1.8.0_402]
at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[?:1.8.0_402]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[?:1.8.0_402]
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
[?:1.8.0_402]
[...]
{code}
> New File Sink end-to-end test timed out
> ---------------------------------------
>
> Key: FLINK-34489
> URL: https://issues.apache.org/jira/browse/FLINK-34489
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.19.0, 1.20.0
> Reporter: Matthias Pohl
> Priority: Critical
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=57707&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=0f3adb59-eefa-51c6-2858-3654d9e0749d&l=3726
> {code}
> eb 21 07:26:03 Number of produced values 10770/60000
> Feb 21 07:39:50 Test (pid: 151375) did not finish after 900 seconds.
> Feb 21 07:39:50 Printing Flink logs and killing it:
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
