Tim created FLINK-14328:
---------------------------
Summary: JobCluster cannot reach TaskManager in K8s
Key: FLINK-14328
URL: https://issues.apache.org/jira/browse/FLINK-14328
Project: Flink
Issue Type: Bug
Reporter: Tim
Fix For: 1.9.0
I have a Job Cluster which I am running in K8s. It consists of
* job manager deployment (1)
* task manager deployment (1)
* service
This is more or less following the standard "Job Cluster" setup.
Additionally, (due to known issues of TMs talking to JMs), I have set
taskmanager.network.bind-policy to "ip", so that the task manager binds on the
IP of the pod rather than the pod name (which is not reachable via DNS). So
far so good.
Once the cluster is started, I can see the job running. I also see that the
JM's resource msnager has registered the TM.
{code:java}
2019-10-05 20:37:14.554 [flink-akka.actor.default-dispatcher-4] DEBUG
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Slot Pool Status:
status: connected to
akka.tcp://flink@data-capture-enrichedtrans-raw-jobcluster:6123/user/resourcemanager
registered TaskManagers: [f34656491b8dfae726d992d276dc6d39]
available slots: []
allocated slots: [[AllocatedSlot a00f44d19f38ca36da3ae5083c2d02ae @
f34656491b8dfae726d992d276dc6d39 @
data-capture-enrichedtrans-raw-taskmanager-674476f57c-26kxr (dataPort=35815) -
0]]
pending requests: []
}
{code}
However, I see several errors like below, before the job eventually fails
(maybe after 5 minutes), and goes into recovery. This happens until all
restarts are exhaused, at which point the cluster completely fails.
{code:java}
2019-10-05 20:42:14.768 [flink-akka.actor.default-dispatcher-19] WARN
akka.remote.ReliableDeliverySupervisor
flink-akka.remote.default-remote-dispatcher-6 - Association with remote system
[akka.tcp://[email protected]:50100] has failed, address is now gated for [50]
ms. Reason: [Association failed with [akka.tcp://[email protected]:50100]]
Caused by: [java.net.ConnectException: Connection refused: /10.107.38.92:50100]
{code}
{{To me it looks like the JM is not able to make a connection on the RPC port
of the taskmanager (50100 is the taskmanager.rpc.port setting).}}
{{Has anyone come across this issue?}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
