Simonas created FLINK-28695:
-------------------------------
Summary: Fail to send partition request to restarted taskmanager
Key: FLINK-28695
URL: https://issues.apache.org/jira/browse/FLINK-28695
Project: Flink
Issue Type: Bug
Components: Deployment / Kubernetes, Runtime / Network
Affects Versions: 1.15.1, 1.15.0
Reporter: Simonas
Attachments: deployment.txt, job_log.txt, jobmanager_config.txt,
jobmanager_logs.txt, pod_restart.txt, taskmanager_config.txt
After upgrade to *1.15.1* we started getting error while deploying JOB
{code:java}
org.apache.flink.runtime.io.network.netty.exception.LocalTransportException:
Sending the partition request to '/XXX.XXX.XX.32:6121 (#0)' failed. at
org.apache.flink.runtime.io.network.netty.NettyPartitionRequestClient$1.operationComplete(NettyPartitionRequestClient.java:145)
.... {code}
{code:java}
Caused by:
org.apache.flink.shaded.netty4.io.netty.channel.StacklessClosedChannelException
atrg.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object,
ChannelPromise)(Unknown Source){code}
After investigation we managed narrow it down to the exact behavior then this
issue happens:
# Deploying JOB on fresh kubernetes session cluster with multiple
TaskManagers: TM1 and TM2 is successful. Job has multiple partitions running on
both TM1 and TM2.
# One TaskManager TM2 (XXX.XXX.XX.32) fails for unrelated issue. For example
OOM exception.
# Kubernetes POD with mentioned TaskManager TM2 is restarted. POD retains same
IP address as before.
# JobManager is able to pickup the restarted TM2 (XXX.XXX.XX.32)
# JOB is restarted because it was running on the failed TaskManager TM2
# TM1 data channel to TM2 is closed and we get LocalTransportException:
Sending the partition request to '/XXX.XXX.XX.32:6121 (#0)' failed during JOB
running stage.
# When we explicitly delete pod with TM2 it creates new POD with different IP
address and JOB is able to start again.
Important to note that we didn't encountered this issue with previous *1.14.4*
version and TaskManager restarts didn't cause such error.
Please note attached kubernetes deployments and reduced logs from JobManager.
TaskManager logs did show errors before error, but doesn't show anything
significant after restart.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
