Stephan Ewen commented on FLINK-8829:

One reason why this could happen that we saw in the past is conflicts between 
Akka's Netty and Netty instances pulled in by Hadoop (through EMR). In some 
cases, that resulted in connections dying even though network connectivity was 

In Flink 1.4.x, we shade and relocate Akka's Netty to ensure such conflicts 
don't happen any more.

You could try to do the following:
  - Upgrade for Flink's 1.4.x line.
  - Try to remove Netty being pulled in via Hadoop. That is not super easy, you 
would need to use a Flink version built against the same Hadoop version as EMR 
runs (Flink should exclude or shade Hadoop's netty) and prevent the Hadoop 
classpath from being added to the Flink classpath.

If you go with option one,  1.4.2 coming out in a few days, 1.4.1 is fine 
except for a classloading bug when using Kafka with a custom watermark 

> Flink in EMR(YARN) is down due to Akka communication issue
> ----------------------------------------------------------
>                 Key: FLINK-8829
>                 URL: https://issues.apache.org/jira/browse/FLINK-8829
>             Project: Flink
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.3.2
>            Reporter: Aleksandr Filichkin
>            Priority: Major
> Hi,
> We have running Flink 1.3.2 app in Amazon EMR with YARN. Every week our Flink 
> job is down due to:
> _2018-02-16 19:00:04,595 WARN akka.remote.ReliableDeliverySupervisor - 
> Association with remote system 
> [akka.tcp://[fl...@ip-10-97-34-209.tr-fr-nonprod.aws-int.thomsonreuters.com:42177]|mailto:fl...@ip-10-97-34-209.tr-fr-nonprod.aws-int.thomsonreuters.com:42177]]
>  has failed, address is now gated for [5000] ms. Reason: [Association failed 
> with 
> [akka.tcp://[fl...@ip-10-97-34-209.tr-fr-nonprod.aws-int.thomsonreuters.com:42177]]|mailto:fl...@ip-10-97-34-209.tr-fr-nonprod.aws-int.thomsonreuters.com:42177]]]
>  Caused by: [Connection refused: 
> ip-10-97-34-209.tr-fr-nonprod.aws-int.thomsonreuters.com/] 
> 2018-02-16 19:00:05,593 WARN akka.remote.RemoteWatcher - Detected 
> unreachable: 
> [akka.tcp://[fl...@ip-10-97-34-209.tr-fr-nonprod.aws-int.thomsonreuters.com:42177]|mailto:fl...@ip-10-97-34-209.tr-fr-nonprod.aws-int.thomsonreuters.com:42177]]
>  2018-02-16 19:00:05,596 INFO 
> org.apache.flink.runtime.client.JobSubmissionClientActor - Lost connection to 
> JobManager 
> akka.tcp://[fl...@ip-10-97-34-209.tr-fr-nonprod.aws-int.thomsonreuters.com:42177/user/jobmanager|mailto:fl...@ip-10-97-34-209.tr-fr-nonprod.aws-int.thomsonreuters.com:42177/user/jobmanager].
>  Triggering connection timeout._
> Do you have any ideas how to troubleshoot it?

This message was sent by Atlassian JIRA

Reply via email to