zhangrucong1982 created FLINK-5770:
--------------------------------------
Summary: Flink yarn session stop in non-detached model
Key: FLINK-5770
URL: https://issues.apache.org/jira/browse/FLINK-5770
Project: Flink
Issue Type: Bug
Components: Client
Affects Versions: 1.2.0
Environment: 1、the cluster contains 4 nodes;
2、every node has 380GB memory, and the CPU has 40 cores;
3、the OS is centOS7.2;
Reporter: zhangrucong1982
1、I user the recent version of flink, and use fink in security mode without
HA.the configurations in flink-conf.yaml are:
security.kerberos.login.keytab:
/home/demo/flink/release/flink-1.2.2/keytab/huawei1.keytab
security.kerberos.login.principal: huawei1
security.kerberos.login.contexts: Client,KafkaClient
2、then I use the command ./yarn-session.sh -n 2 to start the cluster with two
taskmanagers.
3、 But About the 4 hours later, the session is shutting down by itself. the
error stack is following:
2017-02-07 19:27:30,841 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system
[akka.tcp://flink@9-96-101-251:38650] has failed, address is now gated for
[5000] ms. Reason: [Disassociated]
2017-02-07 19:27:42,804 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli
- Exception while running the interactive command line interface
java.lang.RuntimeException: Unable to get ClusterClient status from Application
Client
at
org.apache.flink.yarn.YarnClusterClient.getClusterStatus(YarnClusterClient.java:248)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.runInteractiveCli(FlinkYarnSessionCli.java:410)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:663)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:476)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:473)
at
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:473)
Caused by: org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException:
Could not retrieve the leader gateway
at
org.apache.flink.runtime.util.LeaderRetrievalUtils.retrieveLeaderGateway(LeaderRetrievalUtils.java:142)
at
org.apache.flink.client.program.ClusterClient.getJobManagerGateway(ClusterClient.java:691)
at
org.apache.flink.yarn.YarnClusterClient.getClusterStatus(YarnClusterClient.java:243)
... 10 more
Caused by: java.util.concurrent.TimeoutException: Futures timed out after
[10000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
at
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:190)
at scala.concurrent.Await.result(package.scala)
at
org.apache.flink.runtime.util.LeaderRetrievalUtils.retrieveLeaderGateway(LeaderRetrievalUtils.java:140)
... 12 more
4、the detail log you can see in the following ：
https://docs.google.com/document/d/1mbxrCy6mHHFxcxPv8f7CCA3BI1QVGPeNiHxUQhuZP0o/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
