[ 
https://issues.apache.org/jira/browse/FLINK-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119760#comment-16119760
 ] 

ASF GitHub Bot commented on FLINK-7021:
---------------------------------------

Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/4214#discussion_r132162054
  
    --- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/leaderretrieval/ZooKeeperLeaderRetrievalService.java
 ---
    @@ -196,4 +198,12 @@ protected void handleStateChange(ConnectionState 
newState) {
                                break;
                }
        }
    +
    +   @Override
    +   public void unhandledError(String s, Throwable throwable) {
    +           LOG.warn("Unhandled error encountered while retrieving 
Zookeeper leader.");
    +           final RuntimeException e = new RuntimeException(s, throwable);
    +           this.leaderListener.handleError(e);
    +           throw e;
    --- End diff --
    
    I think it should be enough to call `this.leaderListener.handleError`.


> Flink Task Manager hangs on startup if one Zookeeper node is unresolvable
> -------------------------------------------------------------------------
>
>                 Key: FLINK-7021
>                 URL: https://issues.apache.org/jira/browse/FLINK-7021
>             Project: Flink
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.0, 1.3.0, 1.2.1, 1.3.1
>         Environment: Kubernetes cluster running:
> * Flink 1.3.0 Job Manager & Task Manager on Java 8u131
> * Zookeeper 3.4.10 cluster with 3 nodes
>            Reporter: Scott Kidder
>            Assignee: Scott Kidder
>
> h2. Problem
> Flink Task Manager will hang during startup if one of the Zookeeper nodes in 
> the Zookeeper connection string is unresolvable.
> h2. Expected Behavior
> Flink should retry name resolution & connection to Zookeeper nodes with 
> exponential back-off.
> h2. Environment Details
> We're running Flink and Zookeeper in Kubernetes on CoreOS. CoreOS can run in 
> a configuration that automatically detects and applies operating system 
> updates. We have a Zookeeper node running on the same CoreOS instance as 
> Flink. It's possible that the Zookeeper node will not yet be started when the 
> Flink components are started. This could cause hostname resolution of the 
> Zookeeper nodes to fail.
> h3. Flink Task Manager Logs
> {noformat}
> 2017-06-27 15:38:51,713 INFO  
> org.apache.flink.runtime.taskmanager.TaskManager              - Using 
> configured hostname/address for TaskManager: 10.2.45.11
> 2017-06-27 15:38:51,714 INFO  
> org.apache.flink.runtime.taskmanager.TaskManager              - Starting 
> TaskManager
> 2017-06-27 15:38:51,714 INFO  
> org.apache.flink.runtime.taskmanager.TaskManager              - Starting 
> TaskManager actor system at 10.2.45.11:6122.
> 2017-06-27 15:38:52,950 INFO  akka.event.slf4j.Slf4jLogger                    
>               - Slf4jLogger started
> 2017-06-27 15:38:53,079 INFO  Remoting                                        
>               - Starting remoting
> 2017-06-27 15:38:53,573 INFO  Remoting                                        
>               - Remoting started; listening on addresses 
> :[akka.tcp://flink@10.2.45.11:6122]
> 2017-06-27 15:38:53,576 INFO  
> org.apache.flink.runtime.taskmanager.TaskManager              - Starting 
> TaskManager actor
> 2017-06-27 15:38:53,660 INFO  
> org.apache.flink.runtime.io.network.netty.NettyConfig         - NettyConfig 
> [server address: /10.2.45.11, server port: 6121, ssl enabled: false, memory 
> segment size (bytes): 32768, transport type: NIO, number of server threads: 2 
> (manual), number of client threads: 2 (manual), server connect backlog: 0 
> (use Netty's default), client connect timeout (sec): 120, send/receive buffer 
> size (bytes): 0 (use Netty's default)]
> 2017-06-27 15:38:53,682 INFO  
> org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration  - Messages 
> have a max timeout of 10000 ms
> 2017-06-27 15:38:53,688 INFO  
> org.apache.flink.runtime.taskexecutor.TaskManagerServices     - Temporary 
> file directory '/tmp': total 49 GB, usable 42 GB (85.71% usable)
> 2017-06-27 15:38:54,071 INFO  
> org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated 96 
> MB for network buffer pool (number of memory segments: 3095, bytes per 
> segment: 32768).
> 2017-06-27 15:38:54,564 INFO  
> org.apache.flink.runtime.io.network.NetworkEnvironment        - Starting the 
> network environment and its components.
> 2017-06-27 15:38:54,576 INFO  
> org.apache.flink.runtime.io.network.netty.NettyClient         - Successful 
> initialization (took 4 ms).
> 2017-06-27 15:38:54,677 INFO  
> org.apache.flink.runtime.io.network.netty.NettyServer         - Successful 
> initialization (took 101 ms). Listening on SocketAddress /10.2.45.11:6121.
> 2017-06-27 15:38:54,981 INFO  
> org.apache.flink.runtime.taskexecutor.TaskManagerServices     - Limiting 
> managed memory to 0.7 of the currently free heap space (612 MB), memory will 
> be allocated lazily.
> 2017-06-27 15:38:55,050 INFO  
> org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager 
> uses directory /tmp/flink-io-ca01554d-f25e-4c17-a828-96d82b43d4a7 for spill 
> files.
> 2017-06-27 15:38:55,061 INFO  org.apache.flink.runtime.metrics.MetricRegistry 
>               - Configuring StatsDReporter with {interval=10 SECONDS, 
> port=8125, host=localhost, 
> class=org.apache.flink.metrics.statsd.StatsDReporter}.
> 2017-06-27 15:38:55,065 INFO  org.apache.flink.metrics.statsd.StatsDReporter  
>               - Configured StatsDReporter with {host:localhost, port:8125}
> 2017-06-27 15:38:55,065 INFO  org.apache.flink.runtime.metrics.MetricRegistry 
>               - Periodically reporting metrics in intervals of 10 SECONDS for 
> reporter statsd of type org.apache.flink.metrics.statsd.StatsDReporter.
> 2017-06-27 15:38:55,175 INFO  org.apache.flink.runtime.filecache.FileCache    
>               - User file cache uses directory 
> /tmp/flink-dist-cache-e4c5bcc5-7513-40d9-a665-0d33c80a36ba
> 2017-06-27 15:38:55,187 INFO  org.apache.flink.runtime.filecache.FileCache    
>               - User file cache uses directory 
> /tmp/flink-dist-cache-310ba2f8-f96a-4c3f-b1db-35ac26b83f7e
> 2017-06-27 15:38:55,273 INFO  
> org.apache.flink.runtime.taskmanager.TaskManager              - Starting 
> TaskManager actor at akka://flink/user/taskmanager#207081801.
> 2017-06-27 15:38:55,273 INFO  
> org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager 
> data connection information: 7f86855dac2af4cca9eb2ae4c046630e @ 
> flink-taskmanager-3116622558-sqggc (dataPort=6121)
> 2017-06-27 15:38:55,273 INFO  
> org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager 
> has 2 task slot(s).
> 2017-06-27 15:38:55,276 INFO  
> org.apache.flink.runtime.taskmanager.TaskManager              - Memory usage 
> stats: [HEAP: 124/981/981 MB, NON HEAP: 43/44/-1 MB (used/committed/max)]
> 2017-06-27 15:38:55,276 INFO  
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  - 
> Starting ZooKeeperLeaderRetrievalService.
> 2017-06-27 15:39:10,289 ERROR 
> org.apache.flink.shaded.org.apache.curator.ConnectionState    - Connection 
> timed out for connection string 
> (zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181)
>  and timeout (15000) / elapsed (18617)
> org.apache.flink.shaded.org.apache.curator.CuratorConnectionLossException: 
> KeeperErrorCode = ConnectionLoss
>       at 
> org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
>       at 
> org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
>       at 
> org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
>       at 
> org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.reset(NodeCache.java:242)
>       at 
> org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:175)
>       at 
> org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:154)
>       at 
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService.start(ZooKeeperLeaderRetrievalService.java:100)
>       at 
> org.apache.flink.runtime.taskmanager.TaskManager.preStart(TaskManager.scala:205)
>       at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
>       at 
> org.apache.flink.runtime.taskmanager.TaskManager.aroundPreStart(TaskManager.scala:120)
>       at akka.actor.ActorCell.create(ActorCell.scala:580)
>       at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
>       at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
>       at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
>       at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>       at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>       at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2017-06-27 15:39:30,349 INFO  org.apache.zookeeper.ZooKeeper                  
>               - Initiating client connection, 
> connectString=zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181
>  sessionTimeout=60000 
> watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@16f7b4af
> 2017-06-27 15:40:00,388 WARN  
> org.apache.flink.shaded.org.apache.curator.ConnectionState    - Connection 
> attempt unsuccessful after 68719 (greater than max timeout of 60000). 
> Resetting connection and trying again with a new connection.
> 2017-06-27 15:40:00,388 INFO  org.apache.zookeeper.ZooKeeper                  
>               - Initiating client connection, 
> connectString=zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181
>  sessionTimeout=60000 
> watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@16f7b4af
> 2017-06-27 15:40:00,450 ERROR 
> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl
>   - Ensure path threw exception
> java.net.UnknownHostException: zookeeper-1.zookeeper: Name or service not 
> known
>       at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
>       at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
>       at 
> java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
>       at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
>       at java.net.InetAddress.getAllByName(InetAddress.java:1192)
>       at java.net.InetAddress.getAllByName(InetAddress.java:1126)
>       at 
> org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
>       at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
>       at 
> org.apache.flink.shaded.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
>       at 
> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
>       at 
> org.apache.flink.shaded.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
>       at 
> org.apache.flink.shaded.org.apache.curator.HandleHolder.internalClose(HandleHolder.java:128)
>       at 
> org.apache.flink.shaded.org.apache.curator.HandleHolder.closeAndReset(HandleHolder.java:77)
>       at 
> org.apache.flink.shaded.org.apache.curator.ConnectionState.reset(ConnectionState.java:261)
>       at 
> org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:221)
>       at 
> org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
>       at 
> org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
>       at 
> org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
>       at 
> org.apache.flink.shaded.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
>       at 
> org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
>       at 
> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:594)
>       at 
> org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:158)
>       at 
> org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32)
>       at 
> org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.reset(NodeCache.java:242)
>       at 
> org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:175)
>       at 
> org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:154)
>       at 
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService.start(ZooKeeperLeaderRetrievalService.java:100)
>       at 
> org.apache.flink.runtime.taskmanager.TaskManager.preStart(TaskManager.scala:205)
>       at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
>       at 
> org.apache.flink.runtime.taskmanager.TaskManager.aroundPreStart(TaskManager.scala:120)
>       at akka.actor.ActorCell.create(ActorCell.scala:580)
>       at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
>       at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
>       at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
>       at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>       at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>       at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to