Re: Kylin often hangs up
> 2017-08-10 10:14:35,013 WARN [localhost-startStop-1-EventThread] client.HConnectionManager$HConnectionImplementation:2468 : This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired Obviously the ZK connection is not stable. On Thu, Aug 10, 2017 at 3:39 PM, shendandanwrote: > > > > Thanks for your respond,but Zk is working normally. > > > > > > At 2017-08-10 15:21:49, "Billy Liu" wrote: > >The first WARN happened during ZK timeout. Please check the ZK connection > >first. > > > >2017-08-10 10:58 GMT+08:00 shendandan : > > > >> Hi! > >>The process of kylin often hangs up,and the erroe in kylin,log: > >> > >> > >> 2017-08-10 10:14:22,292 INFO [BadQueryDetector] > >> service.BadQueryDetector:151 : System free memory less than 100 MB. 1 > >> queries running. > >> 2017-08-10 10:14:34,498 INFO [Thread-11-EventThread] > >> state.ConnectionStateManager:228 : State change: SUSPENDED > >> 2017-08-10 10:14:34,506 INFO [pool-7-thread-1] > >> threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 > actual > >> running, 0 ready, 63 already succeed, 3 error, 13 discarded, 0 others > >> 2017-08-10 10:14:34,999 INFO [localhost-startStop-1-SendThread( > >> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening > socket > >> connection to server hadoop-senior01.ctcf.com/10.1.8.90:2181. Will not > >> attempt to authenticate using SASL (unknown error) > >> 2017-08-10 10:14:34,999 INFO [Thread-11-SendThread(hadoop- > >> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket > >> connection to server hadoop-senior03.ctcf.com/10.1.8.92:2181. Will not > >> attempt to authenticate using SASL (unknown error) > >> 2017-08-10 10:14:34,999 INFO [localhost-startStop-1-SendThread( > >> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket > >> connection established, initiating session, client: /10.1.8.90:38229, > >> server: hadoop-senior01.ctcf.com/10.1.8.90:2181 > >> 2017-08-10 10:14:34,999 INFO [Thread-11-SendThread(hadoop- > >> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection > >> established, initiating session, client: /10.1.8.90:32962, server: > >> hadoop-senior03.ctcf.com/10.1.8.92:2181 > >> 2017-08-10 10:14:35,001 INFO [Thread-11-SendThread(hadoop- > >> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to > reconnect > >> to ZooKeeper service, session 0x25dc4bd90cc002a has expired, closing > socket > >> connection > >> 2017-08-10 10:14:35,002 INFO [localhost-startStop-1-SendThread( > >> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to > >> reconnect to ZooKeeper service, session 0x35dc4bd90460028 has expired, > >> closing socket connection > >> 2017-08-10 10:14:35,013 WARN [localhost-startStop-1-EventThread] > >> client.HConnectionManager$HConnectionImplementation:2468 : This client > >> just lost it's session with ZooKeeper, closing it. It will be recreated > >> next time someone needs it > >> org.apache.zookeeper.KeeperException$SessionExpiredException: > >> KeeperErrorCode = Session expired > >> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent( > >> ZooKeeperWatcher.java:517) > >> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher. > >> process(ZooKeeperWatcher.java:435) > >> at org.apache.zookeeper.ClientCnxn$EventThread. > >> processEvent(ClientCnxn.java:522) > >> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) > >> 2017-08-10 10:14:35,013 INFO [Thread-11-EventThread] > >> state.ConnectionStateManager:228 : State change: LOST > >> 2017-08-10 10:14:35,014 INFO [localhost-startStop-1-EventThread] > >> client.HConnectionManager$HConnectionImplementation:1837 : Closing > >> zookeeper sessionid=0x35dc4bd90460028 > >> 2017-08-10 10:14:35,014 WARN [Thread-11-EventThread] > >> curator.ConnectionState:289 : Session expired event received > >> 2017-08-10 10:14:35,014 INFO [localhost-startStop-1-EventThread] > >> zookeeper.ClientCnxn:512 : EventThread shut down > >> 2017-08-10 10:14:35,015 INFO [Thread-11-EventThread] > >> zookeeper.ZooKeeper:438 : Initiating client connection, connectString= > >> hadoop-senior01.ctcf.com:2181,hadoop-senior02.ctcf.com:2181, > >> hadoop-senior03.ctcf.com:2181 sessionTimeout=6 > >> watcher=org.apache.curator.ConnectionState@4d6f4fa7 > >> 2017-08-10 10:14:35,017 INFO [Thread-11-SendThread(hadoop- > >> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket > >> connection to server hadoop-senior02.ctcf.com/10.1.8.91:2181. Will not > >> attempt to authenticate using SASL (unknown error) > >> 2017-08-10 10:14:35,017 INFO [Thread-11-SendThread(hadoop- > >> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection > >> established, initiating session, client: /10.1.8.90:57503, server:
Re: Kylin often hangs up
Thanks for your respond,but Zk is working normally. At 2017-08-10 15:21:49, "Billy Liu"wrote: >The first WARN happened during ZK timeout. Please check the ZK connection >first. > >2017-08-10 10:58 GMT+08:00 shendandan : > >> Hi! >>The process of kylin often hangs up,and the erroe in kylin,log: >> >> >> 2017-08-10 10:14:22,292 INFO [BadQueryDetector] >> service.BadQueryDetector:151 : System free memory less than 100 MB. 1 >> queries running. >> 2017-08-10 10:14:34,498 INFO [Thread-11-EventThread] >> state.ConnectionStateManager:228 : State change: SUSPENDED >> 2017-08-10 10:14:34,506 INFO [pool-7-thread-1] >> threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual >> running, 0 ready, 63 already succeed, 3 error, 13 discarded, 0 others >> 2017-08-10 10:14:34,999 INFO [localhost-startStop-1-SendThread( >> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket >> connection to server hadoop-senior01.ctcf.com/10.1.8.90:2181. Will not >> attempt to authenticate using SASL (unknown error) >> 2017-08-10 10:14:34,999 INFO [Thread-11-SendThread(hadoop- >> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket >> connection to server hadoop-senior03.ctcf.com/10.1.8.92:2181. Will not >> attempt to authenticate using SASL (unknown error) >> 2017-08-10 10:14:34,999 INFO [localhost-startStop-1-SendThread( >> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket >> connection established, initiating session, client: /10.1.8.90:38229, >> server: hadoop-senior01.ctcf.com/10.1.8.90:2181 >> 2017-08-10 10:14:34,999 INFO [Thread-11-SendThread(hadoop- >> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection >> established, initiating session, client: /10.1.8.90:32962, server: >> hadoop-senior03.ctcf.com/10.1.8.92:2181 >> 2017-08-10 10:14:35,001 INFO [Thread-11-SendThread(hadoop- >> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to reconnect >> to ZooKeeper service, session 0x25dc4bd90cc002a has expired, closing socket >> connection >> 2017-08-10 10:14:35,002 INFO [localhost-startStop-1-SendThread( >> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to >> reconnect to ZooKeeper service, session 0x35dc4bd90460028 has expired, >> closing socket connection >> 2017-08-10 10:14:35,013 WARN [localhost-startStop-1-EventThread] >> client.HConnectionManager$HConnectionImplementation:2468 : This client >> just lost it's session with ZooKeeper, closing it. It will be recreated >> next time someone needs it >> org.apache.zookeeper.KeeperException$SessionExpiredException: >> KeeperErrorCode = Session expired >> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent( >> ZooKeeperWatcher.java:517) >> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher. >> process(ZooKeeperWatcher.java:435) >> at org.apache.zookeeper.ClientCnxn$EventThread. >> processEvent(ClientCnxn.java:522) >> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) >> 2017-08-10 10:14:35,013 INFO [Thread-11-EventThread] >> state.ConnectionStateManager:228 : State change: LOST >> 2017-08-10 10:14:35,014 INFO [localhost-startStop-1-EventThread] >> client.HConnectionManager$HConnectionImplementation:1837 : Closing >> zookeeper sessionid=0x35dc4bd90460028 >> 2017-08-10 10:14:35,014 WARN [Thread-11-EventThread] >> curator.ConnectionState:289 : Session expired event received >> 2017-08-10 10:14:35,014 INFO [localhost-startStop-1-EventThread] >> zookeeper.ClientCnxn:512 : EventThread shut down >> 2017-08-10 10:14:35,015 INFO [Thread-11-EventThread] >> zookeeper.ZooKeeper:438 : Initiating client connection, connectString= >> hadoop-senior01.ctcf.com:2181,hadoop-senior02.ctcf.com:2181, >> hadoop-senior03.ctcf.com:2181 sessionTimeout=6 >> watcher=org.apache.curator.ConnectionState@4d6f4fa7 >> 2017-08-10 10:14:35,017 INFO [Thread-11-SendThread(hadoop- >> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket >> connection to server hadoop-senior02.ctcf.com/10.1.8.91:2181. Will not >> attempt to authenticate using SASL (unknown error) >> 2017-08-10 10:14:35,017 INFO [Thread-11-SendThread(hadoop- >> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection >> established, initiating session, client: /10.1.8.90:57503, server: >> hadoop-senior02.ctcf.com/10.1.8.91:2181 >> 2017-08-10 10:14:35,021 INFO [Thread-11-EventThread] >> zookeeper.ClientCnxn:512 : EventThread shut down >> 2017-08-10 10:14:35,021 INFO [Thread-11-SendThread(hadoop- >> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:1235 : Session >> establishment complete on server hadoop-senior02.ctcf.com/10.1.8.91:2181, >> sessionid = 0x25dc4bd90cc002c, negotiated timeout = 4 >> 2017-08-10 10:14:35,021 INFO [Thread-11-EventThread] >> state.ConnectionStateManager:228 : State change: RECONNECTED >> 2017-08-10 10:15:09,204 INFO [pool-7-thread-1] >> threadpool.DefaultScheduler:118 : Job
Re: Kylin often hangs up
The first WARN happened during ZK timeout. Please check the ZK connection first. 2017-08-10 10:58 GMT+08:00 shendandan: > Hi! >The process of kylin often hangs up,and the erroe in kylin,log: > > > 2017-08-10 10:14:22,292 INFO [BadQueryDetector] > service.BadQueryDetector:151 : System free memory less than 100 MB. 1 > queries running. > 2017-08-10 10:14:34,498 INFO [Thread-11-EventThread] > state.ConnectionStateManager:228 : State change: SUSPENDED > 2017-08-10 10:14:34,506 INFO [pool-7-thread-1] > threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual > running, 0 ready, 63 already succeed, 3 error, 13 discarded, 0 others > 2017-08-10 10:14:34,999 INFO [localhost-startStop-1-SendThread( > hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket > connection to server hadoop-senior01.ctcf.com/10.1.8.90:2181. Will not > attempt to authenticate using SASL (unknown error) > 2017-08-10 10:14:34,999 INFO [Thread-11-SendThread(hadoop- > senior03.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket > connection to server hadoop-senior03.ctcf.com/10.1.8.92:2181. Will not > attempt to authenticate using SASL (unknown error) > 2017-08-10 10:14:34,999 INFO [localhost-startStop-1-SendThread( > hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket > connection established, initiating session, client: /10.1.8.90:38229, > server: hadoop-senior01.ctcf.com/10.1.8.90:2181 > 2017-08-10 10:14:34,999 INFO [Thread-11-SendThread(hadoop- > senior03.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection > established, initiating session, client: /10.1.8.90:32962, server: > hadoop-senior03.ctcf.com/10.1.8.92:2181 > 2017-08-10 10:14:35,001 INFO [Thread-11-SendThread(hadoop- > senior03.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to reconnect > to ZooKeeper service, session 0x25dc4bd90cc002a has expired, closing socket > connection > 2017-08-10 10:14:35,002 INFO [localhost-startStop-1-SendThread( > hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to > reconnect to ZooKeeper service, session 0x35dc4bd90460028 has expired, > closing socket connection > 2017-08-10 10:14:35,013 WARN [localhost-startStop-1-EventThread] > client.HConnectionManager$HConnectionImplementation:2468 : This client > just lost it's session with ZooKeeper, closing it. It will be recreated > next time someone needs it > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired > at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent( > ZooKeeperWatcher.java:517) > at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher. > process(ZooKeeperWatcher.java:435) > at org.apache.zookeeper.ClientCnxn$EventThread. > processEvent(ClientCnxn.java:522) > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) > 2017-08-10 10:14:35,013 INFO [Thread-11-EventThread] > state.ConnectionStateManager:228 : State change: LOST > 2017-08-10 10:14:35,014 INFO [localhost-startStop-1-EventThread] > client.HConnectionManager$HConnectionImplementation:1837 : Closing > zookeeper sessionid=0x35dc4bd90460028 > 2017-08-10 10:14:35,014 WARN [Thread-11-EventThread] > curator.ConnectionState:289 : Session expired event received > 2017-08-10 10:14:35,014 INFO [localhost-startStop-1-EventThread] > zookeeper.ClientCnxn:512 : EventThread shut down > 2017-08-10 10:14:35,015 INFO [Thread-11-EventThread] > zookeeper.ZooKeeper:438 : Initiating client connection, connectString= > hadoop-senior01.ctcf.com:2181,hadoop-senior02.ctcf.com:2181, > hadoop-senior03.ctcf.com:2181 sessionTimeout=6 > watcher=org.apache.curator.ConnectionState@4d6f4fa7 > 2017-08-10 10:14:35,017 INFO [Thread-11-SendThread(hadoop- > senior02.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket > connection to server hadoop-senior02.ctcf.com/10.1.8.91:2181. Will not > attempt to authenticate using SASL (unknown error) > 2017-08-10 10:14:35,017 INFO [Thread-11-SendThread(hadoop- > senior02.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection > established, initiating session, client: /10.1.8.90:57503, server: > hadoop-senior02.ctcf.com/10.1.8.91:2181 > 2017-08-10 10:14:35,021 INFO [Thread-11-EventThread] > zookeeper.ClientCnxn:512 : EventThread shut down > 2017-08-10 10:14:35,021 INFO [Thread-11-SendThread(hadoop- > senior02.ctcf.com:2181)] zookeeper.ClientCnxn:1235 : Session > establishment complete on server hadoop-senior02.ctcf.com/10.1.8.91:2181, > sessionid = 0x25dc4bd90cc002c, negotiated timeout = 4 > 2017-08-10 10:14:35,021 INFO [Thread-11-EventThread] > state.ConnectionStateManager:228 : State change: RECONNECTED > 2017-08-10 10:15:09,204 INFO [pool-7-thread-1] > threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual > running, 0 ready, 63 already succeed, 3 error, 13 discarded, 0 others > 2017-08-10 10:15:26,969 INFO [BadQueryDetector] > service.BadQueryDetector:179 : Slow query has