Re: Kylin often hangs up

2017-08-19 Thread Li Yang 
> 2017-08-10 10:14:35,013 WARN  [localhost-startStop-1-EventThread]
client.HConnectionManager$HConnectionImplementation:2468 : This client just
lost it's session with ZooKeeper, closing it. It will be recreated next
time someone needs it
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired

Obviously the ZK connection is not stable.

On Thu, Aug 10, 2017 at 3:39 PM, shendandan 
wrote:

>
>
>
> Thanks for your respond,but Zk is working normally.
>
>
>
>
>
> At 2017-08-10 15:21:49, "Billy Liu"  wrote:
> >The first WARN happened during ZK timeout. Please check the ZK connection
> >first.
> >
> >2017-08-10 10:58 GMT+08:00 shendandan :
> >
> >> Hi!
> >>The process of kylin often hangs up,and the erroe in kylin,log:
> >>
> >>
> >> 2017-08-10 10:14:22,292 INFO  [BadQueryDetector]
> >> service.BadQueryDetector:151 : System free memory less than 100 MB. 1
> >> queries running.
> >> 2017-08-10 10:14:34,498 INFO  [Thread-11-EventThread]
> >> state.ConnectionStateManager:228 : State change: SUSPENDED
> >> 2017-08-10 10:14:34,506 INFO  [pool-7-thread-1]
> >> threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0
> actual
> >> running, 0 ready, 63 already succeed, 3 error, 13 discarded, 0 others
> >> 2017-08-10 10:14:34,999 INFO  [localhost-startStop-1-SendThread(
> >> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening
> socket
> >> connection to server hadoop-senior01.ctcf.com/10.1.8.90:2181. Will not
> >> attempt to authenticate using SASL (unknown error)
> >> 2017-08-10 10:14:34,999 INFO  [Thread-11-SendThread(hadoop-
> >> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket
> >> connection to server hadoop-senior03.ctcf.com/10.1.8.92:2181. Will not
> >> attempt to authenticate using SASL (unknown error)
> >> 2017-08-10 10:14:34,999 INFO  [localhost-startStop-1-SendThread(
> >> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket
> >> connection established, initiating session, client: /10.1.8.90:38229,
> >> server: hadoop-senior01.ctcf.com/10.1.8.90:2181
> >> 2017-08-10 10:14:34,999 INFO  [Thread-11-SendThread(hadoop-
> >> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection
> >> established, initiating session, client: /10.1.8.90:32962, server:
> >> hadoop-senior03.ctcf.com/10.1.8.92:2181
> >> 2017-08-10 10:14:35,001 INFO  [Thread-11-SendThread(hadoop-
> >> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to
> reconnect
> >> to ZooKeeper service, session 0x25dc4bd90cc002a has expired, closing
> socket
> >> connection
> >> 2017-08-10 10:14:35,002 INFO  [localhost-startStop-1-SendThread(
> >> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to
> >> reconnect to ZooKeeper service, session 0x35dc4bd90460028 has expired,
> >> closing socket connection
> >> 2017-08-10 10:14:35,013 WARN  [localhost-startStop-1-EventThread]
> >> client.HConnectionManager$HConnectionImplementation:2468 : This client
> >> just lost it's session with ZooKeeper, closing it. It will be recreated
> >> next time someone needs it
> >> org.apache.zookeeper.KeeperException$SessionExpiredException:
> >> KeeperErrorCode = Session expired
> >> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(
> >> ZooKeeperWatcher.java:517)
> >> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.
> >> process(ZooKeeperWatcher.java:435)
> >> at org.apache.zookeeper.ClientCnxn$EventThread.
> >> processEvent(ClientCnxn.java:522)
> >> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> >> 2017-08-10 10:14:35,013 INFO  [Thread-11-EventThread]
> >> state.ConnectionStateManager:228 : State change: LOST
> >> 2017-08-10 10:14:35,014 INFO  [localhost-startStop-1-EventThread]
> >> client.HConnectionManager$HConnectionImplementation:1837 : Closing
> >> zookeeper sessionid=0x35dc4bd90460028
> >> 2017-08-10 10:14:35,014 WARN  [Thread-11-EventThread]
> >> curator.ConnectionState:289 : Session expired event received
> >> 2017-08-10 10:14:35,014 INFO  [localhost-startStop-1-EventThread]
> >> zookeeper.ClientCnxn:512 : EventThread shut down
> >> 2017-08-10 10:14:35,015 INFO  [Thread-11-EventThread]
> >> zookeeper.ZooKeeper:438 : Initiating client connection, connectString=
> >> hadoop-senior01.ctcf.com:2181,hadoop-senior02.ctcf.com:2181,
> >> hadoop-senior03.ctcf.com:2181 sessionTimeout=6
> >> watcher=org.apache.curator.ConnectionState@4d6f4fa7
> >> 2017-08-10 10:14:35,017 INFO  [Thread-11-SendThread(hadoop-
> >> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket
> >> connection to server hadoop-senior02.ctcf.com/10.1.8.91:2181. Will not
> >> attempt to authenticate using SASL (unknown error)
> >> 2017-08-10 10:14:35,017 INFO  [Thread-11-SendThread(hadoop-
> >> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection
> >> established, initiating session, client: /10.1.8.90:57503, server:

Re: Kylin often hangs up

2017-08-10 Thread shendandan 



Thanks for your respond,but Zk is working normally.





At 2017-08-10 15:21:49, "Billy Liu"  wrote:
>The first WARN happened during ZK timeout. Please check the ZK connection
>first.
>
>2017-08-10 10:58 GMT+08:00 shendandan :
>
>> Hi!
>>The process of kylin often hangs up,and the erroe in kylin,log:
>>
>>
>> 2017-08-10 10:14:22,292 INFO  [BadQueryDetector]
>> service.BadQueryDetector:151 : System free memory less than 100 MB. 1
>> queries running.
>> 2017-08-10 10:14:34,498 INFO  [Thread-11-EventThread]
>> state.ConnectionStateManager:228 : State change: SUSPENDED
>> 2017-08-10 10:14:34,506 INFO  [pool-7-thread-1]
>> threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual
>> running, 0 ready, 63 already succeed, 3 error, 13 discarded, 0 others
>> 2017-08-10 10:14:34,999 INFO  [localhost-startStop-1-SendThread(
>> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket
>> connection to server hadoop-senior01.ctcf.com/10.1.8.90:2181. Will not
>> attempt to authenticate using SASL (unknown error)
>> 2017-08-10 10:14:34,999 INFO  [Thread-11-SendThread(hadoop-
>> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket
>> connection to server hadoop-senior03.ctcf.com/10.1.8.92:2181. Will not
>> attempt to authenticate using SASL (unknown error)
>> 2017-08-10 10:14:34,999 INFO  [localhost-startStop-1-SendThread(
>> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket
>> connection established, initiating session, client: /10.1.8.90:38229,
>> server: hadoop-senior01.ctcf.com/10.1.8.90:2181
>> 2017-08-10 10:14:34,999 INFO  [Thread-11-SendThread(hadoop-
>> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection
>> established, initiating session, client: /10.1.8.90:32962, server:
>> hadoop-senior03.ctcf.com/10.1.8.92:2181
>> 2017-08-10 10:14:35,001 INFO  [Thread-11-SendThread(hadoop-
>> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to reconnect
>> to ZooKeeper service, session 0x25dc4bd90cc002a has expired, closing socket
>> connection
>> 2017-08-10 10:14:35,002 INFO  [localhost-startStop-1-SendThread(
>> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to
>> reconnect to ZooKeeper service, session 0x35dc4bd90460028 has expired,
>> closing socket connection
>> 2017-08-10 10:14:35,013 WARN  [localhost-startStop-1-EventThread]
>> client.HConnectionManager$HConnectionImplementation:2468 : This client
>> just lost it's session with ZooKeeper, closing it. It will be recreated
>> next time someone needs it
>> org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired
>> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(
>> ZooKeeperWatcher.java:517)
>> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.
>> process(ZooKeeperWatcher.java:435)
>> at org.apache.zookeeper.ClientCnxn$EventThread.
>> processEvent(ClientCnxn.java:522)
>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>> 2017-08-10 10:14:35,013 INFO  [Thread-11-EventThread]
>> state.ConnectionStateManager:228 : State change: LOST
>> 2017-08-10 10:14:35,014 INFO  [localhost-startStop-1-EventThread]
>> client.HConnectionManager$HConnectionImplementation:1837 : Closing
>> zookeeper sessionid=0x35dc4bd90460028
>> 2017-08-10 10:14:35,014 WARN  [Thread-11-EventThread]
>> curator.ConnectionState:289 : Session expired event received
>> 2017-08-10 10:14:35,014 INFO  [localhost-startStop-1-EventThread]
>> zookeeper.ClientCnxn:512 : EventThread shut down
>> 2017-08-10 10:14:35,015 INFO  [Thread-11-EventThread]
>> zookeeper.ZooKeeper:438 : Initiating client connection, connectString=
>> hadoop-senior01.ctcf.com:2181,hadoop-senior02.ctcf.com:2181,
>> hadoop-senior03.ctcf.com:2181 sessionTimeout=6
>> watcher=org.apache.curator.ConnectionState@4d6f4fa7
>> 2017-08-10 10:14:35,017 INFO  [Thread-11-SendThread(hadoop-
>> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket
>> connection to server hadoop-senior02.ctcf.com/10.1.8.91:2181. Will not
>> attempt to authenticate using SASL (unknown error)
>> 2017-08-10 10:14:35,017 INFO  [Thread-11-SendThread(hadoop-
>> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection
>> established, initiating session, client: /10.1.8.90:57503, server:
>> hadoop-senior02.ctcf.com/10.1.8.91:2181
>> 2017-08-10 10:14:35,021 INFO  [Thread-11-EventThread]
>> zookeeper.ClientCnxn:512 : EventThread shut down
>> 2017-08-10 10:14:35,021 INFO  [Thread-11-SendThread(hadoop-
>> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:1235 : Session
>> establishment complete on server hadoop-senior02.ctcf.com/10.1.8.91:2181,
>> sessionid = 0x25dc4bd90cc002c, negotiated timeout = 4
>> 2017-08-10 10:14:35,021 INFO  [Thread-11-EventThread]
>> state.ConnectionStateManager:228 : State change: RECONNECTED
>> 2017-08-10 10:15:09,204 INFO  [pool-7-thread-1]
>> threadpool.DefaultScheduler:118 : Job

Re: Kylin often hangs up

2017-08-10 Thread Billy Liu 
The first WARN happened during ZK timeout. Please check the ZK connection
first.

2017-08-10 10:58 GMT+08:00 shendandan :

> Hi!
>The process of kylin often hangs up,and the erroe in kylin,log:
>
>
> 2017-08-10 10:14:22,292 INFO  [BadQueryDetector]
> service.BadQueryDetector:151 : System free memory less than 100 MB. 1
> queries running.
> 2017-08-10 10:14:34,498 INFO  [Thread-11-EventThread]
> state.ConnectionStateManager:228 : State change: SUSPENDED
> 2017-08-10 10:14:34,506 INFO  [pool-7-thread-1]
> threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual
> running, 0 ready, 63 already succeed, 3 error, 13 discarded, 0 others
> 2017-08-10 10:14:34,999 INFO  [localhost-startStop-1-SendThread(
> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket
> connection to server hadoop-senior01.ctcf.com/10.1.8.90:2181. Will not
> attempt to authenticate using SASL (unknown error)
> 2017-08-10 10:14:34,999 INFO  [Thread-11-SendThread(hadoop-
> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket
> connection to server hadoop-senior03.ctcf.com/10.1.8.92:2181. Will not
> attempt to authenticate using SASL (unknown error)
> 2017-08-10 10:14:34,999 INFO  [localhost-startStop-1-SendThread(
> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket
> connection established, initiating session, client: /10.1.8.90:38229,
> server: hadoop-senior01.ctcf.com/10.1.8.90:2181
> 2017-08-10 10:14:34,999 INFO  [Thread-11-SendThread(hadoop-
> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection
> established, initiating session, client: /10.1.8.90:32962, server:
> hadoop-senior03.ctcf.com/10.1.8.92:2181
> 2017-08-10 10:14:35,001 INFO  [Thread-11-SendThread(hadoop-
> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to reconnect
> to ZooKeeper service, session 0x25dc4bd90cc002a has expired, closing socket
> connection
> 2017-08-10 10:14:35,002 INFO  [localhost-startStop-1-SendThread(
> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to
> reconnect to ZooKeeper service, session 0x35dc4bd90460028 has expired,
> closing socket connection
> 2017-08-10 10:14:35,013 WARN  [localhost-startStop-1-EventThread]
> client.HConnectionManager$HConnectionImplementation:2468 : This client
> just lost it's session with ZooKeeper, closing it. It will be recreated
> next time someone needs it
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(
> ZooKeeperWatcher.java:517)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.
> process(ZooKeeperWatcher.java:435)
> at org.apache.zookeeper.ClientCnxn$EventThread.
> processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2017-08-10 10:14:35,013 INFO  [Thread-11-EventThread]
> state.ConnectionStateManager:228 : State change: LOST
> 2017-08-10 10:14:35,014 INFO  [localhost-startStop-1-EventThread]
> client.HConnectionManager$HConnectionImplementation:1837 : Closing
> zookeeper sessionid=0x35dc4bd90460028
> 2017-08-10 10:14:35,014 WARN  [Thread-11-EventThread]
> curator.ConnectionState:289 : Session expired event received
> 2017-08-10 10:14:35,014 INFO  [localhost-startStop-1-EventThread]
> zookeeper.ClientCnxn:512 : EventThread shut down
> 2017-08-10 10:14:35,015 INFO  [Thread-11-EventThread]
> zookeeper.ZooKeeper:438 : Initiating client connection, connectString=
> hadoop-senior01.ctcf.com:2181,hadoop-senior02.ctcf.com:2181,
> hadoop-senior03.ctcf.com:2181 sessionTimeout=6
> watcher=org.apache.curator.ConnectionState@4d6f4fa7
> 2017-08-10 10:14:35,017 INFO  [Thread-11-SendThread(hadoop-
> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket
> connection to server hadoop-senior02.ctcf.com/10.1.8.91:2181. Will not
> attempt to authenticate using SASL (unknown error)
> 2017-08-10 10:14:35,017 INFO  [Thread-11-SendThread(hadoop-
> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection
> established, initiating session, client: /10.1.8.90:57503, server:
> hadoop-senior02.ctcf.com/10.1.8.91:2181
> 2017-08-10 10:14:35,021 INFO  [Thread-11-EventThread]
> zookeeper.ClientCnxn:512 : EventThread shut down
> 2017-08-10 10:14:35,021 INFO  [Thread-11-SendThread(hadoop-
> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:1235 : Session
> establishment complete on server hadoop-senior02.ctcf.com/10.1.8.91:2181,
> sessionid = 0x25dc4bd90cc002c, negotiated timeout = 4
> 2017-08-10 10:14:35,021 INFO  [Thread-11-EventThread]
> state.ConnectionStateManager:228 : State change: RECONNECTED
> 2017-08-10 10:15:09,204 INFO  [pool-7-thread-1]
> threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual
> running, 0 ready, 63 already succeed, 3 error, 13 discarded, 0 others
> 2017-08-10 10:15:26,969 INFO  [BadQueryDetector]
> service.BadQueryDetector:179 : Slow query has