> 2017-08-10 10:14:35,013 WARN [localhost-startStop-1-EventThread] client.HConnectionManager$HConnectionImplementation:2468 : This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
Obviously the ZK connection is not stable. On Thu, Aug 10, 2017 at 3:39 PM, shendandan <shendandan2015y...@163.com> wrote: > > > > Thanks for your respond,but Zk is working normally. > > > > > > At 2017-08-10 15:21:49, "Billy Liu" <billy...@apache.org> wrote: > >The first WARN happened during ZK timeout. Please check the ZK connection > >first. > > > >2017-08-10 10:58 GMT+08:00 shendandan <shendandan2015y...@163.com>: > > > >> Hi! > >> The process of kylin often hangs up,and the erroe in kylin,log: > >> > >> > >> 2017-08-10 10:14:22,292 INFO [BadQueryDetector] > >> service.BadQueryDetector:151 : System free memory less than 100 MB. 1 > >> queries running. > >> 2017-08-10 10:14:34,498 INFO [Thread-11-EventThread] > >> state.ConnectionStateManager:228 : State change: SUSPENDED > >> 2017-08-10 10:14:34,506 INFO [pool-7-thread-1] > >> threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 > actual > >> running, 0 ready, 63 already succeed, 3 error, 13 discarded, 0 others > >> 2017-08-10 10:14:34,999 INFO [localhost-startStop-1-SendThread( > >> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening > socket > >> connection to server hadoop-senior01.ctcf.com/10.1.8.90:2181. Will not > >> attempt to authenticate using SASL (unknown error) > >> 2017-08-10 10:14:34,999 INFO [Thread-11-SendThread(hadoop- > >> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket > >> connection to server hadoop-senior03.ctcf.com/10.1.8.92:2181. Will not > >> attempt to authenticate using SASL (unknown error) > >> 2017-08-10 10:14:34,999 INFO [localhost-startStop-1-SendThread( > >> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket > >> connection established, initiating session, client: /10.1.8.90:38229, > >> server: hadoop-senior01.ctcf.com/10.1.8.90:2181 > >> 2017-08-10 10:14:34,999 INFO [Thread-11-SendThread(hadoop- > >> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection > >> established, initiating session, client: /10.1.8.90:32962, server: > >> hadoop-senior03.ctcf.com/10.1.8.92:2181 > >> 2017-08-10 10:14:35,001 INFO [Thread-11-SendThread(hadoop- > >> senior03.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to > reconnect > >> to ZooKeeper service, session 0x25dc4bd90cc002a has expired, closing > socket > >> connection > >> 2017-08-10 10:14:35,002 INFO [localhost-startStop-1-SendThread( > >> hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to > >> reconnect to ZooKeeper service, session 0x35dc4bd90460028 has expired, > >> closing socket connection > >> 2017-08-10 10:14:35,013 WARN [localhost-startStop-1-EventThread] > >> client.HConnectionManager$HConnectionImplementation:2468 : This client > >> just lost it's session with ZooKeeper, closing it. It will be recreated > >> next time someone needs it > >> org.apache.zookeeper.KeeperException$SessionExpiredException: > >> KeeperErrorCode = Session expired > >> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent( > >> ZooKeeperWatcher.java:517) > >> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher. > >> process(ZooKeeperWatcher.java:435) > >> at org.apache.zookeeper.ClientCnxn$EventThread. > >> processEvent(ClientCnxn.java:522) > >> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) > >> 2017-08-10 10:14:35,013 INFO [Thread-11-EventThread] > >> state.ConnectionStateManager:228 : State change: LOST > >> 2017-08-10 10:14:35,014 INFO [localhost-startStop-1-EventThread] > >> client.HConnectionManager$HConnectionImplementation:1837 : Closing > >> zookeeper sessionid=0x35dc4bd90460028 > >> 2017-08-10 10:14:35,014 WARN [Thread-11-EventThread] > >> curator.ConnectionState:289 : Session expired event received > >> 2017-08-10 10:14:35,014 INFO [localhost-startStop-1-EventThread] > >> zookeeper.ClientCnxn:512 : EventThread shut down > >> 2017-08-10 10:14:35,015 INFO [Thread-11-EventThread] > >> zookeeper.ZooKeeper:438 : Initiating client connection, connectString= > >> hadoop-senior01.ctcf.com:2181,hadoop-senior02.ctcf.com:2181, > >> hadoop-senior03.ctcf.com:2181 sessionTimeout=60000 > >> watcher=org.apache.curator.ConnectionState@4d6f4fa7 > >> 2017-08-10 10:14:35,017 INFO [Thread-11-SendThread(hadoop- > >> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket > >> connection to server hadoop-senior02.ctcf.com/10.1.8.91:2181. Will not > >> attempt to authenticate using SASL (unknown error) > >> 2017-08-10 10:14:35,017 INFO [Thread-11-SendThread(hadoop- > >> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection > >> established, initiating session, client: /10.1.8.90:57503, server: > >> hadoop-senior02.ctcf.com/10.1.8.91:2181 > >> 2017-08-10 10:14:35,021 INFO [Thread-11-EventThread] > >> zookeeper.ClientCnxn:512 : EventThread shut down > >> 2017-08-10 10:14:35,021 INFO [Thread-11-SendThread(hadoop- > >> senior02.ctcf.com:2181)] zookeeper.ClientCnxn:1235 : Session > >> establishment complete on server hadoop-senior02.ctcf.com/10.1. > 8.91:2181, > >> sessionid = 0x25dc4bd90cc002c, negotiated timeout = 40000 > >> 2017-08-10 10:14:35,021 INFO [Thread-11-EventThread] > >> state.ConnectionStateManager:228 : State change: RECONNECTED > >> 2017-08-10 10:15:09,204 INFO [pool-7-thread-1] > >> threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 > actual > >> running, 0 ready, 63 already succeed, 3 error, 13 discarded, 0 others > >> 2017-08-10 10:15:26,969 INFO [BadQueryDetector] > >> service.BadQueryDetector:179 : Slow query has been running 551.737 > seconds > >> (project:eam_contract_baseinfo, thread: 0x4a, user:ADMIN) -- SELECT * > >> FROM "TN_QUERY"."BASEINFO_REPAY_HIS_FACT_PARTATION" > >> 2017-08-10 10:15:26,971 DEBUG [BadQueryDetector] badquery. > BadQueryHistoryManager:84 > >> : Loaded 10 Bad Query(s) > >> 2017-08-10 10:15:26,976 DEBUG [BadQueryDetector] > >> hbase.HBaseResourceStore:262 : Update row /bad_query/eam_contract_ > baseinfo.json > >> from oldTs: 1502331183691, to newTs: 1502331326972, operation result: > true > >> 2017-08-10 10:15:26,977 INFO [BadQueryDetector] > >> service.BadQueryDetector:169 : Problematic thread 0x4a > >> at org.apache.kylin.metadata.datatype.BigDecimalSerializer.deserialize( > >> BigDecimalSerializer.java:74) > >> at org.apache.kylin.metadata.datatype.BigDecimalSerializer.deserialize( > >> BigDecimalSerializer.java:33) > >> at org.apache.kylin.cube.gridtable.CubeCodeSystem.decodeColumnValue( > >> CubeCodeSystem.java:133) > >> at org.apache.kylin.gridtable.GTRecord.getValues(GTRecord.java:129) > >> at org.apache.kylin.storage.gtrecord.CubeTupleConverter. > translateResult( > >> CubeTupleConverter.java:207) > >> at org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext( > >> SegmentCubeTupleIterator.java:100) > >> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:596) > >> at org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator. > hasNext( > >> SequentialCubeTupleIterator.java:84) > >> at org.apache.kylin.query.enumerator.OLAPEnumerator. > >> moveNext(OLAPEnumerator.java:68) > >> at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator. > >> next(Linq4j.java:673) > >> > >> > >> How can I solve the problem,thank you! >