The first WARN happened during ZK timeout. Please check the ZK connection first.
2017-08-10 10:58 GMT+08:00 shendandan <shendandan2015y...@163.com>: > Hi! > The process of kylin often hangs up,and the erroe in kylin,log: > > > 2017-08-10 10:14:22,292 INFO [BadQueryDetector] > service.BadQueryDetector:151 : System free memory less than 100 MB. 1 > queries running. > 2017-08-10 10:14:34,498 INFO [Thread-11-EventThread] > state.ConnectionStateManager:228 : State change: SUSPENDED > 2017-08-10 10:14:34,506 INFO [pool-7-thread-1] > threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual > running, 0 ready, 63 already succeed, 3 error, 13 discarded, 0 others > 2017-08-10 10:14:34,999 INFO [localhost-startStop-1-SendThread( > hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket > connection to server hadoop-senior01.ctcf.com/10.1.8.90:2181. Will not > attempt to authenticate using SASL (unknown error) > 2017-08-10 10:14:34,999 INFO [Thread-11-SendThread(hadoop- > senior03.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket > connection to server hadoop-senior03.ctcf.com/10.1.8.92:2181. Will not > attempt to authenticate using SASL (unknown error) > 2017-08-10 10:14:34,999 INFO [localhost-startStop-1-SendThread( > hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket > connection established, initiating session, client: /10.1.8.90:38229, > server: hadoop-senior01.ctcf.com/10.1.8.90:2181 > 2017-08-10 10:14:34,999 INFO [Thread-11-SendThread(hadoop- > senior03.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection > established, initiating session, client: /10.1.8.90:32962, server: > hadoop-senior03.ctcf.com/10.1.8.92:2181 > 2017-08-10 10:14:35,001 INFO [Thread-11-SendThread(hadoop- > senior03.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to reconnect > to ZooKeeper service, session 0x25dc4bd90cc002a has expired, closing socket > connection > 2017-08-10 10:14:35,002 INFO [localhost-startStop-1-SendThread( > hadoop-senior01.ctcf.com:2181)] zookeeper.ClientCnxn:1094 : Unable to > reconnect to ZooKeeper service, session 0x35dc4bd90460028 has expired, > closing socket connection > 2017-08-10 10:14:35,013 WARN [localhost-startStop-1-EventThread] > client.HConnectionManager$HConnectionImplementation:2468 : This client > just lost it's session with ZooKeeper, closing it. It will be recreated > next time someone needs it > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired > at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent( > ZooKeeperWatcher.java:517) > at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher. > process(ZooKeeperWatcher.java:435) > at org.apache.zookeeper.ClientCnxn$EventThread. > processEvent(ClientCnxn.java:522) > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) > 2017-08-10 10:14:35,013 INFO [Thread-11-EventThread] > state.ConnectionStateManager:228 : State change: LOST > 2017-08-10 10:14:35,014 INFO [localhost-startStop-1-EventThread] > client.HConnectionManager$HConnectionImplementation:1837 : Closing > zookeeper sessionid=0x35dc4bd90460028 > 2017-08-10 10:14:35,014 WARN [Thread-11-EventThread] > curator.ConnectionState:289 : Session expired event received > 2017-08-10 10:14:35,014 INFO [localhost-startStop-1-EventThread] > zookeeper.ClientCnxn:512 : EventThread shut down > 2017-08-10 10:14:35,015 INFO [Thread-11-EventThread] > zookeeper.ZooKeeper:438 : Initiating client connection, connectString= > hadoop-senior01.ctcf.com:2181,hadoop-senior02.ctcf.com:2181, > hadoop-senior03.ctcf.com:2181 sessionTimeout=60000 > watcher=org.apache.curator.ConnectionState@4d6f4fa7 > 2017-08-10 10:14:35,017 INFO [Thread-11-SendThread(hadoop- > senior02.ctcf.com:2181)] zookeeper.ClientCnxn:975 : Opening socket > connection to server hadoop-senior02.ctcf.com/10.1.8.91:2181. Will not > attempt to authenticate using SASL (unknown error) > 2017-08-10 10:14:35,017 INFO [Thread-11-SendThread(hadoop- > senior02.ctcf.com:2181)] zookeeper.ClientCnxn:852 : Socket connection > established, initiating session, client: /10.1.8.90:57503, server: > hadoop-senior02.ctcf.com/10.1.8.91:2181 > 2017-08-10 10:14:35,021 INFO [Thread-11-EventThread] > zookeeper.ClientCnxn:512 : EventThread shut down > 2017-08-10 10:14:35,021 INFO [Thread-11-SendThread(hadoop- > senior02.ctcf.com:2181)] zookeeper.ClientCnxn:1235 : Session > establishment complete on server hadoop-senior02.ctcf.com/10.1.8.91:2181, > sessionid = 0x25dc4bd90cc002c, negotiated timeout = 40000 > 2017-08-10 10:14:35,021 INFO [Thread-11-EventThread] > state.ConnectionStateManager:228 : State change: RECONNECTED > 2017-08-10 10:15:09,204 INFO [pool-7-thread-1] > threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual > running, 0 ready, 63 already succeed, 3 error, 13 discarded, 0 others > 2017-08-10 10:15:26,969 INFO [BadQueryDetector] > service.BadQueryDetector:179 : Slow query has been running 551.737 seconds > (project:eam_contract_baseinfo, thread: 0x4a, user:ADMIN) -- SELECT * > FROM "TN_QUERY"."BASEINFO_REPAY_HIS_FACT_PARTATION" > 2017-08-10 10:15:26,971 DEBUG [BadQueryDetector] > badquery.BadQueryHistoryManager:84 > : Loaded 10 Bad Query(s) > 2017-08-10 10:15:26,976 DEBUG [BadQueryDetector] > hbase.HBaseResourceStore:262 : Update row > /bad_query/eam_contract_baseinfo.json > from oldTs: 1502331183691, to newTs: 1502331326972, operation result: true > 2017-08-10 10:15:26,977 INFO [BadQueryDetector] > service.BadQueryDetector:169 : Problematic thread 0x4a > at org.apache.kylin.metadata.datatype.BigDecimalSerializer.deserialize( > BigDecimalSerializer.java:74) > at org.apache.kylin.metadata.datatype.BigDecimalSerializer.deserialize( > BigDecimalSerializer.java:33) > at org.apache.kylin.cube.gridtable.CubeCodeSystem.decodeColumnValue( > CubeCodeSystem.java:133) > at org.apache.kylin.gridtable.GTRecord.getValues(GTRecord.java:129) > at org.apache.kylin.storage.gtrecord.CubeTupleConverter.translateResult( > CubeTupleConverter.java:207) > at org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext( > SegmentCubeTupleIterator.java:100) > at com.google.common.collect.Iterators$5.hasNext(Iterators.java:596) > at org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext( > SequentialCubeTupleIterator.java:84) > at org.apache.kylin.query.enumerator.OLAPEnumerator. > moveNext(OLAPEnumerator.java:68) > at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator. > next(Linq4j.java:673) > > > How can I solve the problem,thank you!
