Looks like tomcat crashed after running out of memory. I saw this in "kylin.out" :
# # java.lang.OutOfMemoryError: Java heap space # -XX:OnOutOfMemoryError="kill -9 %p" # Executing /bin/sh -c "kill -9 12727"... Before the crash , "kylin.log" file shows following lines. Seems it keep trying to reconnect to ZooKeeper. What the reason for Kylin to communicate with ZK ? I see the line "System free memory less than 100 MB." ---- kylin.log ---- 2016-09-28 06:50:02,495 ERROR [Curator-Framework-0] curator.ConnectionState:200 : Connection timed out for connection string (hdp-jz5001.hadoop.local:2181,hdp-jz5002.hadoop.local:2181,hdp-jz5003.hadoop.local:2181) and timeout (15000) / elapsed (28428) org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87) at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:806) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-09-28 06:50:02,495 INFO [Thread-10-SendThread(hdp-jz5001.hadoop.local:2181)] zookeeper.ClientCnxn:1279 : Session establishment complete on server hdp-jz5001.hadoop.local/100.78.7.155:2181, sessionid = 0x156d401adb1701a, negotiated timeout = 40000 2016-09-28 06:50:02,495 INFO [localhost-startStop-1-SendThread(hdp-jz5003.hadoop.local:2181)] zookeeper.ClientCnxn:1019 : Opening socket connection to server hdp-jz5003.hadoop.local/100.78.8.153:2181. Will not attempt to authenticate using SASL (unknown error) 2016-09-28 06:50:02,495 ERROR [Curator-Framework-0] curator.ConnectionState:200 : Connection timed out for connection string (hdp-jz5001.hadoop.local:2181,hdp-jz5002.hadoop.local:2181,hdp-jz5003.hadoop.local:2181) and timeout (15000) / elapsed (28429) org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87) at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:806) at org.apache.curator.framework.imps.CuratorFrameworkImpl.doSyncForSuspendedConnection(CuratorFrameworkImpl.java:681) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$700(CuratorFrameworkImpl.java:62) at org.apache.curator.framework.imps.CuratorFrameworkImpl$7.retriesExhausted(CuratorFrameworkImpl.java:677) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:696) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:826) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-09-28 06:50:02,495 INFO [localhost-startStop-1-SendThread(hdp-jz5003.hadoop.local:2181)] zookeeper.ClientCnxn:864 : Socket connection established to hdp-jz5003.hadoop.local/100.78.8.153:2181, initiating session 2016-09-28 06:50:15,060 INFO [localhost-startStop-1-SendThread(hdp-jz5003.hadoop.local:2181)] zookeeper.ClientCnxn:1140 : Client session timed out, have not heard from server in 12565ms for sessionid 0x356d401ac017143, closing socket connection and attempting reconnect 2016-09-28 06:50:02,495 INFO [Thread-10-EventThread] state.ConnectionStateManager:228 : State change: RECONNECTED 2016-09-28 06:50:31,040 INFO [Thread-10-SendThread(hdp-jz5001.hadoop.local:2181)] zookeeper.ClientCnxn:1140 : Client session timed out, have not heard from server in 28544ms for sessionid 0x156d401adb1701a, closing socket connection and attempting reconnect 2016-09-28 06:50:31,042 DEBUG [http-bio-7070-exec-7] service.AdminService:89 : Get Kylin Runtime Config 2016-09-28 06:50:31,043 DEBUG [http-bio-7070-exec-1] controller.UserController:64 : authentication.getPrincipal() is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER 2016-09-28 06:50:43,799 INFO [localhost-startStop-1-SendThread(hdp-jz5002.hadoop.local:2181)] zookeeper.ClientCnxn:1019 : Opening socket connection to server hdp-jz5002.hadoop.local/100.78.8.20:2181. Will not attempt to authenticate using SASL (unknown error) 2016-09-28 06:50:43,799 INFO [Thread-10-EventThread] state.ConnectionStateManager:228 : State change: SUSPENDED 2016-09-28 06:50:59,925 INFO [BadQueryDetector] service.BadQueryDetector:151 : System free memory less than 100 MB. 0 queries running. 2016-09-28 06:50:59,926 INFO [localhost-startStop-1-SendThread(hdp-jz5002.hadoop.local:2181)] zookeeper.ClientCnxn:864 : Socket connection established to hdp-jz5002.hadoop.local/100.78.8.20:2181, initiating session 2016-09-28 06:51:28,723 INFO [localhost-startStop-1-SendThread(hdp-jz5002.hadoop.local:2181)] zookeeper.ClientCnxn:1140 : Client session timed out, have not heard from server in 28798ms for sessionid 0x356d401ac017143, closing socket connection and attempting reconnect 2016-09-28 06:51:41,129 INFO [pool-8-thread-10-SendThread(hdp-jz5001.hadoop.local:2181)] zookeeper.ClientCnxn:1142 : Unable to read additional data from server sessionid 0x356d401ac01714a, likely server has closed socket, closing socket connection and attempting reconnect 2016-09-28 06:51:53,474 INFO [Thread-10-SendThread(hdp-jz5003.hadoop.local:2181)] zookeeper.ClientCnxn:1019 : Opening socket connection to server hdp-jz5003.hadoop.local/100.78.8.153:2181. Will not attempt to authenticate using SASL (unknown error) 2016-09-28 06:51:12,316 INFO [pool-8-thread-10-SendThread(hdp-jz5003.hadoop.local:2181)] zookeeper.ClientCnxn:1140 : Client session timed out, have not heard from server in 28517ms for sessionid 0x256d401adbf6f77, closing socket connection and attempting reconnect 2016-09-28 06:54:29,304 INFO [localhost-startStop-1-SendThread(hdp-jz5001.hadoop.local:2181)] zookeeper.ClientCnxn:1019 : Opening socket connection to server hdp-jz5001.hadoop.local/100.78.7.155:2181. Will not attempt to authenticate using SASL (unknown error) 2016-09-28 06:52:05,570 INFO [BadQueryDetector] service.BadQueryDetector:151 : System free memory less than 100 MB. 0 queries running. 2016-09-28 06:56:29,665 ERROR [Curator-Framework-0] imps.CuratorFrameworkImpl:537 : Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:708) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:826) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-09-28 06:57:31,275 INFO [BadQueryDetector] service.BadQueryDetector:151 : System free memory less than 100 MB. 0 queries running. 2016-09-28 06:56:29,665 INFO [pool-8-thread-10-SendThread(hdp-jz5001.hadoop.local:2181)] zookeeper.ClientCnxn:1019 : Opening socket connection to server hdp-jz5001.hadoop.local/100.78.7.155:2181. Will not attempt to authenticate using SASL (unknown error) # # java.lang.OutOfMemoryError: Java heap space # -XX:OnOutOfMemoryError="kill -9 %p" # Executing /bin/sh -c "kill -9 12727"...
