Hi, Then this seems to be the Hazlecast OOM issue. Please report a jira for this.
Regards, Firzhan -- *Firzhan Naqash* Senior Software Engineer - Integration Platform Team WSO2 Inc. http://wso2.com email: [email protected] <[email protected]> mobile: (+94) 77 9785674 <%28%2B94%29%2071%205247551>*| blog: http://firzhanblogger.blogspot.com/ <http://firzhanblogger.blogspot.com/> <http://suhothayan.blogspot.com/>* *twitter: https://twitter.com/firzhan007 <https://twitter.com/firzhan007> | linked-in: **https://www.linkedin.com/in/firzhan <https://www.linkedin.com/in/firzhan>* On Mon, Oct 12, 2015 at 11:11 AM, Chathurika De Silva <[email protected]> wrote: > Hi Firzhan > > The cluster have the following configuration currently. The default value > is 256m > > Xms512m -Xmx2048m -XX:MaxPermSize=512m > > Please advice with the recommended value and please note that this > configuration is not given in the documentation > > On Mon, Oct 12, 2015 at 10:56 AM, Firzhan Naqash <[email protected]> wrote: > >> Hi Chathiruka, >> >> This seems to be the hazlecast OOM error observed with kernel 4.4.1. Can >> you increase heap size and check this scenario again. >> >> >> *TID: [-1] [] [2015-10-09 18:49:36,827] ERROR {org.apache.tomcat.util.net >> <http://org.apache.tomcat.util.net>.NioEndpoint$Poller} - >> {org.apache.tomcat.util.net >> <http://org.apache.tomcat.util.net>.NioEndpoint$Poller}java.lang.OutOfMemoryError: >> GC overhead limit exceeded* >> >> Regards, >> Firzhan >> >> >> -- >> *Firzhan Naqash* >> Senior Software Engineer - Integration Platform Team >> WSO2 Inc. http://wso2.com >> >> email: [email protected] <[email protected]> >> mobile: (+94) 77 9785674 <%28%2B94%29%2071%205247551>*| >> blog: http://firzhanblogger.blogspot.com/ >> <http://firzhanblogger.blogspot.com/> <http://suhothayan.blogspot.com/>* >> *twitter: https://twitter.com/firzhan007 >> <https://twitter.com/firzhan007> | linked-in: >> **https://www.linkedin.com/in/firzhan >> <https://www.linkedin.com/in/firzhan>* >> >> On Mon, Oct 12, 2015 at 10:51 AM, Chathurika De Silva < >> [email protected]> wrote: >> >>> Hi All >>> >>> We have configured a SchedulerTask for the BPS cluster (2 workers and >>> manager fronted by nginx). The task runs every day 5.10 pm and deletes >>> FAILED instances. The cluster was up and running in the weekend but was >>> idle. Following error was observed. >>> >>> >>> TID: [-1] [] [2015-10-09 18:49:23,157] ERROR >>> {org.apache.ode.scheduler.simple.SimpleScheduler} - Error during >>> SchedulerTask execution {org.apache.ode.scheduler.simple.SimpleScheduler} >>> com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast >>> instance is not active! >>> at >>> com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.getService(ProxyRegistry.java:65) >>> at >>> com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.<init>(ProxyRegistry.java:53) >>> at >>> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:74) >>> at >>> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:72) >>> at >>> com.hazelcast.util.ConcurrencyUtil.getOrPutIfAbsent(ConcurrencyUtil.java:51) >>> at >>> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.getOrCreateRegistry(ProxyServiceImpl.java:110) >>> at >>> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:118) >>> at >>> com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:390) >>> at >>> com.hazelcast.instance.HazelcastInstanceImpl.getMap(HazelcastInstanceImpl.java:192) >>> at >>> com.hazelcast.instance.HazelcastInstanceProxy.getMap(HazelcastInstanceProxy.java:75) >>> at >>> org.wso2.carbon.bpel.core.ode.integration.BPELServerImpl$ODEClusterImpl.getKnownNodes(BPELServerImpl.java:1009) >>> at >>> org.apache.ode.scheduler.simple.SimpleScheduler.updateHeartBeatAndGetStaleNodes(SimpleScheduler.java:848) >>> at >>> org.apache.ode.scheduler.simple.SimpleScheduler.access$800(SimpleScheduler.java:53) >>> at >>> org.apache.ode.scheduler.simple.SimpleScheduler$CheckStaleNodes.run(SimpleScheduler.java:1226) >>> at >>> org.apache.ode.scheduler.simple.SimpleScheduler$8.call(SimpleScheduler.java:810) >>> at >>> org.apache.ode.scheduler.simple.SimpleScheduler$8.call(SimpleScheduler.java:807) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> TID: [-1] [] [2015-10-09 18:49:36,827] ERROR >>> {org.apache.tomcat.util.net.NioEndpoint$Poller} - >>> {org.apache.tomcat.util.net.NioEndpoint$Poller} >>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>> TID: [-1] [] [2015-10-09 18:49:36,827] ERROR >>> {org.apache.tomcat.util.net.NioEndpoint$Poller} - >>> {org.apache.tomcat.util.net.NioEndpoint$Poller} >>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>> TID: [-1] [] [2015-10-09 18:49:41,310] ERROR >>> {org.apache.tomcat.util.net.NioEndpoint$Poller} - >>> {org.apache.tomcat.util.net.NioEndpoint$Poller} >>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>> TID: [-1] [] [2015-10-09 18:49:53,704] ERROR >>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} - >>> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous >>> execution of execution callback: >>> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@6adde0a6for >>> call Invocation{ serviceName='hz:impl:mapService', >>> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService', >>> partitionId=-1, callId=0, invocationTime=-1, waitTimeout=-1, >>> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, >>> tryPauseMillis=0, invokeCount=0, callTimeout=5000, target=null, >>> backupsExpected=0, backupsCompleted=0} >>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} >>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>> TID: [-1] [] [2015-10-09 18:50:06,800] ERROR >>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} - >>> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous >>> execution of execution callback: >>> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@295de461for >>> call Invocation{ serviceName='hz:impl:mapService', >>> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService', >>> partitionId=-1, callId=0, invocationTime=1444415365856, waitTimeout=-1, >>> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, >>> tryPauseMillis=0, invokeCount=1, callTimeout=5000, >>> target=Address[192.168.48.2]:4000, backupsExpected=0, backupsCompleted=0} >>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} >>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>> TID: [-1] [] [2015-10-09 18:50:30,826] ERROR >>> {org.wso2.carbon.humantask.core.scheduler.SimpleScheduler} - Error loading >>> immediate jobs from database. >>> {org.wso2.carbon.humantask.core.scheduler.SimpleScheduler} >>> <openjpa-2.2.0-wso2v1-r128166:129108 fatal general error> >>> org.apache.openjpa.persistence.PersistenceException: No more data to read >>> from socket >>> FailedObject: SELECT DISTINCT t FROM >>> org.wso2.carbon.humantask.core.dao.jpa.openjpa.model.HumanTaskJob t WHERE >>> t.nodeId = ?1 AND t.time < ?2 order by t.time [java.lang.String] >>> at >>> org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:4918) >>> at >>> org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(DBDictionary.java:4878) >>> at >>> org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:136) >>> at >>> org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:110) >>> at >>> org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:62) >>> at >>> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:949) >>> at >>> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getConnection(JDBCStoreManager.java:240) >>> at >>> org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:413) >>> at >>> org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:382) >>> at >>> org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.open(SelectResultObjectProvider.java:94) >>> at >>> org.apache.openjpa.lib.rop.EagerResultList.<init>(EagerResultList.java:34) >>> at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1251) >>> at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1007) >>> at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863) >>> at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794) >>> at >>> org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542) >>> at >>> org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:286) >>> at >>> org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302) >>> at >>> org.wso2.carbon.humantask.core.dao.jpa.openjpa.HumanTaskDAOConnectionImpl.dequeueImmediate(HumanTaskDAOConnectionImpl.java:248) >>> at >>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$2.call(SimpleScheduler.java:344) >>> at >>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$2.call(SimpleScheduler.java:342) >>> at >>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.execTransaction(SimpleScheduler.java:879) >>> at >>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.execTransaction(SimpleScheduler.java:828) >>> at >>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.doLoadImmediate(SimpleScheduler.java:342) >>> at >>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$LoadImmediateTask.run(SimpleScheduler.java:297) >>> at >>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$1.call(SimpleScheduler.java:149) >>> at >>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$1.call(SimpleScheduler.java:146) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> Caused by: java.sql.SQLRecoverableException: No more data to read from >>> socket >>> at >>> oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:456) >>> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397) >>> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257) >>> at >>> oracle.jdbc.driver.T4CTTIoauthenticate.doOSESSKEY(T4CTTIoauthenticate.java:460) >>> at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:594) >>> at >>> oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666) >>> at >>> oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32) >>> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566) >>> at >>> org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278) >>> at >>> org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182) >>> at >>> org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701) >>> at >>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635) >>> at >>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>> at >>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>> at >>> org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110) >>> at >>> org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:87) >>> at >>> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreManager.java:960) >>> at >>> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:945) >>> ... 25 more >>> TID: [-1] [] [2015-10-09 18:51:57,344] ERROR >>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} - >>> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous >>> execution of execution callback: >>> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@1c5fe7d7for >>> call Invocation{ serviceName='hz:impl:mapService', >>> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService', >>> partitionId=-1, callId=0, invocationTime=-1, waitTimeout=-1, >>> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, >>> tryPauseMillis=0, invokeCount=0, callTimeout=5000, target=null, >>> backupsExpected=0, backupsCompleted=0} >>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} >>> TID: [-1] [] [2015-10-09 18:54:53,738] ERROR >>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} - >>> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous >>> execution of execution callback: >>> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@3c90fdbffor >>> call Invocation{ serviceName='hz:impl:mapService', >>> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService', >>> partitionId=-1, callId=0, invocationTime=1444415373639, waitTimeout=-1, >>> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, >>> tryPauseMillis=0, invokeCount=1, callTimeout=5000, >>> target=Address[192.168.48.2]:4000, backupsExpected=0, backupsCompleted=0} >>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} >>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>> >>> >>> What is the reason for such an error? Please note the error is >>> continuously coming in the cluster. Any help will be highly appreciated. >>> -- >>> *Chathurika Erandi* >>> Software Engineer, >>> WSO2 Inc. >>> >>> *E-mail:* [email protected] >>> *Mobile: *+94714 328 612 >>> *Blog*: chathurikaerandi.blogspot.com >>> >>> _______________________________________________ >>> Dev mailing list >>> [email protected] >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> > > > -- > *Chathurika Erandi* > Software Engineer, > WSO2 Inc. > > *E-mail:* [email protected] > *Mobile: *+94714 328 612 > *Blog*: chathurikaerandi.blogspot.com >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
