The cause of this issue is the deployment task not being properly terminated before the server shutdown is initialized. I don't think Supun's fix to persistence will fix this. It is a different issue.
On Thu, May 31, 2012 at 1:49 PM, Hiranya Jayathilaka <[email protected]>wrote: > I got the following error when shutting the server down: > > [2012-05-31 13:46:55,603] ERROR - JDBCResourceDAO Failed to get the > resource at path > /_system/local/repository/components/org.wso2.carbon.system.statistics/services/WSRegistryService/getChildCount. > Database is already closed (to disable automatic closing at VM shutdown, > add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL) [90121-140] > org.h2.jdbc.JdbcSQLException: Database is already closed (to disable > automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db > URL) [90121-140] > at org.h2.message.DbException.getJdbcSQLException(DbException.java:327) > at org.h2.message.DbException.get(DbException.java:167) > at org.h2.message.DbException.get(DbException.java:144) > at org.h2.message.DbException.get(DbException.java:133) > at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1348) > at org.h2.jdbc.JdbcStatement.checkClosed(JdbcStatement.java:886) > at > org.h2.jdbc.JdbcPreparedStatement.checkClosed(JdbcPreparedStatement.java:1487) > at org.h2.jdbc.JdbcStatement.checkClosed(JdbcStatement.java:860) > at org.h2.jdbc.JdbcResultSet.checkClosed(JdbcResultSet.java:2928) > at org.h2.jdbc.JdbcResultSet.getColumnIndex(JdbcResultSet.java:2852) > at org.h2.jdbc.JdbcResultSet.get(JdbcResultSet.java:2959) > at org.h2.jdbc.JdbcResultSet.getString(JdbcResultSet.java:283) > at > org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:263) > at > org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:263) > at > org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getResourceMetaData(JDBCResourceDAO.java:1146) > at > org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getResourceMetaData(JDBCResourceDAO.java:1084) > at org.wso2.carbon.registry.core.jdbc.Repository.get(Repository.java:209) > at > org.wso2.carbon.registry.core.jdbc.handlers.filters.MediaTypeMatcher.handleGet(MediaTypeMatcher.java:130) > at > org.wso2.carbon.registry.core.jdbc.handlers.HandlerManager.get(HandlerManager.java:2439) > at > org.wso2.carbon.registry.core.jdbc.handlers.HandlerLifecycleManager.get(HandlerLifecycleManager.java:955) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:512) > at > org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:133) > at > org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:516) > at > org.wso2.carbon.statistics.persistance.StatisticsPersistenceScheduler.persistOperationStatistics(StatisticsPersistenceScheduler.java:171) > at > org.wso2.carbon.statistics.persistance.StatisticsPersistenceScheduler.invoke(StatisticsPersistenceScheduler.java:37) > at > org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.invokeCarbonDeploymentSchedulerExtenders(CarbonDeploymentSchedulerTask.java:101) > at > org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:71) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-05-31 13:46:55,606] ERROR - TransactionManager Failed to rollback > transaction. > org.h2.jdbc.JdbcSQLException: Database is already closed (to disable > automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db > URL) [90121-140] > at org.h2.message.DbException.getJdbcSQLException(DbException.java:327) > at org.h2.message.DbException.get(DbException.java:167) > at org.h2.message.DbException.get(DbException.java:144) > at org.h2.message.DbException.get(DbException.java:133) > at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1348) > at org.h2.jdbc.JdbcConnection.checkClosedForWrite(JdbcConnection.java:1333) > at org.h2.jdbc.JdbcConnection.rollback(JdbcConnection.java:435) > at > org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:368) > at > org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:323) > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCDatabaseTransaction$ManagedRegistryConnection.rollback(JDBCDatabaseTransaction.java:1244) > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.rollbackTransaction(JDBCTransactionManager.java:120) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.rollbackTransaction(EmbeddedRegistry.java:447) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:552) > at > org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:133) > at > org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:516) > at > org.wso2.carbon.statistics.persistance.StatisticsPersistenceScheduler.persistOperationStatistics(StatisticsPersistenceScheduler.java:171) > at > org.wso2.carbon.statistics.persistance.StatisticsPersistenceScheduler.invoke(StatisticsPersistenceScheduler.java:37) > at > org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.invokeCarbonDeploymentSchedulerExtenders(CarbonDeploymentSchedulerTask.java:101) > at > org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:71) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-05-31 13:46:55,607] ERROR - TransactionManager Failed to close > transaction. > java.sql.SQLException: Already closed. > at > org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:114) > at > org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:191) > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCDatabaseTransaction$ManagedRegistryConnection.close(JDBCDatabaseTransaction.java:1309) > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.endTransaction(JDBCTransactionManager.java:178) > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.rollbackTransaction(JDBCTransactionManager.java:128) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.rollbackTransaction(EmbeddedRegistry.java:447) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:552) > at > org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:133) > at > org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:516) > at > org.wso2.carbon.statistics.persistance.StatisticsPersistenceScheduler.persistOperationStatistics(StatisticsPersistenceScheduler.java:171) > at > org.wso2.carbon.statistics.persistance.StatisticsPersistenceScheduler.invoke(StatisticsPersistenceScheduler.java:37) > at > org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.invokeCarbonDeploymentSchedulerExtenders(CarbonDeploymentSchedulerTask.java:101) > at > org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:71) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to > close transaction. > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.endTransaction(JDBCTransactionManager.java:183) > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.rollbackTransaction(JDBCTransactionManager.java:128) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.rollbackTransaction(EmbeddedRegistry.java:447) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:552) > at > org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:133) > at > org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:516) > at > org.wso2.carbon.statistics.persistance.StatisticsPersistenceScheduler.persistOperationStatistics(StatisticsPersistenceScheduler.java:171) > at > org.wso2.carbon.statistics.persistance.StatisticsPersistenceScheduler.invoke(StatisticsPersistenceScheduler.java:37) > at > org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.invokeCarbonDeploymentSchedulerExtenders(CarbonDeploymentSchedulerTask.java:101) > at > org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:71) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.sql.SQLException: Already closed. > at > org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:114) > at > org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:191) > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCDatabaseTransaction$ManagedRegistryConnection.close(JDBCDatabaseTransaction.java:1309) > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.endTransaction(JDBCTransactionManager.java:178) > ... 18 more > > Is this fixed in latest builds? > > Thanks, > Hiranya > > On Wed, May 30, 2012 at 10:07 PM, Supun Malinga <[email protected]> wrote: > >> Hi azeez, >> >> I have fixed the tenant registry issue. Now the stats are persisted >> correctly. So both this and the previous issue you reported should be >> solved by that. Will commit the changes shortly. Still running a build with >> tests. >> >> thanks, >> >> >> On Wed, May 30, 2012 at 9:51 PM, Afkham Azeez <[email protected]> wrote: >> >>> I think I know the reason for this issue. The code had an error where >>> all tenant's stuff are persisted into the STs registry. The deployment >>> tasks of the ST & tenant azeez.org would have clashed causing this >>> issue. >>> >>> >>> On Wed, May 30, 2012 at 9:47 PM, Afkham Azeez <[email protected]> wrote: >>> >>>> I ran a distributed load test today on a tenant in AppServer, and under >>>> load, the following exception was being printed from time to time. Possibly >>>> a registry issue or an issue with the stats persistence code. This >>>> deployment task runs every 15 seconds or so. I'd advice that we remove the >>>> stat persistence code until it is properly reviewed & tested. >>>> >>>> >>>> [2012-05-30 21:42:28,675] ERROR >>>> {org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO} - Failed to get >>>> the resource at path >>>> /_system/local/repository/components/org.wso2.carbon.system.statistics/services/echo. >>>> Timeout trying to lock table "REG_RESOURCE"; SQL statement: >>>> SELECT REG_MEDIA_TYPE, REG_CREATOR, REG_CREATED_TIME, REG_LAST_UPDATOR, >>>> REG_LAST_UPDATED_TIME, REG_VERSION, REG_DESCRIPTION, REG_CONTENT_ID, >>>> REG_UUID FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME IS NULL AND >>>> REG_TENANT_ID=? [50200-140] >>>> org.h2.jdbc.JdbcSQLException: Timeout trying to lock table >>>> "REG_RESOURCE"; SQL statement: >>>> SELECT REG_MEDIA_TYPE, REG_CREATOR, REG_CREATED_TIME, REG_LAST_UPDATOR, >>>> REG_LAST_UPDATED_TIME, REG_VERSION, REG_DESCRIPTION, REG_CONTENT_ID, >>>> REG_UUID FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME IS NULL AND >>>> REG_TENANT_ID=? [50200-140] >>>> at >>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:327) >>>> at org.h2.message.DbException.get(DbException.java:167) >>>> at org.h2.message.DbException.get(DbException.java:144) >>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466) >>>> at org.h2.table.RegularTable.lock(RegularTable.java:404) >>>> at org.h2.table.TableFilter.lock(TableFilter.java:139) >>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554) >>>> at org.h2.command.dml.Query.query(Query.java:241) >>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80) >>>> at org.h2.command.Command.executeQuery(Command.java:132) >>>> at >>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96) >>>> at >>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) >>>> at >>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) >>>> at >>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getResourceMetaData(JDBCResourceDAO.java:1131) >>>> at >>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getResourceMetaData(JDBCResourceDAO.java:1084) >>>> at >>>> org.wso2.carbon.registry.core.jdbc.Repository.get(Repository.java:209) >>>> at >>>> org.wso2.carbon.registry.core.jdbc.handlers.filters.MediaTypeMatcher.handleGet(MediaTypeMatcher.java:130) >>>> at >>>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerManager.get(HandlerManager.java:2439) >>>> at >>>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerLifecycleManager.get(HandlerLifecycleManager.java:955) >>>> at >>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:512) >>>> at >>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:133) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:516) >>>> at >>>> org.wso2.carbon.statistics.persistance.StatisticsPersistenceScheduler.persistServiceStatistics(StatisticsPersistenceScheduler.java:101) >>>> at >>>> org.wso2.carbon.statistics.persistance.StatisticsPersistenceScheduler.invoke(StatisticsPersistenceScheduler.java:38) >>>> at >>>> org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.invokeCarbonDeploymentSchedulerExtenders(CarbonDeploymentSchedulerTask.java:101) >>>> >>>> -- >>>> *Afkham Azeez* >>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>> Member; Apache Software Foundation; http://www.apache.org/ >>>> * <http://www.apache.org/>** >>>> email: **[email protected]* <[email protected]>* cell: +94 77 3320919 >>>> blog: **http://blog.afkham.org* <http://blog.afkham.org>* >>>> twitter: >>>> **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> >>>> * >>>> linked-in: **http://lk.linkedin.com/in/afkhamazeez* >>>> * >>>> * >>>> *Lean . Enterprise . Middleware* >>>> >>>> >>> >>> >>> -- >>> *Afkham Azeez* >>> Director of Architecture; WSO2, Inc.; http://wso2.com >>> Member; Apache Software Foundation; http://www.apache.org/ >>> * <http://www.apache.org/>** >>> email: **[email protected]* <[email protected]>* cell: +94 77 3320919 >>> blog: **http://blog.afkham.org* <http://blog.afkham.org>* >>> twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> >>> * >>> linked-in: **http://lk.linkedin.com/in/afkhamazeez* >>> * >>> * >>> *Lean . Enterprise . Middleware* >>> >>> >> >> >> -- >> Supun Malinga, >> >> Software Engineer, >> WSO2 Inc. >> http://wso2.com >> http://wso2.org >> email - [email protected] <[email protected]> >> mobile - 071 56 91 321 >> >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Hiranya Jayathilaka > Senior Technical Lead; > WSO2 Inc.; http://wso2.org > E-mail: [email protected]; Mobile: +94 77 633 3491 > Blog: http://techfeast-hiranya.blogspot.com > -- *Afkham Azeez* Director of Architecture; WSO2, Inc.; http://wso2.com Member; Apache Software Foundation; http://www.apache.org/ * <http://www.apache.org/>** email: **[email protected]* <[email protected]>* cell: +94 77 3320919 blog: **http://blog.afkham.org* <http://blog.afkham.org>* twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> * linked-in: **http://lk.linkedin.com/in/afkhamazeez* * * *Lean . Enterprise . Middleware*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
