When we startup another one of our web applications on a second instance in a
cluster we get an org.jboss.cache.lock.TimeoutException thrown by
org.jboss.cache.lock.IdentityLock. It seems that IdentityLock is attempting to
acquire a WL when attempting to cache the query below but there is already a
lock on the node /org/hibernate/cache/StandardQueryCache/sql, probably in the
first instance. This application is running on Resin 3.1.2. We are using
Hibernate 3.2.3 and JBoss cache 1.4.0.SP1. We have Hibernate configured to use
a query and second level cache which is org.hibernate.cache.TreeCacheProvider,
the JBoss TreeCache. We have the JBoss TreeCache to use the pessimistic
locking scheme, the node isolation level is REPEATABLE_READ and the cache mode
is LOCAL.
How do we resolve this problem?
Please let me know if you need further information from me.
Stack trace:
[14:10:41,302] ERROR [pool-9-thread-8] IdentityLock : write lock for
/org/hibernate/cache/StandardQueryCache/sql: select searchro
ut0_.ID as ID28_, searchrout0_.hotel_code as hotel2_28_, searchrout0_.hotel_lat
as hotel3_28_, searchrout0_.hotel_long as hotel4_28_
, searchrout0_.destination_name as destinat5_28_, searchrout0_.destination_lat
as destinat6_28_, searchrout0_.destination_long as de
stinat7_28_, searchrout0_.distance_miles as distance8_28_,
searchrout0_.distance_km as distance9_28_, searchrout0_.route_type as rou
te10_28_ from UA_MQ_HOTEL_DESTINATION_ROUTE searchrout0_ where
searchrout0_.hotel_code=? and searchrout0_.hotel_lat=? and searchrout
0_.hotel_long=? and searchrout0_.destination_lat=? and
searchrout0_.destination_long=? and searchrout0_.route_type=?; parameters: ;
named parameters: {hotelCode=JMBNJ, hotelLong=-74.4783, routeType=2,
hotelLat=40.3444, destinationLong=-75.164199, destinationLat=39
.952201} could not be acquired after 0 ms. Locks: Read lock owners: []
Write lock owner: null
(caller=Thread[pool-9-thread-8,5,main], lock info: (activeReaders=0,
activeWriter=null, waitingReaders=0, waitingWriters
=0, waitingUpgrader=0))
[14:10:41,305] INFO [pool-9-thread-8] TxInterceptor : There was a problem
handling this request
org.jboss.cache.lock.TimeoutException: failure acquiring lock:
fqn=/org/hibernate/cache/StandardQueryCache/sql: select searchrout0_.
ID as ID28_, searchrout0_.hotel_code as hotel2_28_, searchrout0_.hotel_lat as
hotel3_28_, searchrout0_.hotel_long as hotel4_28_, sea
rchrout0_.destination_name as destinat5_28_, searchrout0_.destination_lat as
destinat6_28_, searchrout0_.destination_long as destina
t7_28_, searchrout0_.distance_miles as distance8_28_, searchrout0_.distance_km
as distance9_28_, searchrout0_.route_type as route10_
28_ from UA_MQ_HOTEL_DESTINATION_ROUTE searchrout0_ where
searchrout0_.hotel_code=? and searchrout0_.hotel_lat=? and searchrout0_.ho
tel_long=? and searchrout0_.destination_lat=? and
searchrout0_.destination_long=? and searchrout0_.route_type=?; parameters: ;
named
parameters: {hotelCode=JMBNJ, hotelLong=-74.4783, routeType=2,
hotelLat=40.3444, destinationLong=-75.164199, destinationLat=39.9522
01}, caller=Thread[pool-9-thread-8,5,main], lock= (activeReaders=0,
activeWriter=null, waitingReaders=0, waitingWriters=0,
waitingUpgrader=0)
at org.jboss.cache.Node.acquire(Node.java:407)
at
org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:228)
at
org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:160)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
at
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
at
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
at
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
at
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
at org.jboss.cache.TreeCache.putFailFast(TreeCache.java:3643)
at org.hibernate.cache.TreeCache.put(TreeCache.java:74)
at
org.hibernate.cache.StandardQueryCache.put(StandardQueryCache.java:87)
at org.hibernate.loader.Loader.putResultInQueryCache(Loader.java:2194)
at org.hibernate.loader.Loader.listUsingQueryCache(Loader.java:2138)
at org.hibernate.loader.Loader.list(Loader.java:2096)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:378)
at
org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)
at
org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121)
at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
at
com.ihg.dec.framework.dataAccessServices.hibernate.HibernateOperation.execGetListNamedQuery(HibernateOperation.java:208)
at
com.ihg.dec.framework.dataAccessServices.hibernate.destination.search.HibernateSearchResultDAO.getSearchRoute(HibernateSe
archResultDAO.java:219)
at
com.ihg.dec.framework.dataAccessServices.cacheable.destination.search.CacheableSearchResultDAO$4.retreiveCachedObject(Cac
heableSearchResultDAO.java:156)
at
com.ihg.dec.framework.dataAccessServices.cacheable.destination.search.CacheableSearchResultDAO$4.retreiveCachedObject(Cac
heableSearchResultDAO.java:153)
at
com.ihg.dec.framework.dataAccessServices.cacheable.AbstractCacheableDAO.getCachedObject(AbstractCacheableDAO.java:429)
at
com.ihg.dec.framework.dataAccessServices.cacheable.AbstractProxiedCacheableDAO.getProxiedCachedObject(AbstractProxiedCach
eableDAO.java:56)
at
com.ihg.dec.framework.dataAccessServices.cacheable.destination.search.CacheableSearchResultDAO.getSearchRoute(CacheableSe
archResultDAO.java:150)
at
com.ihg.dec.framework.businessServices.hotel.delegates.SearchDelegate.fillDistanceFromSearchResultDAO(SearchDelegate.java
:975)
at
com.ihg.dec.framework.businessServices.hotel.delegates.SearchDelegate.fillDistanceInformation(SearchDelegate.java:904)
at
com.ihg.dec.framework.businessServices.hotel.delegates.SearchDelegate.fillHotelInformation(SearchDelegate.java:792)
at
com.ihg.dec.framework.businessServices.hotel.delegates.SearchDelegate.constructHotelSearchResultSet(SearchDelegate.java:1
250)
at
com.ihg.dec.framework.businessServices.hotel.delegates.SearchDelegate.search(SearchDelegate.java:159)
at
com.ihg.dec.framework.businessServices.hotel.delegates.SearchDelegate.searchWithFilterAndSort(SearchDelegate.java:1448)
at
com.ihg.dec.framework.businessServices.hotel.HotelSearchBO.findHotels(HotelSearchBO.java:65)
at sun.reflect.GeneratedMethodAccessor195.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
com.ihg.dec.framework.businessServices.BusinessObjectProxy.invoke_aroundBody0(BusinessObjectProxy.java:111)
at
com.ihg.dec.framework.businessServices.BusinessObjectProxy$AjcClosure1.run(BusinessObjectProxy.java:1)
at
com.ihg.dec.aop.statistics.businessServices.BusinessObjectProxyStatisticCollector.ajc$around$com_ihg_dec_aop_statistics_b
usinessServices_BusinessObjectProxyStatisticCollector$1$640f34fproceed(BusinessObjectProxyStatisticCollector.aj:1)
at
com.ihg.dec.aop.statistics.businessServices.BusinessObjectProxyStatisticCollector$1.callProceed(BusinessObjectProxyStatis
ticCollector.aj:39)
at
com.ihg.dec.aop.statistics.AbstractStatisticCollector.callObjectProxy(AbstractStatisticCollector.aj:49)
at
com.ihg.dec.aop.statistics.businessServices.BusinessObjectProxyStatisticCollector.ajc$around$com_ihg_dec_aop_statistics_b
usinessServices_BusinessObjectProxyStatisticCollector$1$640f34f(BusinessObjectProxyStatisticCollector.aj:36)
at
com.ihg.dec.framework.businessServices.BusinessObjectProxy.invoke(BusinessObjectProxy.java:94)
at $Proxy15.findHotels(Unknown Source)
at
com.ihg.dec.framework.runtime.cacheWarmer.HotelSearchCacheWarmer$HotelSearchWarmTask.call(HotelSearchCacheWarmer.java:168
)
at
com.ihg.dec.framework.runtime.cacheWarmer.HotelSearchCacheWarmer$HotelSearchWarmTask.call(HotelSearchCacheWarmer.java:128
)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Caused by: org.jboss.cache.lock.TimeoutException: write lock for
/org/hibernate/cache/StandardQueryCache/sql: select searchrout0_.ID
as ID28_, searchrout0_.hotel_code as hotel2_28_, searchrout0_.hotel_lat as
hotel3_28_, searchrout0_.hotel_long as hotel4_28_, searc
hrout0_.destination_name as destinat5_28_, searchrout0_.destination_lat as
destinat6_28_, searchrout0_.destination_long as destinat7
_28_, searchrout0_.distance_miles as distance8_28_, searchrout0_.distance_km as
distance9_28_, searchrout0_.route_type as route10_28
_ from UA_MQ_HOTEL_DESTINATION_ROUTE searchrout0_ where
searchrout0_.hotel_code=? and searchrout0_.hotel_lat=? and searchrout0_.hote
l_long=? and searchrout0_.destination_lat=? and searchrout0_.destination_long=?
and searchrout0_.route_type=?; parameters: ; named p
arameters: {hotelCode=JMBNJ, hotelLong=-74.4783, routeType=2, hotelLat=40.3444,
destinationLong=-75.164199, destinationLat=39.952201
} could not be acquired after 0 ms. Locks: Read lock owners: []
Write lock owner: null
(caller=Thread[pool-9-thread-8,5,main], lock info: (activeReaders=0,
activeWriter=null, waitingReaders=0, waitingWriters
=0, waitingUpgrader=0))
at
org.jboss.cache.lock.IdentityLock.acquireWriteLock(IdentityLock.java:202)
at org.jboss.cache.Node.acquireWriteLock(Node.java:431)
at org.jboss.cache.Node.acquire(Node.java:386)
Hibernate configuration:
| <!DOCTYPE hibernate-configuration PUBLIC
| "-//Hibernate/Hibernate Configuration DTD 3.0//EN"
| "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd";>
|
| <hibernate-configuration>
| <session-factory>
| <property name="show_sql">false</property>
| <property
name="connection.datasource">java:comp/env/jdbc/oracle</property>
| <property name="dialect">
| org.hibernate.dialect.Oracle9Dialect
| </property>
| <property name="cache.provider_class">
| org.hibernate.cache.TreeCacheProvider
| </property>
| <property name="cache.use_query_cache">true</property>
| <property name="cache.use_second_level_cache">true</property>
| <property name="transaction.manager_lookup_class">
| org.hibernate.transaction.ResinTransactionManagerLookup
| </property>
| <property name="transaction.factory_class">
| org.hibernate.transaction.JTATransactionFactory
| </property>
| </session-factory>
| </hibernate-configuration>
|
JBoss TreeCache configuration:
|
| <?xml version="1.0" encoding="UTF-8"?>
|
| <!-- =====================================================================
-->
| <!--
-->
| <!-- TreeCache Service Configuration -->
| <!--
-->
| <!-- =====================================================================
-->
|
| <server>
|
| <classpath codebase="./lib" archives="jboss-cache.jar" />
|
|
| <!--
==================================================================== -->
| <!-- Defines TreeCache configuration
-->
| <!--
==================================================================== -->
|
| <mbean code="org.jboss.cache.TreeCache"
| name="jboss.cache:service=TreeCache">
|
| <depends>jboss:service=Naming</depends>
|
| <!--
| Configure the TransactionManager
| -->
| <!--
| Node locking scheme:
| OPTIMISTIC
| PESSIMISTIC (default)
| -->
| <attribute name="NodeLockingScheme">PESSIMISTIC</attribute>
|
| <attribute name="FetchInMemoryState">false</attribute>
| <attribute name="FetchPersistentState">false</attribute>
|
| <!--
| Node isolation level : SERIALIZABLE
| REPEATABLE_READ (default)
| READ_COMMITTED
| READ_UNCOMMITTED
| NONE
| -->
| <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
| <!--
| Valid modes are LOCAL
| REPL_ASYNC
| REPL_SYNC
| -->
| <attribute name="CacheMode">LOCAL</attribute>
|
| <!-- Whether each interceptor should have an mbean
| registered to capture and display its statistics. -->
| <attribute name="UseInterceptorMbeans">true</attribute>
|
| <!--
| The max amount of time (in milliseconds) we wait until
the
| initial state (ie. the contents of the cache) are
retrieved from
| existing members in a clustered environment
| -->
| <attribute name="InitialStateRetrievalTimeout">
| 100000
| </attribute>
|
| <!--
| Number of milliseconds to wait until all responses for a
| synchronous call have been received.
| -->
| <attribute name="SyncReplTimeout">10000</attribute>
|
| <!-- Max number of milliseconds to wait for a lock acquisition
-->
| <attribute name="LockAcquisitionTimeout">60000</attribute>
|
| <!-- Name of the eviction policy class. -->
| <attribute name="EvictionPolicyClass">
| org.jboss.cache.eviction.LRUPolicy
| </attribute>
|
| <!-- Specific eviction policy configurations. This is LRU -->
| <attribute name="EvictionPolicyConfig">
| <config>
| <attribute
name="wakeUpIntervalSeconds">5</attribute>
| <!-- Cache wide default -->
| <region name="/_default_">
| <attribute
name="maxNodes">5000</attribute>
| <attribute
name="timeToLiveSeconds">3600</attribute>
| <attribute
name="maxAgeSeconds">43200</attribute>
| </region>
|
| <region name="/cro/numbers">
| <attribute
name="maxNodes">50000</attribute>
| <attribute
name="timeToLiveSeconds">3600</attribute>
| <attribute
name="maxAgeSeconds">43200</attribute>
| </region>
|
| <region name="/hotels/detail">
| <attribute
name="maxNodes">2000</attribute>
| <attribute name="timeToLiveSeconds">
| 14400
| </attribute>
| <attribute
name="maxAgeSeconds">43200</attribute>
| </region>
|
| <!-- Vales based on that we are not properly
clearing the cache after a data load -->
| <region name="/hotels/sales/features">
| <attribute
name="maxNodes">5000</attribute>
| <attribute name="timeToLiveSeconds">
| 14400
| </attribute>
| <attribute
name="maxAgeSeconds">21600</attribute>
| </region>
|
| <region name="/hotels/sales/help">
| <attribute
name="maxNodes">5000</attribute>
| <attribute name="timeToLiveSeconds">
| 14400
| </attribute>
| <attribute
name="maxAgeSeconds">21600</attribute>
| </region>
|
| <region name="/hotels/geocode">
| <attribute
name="maxNodes">5000</attribute>
| <attribute
name="timeToLiveSeconds">86400</attribute>
| <attribute
name="maxAgeSeconds">86400</attribute>
| </region>
|
| <region name="/hotels/search/addresses">
| <attribute
name="maxNodes">5000</attribute>
| <attribute name="timeToLiveSeconds">
| 14400
| </attribute>
| <attribute
name="maxAgeSeconds">43200</attribute>
| </region>
|
| <region name="/destination/searchResult">
| <attribute
name="maxNodes">5000</attribute>
| <attribute name="timeToLiveSeconds">
| 14400
| </attribute>
| <attribute
name="maxAgeSeconds">43200</attribute>
| </region>
|
| <region name="/destination/searchRoute">
| <attribute
name="maxNodes">5000</attribute>
| <attribute name="timeToLiveSeconds">
| 14400
| </attribute>
| <attribute
name="maxAgeSeconds">43200</attribute>
| </region>
|
| <region name="/hotels/search/hotels">
| <attribute
name="maxNodes">50000</attribute>
| <attribute name="timeToLiveSeconds">
| 14400
| </attribute>
| <attribute
name="maxAgeSeconds">86400</attribute>
| </region>
|
| <region name="/lookup/countryCollections">
| <attribute
name="maxNodes">50000</attribute>
| <attribute name="timeToLiveSeconds">
| 21600
| </attribute>
| <attribute
name="maxAgeSeconds">86400</attribute>
| </region>
|
| <region name="/lookup/stateCollections">
| <attribute
name="maxNodes">50000</attribute>
| <attribute name="timeToLiveSeconds">
| 21600
| </attribute>
| <attribute
name="maxAgeSeconds">86400</attribute>
| </region>
|
| <region name="/lookup/pointsOfInterests">
| <attribute
name="maxNodes">50000</attribute>
| <attribute name="timeToLiveSeconds">
| 21600
| </attribute>
| <attribute
name="maxAgeSeconds">86400</attribute>
| </region>
|
| <region name="/teasers">
| <attribute
name="maxNodes">50000</attribute>
| <attribute
name="timeToLiveSeconds">3600</attribute>
| <attribute
name="maxAgeSeconds">259200</attribute>
| </region>
|
| <region name="/urlmanager">
| <attribute
name="maxNodes">50000</attribute>
| <attribute
name="timeToLiveSeconds">3600</attribute>
| <attribute
name="maxAgeSeconds">43200</attribute>
| </region>
|
|
| <region name="/rate/RateCategory">
| <attribute
name="maxNodes">1500</attribute>
| <attribute name="timeToLiveSeconds">
| 604800
| </attribute>
| <attribute
name="maxAgeSeconds">604800</attribute>
| </region>
| <region name="/b2b/displayableText">
| <attribute
name="maxNodes">1000</attribute>
| <attribute name="timeToLiveSeconds">
| 604800
| </attribute>
| <attribute
name="maxAgeSeconds">604800</attribute>
| </region>
| <region name="/b2b/client">
| <attribute
name="maxNodes">1000</attribute>
| <attribute name="timeToLiveSeconds">
| 604800
| </attribute>
| <attribute
name="maxAgeSeconds">604800</attribute>
| </region>
| <region name="/b2b/codes">
| <attribute
name="maxNodes">1000</attribute>
| <attribute name="timeToLiveSeconds">
| 604800
| </attribute>
| <attribute
name="maxAgeSeconds">604800</attribute>
| </region>
| <region name="/b2b/errors">
| <attribute
name="maxNodes">1000</attribute>
| <attribute name="timeToLiveSeconds">
| 604800
| </attribute>
| <attribute
name="maxAgeSeconds">604800</attribute>
| </region>
| <!--
| This region should be used for caching
transaction errors from
| external systems such as 500 errors
from Content servres. Nodes should
| not be stored directly to this region
but should be stored in sub-regions
| such as /transactionErrors/
| -->
| <region name="/transactionErrors">
| <attribute
name="maxNodes">1000</attribute>
| <attribute
name="timeToLiveSeconds">60</attribute>
| <attribute
name="maxAgeSeconds">60</attribute>
| </region>
|
| <!--
| This region is used as an in-memory
cache for avail cache. Items in avail cache should
| not live beyond 15 minutes.
| -->
| <region name="/availCache">
| <attribute
name="maxNodes">5000</attribute>
| <attribute
name="timeToLiveSeconds">900</attribute>
| <attribute
name="maxAgeSeconds">900</attribute>
| </region>
|
| <!-- Region to store the collection of Brand
objects -->
| <region name="/lookup/brand">
| <attribute
name="maxNodes">50</attribute>
| <attribute
name="timeToLiveSeconds">0</attribute>
| <attribute
name="maxAgeSeconds">0</attribute>
| </region>
|
| <!-- Region to store the collection of
FeatureType objects -->
| <region name="/lookup/featureType">
| <attribute
name="maxNodes">2000</attribute>
| <attribute name="timeToLiveSeconds">
| 21600
| </attribute>
| <attribute
name="maxAgeSeconds">86400</attribute>
| </region>
|
| <!-- Region to store geo destination objects -->
| <region name="/hotels/search/geoDestinations">
| <attribute
name="maxNodes">5000</attribute>
| <attribute name="timeToLiveSeconds">
| 14400
| </attribute>
| <attribute
name="maxAgeSeconds">43200</attribute>
| </region>
|
| <!-- Region to store the RoomType objects -->
| <region name="/lookup/roomType">
| <attribute
name="maxNodes">100</attribute>
| <attribute name="timeToLiveSeconds">
| 21600
| </attribute>
| <attribute
name="maxAgeSeconds">86400</attribute>
| </region>
|
| <!-- Region to store hotel name objects, the
maxAge is 24 hours and timeToLive is 12 hours -->
| <region name="/hotels/search/hotelName">
| <attribute
name="maxNodes">50000</attribute>
| <attribute name="timeToLiveSeconds">
| 43200
| </attribute>
| <attribute
name="maxAgeSeconds">86400</attribute>
| </region>
|
| <!-- Region to store city nick name, the maxAge
is 24 hours and timeToLive is 12 hours -->
| <region name="/hotels/search/cityNickname">
| <attribute
name="maxNodes">50000</attribute>
| <attribute name="timeToLiveSeconds">
| 43200
| </attribute>
| <attribute
name="maxAgeSeconds">86400</attribute>
| </region>
|
| <!-- Region to store the Currency objects -->
| <region name="/lookup/currency">
| <attribute
name="maxNodes">200</attribute>
| <attribute name="timeToLiveSeconds">
| 21600
| </attribute>
| <attribute
name="maxAgeSeconds">86400</attribute>
| </region>
|
| <!-- Region to store the CurrencyDetail objects
-->
| <region name="/lookup/currencyDetail">
| <attribute
name="maxNodes">200</attribute>
| <attribute
name="timeToLiveSeconds">3600</attribute>
| <attribute
name="maxAgeSeconds">7200</attribute>
| </region>
|
| <!-- Region to store credit card identifiers -->
| <region name="/lookup/creditCard">
| <attribute
name="maxNodes">10</attribute>
| <attribute name="timeToLiveSeconds">
| 86400
| </attribute>
| <attribute
name="maxAgeSeconds">86400</attribute>
| </region>
|
| <!-- Region to store contact centers -->
| <region name="/lookup/contactCenter">
| <attribute
name="maxNodes">20</attribute>
| <attribute name="timeToLiveSeconds">
| 86400
| </attribute>
| <attribute
name="maxAgeSeconds">86400</attribute>
| </region>
|
| <!-- Region to store contact centers -->
| <region name="/proxy/proxyClasses">
| <attribute
name="maxNodes">2000</attribute>
| <attribute name="timeToLiveSeconds">
| 86400
| </attribute>
| <attribute
name="maxAgeSeconds">86400</attribute>
| </region>
|
| </config>
| </attribute>
|
| </mbean>
| </server>
|
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4114227#4114227
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4114227
_______________________________________________
jboss-user mailing list
[email protected]
https://lists.jboss.org/mailman/listinfo/jboss-user
