Deadlock on cluster during making changes on content repository
---------------------------------------------------------------
Key: JCR-2822
URL: https://issues.apache.org/jira/browse/JCR-2822
Project: Jackrabbit Content Repository
Issue Type: Bug
Components: clustering, jackrabbit-core
Affects Versions: 1.6.4
Environment: - JEE Web application with parallel web sessions
- Several clusters of JEE server
- Oracle database
Reporter: aliaksandr shpak
Deadlock is possible on cluster during making changes on content repository.
The steps for deadlock are following:
Thread 1 - making some changes on content repository. Code:
public static void move(final Node node, final Node destNode, final Session
session,
final Boolean versionable) throws ItemNotFoundException,
RepositoryException {
final String path = node.getPath();
final String name = node.getName();
final String destPath = destNode.getPath() + FileSystem.SEPARATOR_CHAR
+ name;
session.move(path, destPath);
session.save();
Thread 2 - tries to commit transaction and executes
org.apache.jackrabbit.core.TransactionContext.prepare()
What's happened:
Thread 1:
- Call session.move
- ..., calls CachingHierarchyManager.nodeRemoved, it locks cacheMonitor
- thread 2 in this time does sync and executes
SharedItemStateManager.externalUpdate, which occurs write lock
- thread 1 continues working and tries to occur read lock on
SharedItemStateManager.hasItemState, but write lock is already on thread 2, so
thread 1 starts wait
Thread 2:
- Calls TransactionBoundXAResource.prepare
- comes to ClusterNode.consume
- Executes SharedItemStateManager.externalUpdate, where occurs the write lock
- And finally comes to CachingHierarchyManager.nodeModified, but can't
proceed because of cacheMonitor is already locked by Thread1
Thrace from Thread1:
null at java.lang.Object.wait(J)V(Native Method)
- waiting on <0xfffffffe7e84c348> (a
EDU.oswego.cs.dl.util.concurrent.Latch)
at java.lang.Object.wait()V(Object.java:474)
at EDU.oswego.cs.dl.util.concurrent.Latch.acquire()V(Unknown Source)
- locked <0xfffffffe7e84c348> (a EDU.oswego.cs.dl.util.concurrent.Latch)
at
org.apache.jackrabbit.core.state.FineGrainedISMLocking.acquireReadLock(Lorg/apache/jackrabbit/core/ItemId;)Lorg/apache/jackrabbit/core/state/ISMLocking$ReadLock;(FineGrainedISMLocking.java:112)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.acquireReadLock(Lorg/apache/jackrabbit/core/ItemId;)Lorg/apache/jackrabbit/core/state/ISMLocking$ReadLock;(SharedItemStateManager.java:1432)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.hasItemState(Lorg/apache/jackrabbit/core/ItemId;)Z(SharedItemStateManager.java:286)
at
org.apache.jackrabbit.core.state.XAItemStateManager.hasItemState(Lorg/apache/jackrabbit/core/ItemId;)Z(XAItemStateManager.java:302)
at
org.apache.jackrabbit.core.state.SessionItemStateManager.hasItemState(Lorg/apache/jackrabbit/core/ItemId;)Z(SessionItemStateManager.java:218)
at
org.apache.jackrabbit.core.HierarchyManagerImpl.hasItemState(Lorg/apache/jackrabbit/core/ItemId;)Z(HierarchyManagerImpl.java:164)
at
org.apache.jackrabbit.core.CachingHierarchyManager.nodeRemoved(Lorg/apache/jackrabbit/core/state/NodeState;Lorg/apache/jackrabbit/spi/Path;Lorg/apache/jackrabbit/core/NodeId;)V(CachingHierarchyManager.java:725)
at
org.apache.jackrabbit.core.CachingHierarchyManager.nodeRemoved(Lorg/apache/jackrabbit/core/state/NodeState;Lorg/apache/jackrabbit/spi/Name;ILorg/apache/jackrabbit/core/NodeId;)V(CachingHierarchyManager.java:455)
- locked <0xfffffffe63626110> (a java.lang.Object)
at
org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyNodeRemoved(Lorg/apache/jackrabbit/core/state/NodeState;Lorg/apache/jackrabbit/spi/Name;ILorg/apache/jackrabbit/core/NodeId;)V(StateChangeDispatcher.java:207)
at
org.apache.jackrabbit.core.state.SessionItemStateManager.nodeRemoved(Lorg/apache/jackrabbit/core/state/NodeState;Lorg/apache/jackrabbit/spi/Name;ILorg/apache/jackrabbit/core/NodeId;)V(SessionItemStateManager.java:983)
at
org.apache.jackrabbit.core.state.NodeState.notifyNodeRemoved(Lorg/apache/jackrabbit/core/state/ChildNodeEntry;)V(NodeState.java:924)
at
org.apache.jackrabbit.core.state.NodeState.removeChildNodeEntry(Lorg/apache/jackrabbit/spi/Name;I)Z(NodeState.java:403)
at
org.apache.jackrabbit.core.SessionImpl.move(Ljava/lang/String;Ljava/lang/String;)V(SessionImpl.java:1118)
at
org.apache.jackrabbit.jca.JCASessionHandle.move(Ljava/lang/String;Ljava/lang/String;)V(JCASessionHandle.java:170)
at
mypackage.ContentRepositoryUtil.move(Ljavax/jcr/Node;Ljavax/jcr/Node;Ljavax/jcr/Session;)V(ContentRepositoryUtil.java:163)
Thread 2:
null at
org.apache.jackrabbit.core.CachingHierarchyManager.nodeModified(Lorg/apache/jackrabbit/core/state/NodeState;)V(CachingHierarchyManager.java:306)
- waiting to lock <0xfffffffe63626110> (a java.lang.Object)
at
org.apache.jackrabbit.core.CachingHierarchyManager.stateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(CachingHierarchyManager.java:293)
at
org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(StateChangeDispatcher.java:111)
at
org.apache.jackrabbit.core.state.SessionItemStateManager.stateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(SessionItemStateManager.java:889)
at
org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(StateChangeDispatcher.java:111)
at
org.apache.jackrabbit.core.state.LocalItemStateManager.stateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(LocalItemStateManager.java:452)
at
org.apache.jackrabbit.core.state.XAItemStateManager.stateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(XAItemStateManager.java:602)
at
org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(StateChangeDispatcher.java:111)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.stateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(SharedItemStateManager.java:400)
at
org.apache.jackrabbit.core.state.ItemState.notifyStateUpdated()V(ItemState.java:244)
at
org.apache.jackrabbit.core.state.ChangeLog.persisted()V(ChangeLog.java:297)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.doExternalUpdate(Lorg/apache/jackrabbit/core/state/ChangeLog;)V(SharedItemStateManager.java:1209)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.externalUpdate(Lorg/apache/jackrabbit/core/state/ChangeLog;Lorg/apache/jackrabbit/core/observation/EventStateCollection;)V(SharedItemStateManager.java:1132)
at
org.apache.jackrabbit.core.RepositoryImpl$WorkspaceInfo.externalUpdate(Lorg/apache/jackrabbit/core/state/ChangeLog;Ljava/util/List;JLjava/lang/String;)V(RepositoryImpl.java:2193)
at
org.apache.jackrabbit.core.cluster.ClusterNode.process(Lorg/apache/jackrabbit/core/cluster/ChangeLogRecord;)V(ClusterNode.java:869)
at
org.apache.jackrabbit.core.cluster.ChangeLogRecord.process(Lorg/apache/jackrabbit/core/cluster/ClusterRecordProcessor;)V(ChangeLogRecord.java:507)
at
org.apache.jackrabbit.core.cluster.ClusterNode.consume(Lorg/apache/jackrabbit/core/journal/Record;)V(ClusterNode.java:815)
at
org.apache.jackrabbit.core.journal.AbstractJournal.doSync(J)V(AbstractJournal.java:214)
at
org.apache.jackrabbit.core.journal.AbstractJournal.lockAndSync()V(AbstractJournal.java:258)
at
org.apache.jackrabbit.core.journal.DefaultRecordProducer.append()Lorg/apache/jackrabbit/core/journal/Record;(DefaultRecordProducer.java:51)
at
org.apache.jackrabbit.core.cluster.ClusterNode$WorkspaceUpdateChannel.updateCreated(Lorg/apache/jackrabbit/core/cluster/Update;)V(ClusterNode.java:602)
at
org.apache.jackrabbit.core.state.SharedItemStateManager$Update.begin()V(SharedItemStateManager.java:554)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.beginUpdate(Lorg/apache/jackrabbit/core/state/ChangeLog;Lorg/apache/jackrabbit/core/observation/EventStateCollectionFactory;Lorg/apache/jackrabbit/core/virtual/VirtualItemStateProvider;)Lorg/apache/jackrabbit/core/state/SharedItemStateManager$Update;(SharedItemStateManager.java:1085)
at
org.apache.jackrabbit.core.state.XAItemStateManager.prepare(Lorg/apache/jackrabbit/core/TransactionContext;)V(XAItemStateManager.java:163)
at
org.apache.jackrabbit.core.TransactionContext.prepare()V(TransactionContext.java:153)
- locked <0xfffffffe7e4b17a8> (a
org.apache.jackrabbit.core.TransactionContext)
at
org.apache.jackrabbit.core.XASessionImpl.prepare(Ljavax/transaction/xa/Xid;)I(XASessionImpl.java:331)
at
org.apache.jackrabbit.jca.TransactionBoundXAResource.prepare(Ljavax/transaction/xa/Xid;)I(TransactionBoundXAResource.java:68)
at mypackage.XAResourceWrapper.prepare()I(XAResourceWrapper.java:61)
repository.xml:
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or
more
contributor license agreements. See the NOTICE file distributed
with
this work for additional information regarding copyright
ownership.
The ASF licenses this file to You under the Apache License,
Version
2.0 (the "License"); you may not use this file except in
compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0 Unless required by
applicable law or agreed to in writing, software distributed
under the
License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
OR
CONDITIONS OF ANY KIND, either express or implied. See the
License for
the specific language governing permissions and limitations
under the
License.
-->
<!DOCTYPE Repository
PUBLIC "-//The Apache Software Foundation//DTD Jackrabbit 1.5//EN"
"http://jackrabbit.apache.org/dtd/repository-1.5.dtd";>
<!-- Example Repository Configuration File -->
<Repository>
<!--
virtual file system where the repository stores global state
(e.g.
registered namespaces, custom node types, etc.)
-->
<FileSystem
class="com.supplyon.cc.cm.custom.filesystem.OracleFileSystem">
<param name="driver" value="oracle.jdbc.driver.OracleDriver" />
<param name="url"
value="jdbc:oracle:thin:@EPMSD100.minsk.epam.com:1512:SLONINIT" />
<param name="user" value="TEST" />
<param name="password" value="abc123" />
<param name="schema" value="oracle" />
<param name="schemaObjectPrefix" value="SO_CM_REP_" />
</FileSystem>
<!--
security configuration
-->
<Security appName="Jackrabbit">
<!--
access manager: class: FQN of class implementing the
AccessManager
interface
-->
<AccessManager
class="org.apache.jackrabbit.core.security.simple.SimpleAccessManager">
<!-- <param name="config" value="${rep.home}/access.xml"/> -->
</AccessManager>
<LoginModule
class="org.apache.jackrabbit.core.security.SimpleLoginModule">
<!-- anonymous user name ('anonymous' is the default value) -->
<param name="anonymousId" value="anonymous" />
<!--
default user name to be used instead of the
anonymous user when no
login credentials are provided (unset by
default)
-->
<!-- <param name="defaultUserId" value="superuser"/> -->
</LoginModule>
</Security>
<!--
location of workspaces root directory and name of default
workspace
-->
<Workspaces rootPath="${rep.home}/workspaces"
defaultWorkspace="default" />
<!--
workspace configuration template: used to create the initial
workspace
if there's no workspace yet
-->
<Workspace name="${wsp.name}">
<!--
virtual file system of the workspace: class: FQN of
class
implementing the FileSystem interface <FileSystem
class="org.apache.jackrabbit.core.fs.local.LocalFileSystem"> <param
name="path" value="${rep.home}/repository" />
</FileSystem>
-->
<FileSystem
class="com.supplyon.cc.cm.custom.filesystem.OracleFileSystem">
<param name="driver"
value="oracle.jdbc.driver.OracleDriver" />
<param name="url"
value="jdbc:oracle:thin:@EPMSD100.minsk.epam.com:1512:SLONINIT" />
<param name="user" value="TEST" />
<param name="password" value="abc123" />
<param name="schema" value="oracle" />
<param name="schemaObjectPrefix"
value="SO_CM_${wsp.name}_" />
</FileSystem>
<!---->
<!--
persistence manager of the workspace: class: FQN of
class
implementing the PersistenceManager interface
org.apache.jackrabbit.core.persistence.bundle.OraclePersistenceManager
-->
<PersistenceManager
class="com.supplyon.cc.cm.custom.persistence.OracleBundlePersistenceManager">
<param name="driver"
value="oracle.jdbc.driver.OracleDriver" />
<param name="url"
value="jdbc:oracle:thin:@EPMSD100.minsk.epam.com:1512:SLONINIT" />
<param name="user" value="TEST" />
<param name="password" value="abc123" />
<param name="schema" value="oracle" />
<param name="schemaObjectPrefix"
value="SO_CM_${wsp.name}_" />
<!-- <param name="consistencyCheck" value="true"/> -->
<!-- <param name="consistencyFix" value="true"/> -->
<!-- <param name="externalBLOBs" value="false" /> -->
</PersistenceManager>
<!--
Search index and the file system it uses. class: FQN of
class
implementing the QueryHandler interface
<SearchIndex
class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
<param name="path" value="${wsp.home}/index" />
</SearchIndex>
-->
<ISMLocking
class="org.apache.jackrabbit.core.state.FineGrainedISMLocking"/>
</Workspace>
<!--
Configures the versioning
-->
<Versioning rootPath="${rep.home}/version">
<!--
Configures the filesystem to use for versioning for the
respective
persistence manager
-->
<FileSystem
class="com.supplyon.cc.cm.custom.filesystem.OracleFileSystem">
<param name="driver"
value="oracle.jdbc.driver.OracleDriver" />
<param name="url"
value="jdbc:oracle:thin:@EPMSD100.minsk.epam.com:1512:SLONINIT" />
<param name="user" value="TEST" />
<param name="password" value="abc123" />
<param name="schema" value="oracle" />
<param name="schemaObjectPrefix" value="SO_CM_VERSION_"
/>
</FileSystem>
<!--
Configures the persistence manager to be used for
persisting version
state. Please note that the current versioning
implementation is
based on a 'normal' persistence manager, but this could
change in
future implementations.
-->
<PersistenceManager
class="com.supplyon.cc.cm.custom.persistence.OracleBundlePersistenceManager">
<param name="driver"
value="oracle.jdbc.driver.OracleDriver" />
<param name="url"
value="jdbc:oracle:thin:@EPMSD100.minsk.epam.com:1512:SLONINIT" />
<param name="user" value="TEST" />
<param name="password" value="abc123" />
<param name="schema" value="oracle" />
<param name="schemaObjectPrefix" value="SO_CM_VERSION_"
/>
<!-- <param name="consistencyCheck" value="true"/> -->
<!-- <param name="consistencyFix" value="true"/> -->
<!-- <param name="externalBLOBs" value="false" /> -->
</PersistenceManager>
<ISMLocking
class="org.apache.jackrabbit.core.state.FineGrainedISMLocking"/>
</Versioning>
<!--
Search index for content that is shared repository wide
(/jcr:system
tree, contains mainly versions)
<SearchIndex
class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
<param name="path" value="${rep.home}/repository/index" />
</SearchIndex>
-->
<DataStore class="org.apache.jackrabbit.core.data.db.DbDataStore">
<param name="driver" value="oracle.jdbc.driver.OracleDriver" />
<param name="url"
value="jdbc:oracle:thin:@EPMSD100.minsk.epam.com:1512:SLONINIT" />
<param name="user" value="TEST" />
<param name="password" value="abc123" />
<param name="databaseType" value="oracle" />
<param name="maxConnections" value="50" />
<param name="tablePrefix" value="SO_CM_"/>
</DataStore>
<!-- Cluster configuration -->
<!--
<Cluster id="node1" syncDelay="2000">
<Journal
class="org.apache.jackrabbit.core.journal.OracleDatabaseJournal">
<param name="revision" value="${rep.home}/revision.log"
/>
<param name="driver"
value="oracle.jdbc.driver.OracleDriver" />
<param name="url"
value="jdbc:oracle:thin:@EPMSD100.minsk.epam.com:1521:SLONINIT" />
<param name="user" value="TEST" />
<param name="password" value="abc123"/>
<param name="schemaObjectPrefix" value="SO_CM_" />
</Journal>
</Cluster>
-->
</Repository>
<?xml version="1.0"?>
<Repository>
<FileSystem class="mypackage.OracleFileSystem">
<param name="driver" value="oracle.jdbc.driver.OracleDriver" />
<param name="url"
value="jdbc:oracle:thin:@loclhost" />
<param name="user" value="" />
<param name="password" value="" />
<param name="schema" value="oracle" />
<param name="schemaObjectPrefix" value="PREF_" />
</FileSystem>
<Security appName="Jackrabbit">
<AccessManager
class="org.apache.jackrabbit.core.security.simple.SimpleAccessManager">
</AccessManager>
<LoginModule
class="org.apache.jackrabbit.core.security.SimpleLoginModule">
<param name="anonymousId" value="anonymous" />
</LoginModule>
</Security>
<Workspaces rootPath="${rep.home}/workspaces"
defaultWorkspace="default" />
<Workspace name="${wsp.name}">
<FileSystem class="mypackage.OracleFileSystem">
<param name="driver"
value="oracle.jdbc.driver.OracleDriver" />
<param name="url"
value="jdbc:oracle:thin:@loclhost" />
<param name="user" value="" />
<param name="password" value="" />
<param name="schema" value="oracle" />
<param name="schemaObjectPrefix" value="PREF_" />
</FileSystem>
<PersistenceManager
class="mypackage.OracleBundlePersistenceManager">
<param name="driver"
value="oracle.jdbc.driver.OracleDriver" />
<param name="url"
value="jdbc:oracle:thin:@loclhost" />
<param name="user" value="" />
<param name="password" value="" />
<param name="schema" value="oracle" />
<param name="schemaObjectPrefix" value="P_${wsp.name}_"
/>
</PersistenceManager>
<ISMLocking
class="org.apache.jackrabbit.core.state.FineGrainedISMLocking"/>
</Workspace>
<Versioning rootPath="${rep.home}/version">
<FileSystem class="mypackage.OracleFileSystem">
<param name="driver"
value="oracle.jdbc.driver.OracleDriver" />
<param name="url"
value="jdbc:oracle:thin:@localhost" />
<param name="user" value="" />
<param name="password" value="" />
<param name="schema" value="oracle" />
<param name="schemaObjectPrefix" value="P_V_" />
</FileSystem>
<PersistenceManager
class="mypackage.OracleBundlePersistenceManager">
<param name="driver"
value="oracle.jdbc.driver.OracleDriver" />
<param name="url"
value="jdbc:oracle:thin:@localhost" />
<param name="user" value="" />
<param name="password" value="" />
<param name="schema" value="oracle" />
<param name="schemaObjectPrefix" value="P_V_" />
</PersistenceManager>
<ISMLocking
class="org.apache.jackrabbit.core.state.FineGrainedISMLocking"/>
</Versioning>
<DataStore class="org.apache.jackrabbit.core.data.db.DbDataStore">
<param name="driver" value="oracle.jdbc.driver.OracleDriver" />
<param name="url"
value="jdbc:oracle:thin:@localhost" />
<param name="user" value="TEST" />
<param name="password" value="abc123" />
<param name="databaseType" value="oracle" />
<param name="maxConnections" value="50" />
<param name="tablePrefix" value="SO_CM_"/>
</DataStore>
<Cluster id="node1" syncDelay="2000">
<Journal
class="org.apache.jackrabbit.core.journal.OracleDatabaseJournal">
<param name="revision" value="${rep.home}/revision.log"
/>
<param name="driver"
value="oracle.jdbc.driver.OracleDriver" />
<param name="url" value="jdbc:oracle:thin:@localhost" />
<param name="user" value="" />
<param name="password" value=""/>
<param name="schemaObjectPrefix" value="PREF_" />
</Journal>
</Cluster>
</Repository>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
