[jira] [Commented] (JCR-3226) stateCreated deadlock
[ https://issues.apache.org/jira/browse/JCR-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644006#comment-14644006 ] Thomas Mueller commented on JCR-3226: - http://svn.apache.org/r1693011 (trunk) stateCreated deadlock - Key: JCR-3226 URL: https://issues.apache.org/jira/browse/JCR-3226 Project: Jackrabbit Content Repository Issue Type: Bug Components: jackrabbit-core Affects Versions: 2.2, 2.4 Reporter: Jukka Zitting Assignee: Thomas Mueller Labels: deadlock Attachments: JCR-3226-test-2.patch, JCR-3226-test.patch In JCR-2650 a potential deadlock in SessionItemStateManager.stateModified() got fixed by postponing the work to the save() call. Unfortunately this still leaves the stateCreated() method vulnerable to a similar (though much less likely) deadlock scenario (observed in Jackrabbit 2.2.x): Thread A: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:118) - waiting to lock 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.ItemState.pull(ItemState.java:152) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.SessionItemStateManager.stateCreated(SessionItemStateManager.java:791) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.LocalItemStateManager.stateCreated(LocalItemStateManager.java:428) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:398) at org.apache.jackrabbit.core.state.ItemState.notifyStateCreated(ItemState.java:231) at org.apache.jackrabbit.core.state.ChangeLog.persisted(ChangeLog.java:309) at org.apache.jackrabbit.core.state.SharedItemStateManager$Update.end(SharedItemStateManager.java:777) at org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:1488) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:351) at org.apache.jackrabbit.core.state.XAItemStateManager.update(XAItemStateManager.java:354) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:326) at org.apache.jackrabbit.core.state.SessionItemStateManager.update(SessionItemStateManager.java:289) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:258) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) Thread B: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:119) - waiting to lock 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.NodeImpl.makePersistent(NodeImpl.java:869) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.ItemSaveOperation.persistTransientItems(ItemSaveOperation.java:836) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:243) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) at org.apache.jackrabbit.core.session.SessionSaveOperation.perform(SessionSaveOperation.java:42) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.SessionImpl.perform(SessionImpl.java:355) at org.apache.jackrabbit.core.SessionImpl.save(SessionImpl.java:758) IIUC this can only occur when two sessions are concurrently importing a node with the same UUID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCR-3226) stateCreated deadlock
[ https://issues.apache.org/jira/browse/JCR-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640387#comment-14640387 ] Dominique Pfister commented on JCR-3226: Looks good to me, Thomas, +1 stateCreated deadlock - Key: JCR-3226 URL: https://issues.apache.org/jira/browse/JCR-3226 Project: Jackrabbit Content Repository Issue Type: Bug Components: jackrabbit-core Affects Versions: 2.2, 2.4 Reporter: Jukka Zitting Assignee: Thomas Mueller Labels: deadlock Attachments: JCR-3226-test-2.patch, JCR-3226-test.patch In JCR-2650 a potential deadlock in SessionItemStateManager.stateModified() got fixed by postponing the work to the save() call. Unfortunately this still leaves the stateCreated() method vulnerable to a similar (though much less likely) deadlock scenario (observed in Jackrabbit 2.2.x): Thread A: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:118) - waiting to lock 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.ItemState.pull(ItemState.java:152) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.SessionItemStateManager.stateCreated(SessionItemStateManager.java:791) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.LocalItemStateManager.stateCreated(LocalItemStateManager.java:428) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:398) at org.apache.jackrabbit.core.state.ItemState.notifyStateCreated(ItemState.java:231) at org.apache.jackrabbit.core.state.ChangeLog.persisted(ChangeLog.java:309) at org.apache.jackrabbit.core.state.SharedItemStateManager$Update.end(SharedItemStateManager.java:777) at org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:1488) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:351) at org.apache.jackrabbit.core.state.XAItemStateManager.update(XAItemStateManager.java:354) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:326) at org.apache.jackrabbit.core.state.SessionItemStateManager.update(SessionItemStateManager.java:289) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:258) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) Thread B: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:119) - waiting to lock 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.NodeImpl.makePersistent(NodeImpl.java:869) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.ItemSaveOperation.persistTransientItems(ItemSaveOperation.java:836) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:243) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) at org.apache.jackrabbit.core.session.SessionSaveOperation.perform(SessionSaveOperation.java:42) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.SessionImpl.perform(SessionImpl.java:355) at org.apache.jackrabbit.core.SessionImpl.save(SessionImpl.java:758) IIUC this can only occur when two sessions are concurrently importing a node with the same UUID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCR-3226) stateCreated deadlock
[ https://issues.apache.org/jira/browse/JCR-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14635015#comment-14635015 ] Thomas Mueller commented on JCR-3226: - [~dpfister], [~stefan@jira], [~tripod], could you please review the patch above? stateCreated deadlock - Key: JCR-3226 URL: https://issues.apache.org/jira/browse/JCR-3226 Project: Jackrabbit Content Repository Issue Type: Bug Components: jackrabbit-core Affects Versions: 2.2, 2.4 Reporter: Jukka Zitting Assignee: Thomas Mueller Labels: deadlock Attachments: JCR-3226-test-2.patch, JCR-3226-test.patch In JCR-2650 a potential deadlock in SessionItemStateManager.stateModified() got fixed by postponing the work to the save() call. Unfortunately this still leaves the stateCreated() method vulnerable to a similar (though much less likely) deadlock scenario (observed in Jackrabbit 2.2.x): Thread A: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:118) - waiting to lock 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.ItemState.pull(ItemState.java:152) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.SessionItemStateManager.stateCreated(SessionItemStateManager.java:791) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.LocalItemStateManager.stateCreated(LocalItemStateManager.java:428) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:398) at org.apache.jackrabbit.core.state.ItemState.notifyStateCreated(ItemState.java:231) at org.apache.jackrabbit.core.state.ChangeLog.persisted(ChangeLog.java:309) at org.apache.jackrabbit.core.state.SharedItemStateManager$Update.end(SharedItemStateManager.java:777) at org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:1488) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:351) at org.apache.jackrabbit.core.state.XAItemStateManager.update(XAItemStateManager.java:354) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:326) at org.apache.jackrabbit.core.state.SessionItemStateManager.update(SessionItemStateManager.java:289) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:258) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) Thread B: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:119) - waiting to lock 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.NodeImpl.makePersistent(NodeImpl.java:869) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.ItemSaveOperation.persistTransientItems(ItemSaveOperation.java:836) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:243) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) at org.apache.jackrabbit.core.session.SessionSaveOperation.perform(SessionSaveOperation.java:42) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.SessionImpl.perform(SessionImpl.java:355) at org.apache.jackrabbit.core.SessionImpl.save(SessionImpl.java:758) IIUC this can only occur when two sessions are concurrently importing a node with the same UUID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCR-3226) stateCreated deadlock
[ https://issues.apache.org/jira/browse/JCR-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14634838#comment-14634838 ] Thomas Mueller commented on JCR-3226: - Potential patch to fix the problem. This requires a code review. {noformat} --- src/main/java/org/apache/jackrabbit/core/state/SessionItemStateManager.java (revision 1691398) +++ src/main/java/org/apache/jackrabbit/core/state/SessionItemStateManager.java (working copy) @@ -803,7 +803,19 @@ visibleState = transientState; } } -dispatcher.notifyStateCreated(visibleState); +boolean notifyTransientSpace; +if (visibleState instanceof NodeState) { +// No need to push node created to transient space: +// either the transient already knows about this state, or it doesn't. +// If we notify in this case, this can lead to a deadlock, +// see JCR-3226. +notifyTransientSpace = false; +} else { +notifyTransientSpace = true; +} +if (notifyTransientSpace) { +dispatcher.notifyStateCreated(visibleState); +} } /** {noformat} stateCreated deadlock - Key: JCR-3226 URL: https://issues.apache.org/jira/browse/JCR-3226 Project: Jackrabbit Content Repository Issue Type: Bug Components: jackrabbit-core Affects Versions: 2.2, 2.4 Reporter: Jukka Zitting Assignee: Thomas Mueller Labels: deadlock Attachments: JCR-3226-test-2.patch, JCR-3226-test.patch In JCR-2650 a potential deadlock in SessionItemStateManager.stateModified() got fixed by postponing the work to the save() call. Unfortunately this still leaves the stateCreated() method vulnerable to a similar (though much less likely) deadlock scenario (observed in Jackrabbit 2.2.x): Thread A: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:118) - waiting to lock 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.ItemState.pull(ItemState.java:152) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.SessionItemStateManager.stateCreated(SessionItemStateManager.java:791) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.LocalItemStateManager.stateCreated(LocalItemStateManager.java:428) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:398) at org.apache.jackrabbit.core.state.ItemState.notifyStateCreated(ItemState.java:231) at org.apache.jackrabbit.core.state.ChangeLog.persisted(ChangeLog.java:309) at org.apache.jackrabbit.core.state.SharedItemStateManager$Update.end(SharedItemStateManager.java:777) at org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:1488) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:351) at org.apache.jackrabbit.core.state.XAItemStateManager.update(XAItemStateManager.java:354) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:326) at org.apache.jackrabbit.core.state.SessionItemStateManager.update(SessionItemStateManager.java:289) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:258) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) Thread B: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:119) - waiting to lock 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.NodeImpl.makePersistent(NodeImpl.java:869) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.ItemSaveOperation.persistTransientItems(ItemSaveOperation.java:836) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:243) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at
[jira] [Commented] (JCR-3226) stateCreated deadlock
[ https://issues.apache.org/jira/browse/JCR-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14634831#comment-14634831 ] Thomas Mueller commented on JCR-3226: - Please note the repository.xml file in the patch needs to be copied to ./target/repository before running the test. This is needed to disable the cluster config (I couldn't reproduce the problem with the cluster enabled). stateCreated deadlock - Key: JCR-3226 URL: https://issues.apache.org/jira/browse/JCR-3226 Project: Jackrabbit Content Repository Issue Type: Bug Components: jackrabbit-core Affects Versions: 2.2, 2.4 Reporter: Jukka Zitting Assignee: Thomas Mueller Labels: deadlock Attachments: JCR-3226-test-2.patch, JCR-3226-test.patch In JCR-2650 a potential deadlock in SessionItemStateManager.stateModified() got fixed by postponing the work to the save() call. Unfortunately this still leaves the stateCreated() method vulnerable to a similar (though much less likely) deadlock scenario (observed in Jackrabbit 2.2.x): Thread A: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:118) - waiting to lock 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.ItemState.pull(ItemState.java:152) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.SessionItemStateManager.stateCreated(SessionItemStateManager.java:791) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.LocalItemStateManager.stateCreated(LocalItemStateManager.java:428) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:398) at org.apache.jackrabbit.core.state.ItemState.notifyStateCreated(ItemState.java:231) at org.apache.jackrabbit.core.state.ChangeLog.persisted(ChangeLog.java:309) at org.apache.jackrabbit.core.state.SharedItemStateManager$Update.end(SharedItemStateManager.java:777) at org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:1488) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:351) at org.apache.jackrabbit.core.state.XAItemStateManager.update(XAItemStateManager.java:354) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:326) at org.apache.jackrabbit.core.state.SessionItemStateManager.update(SessionItemStateManager.java:289) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:258) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) Thread B: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:119) - waiting to lock 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.NodeImpl.makePersistent(NodeImpl.java:869) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.ItemSaveOperation.persistTransientItems(ItemSaveOperation.java:836) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:243) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) at org.apache.jackrabbit.core.session.SessionSaveOperation.perform(SessionSaveOperation.java:42) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.SessionImpl.perform(SessionImpl.java:355) at org.apache.jackrabbit.core.SessionImpl.save(SessionImpl.java:758) IIUC this can only occur when two sessions are concurrently importing a node with the same UUID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCR-3226) stateCreated deadlock
[ https://issues.apache.org/jira/browse/JCR-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628128#comment-14628128 ] Thomas Mueller commented on JCR-3226: - I have a much simpler test case now. 3 threads are needed (even thought one thread doesn't do much - it is blocked when trying to save the created node), and node type nt:unstructured (can not reproduce with nt:base). Each thread does this just once, concurrently: {noformat} if (node.hasNode(test)) { Node t = node.getNode(test); t.setProperty(x, 1); s.save(); } else { // can not reproduce with nt:base // node.addNode(test, nt:base); node.addNode(test, nt:unstructured); s.save(); } {noformat} stateCreated deadlock - Key: JCR-3226 URL: https://issues.apache.org/jira/browse/JCR-3226 Project: Jackrabbit Content Repository Issue Type: Bug Components: jackrabbit-core Affects Versions: 2.2, 2.4 Reporter: Jukka Zitting Assignee: Jukka Zitting Labels: deadlock Attachments: JCR-3226-test.patch In JCR-2650 a potential deadlock in SessionItemStateManager.stateModified() got fixed by postponing the work to the save() call. Unfortunately this still leaves the stateCreated() method vulnerable to a similar (though much less likely) deadlock scenario (observed in Jackrabbit 2.2.x): Thread A: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:118) - waiting to lock 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.ItemState.pull(ItemState.java:152) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.SessionItemStateManager.stateCreated(SessionItemStateManager.java:791) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.LocalItemStateManager.stateCreated(LocalItemStateManager.java:428) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:398) at org.apache.jackrabbit.core.state.ItemState.notifyStateCreated(ItemState.java:231) at org.apache.jackrabbit.core.state.ChangeLog.persisted(ChangeLog.java:309) at org.apache.jackrabbit.core.state.SharedItemStateManager$Update.end(SharedItemStateManager.java:777) at org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:1488) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:351) at org.apache.jackrabbit.core.state.XAItemStateManager.update(XAItemStateManager.java:354) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:326) at org.apache.jackrabbit.core.state.SessionItemStateManager.update(SessionItemStateManager.java:289) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:258) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) Thread B: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:119) - waiting to lock 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.NodeImpl.makePersistent(NodeImpl.java:869) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.ItemSaveOperation.persistTransientItems(ItemSaveOperation.java:836) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:243) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) at org.apache.jackrabbit.core.session.SessionSaveOperation.perform(SessionSaveOperation.java:42) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.SessionImpl.perform(SessionImpl.java:355) at org.apache.jackrabbit.core.SessionImpl.save(SessionImpl.java:758) IIUC this can only occur when two
[jira] [Commented] (JCR-3226) stateCreated deadlock
[ https://issues.apache.org/jira/browse/JCR-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622347#comment-14622347 ] Thomas Mueller commented on JCR-3226: - So the output of the test is for example: i: 0 i: 1 ... i: 40 i: 41 and then it stops there and doesn't continue. If it doesn't continue for a few seconds, it was always a deadlock, and I think always the same type of deadlock. jstack -l pid, or pausing in Eclipse will show that. Some observation below. I don't currently understand the algorithm that is used, how this is supposed to work, so I can't say if some state is a localState (or transient) while it should not longer be one, or before it should be one, or if there is a problem with the overall algorithm. {noformat} NodeState #241: isTransient = false overlayedState = #247 status = 1 propertyNames = primaryType NodeState #242: isTransient = true listener = SessionItemStateManager #260 overlayedState = #241 status = 2 propertyNames = primaryType, x Java stack information for the threads listed above: === Thread-179: * the sessions SessionItemStateManager is #260 * locked the transient.overlayedState, which is called localState here, and waits to lock the transient at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:153) - waiting to lock 7f42b1b10 (a org.apache.jackrabbit.core.state.NodeState) #242 - locked 7f42b (a org.apache.jackrabbit.core.state.NodeState) #241 at org.apache.jackrabbit.core.state.SessionItemStateManager.makePersistent(SessionItemStateManager.java:984) - locked 7f42b (a org.apache.jackrabbit.core.state.NodeState) #241 at org.apache.jackrabbit.core.NodeImpl.makePersistent(NodeImpl.java:867) at org.apache.jackrabbit.core.ItemSaveOperation.persistTransientItems(ItemSaveOperation.java:849) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:243) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:216) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) at org.apache.jackrabbit.core.session.SessionSaveOperation.perform(SessionSaveOperation.java:65) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:216) at org.apache.jackrabbit.core.SessionImpl.perform(SessionImpl.java:361) at org.apache.jackrabbit.core.SessionImpl.save(SessionImpl.java:812) at org.apache.jackrabbit.core.deadlock.RandomChange.randomOp(RandomChange.java:96) at org.apache.jackrabbit.core.deadlock.RandomChange$1.run(RandomChange.java:58) Thread-180: * the sessions SessionItemStateManager is #283 * locked the transient of the other session, and waits to lock the transient.overlayedState at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:153) - waiting to lock 7f42b (a org.apache.jackrabbit.core.state.NodeState) #241 - locked 7f42b1b10 (a org.apache.jackrabbit.core.state.NodeState) #242 at org.apache.jackrabbit.core.state.ItemState.pull(ItemState.java:152) - locked 7f42b1b10 (a org.apache.jackrabbit.core.state.NodeState) #242 at org.apache.jackrabbit.core.state.SessionItemStateManager.stateCreated(SessionItemStateManager.java:788) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.LocalItemStateManager.stateCreated(LocalItemStateManager.java:487) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:407) at org.apache.jackrabbit.core.state.ItemState.notifyStateCreated(ItemState.java:233) at org.apache.jackrabbit.core.state.ChangeLog.persisted(ChangeLog.java:323) at org.apache.jackrabbit.core.state.SharedItemStateManager$Update.end(SharedItemStateManager.java:831) at org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:1541) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:400) at org.apache.jackrabbit.core.state.XAItemStateManager.update(XAItemStateManager.java:354) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:375) at org.apache.jackrabbit.core.state.SessionItemStateManager.update(SessionItemStateManager.java:275) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:258) at
[jira] [Commented] (JCR-3226) stateCreated deadlock
[ https://issues.apache.org/jira/browse/JCR-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622383#comment-14622383 ] Thomas Mueller commented on JCR-3226: - I'm not an expert in this area, maybe [~stefan@jira], [~dpfister], or [~tripod] can help here? stateCreated deadlock - Key: JCR-3226 URL: https://issues.apache.org/jira/browse/JCR-3226 Project: Jackrabbit Content Repository Issue Type: Bug Components: jackrabbit-core Affects Versions: 2.2, 2.4 Reporter: Jukka Zitting Assignee: Jukka Zitting Labels: deadlock Attachments: JCR-3226-test.patch In JCR-2650 a potential deadlock in SessionItemStateManager.stateModified() got fixed by postponing the work to the save() call. Unfortunately this still leaves the stateCreated() method vulnerable to a similar (though much less likely) deadlock scenario (observed in Jackrabbit 2.2.x): Thread A: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:118) - waiting to lock 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.ItemState.pull(ItemState.java:152) - locked 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.state.SessionItemStateManager.stateCreated(SessionItemStateManager.java:791) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.LocalItemStateManager.stateCreated(LocalItemStateManager.java:428) at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:94) at org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:398) at org.apache.jackrabbit.core.state.ItemState.notifyStateCreated(ItemState.java:231) at org.apache.jackrabbit.core.state.ChangeLog.persisted(ChangeLog.java:309) at org.apache.jackrabbit.core.state.SharedItemStateManager$Update.end(SharedItemStateManager.java:777) at org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:1488) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:351) at org.apache.jackrabbit.core.state.XAItemStateManager.update(XAItemStateManager.java:354) at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:326) at org.apache.jackrabbit.core.state.SessionItemStateManager.update(SessionItemStateManager.java:289) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:258) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) Thread B: at org.apache.jackrabbit.core.state.NodeState.copy(NodeState.java:119) - waiting to lock 0x7ffec8ddf9d8 (a org.apache.jackrabbit.core.state.NodeState) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.NodeImpl.makePersistent(NodeImpl.java:869) - locked 0x7ffec8ddfa18 (a org.apache.jackrabbit.core.state.NodeState) at org.apache.jackrabbit.core.ItemSaveOperation.persistTransientItems(ItemSaveOperation.java:836) at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:243) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329) at org.apache.jackrabbit.core.session.SessionSaveOperation.perform(SessionSaveOperation.java:42) at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:200) at org.apache.jackrabbit.core.SessionImpl.perform(SessionImpl.java:355) at org.apache.jackrabbit.core.SessionImpl.save(SessionImpl.java:758) IIUC this can only occur when two sessions are concurrently importing a node with the same UUID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)