Hi,
we just ran into a deadlock in OpenJPA. We do use `openjpa.Multithreaded=true` (as per examples from
James deployments). Threaddump below (full:
https://gist.github.com/woj-tek/f4571031f57e4e70de2bda56c460f61d).
Andy ideas? Missing blocking in reactor somewhere maybe (very wild guess)?
```
Found one Java-level deadlock:
=============================
"parallel-1":
waiting for ownable synchronizer 0x00000000ed900438, (a
java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "imapserver-ssl-io-1"
"imapserver-ssl-io-1":
waiting for ownable synchronizer 0x00000000ed980198, (a
java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "parallel-1"
Java stack information for the threads listed above:
===================================================
"parallel-1":
at jdk.internal.misc.Unsafe.park(java.base@21.0.3/Native Method)
- parking to wait for <0x00000000ed900438> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@21.0.3/Unknown
Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.3/Unknown
Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.3/Unknown
Source)
at
java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@21.0.3/Unknown
Source)
at
java.util.concurrent.locks.ReentrantLock.lock(java.base@21.0.3/Unknown Source)
at org.apache.openjpa.kernel.BrokerImpl.lock(BrokerImpl.java:4655)
at
org.apache.openjpa.jdbc.kernel.JDBCStoreManager$RefCountConnection.close(JDBCStoreManager.java:1653)
at
org.apache.openjpa.jdbc.sql.ResultSetResult.close(ResultSetResult.java:205)
at
org.apache.openjpa.jdbc.kernel.JDBCStoreManager.load(JDBCStoreManager.java:688)
at
org.apache.openjpa.kernel.DelegatingStoreManager.load(DelegatingStoreManager.java:129)
at
org.apache.openjpa.kernel.ROPStoreManager.load(ROPStoreManager.java:82)
at
org.apache.openjpa.kernel.StateManagerImpl.loadFields(StateManagerImpl.java:3278)
at
org.apache.openjpa.kernel.StateManagerImpl.loadField(StateManagerImpl.java:3358)
at
org.apache.openjpa.kernel.StateManagerImpl.beforeAccessField(StateManagerImpl.java:1747)
at
org.apache.openjpa.kernel.StateManagerImpl.accessingField(StateManagerImpl.java:1678)
at
org.apache.james.mailbox.jpa.mail.model.openjpa.JPAMailboxMessage.pcGetbody(JPAMailboxMessage.java)
at
org.apache.james.mailbox.jpa.mail.model.openjpa.JPAMailboxMessage.getBodyContent(JPAMailboxMessage.java:109)
at
org.apache.james.mailbox.jpa.mail.model.openjpa.AbstractJPAMailboxMessage.getFullContent(AbstractJPAMailboxMessage.java:520)
at
org.apache.james.mailbox.store.streaming.InputStreamContent.getInputStream(InputStreamContent.java:64)
at
org.apache.james.imap.processor.fetch.ContentBodyElement.getInputStream(ContentBodyElement.java:63)
at
org.apache.james.imap.processor.fetch.PartialFetchBodyElement.getInputStream(PartialFetchBodyElement.java:65)
at
org.apache.james.imapserver.netty.ChannelImapResponseWriter.write(ChannelImapResponseWriter.java:84)
at
org.apache.james.imap.encode.base.ImapResponseComposerImpl.literal(ImapResponseComposerImpl.java:320)
at
org.apache.james.imap.encode.FetchResponseEncoder.encodeBodyElements(FetchResponseEncoder.java:281)
at
org.apache.james.imap.encode.FetchResponseEncoder.encode(FetchResponseEncoder.java:96)
at
org.apache.james.imap.encode.FetchResponseEncoder.encode(FetchResponseEncoder.java:51)
at
org.apache.james.imap.encode.main.DefaultImapEncoderFactory$DefaultImapEncoder.encode(DefaultImapEncoderFactory.java:83)
at
org.apache.james.imap.main.ResponseEncoder.respond(ResponseEncoder.java:43)
at
org.apache.james.imap.processor.fetch.FetchProcessor$FetchSubscriber.onNext(FetchProcessor.java:100)
at
org.apache.james.imap.processor.fetch.FetchProcessor$FetchSubscriber.onNext(FetchProcessor.java:80)
at
reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89)
at
reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:258)
- locked <0x00000000ed900798> (a
reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber)
at
reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:863)
at
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:293)
at
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:188)
at
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:237)
at
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:204)
at
reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:89)
at
reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onComplete(FluxPeekFuseable.java:277)
at
reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2573)
at
reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:144)
at
reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onSubscribe(MonoIgnoreElements.java:72)
at
reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:178)
at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
at reactor.core.publisher.Mono.subscribe(Mono.java:4512)
at
reactor.core.publisher.MonoIgnorePublisher.subscribe(MonoIgnorePublisher.java:57)
at
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:264)
at
reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
at
reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onNext(FluxConcatMapNoPrefetch.java:206)
at
reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:113)
at
reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:258)
- locked <0x00000000ed900880> (a
reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber)
at
reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:863)
at
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
at
reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:440)
at
reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:527)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
at java.util.concurrent.FutureTask.run(java.base@21.0.3/Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@21.0.3/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.3/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.3/Unknown
Source)
at java.lang.Thread.runWith(java.base@21.0.3/Unknown Source)
at java.lang.Thread.run(java.base@21.0.3/Unknown Source)
"imapserver-ssl-io-1":
at jdk.internal.misc.Unsafe.park(java.base@21.0.3/Native Method)
- parking to wait for <0x00000000ed980198> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@21.0.3/Unknown
Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.3/Unknown
Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.3/Unknown
Source)
at
java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@21.0.3/Unknown
Source)
at
java.util.concurrent.locks.ReentrantLock.lock(java.base@21.0.3/Unknown Source)
at
org.apache.openjpa.kernel.StateManagerImpl.lock(StateManagerImpl.java:3553)
at
org.apache.openjpa.kernel.StateManagerImpl.assignObjectId(StateManagerImpl.java:597)
at
org.apache.openjpa.kernel.StateManagerImpl.fetchObjectId(StateManagerImpl.java:1617)
at
org.apache.openjpa.kernel.DetachedStateManager.<init>(DetachedStateManager.java:87)
at
org.apache.openjpa.kernel.DetachManager.detachInternal(DetachManager.java:469)
at
org.apache.openjpa.kernel.DetachManager.detachAll(DetachManager.java:333)
at
org.apache.openjpa.kernel.BrokerImpl.detachAllInternal(BrokerImpl.java:3608)
at org.apache.openjpa.kernel.BrokerImpl.free(BrokerImpl.java:4579)
at org.apache.openjpa.kernel.BrokerImpl.close(BrokerImpl.java:4566)
at
org.apache.openjpa.kernel.DelegatingBroker.close(DelegatingBroker.java:1508)
at
org.apache.openjpa.persistence.EntityManagerImpl.close(EntityManagerImpl.java:1598)
at
org.apache.james.backends.jpa.EntityManagerUtils.safelyClose(EntityManagerUtils.java:40)
at
org.apache.james.mailbox.jpa.JPATransactionalMapper.endRequest(JPATransactionalMapper.java:91)
at
org.apache.james.mailbox.store.MailboxSessionMapperFactory.endProcessingRequest(MailboxSessionMapperFactory.java:143)
at
org.apache.james.mailbox.store.StoreMailboxManager.endProcessingRequest(StoreMailboxManager.java:915)
at
org.apache.james.mailbox.MailboxManager.lambda$manageProcessing$10(MailboxManager.java:436)
at
org.apache.james.mailbox.MailboxManager$$Lambda/0x00007f4b3ca926e8.run(Unknown
Source)
at
org.apache.james.mailbox.MailboxManager$$Lambda/0x00007f4b3c408000.accept(Unknown
Source)
at
reactor.core.publisher.MonoUsing$MonoUsingSubscriber.cleanup(MonoUsing.java:191)
at
reactor.core.publisher.MonoUsing$MonoUsingSubscriber.cancel(MonoUsing.java:185)
at
reactor.core.publisher.MonoUsing$MonoUsingSubscriber.cancel(MonoUsing.java:183)
at
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.cancel(MonoIgnoreThen.java:144)
at
reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
at
reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.cancel(FluxDoOnEach.java:113)
at
reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
at
reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.cancel(FluxDoFinally.java:134)
at
reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.cancel(MonoPeekTerminal.java:144)
at
reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.cancel(FluxDoFinally.java:134)
at
reactor.core.publisher.LambdaMonoSubscriber.dispose(LambdaMonoSubscriber.java:217)
at
org.apache.james.imapserver.netty.NettyImapSession$$Lambda/0x00007f4b3ca41b90.accept(Unknown Source)
at java.util.Optional.ifPresent(java.base@21.0.3/Unknown Source)
at
org.apache.james.imapserver.netty.NettyImapSession.cancelOngoingProcessing(NettyImapSession.java:123)
at
org.apache.james.imapserver.netty.ImapChannelUpstreamHandler$$Lambda/0x00007f4b3ca41968.accept(Unknown
Source)
at java.util.Optional.ifPresent(java.base@21.0.3/Unknown Source)
at
org.apache.james.imapserver.netty.ImapChannelUpstreamHandler.channelInactive(ImapChannelUpstreamHandler.java:275)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at
io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:411)
at
io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:376)
at
org.apache.james.imapserver.netty.ImapRequestFrameDecoder.channelInactive(ImapRequestFrameDecoder.java:101)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at
io.netty.handler.stream.ChunkedWriteHandler.channelInactive(ChunkedWriteHandler.java:137)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at
io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:411)
at
io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:376)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
at
io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at
io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:411)
at
io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:376)
at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1174)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at
io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
at
io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)
at
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.runWith(java.base@21.0.3/Unknown Source)
at java.lang.Thread.run(java.base@21.0.3/Unknown Source)
```
--
Wojtek
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org
