Hello Pierre, Thanks for the report! Don't you mind creating a JIRA ticket and proposing a PR to fix this? Otherwise, we would try to have a look at this soon (our time budget has been a bit limited recently on our side). Regards, Quan
On Sun, Nov 15, 2026 at 4:30 AM Pierre Grimaud <[email protected]> wrote: > Hello, > > After upgrading a single-node JPA server from 3.8.0 to 3.9.x, the full > reindex required by the Lucene 3 -> 9 migration (JAMES-4046) deadlocks > inside Derby after a few hundred messages. I couldn't find an existing JIRA > for it, so I'm reporting it here. > > Environment > ----------- > - Distribution: james-server-jpa-guice (zip), stock launch command > - Seen on 3.9.0. I then moved to 3.9.1, which ships the same Derby, > OpenJPA and Lucene jars; there I've only run the throttled workaround > below, so I haven't confirmed the deadlock on 3.9.1 itself. > - Embedded Derby 10.17.1.0 (database created by 3.8.0 / Derby 10.14) > - OpenJPA 4.1.1, lucene-core 10.2.1 > - OpenJDK 21.0.7 (Ubuntu 20.04) > - jvm.properties as in the 3.9 distribution, including > openjpa.Multithreaded=true > - About 3.6 GB in Derby, mailbox UIDs up to about 150,000 > > Steps to reproduce > ------------------ > 1. Start the server on an existing Derby store containing mail. > 2. curl -XPOST 'localhost:8000/mailboxes?task=reIndex' (defaults: > messagesPerSecond=50, mode=REBUILD_ALL) > > What happens > ------------ > After about 200 messages (189 succeeded, 12 failed), the task stops > progressing. GET /tasks/{id} reports "inProgress" indefinitely and the > counters stay the same. CPU and disk I/O drop to zero. The console then > shows this every minute: > > [ERROR] o.a.j.u.ReactorUtils - Error encountered while throttling > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 60000ms in 'onErrorResume' (and no fallback has been > configured) > > A jcmd Thread.print shows a JVM-detected deadlock between three > boundedElastic threads, entirely within Derby: > > "boundedElastic-4": waiting to lock EmbedConnection, held by > "boundedElastic-42" > "boundedElastic-42": waiting to lock StoredPage, held by > "boundedElastic-30" > "boundedElastic-30": waiting to lock BaseContainerHandle, held by > "boundedElastic-42" > > All three come from the same path, lazily loading the message content > while building the Lucene document: > > LuceneMessageSearchIndex.lambda$add$10 ( > LuceneMessageSearchIndex.java:755) > -> LuceneIndexableDocument.createMessageDocument (:119) > -> AbstractJPAMailboxMessage.getFullContent (:521) > -> JPAMailboxMessage.getBodyContent (:109) / pcGetbody > -> OpenJPA StateManagerImpl.loadField -> JDBCStoreManager.load > -> Derby EmbedConnection.prepareStatement / EmbedResultSet.getBytes > (OverflowInputStream, i.e. reading a LOB spanning overflow pages) > > At the time of the dump, 38 threads were inside > LuceneMessageSearchIndex.add. Threads 4 and 42 are blocked on the very same > EmbedConnection instance. So it looks like several threads lazily load > entities through one shared EntityManager / JDBC connection at the same > time. An embedded Derby connection doesn't seem to support concurrent use > while LOB overflow pages are being streamed, and it ends in a lock-ordering > deadlock between the page latch and the container handle observer. > > The deadlocked threads never recover. Mail delivery and IMAP keep working > partially, but a restart is needed. > > Workaround > ---------- > curl -XPOST 'localhost:8000/mailboxes?task=reIndex&messagesPerSecond=1' > progresses steadily with no deadlock. It is slow, though. > > Possible directions (only suggestions, I don't know the code well): > - don't share one EntityManager/connection across the concurrent indexing > operations (one per operation, or load the content eagerly before handing > the messages to the parallel stage); > - or cap the indexing concurrency to 1 for the JPA/Derby backend. > > The deadlock section of the thread dump is attached. I can provide the > full dump, or test a patch, if that helps. > > Thanks for James! > > Best regards, > Pierre Grimaud > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected]
