Hello Pierre,
Thanks for the report!
Don't you mind creating a JIRA ticket and proposing a PR to fix this?
Otherwise, we would try to have a look at this soon (our time budget has
been a bit limited recently on our side).
Regards,
Quan

On Sun, Nov 15, 2026 at 4:30 AM Pierre Grimaud <[email protected]> wrote:

> Hello,
>
> After upgrading a single-node JPA server from 3.8.0 to 3.9.x, the full
> reindex required by the Lucene 3 -> 9 migration (JAMES-4046) deadlocks
> inside Derby after a few hundred messages. I couldn't find an existing JIRA
> for it, so I'm reporting it here.
>
> Environment
> -----------
> - Distribution: james-server-jpa-guice (zip), stock launch command
> - Seen on 3.9.0. I then moved to 3.9.1, which ships the same Derby,
> OpenJPA and Lucene jars; there I've only run the throttled workaround
> below, so I haven't confirmed the deadlock on 3.9.1 itself.
> - Embedded Derby 10.17.1.0 (database created by 3.8.0 / Derby 10.14)
> - OpenJPA 4.1.1, lucene-core 10.2.1
> - OpenJDK 21.0.7 (Ubuntu 20.04)
> - jvm.properties as in the 3.9 distribution, including
> openjpa.Multithreaded=true
> - About 3.6 GB in Derby, mailbox UIDs up to about 150,000
>
> Steps to reproduce
> ------------------
> 1. Start the server on an existing Derby store containing mail.
> 2. curl -XPOST 'localhost:8000/mailboxes?task=reIndex' (defaults:
> messagesPerSecond=50, mode=REBUILD_ALL)
>
> What happens
> ------------
> After about 200 messages (189 succeeded, 12 failed), the task stops
> progressing. GET /tasks/{id} reports "inProgress" indefinitely and the
> counters stay the same. CPU and disk I/O drop to zero. The console then
> shows this every minute:
>
>   [ERROR] o.a.j.u.ReactorUtils - Error encountered while throttling
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 60000ms in 'onErrorResume' (and no fallback has been
> configured)
>
> A jcmd Thread.print shows a JVM-detected deadlock between three
> boundedElastic threads, entirely within Derby:
>
>   "boundedElastic-4":  waiting to lock EmbedConnection, held by
> "boundedElastic-42"
>   "boundedElastic-42": waiting to lock StoredPage, held by
> "boundedElastic-30"
>   "boundedElastic-30": waiting to lock BaseContainerHandle, held by
> "boundedElastic-42"
>
> All three come from the same path, lazily loading the message content
> while building the Lucene document:
>
>   LuceneMessageSearchIndex.lambda$add$10 (
> LuceneMessageSearchIndex.java:755)
>   -> LuceneIndexableDocument.createMessageDocument (:119)
>   -> AbstractJPAMailboxMessage.getFullContent (:521)
>   -> JPAMailboxMessage.getBodyContent (:109) / pcGetbody
>   -> OpenJPA StateManagerImpl.loadField -> JDBCStoreManager.load
>   -> Derby EmbedConnection.prepareStatement / EmbedResultSet.getBytes
>      (OverflowInputStream, i.e. reading a LOB spanning overflow pages)
>
> At the time of the dump, 38 threads were inside
> LuceneMessageSearchIndex.add. Threads 4 and 42 are blocked on the very same
> EmbedConnection instance. So it looks like several threads lazily load
> entities through one shared EntityManager / JDBC connection at the same
> time. An embedded Derby connection doesn't seem to support concurrent use
> while LOB overflow pages are being streamed, and it ends in a lock-ordering
> deadlock between the page latch and the container handle observer.
>
> The deadlocked threads never recover. Mail delivery and IMAP keep working
> partially, but a restart is needed.
>
> Workaround
> ----------
> curl -XPOST 'localhost:8000/mailboxes?task=reIndex&messagesPerSecond=1'
> progresses steadily with no deadlock. It is slow, though.
>
> Possible directions (only suggestions, I don't know the code well):
> - don't share one EntityManager/connection across the concurrent indexing
> operations (one per operation, or load the content eagerly before handing
> the messages to the parallel stage);
> - or cap the indexing concurrency to 1 for the JPA/Derby backend.
>
> The deadlock section of the thread dump is attached. I can provide the
> full dump, or test a patch, if that helps.
>
> Thanks for James!
>
> Best regards,
> Pierre Grimaud
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]

Reply via email to