I see. So the issue is that we shall buffer the page of result from PG in James memory and replay it so that we are sure to release the underlying connection if I get it right.
I think I found something which collectList() withholds upstream requests until downstream demand exists, so expand's page N+1 subscription arrives with zero demand and just idles on its connection. I'll open a draft PR: https://github.com/apache/james-project/pull/3215 I'm happy if you can test it and see if it fixes the issue. Sorry for missing this - I thought the existing test suite covered this. -- Best regards, Benoit TELLIER General manager of Linagora VIETNAM. Product owner for Twake-Mail product. Chairman of the Apache James project. Mail: [email protected] Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) Le sept. 28, 2026 9:53 AM, de Felix Auringer <[email protected]>Hey Benoit, thanks for the very fast response! I only managed to test your changes today and they definitely improved the issue. However, I can still reproduce it in a similar way. When using opensearch as the search backend and triggering a full reindex with curl -XPOST 'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=500', the reindexing fails for some folders: 2026-09-28T07:22:25.769Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2336 2026-09-28T07:22:25.778Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2339 2026-09-28T07:22:25.766Z [1;31m[ERROR][0;39m o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres query. May need to check either jOOQ reactive issue or Postgres DB performance. java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in 'flatMapMany' (and no fallback has been configured) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281) at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420) at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162) at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270) at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) 2026-09-28T07:22:25.780Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2346 2026-09-28T07:22:25.783Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2342 2026-09-28T07:22:25.784Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2345 2026-09-28T07:22:25.786Z [31m[WARN ][0;39m o.a.m.tools.indexer.ReIndexerPerformer - ReIndexing failed for #private:<user>:Sent java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in 'flatMapMany' (and no fallback has been configured) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281) at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420) at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162) at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270) at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) 2026-09-28T07:22:25.789Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2343 2026-09-28T07:22:25.792Z [34m[INFO ][0;39m o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox Sent-<uuid> of user re-indexing on message 2341 In total, the reindexing now processed a few dozen thousand emails and terminated in a short time. The folders the process fails for all have more than 5000 emails, so I guess James currently opens one streaming connection per mailbox it indexes and then fails after processing 500 (messages per seconds) * 10 (seconds timeout) ≈ 5000 emails. Best regards, Felix On 25/09/2026 15.17, Benoit TELLIER via server-dev wrote: > Thanks for the look @Quan but I'd rather address the root cause: the listing > should not hold a permanent connection. It should use one every time it > fetches one page. > > I did do an alternative proposition here: > github.com/apache/james-project/pull/3209 > > -- > > > Best regards, > > Benoit TELLIER > > General manager of Linagora VIETNAM. > Product owner for Twake-Mail product. > Chairman of the Apache James project. > > Mail: [email protected] > Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) > > > > On Sep 25, 2026 3:02 PM, from Quan Tran Hong <[email protected]>Hi > Benoit, > > Food for thought: github.com/apache/james-project/pull/3208 > > I did a quick review of it, and the fixes make sense to me (I asked the > agent to do TDD: reproduce the reported issue first). > > I hope it helps. I will properly double check the fixes later. > > Quan > > On Fri, Sep 25, 2026 at 7:42 PM Benoit TELLIER via server-dev < > [email protected]> wrote: > >> Hello Felix >> >> That's a classic Jooq / R2DBC issue: it badly support streaming as it hold >> a live connection. >> >> The reindexing read live data as it needs it (so 2 message per second for >> instance) and keep it opens as it iterates/ >> >> The listing query then times out after 10s, thus the symptoms. >> >> We encountered that in over parts of the PG implem and used buffering ( >> intermediate .collectList ) to pull results in memory, and return the >> driver thread as fast as we could. >> >> The tricky point is that EG Cassandra driver supports reading large result >> sets without monopolizing driver resources. It thus don't need that memory >> buffering. Any fix doing this would then need not to impact Cassandra >> implementation that could be doing the optimal list-as-it-comes operation. >> >> Another solution is to rethink "streaming" and switch it to explicit >> separated request per page size. This lift the time limit as we no longer >> hold a collection at the cost of repeated queries. A sample can be found >> here >> github.com/apache/james-project/blob/bf100353ac3c4fc76d9ecced7472528b43e1ca21/mailbox/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/dao/PostgresMailboxMessageDAO.java#L487 >> >> A generic adoption of this pattern can solve the problem you mention (at >> the cost of snapshot isolation which I believe we do not really care. >> I'll try to propose a little something in that direction. >> >> -- >> >> >> Best regards, >> >> Benoit TELLIER >> >> General manager of Linagora VIETNAM. >> Product owner for Twake-Mail product. >> Chairman of the Apache James project. >> >> Mail: [email protected] >> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) >> >> >> >> On Sep 25, 2026 12:51 PM, from Felix Auringer <[email protected] >>> Hello, >> >> For some time now, I have seen a timeout error in the postgres executor: >> o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres >> query. May need to check either jOOQ reactive issue or Postgres DB >> performance. >> This was mainly happening in TMail and was reported here: >> github.com/linagora/tmail-backend/issues/1599 >> >> I have now found a way to reproduce this reliably with the Docker image >> built from the current James master >> (bf100353ac3c4fc76d9ecced7472528b43e1ca21). >> When using opensearch as the search backend and triggering a full reindex >> with curl -XPOST >> 'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=1', >> I observe the following behavior: >> >> 2026-09-25T10:29:41.968Z [INFO ] o.a.j.webadmin.mdc.LoggingRequestFilter - >> WebAdmin request received >> 2026-09-25T10:29:41.974Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer - >> Starting a full reindex >> 2026-09-25T10:29:41.985Z [INFO ] o.a.j.webadmin.mdc.LoggingResponseFilter >> - WebAdmin response received >> 2026-09-25T10:29:43.178Z [INFO ] >> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox >> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 1 >> 2026-09-25T10:29:44.120Z [INFO ] >> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox >> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 2 >> 2026-09-25T10:29:45.125Z [INFO ] >> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox >> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 3 >> 2026-09-25T10:29:46.128Z [INFO ] >> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox >> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 4 >> 2026-09-25T10:29:47.131Z [INFO ] >> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox >> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 5 >> 2026-09-25T10:29:48.135Z [INFO ] >> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox >> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 6 >> 2026-09-25T10:29:49.137Z [INFO ] >> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox >> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 7 >> 2026-09-25T10:29:50.147Z [INFO ] >> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox >> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 8 >> 2026-09-25T10:29:51.143Z [INFO ] >> o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox >> email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 9 >> 2026-09-25T10:29:52.052Z [ERROR] o.a.j.b.postgres.utils.PostgresExecutor - >> Time out executing Postgres query. May need to check either jOOQ reactive >> issue or Postgres DB performance. >> java.util.concurrent.TimeoutException: Did not observe any item or >> terminal signal within 10000ms in 'flatMapMany' (and no fallback has been >> configured) >> at >> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296) >> at >> reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281) >> at >> reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420) >> at >> reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162) >> at >> reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270) >> at >> reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285) >> at >> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) >> at >> reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) >> at java.base/java.util.concurrent.FutureTask.run(Unknown Source) >> at >> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown >> Source) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >> at java.base/java.lang.Thread.run(Unknown Source) >> 2026-09-25T10:29:52.058Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer - >> Full reindex finished >> 2026-09-25T10:29:52.059Z [ERROR] o.a.james.task.SerialTaskManagerWorker - >> Task was partially performed. Check logs for more details. Taskid : >> TaskId{value=9553e1df-9ad0-42d6-b90d-30c5882b8d17} >> >> This error always happens 10 seconds after starting the reindex task. >> Therefore, exactly 9 batches of the size specified with messagesPerSecond >> are processed. >> When indexing 2 messages per second, the indexer manages to index the >> first 18 messages, when indexing 5 messages per second, the first 45 >> messages are indexed, and so on... >> >> I don't know much about this part of the codebase and would be happy about >> help to debug this. >> Did somebody else ever see this error or can somebody else reproduce it? >> >> Best regards, >> Felix >> --- >> Gesellschaft für interkulturelles >> Zusammenleben gGmbH (GIZ) >> Felix Auringer >> IT >> Reformationsplatz 2 >> 13597 Berlin >> >> Tel: 030/513 0100 00; Fax: 030/513 0100 09 >> giz.berlin; [email protected] >> >> Amtsgericht Charlottenburg HRB 200872 B >> Geschäftsführerin: Dr. Britta Marschke >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > --- Gesellschaft für interkulturelles Zusammenleben gGmbH (GIZ) Felix Auringer IT Reformationsplatz 2 13597 Berlin Tel: 030/513 0100 00; Fax: 030/513 0100 09 giz.berlin; [email protected] Amtsgericht Charlottenburg HRB 200872 B Geschäftsführerin: Dr. Britta Marschke --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
