Hey Benoit,
thanks for the quick fix! I reviewed your MR on GitHub.
Sorry for missing this - I thought the existing test suite covered this.
No, problem, I also thought the test was covering this.
Best regards,
Felix
On 28/09/2026 17.01, Benoit TELLIER via server-dev wrote:
I see.
So the issue is that we shall buffer the page of result from PG in James memory
and replay it so that we are sure to release the underlying connection if I get
it right.
I think I found something which collectList() withholds upstream requests until
downstream demand exists, so expand's page N+1 subscription arrives with zero
demand and just idles on its connection.
I'll open a draft PR: https://github.com/apache/james-project/pull/3215
I'm happy if you can test it and see if it fixes the issue.
Sorry for missing this - I thought the existing test suite covered this.
--
Best regards,
Benoit TELLIER
General manager of Linagora VIETNAM.
Product owner for Twake-Mail product.
Chairman of the Apache James project.
Mail: [email protected]
Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
Le sept. 28, 2026 9:53 AM, de Felix Auringer <[email protected]>Hey
Benoit,
thanks for the very fast response!
I only managed to test your changes today and they definitely improved the
issue.
However, I can still reproduce it in a similar way.
When using opensearch as the search backend and triggering a full reindex with curl -XPOST
'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=500', the
reindexing fails for some folders:
2026-09-28T07:22:25.769Z [34m[INFO ][0;39m
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
Sent-<uuid> of user re-indexing on message 2336
2026-09-28T07:22:25.778Z [34m[INFO ][0;39m
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
Sent-<uuid> of user re-indexing on message 2339
2026-09-28T07:22:25.766Z [1;31m[ERROR][0;39m
o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres query.
May need to check either jOOQ reactive issue or Postgres DB performance.
java.util.concurrent.TimeoutException: Did not observe any item or terminal
signal within 10000ms in 'flatMapMany' (and no fallback has been configured)
at
reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
at
reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
at
reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
at
reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at
reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
at
reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
2026-09-28T07:22:25.780Z [34m[INFO ][0;39m
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
Sent-<uuid> of user re-indexing on message 2346
2026-09-28T07:22:25.783Z [34m[INFO ][0;39m
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
Sent-<uuid> of user re-indexing on message 2342
2026-09-28T07:22:25.784Z [34m[INFO ][0;39m
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
Sent-<uuid> of user re-indexing on message 2345
2026-09-28T07:22:25.786Z [31m[WARN ][0;39m o.a.m.tools.indexer.ReIndexerPerformer -
ReIndexing failed for #private:<user>:Sent
java.util.concurrent.TimeoutException: Did not observe any item or terminal
signal within 10000ms in 'flatMapMany' (and no fallback has been configured)
at
reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
at
reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
at
reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
at
reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at
reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
at
reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
2026-09-28T07:22:25.789Z [34m[INFO ][0;39m
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
Sent-<uuid> of user re-indexing on message 2343
2026-09-28T07:22:25.792Z [34m[INFO ][0;39m
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
Sent-<uuid> of user re-indexing on message 2341
In total, the reindexing now processed a few dozen thousand emails and
terminated in a short time.
The folders the process fails for all have more than 5000 emails, so I guess
James currently opens one streaming connection per mailbox it indexes and then
fails after processing 500 (messages per seconds) * 10 (seconds timeout) ≈ 5000
emails.
Best regards,
Felix
On 25/09/2026 15.17, Benoit TELLIER via server-dev wrote:
Thanks for the look @Quan but I'd rather address the root cause: the listing
should not hold a permanent connection. It should use one every time it fetches
one page.
I did do an alternative proposition here:
github.com/apache/james-project/pull/3209
--
Best regards,
Benoit TELLIER
General manager of Linagora VIETNAM.
Product owner for Twake-Mail product.
Chairman of the Apache James project.
Mail: [email protected]
Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
On Sep 25, 2026 3:02 PM, from Quan Tran Hong <[email protected]>Hi
Benoit,
Food for thought: github.com/apache/james-project/pull/3208
I did a quick review of it, and the fixes make sense to me (I asked the
agent to do TDD: reproduce the reported issue first).
I hope it helps. I will properly double check the fixes later.
Quan
On Fri, Sep 25, 2026 at 7:42 PM Benoit TELLIER via server-dev <
[email protected]> wrote:
Hello Felix
That's a classic Jooq / R2DBC issue: it badly support streaming as it hold
a live connection.
The reindexing read live data as it needs it (so 2 message per second for
instance) and keep it opens as it iterates/
The listing query then times out after 10s, thus the symptoms.
We encountered that in over parts of the PG implem and used buffering (
intermediate .collectList ) to pull results in memory, and return the
driver thread as fast as we could.
The tricky point is that EG Cassandra driver supports reading large result
sets without monopolizing driver resources. It thus don't need that memory
buffering. Any fix doing this would then need not to impact Cassandra
implementation that could be doing the optimal list-as-it-comes operation.
Another solution is to rethink "streaming" and switch it to explicit
separated request per page size. This lift the time limit as we no longer
hold a collection at the cost of repeated queries. A sample can be found
here
github.com/apache/james-project/blob/bf100353ac3c4fc76d9ecced7472528b43e1ca21/mailbox/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/dao/PostgresMailboxMessageDAO.java#L487
A generic adoption of this pattern can solve the problem you mention (at
the cost of snapshot isolation which I believe we do not really care.
I'll try to propose a little something in that direction.
--
Best regards,
Benoit TELLIER
General manager of Linagora VIETNAM.
Product owner for Twake-Mail product.
Chairman of the Apache James project.
Mail: [email protected]
Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
On Sep 25, 2026 12:51 PM, from Felix Auringer <[email protected]
Hello,
For some time now, I have seen a timeout error in the postgres executor:
o.a.j.b.postgres.utils.PostgresExecutor - Time out executing Postgres
query. May need to check either jOOQ reactive issue or Postgres DB
performance.
This was mainly happening in TMail and was reported here:
github.com/linagora/tmail-backend/issues/1599
I have now found a way to reproduce this reliably with the Docker image
built from the current James master
(bf100353ac3c4fc76d9ecced7472528b43e1ca21).
When using opensearch as the search backend and triggering a full reindex
with curl -XPOST
'http://<IP>:<port>/mailboxes?task=reIndex&messagesPerSecond=1',
I observe the following behavior:
2026-09-25T10:29:41.968Z [INFO ] o.a.j.webadmin.mdc.LoggingRequestFilter -
WebAdmin request received
2026-09-25T10:29:41.974Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
Starting a full reindex
2026-09-25T10:29:41.985Z [INFO ] o.a.j.webadmin.mdc.LoggingResponseFilter
- WebAdmin response received
2026-09-25T10:29:43.178Z [INFO ]
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 1
2026-09-25T10:29:44.120Z [INFO ]
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 2
2026-09-25T10:29:45.125Z [INFO ]
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 3
2026-09-25T10:29:46.128Z [INFO ]
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 4
2026-09-25T10:29:47.131Z [INFO ]
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 5
2026-09-25T10:29:48.135Z [INFO ]
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 6
2026-09-25T10:29:49.137Z [INFO ]
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 7
2026-09-25T10:29:50.147Z [INFO ]
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 8
2026-09-25T10:29:51.143Z [INFO ]
o.a.j.m.o.e.OpenSearchListeningMessageSearchIndex - Indexing mailbox
email-88f7e4e1-cb74-47fb-a4f6-3046afc29714 of user re-indexing on message 9
2026-09-25T10:29:52.052Z [ERROR] o.a.j.b.postgres.utils.PostgresExecutor -
Time out executing Postgres query. May need to check either jOOQ reactive
issue or Postgres DB performance.
java.util.concurrent.TimeoutException: Did not observe any item or
terminal signal within 10000ms in 'flatMapMany' (and no fallback has been
configured)
at
reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:296)
at
reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:281)
at
reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:420)
at
reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at
reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
at
reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
at
reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at
reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2026-09-25T10:29:52.058Z [INFO ] o.a.m.tools.indexer.ReIndexerPerformer -
Full reindex finished
2026-09-25T10:29:52.059Z [ERROR] o.a.james.task.SerialTaskManagerWorker -
Task was partially performed. Check logs for more details. Taskid :
TaskId{value=9553e1df-9ad0-42d6-b90d-30c5882b8d17}
This error always happens 10 seconds after starting the reindex task.
Therefore, exactly 9 batches of the size specified with messagesPerSecond
are processed.
When indexing 2 messages per second, the indexer manages to index the
first 18 messages, when indexing 5 messages per second, the first 45
messages are indexed, and so on...
I don't know much about this part of the codebase and would be happy about
help to debug this.
Did somebody else ever see this error or can somebody else reproduce it?
Best regards,
Felix
---
Gesellschaft für interkulturelles
Zusammenleben gGmbH (GIZ)
Felix Auringer
IT
Reformationsplatz 2
13597 Berlin
Tel: 030/513 0100 00; Fax: 030/513 0100 09
giz.berlin; [email protected]
Amtsgericht Charlottenburg HRB 200872 B
Geschäftsführerin: Dr. Britta Marschke
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---
Gesellschaft für interkulturelles
Zusammenleben gGmbH (GIZ)
Felix Auringer
IT
Reformationsplatz 2
13597 Berlin
Tel: 030/513 0100 00; Fax: 030/513 0100 09
giz.berlin; [email protected]
Amtsgericht Charlottenburg HRB 200872 B
Geschäftsführerin: Dr. Britta Marschke
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---
Gesellschaft für interkulturelles
Zusammenleben gGmbH (GIZ)
Felix Auringer
IT
Reformationsplatz 2
13597 Berlin
Tel: 030/513 0100 00; Fax: 030/513 0100 09
www.giz.berlin; [email protected]
Amtsgericht Charlottenburg HRB 200872 B
Geschäftsführerin: Dr. Britta Marschke
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
