He-Pin opened a new issue, #3241:
URL: https://github.com/apache/pekko/issues/3241
### Description
The event replay mechanism in `AsyncWriteJournal` sends `ReplayedMessage`
events directly to the persistent actor's mailbox via `tell`, with no
backpressure or batch size limit. For persistent actors with millions of
events, this can overflow the mailbox and cause OOM.
### Affected code
`persistence/src/main/scala/org/apache/pekko/persistence/journal/AsyncWriteJournal.scala`
(lines 190-195):
```scala
asyncReplayMessages(persistenceId, fromSequenceNr, toSeqNr, max) { p =>
if (!p.deleted)
adaptFromJournal(p).foreach { adaptedPersistentRepr =>
replyTo.tell(ReplayedMessage(adaptedPersistentRepr), Actor.noSender)
}
}
```
### Analysis
- The replay callback invokes `tell` for every event with no rate limiting
- The replay runs on a separate dispatcher thread, pushing into the actor's
mailbox faster than the actor can process events
- No `replay-batch-size` or memory-bounding configuration exists in
`reference.conf`
- Even streaming journal plugins (e.g., LevelDB's `DBIterator`) cannot
prevent this because the bottleneck is the mailbox, not the journal
- The `InmemJournal` (test-only) additionally loads all matching events into
memory via `immutable.Seq`
### Impact
Persistent actors with millions of events risk OOM during recovery. This is
a production concern for any deployment using large event logs.
### Suggested approach
Introduce bounded streaming between the replay callback and the persistent
actor — either via a batched `ReplayedMessages` protocol with flow control, or
a bounded buffer between the journal callback and the actor's mailbox.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
