Chia Chien Liu created FINERACT-2900:
----------------------------------------
Summary: Send Asynchronous Events job sends at most one batch per
run: default ceiling is 1000 events per minute
Key: FINERACT-2900
URL: https://issues.apache.org/jira/browse/FINERACT-2900
Project: Apache Fineract
Issue Type: Improvement
Reporter: Chia Chien Liu
Measured on current develop (4684c66), PostgreSQL, Kafka producer enabled.
SavingsDepositBusinessEvent enabled, with a backlog of 12,488 TO_BE_SENT events.
Defaults (job "Send Asynchronous Events" with cron 0 0/1 * * * ?,
external-event-batch-size = 1000): exactly 1000 events were sent per run, one
run per minute, i.e. about 17 events/s or 1.4M events/day. A run takes only
0.12-0.28 s. With external-event-batch-size = 10000 the remaining 7,488 events
were sent in a single run in 1.08 s (about 6,900 events/s).
So the ceiling comes from the configuration, not from processing cost:
SendAsynchronousEventsTasklet reads one batch per execution and returns
RepeatStatus.FINISHED. Operators have to know to raise the batch size or the
schedule.
Proposal: add fineract.events.external.max-batches-per-run (default 1, so the
current behavior is unchanged). A run reads up to that many batches, marks each
batch as sent before reading the next one, and stops as soon as a batch is not
full. A failed send leaves the failed batch TO_BE_SENT and stops the run, as
today.
Question for maintainers: is the one-batch-per-run behavior intentional (for
example to bound the run time per scheduler slot)? If not, should the default
be raised later?
I have a branch with the change and unit tests, and a benchmark harness (docker
compose override plus a k6 script). I will open a pull request once this ticket
exists, and I will add before/after numbers measured on a build of the change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
