Francesco Nigro created ARTEMIS-3289:
----------------------------------------
Summary: Reduce journal appender executor Thread wakeup cost
Key: ARTEMIS-3289
URL: https://issues.apache.org/jira/browse/ARTEMIS-3289
Project: ActiveMQ Artemis
Issue Type: Improvement
Reporter: Francesco Nigro
Assignee: Francesco Nigro
Attachments: image-2021-05-11-09-32-15-538.png
As mentioned in https://issues.apache.org/jira/browse/ARTEMIS-2877 one of the
major factors that contribute to reduce the scalability of a shared-nothing
replication setup is the thread wake-up cost of the {{JournalImpl}}'s
{{appendExecutor}} I/O threads.
See the flamegraph below for a busy replica while appending replicated journal
record:
!image-2021-05-11-09-32-15-538.png!
The violet bars represent the CPU spent while awaking the Journal appender
thread(s): despite https://issues.apache.org/jira/browse/ARTEMIS-2877 allow
backup to batch append tasks as much as possible, it seems the signaling cost
is still too high, if compared with the rest of replica packet processing.
Given that the append executor is an ordered executor built on top of I/O
thread pool, see {{ActiveMQServerImpl}}:
{code:java}
if (serviceRegistry.getIOExecutorService() != null) {
this.ioExecutorFactory = new
OrderedExecutorFactory(serviceRegistry.getIOExecutorService());
} else {
ThreadFactory tFactory = AccessController.doPrivileged(new
PrivilegedAction<ThreadFactory>() {
@Override
public ThreadFactory run() {
return new ActiveMQThreadFactory("ActiveMQ-IO-server-" +
this.toString(), false, ClientSessionFactoryImpl.class.getClassLoader());
}
});
this.ioExecutorPool = new ThreadPoolExecutor(0, Integer.MAX_VALUE,
60L, TimeUnit.SECONDS, new SynchronousQueue<Runnable>(), tFactory);
this.ioExecutorFactory = new OrderedExecutorFactory(ioExecutorPool);
}
{code}
And it's using a {{SynchronousQueue}} to submit/take new wakeup tasks, using a
different thread pool, executor or a different "sleeping" strategy could
dramatically reduce such cost under heavy load eg both on replication and
without replication too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
