[
https://issues.apache.org/jira/browse/ARTEMIS-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francesco Nigro updated ARTEMIS-3289:
-------------------------------------
Description:
As mentioned in https://issues.apache.org/jira/browse/ARTEMIS-2877 one of the
major factors that contribute to reduce the scalability of a shared-nothing
replication setup is the thread wake-up cost of the {{JournalImpl}}'s
{{appendExecutor}} I/O threads.
See the flamegraph below for a busy replica while appending replicated journal
record:
!image-2021-05-11-09-32-15-538.png|width=966,height=313!
The violet bars represent the CPU cycles spent to awake the Journal appender
thread(s): despite https://issues.apache.org/jira/browse/ARTEMIS-2877 allow
backup to batch append tasks as much as possible, it seems the signaling cost
is still too high, if compared with the rest of replica packet processing.
Given that the append executor is an ordered executor built on top of I/O
thread pool, see {{ActiveMQServerImpl}}:
{code:java}
if (serviceRegistry.getIOExecutorService() != null) {
this.ioExecutorFactory = new
OrderedExecutorFactory(serviceRegistry.getIOExecutorService());
} else {
ThreadFactory tFactory = AccessController.doPrivileged(new
PrivilegedAction<ThreadFactory>() {
@Override
public ThreadFactory run() {
return new ActiveMQThreadFactory("ActiveMQ-IO-server-" +
this.toString(), false, ClientSessionFactoryImpl.class.getClassLoader());
}
});
this.ioExecutorPool = new ThreadPoolExecutor(0, Integer.MAX_VALUE,
60L, TimeUnit.SECONDS, new SynchronousQueue<Runnable>(), tFactory);
this.ioExecutorFactory = new OrderedExecutorFactory(ioExecutorPool);
}
{code}
And it's using a {{SynchronousQueue}} to submit/take new wakeup tasks, it
worths investigate if using a different thread pool, executor or a different
"sleeping" strategy could reduce such cost under heavy load and improve
response time with/without replication.
Most of the problems of the existing implementation seems related to how
ThreadPoolExecutor + SynchrnoousQueue works in tandem with ArtemisExecutor.
This small program print, on my machine:
{code:java}
public static void main(String[] args) throws InterruptedException {
ThreadPoolExecutor executor = new ThreadPoolExecutor(0,
Integer.MAX_VALUE, 60L, TimeUnit.SECONDS, new SynchronousQueue(), new
ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
Thread t = new Thread(r);
System.err.println("created new thread: " + t);
return t;
}
});
ExecutorFactory factory = new OrderedExecutorFactory(executor);
ArtemisExecutor artemisExecutor = factory.getExecutor();
ConcurrentSet<Thread> executingT = new ConcurrentHashSet<>();
for (int j = 0; j< 100;j++) {
for (int i = 0; i < 10; i++) {
artemisExecutor.execute(() -> {
executingT.add(Thread.currentThread());
try {
TimeUnit.MILLISECONDS.sleep(10);
} catch (InterruptedException e) {
e.printStackTrace();
}
});
}
Thread.sleep(100*10);
}
executor.shutdown();
executor.awaitTermination(70, TimeUnit.SECONDS);
System.out.println("Executing threads: " + executingT);
}
{code}
{code:bash}
created new thread: Thread[Thread-1,5,main]
created new thread: Thread[Thread-2,5,main]
created new thread: Thread[Thread-3,5,main]
created new thread: Thread[Thread-4,5,main]
created new thread: Thread[Thread-5,5,main]
created new thread: Thread[Thread-6,5,main]
created new thread: Thread[Thread-7,5,main]
created new thread: Thread[Thread-8,5,main]
created new thread: Thread[Thread-9,5,main]
created new thread: Thread[Thread-10,5,main]
Executing threads: [Thread[Thread-5,5,], Thread[Thread-10,5,],
Thread[Thread-8,5,], Thread[Thread-3,5,], Thread[Thread-2,5,],
Thread[Thread-1,5,], Thread[Thread-9,5,], Thread[Thread-6,5,],
Thread[Thread-4,5,]]
{code}
Meaning that bursts of short running tasks can still create many different
threads (and they won't use'em all too - see {{Thread[Thread-7,5,main]}})
introducing the cost to create and awake them.
There are cases (like the message journal) where keep reusing the same thread
is highly beneficial and the "special" exclusive executor seems a good choice
to express the single-threaded affinity of the appending tasks.
was:
As mentioned in https://issues.apache.org/jira/browse/ARTEMIS-2877 one of the
major factors that contribute to reduce the scalability of a shared-nothing
replication setup is the thread wake-up cost of the {{JournalImpl}}'s
{{appendExecutor}} I/O threads.
See the flamegraph below for a busy replica while appending replicated journal
record:
!image-2021-05-11-09-32-15-538.png|width=966,height=313!
The violet bars represent the CPU cycles spent to awake the Journal appender
thread(s): despite https://issues.apache.org/jira/browse/ARTEMIS-2877 allow
backup to batch append tasks as much as possible, it seems the signaling cost
is still too high, if compared with the rest of replica packet processing.
Given that the append executor is an ordered executor built on top of I/O
thread pool, see {{ActiveMQServerImpl}}:
{code:java}
if (serviceRegistry.getIOExecutorService() != null) {
this.ioExecutorFactory = new
OrderedExecutorFactory(serviceRegistry.getIOExecutorService());
} else {
ThreadFactory tFactory = AccessController.doPrivileged(new
PrivilegedAction<ThreadFactory>() {
@Override
public ThreadFactory run() {
return new ActiveMQThreadFactory("ActiveMQ-IO-server-" +
this.toString(), false, ClientSessionFactoryImpl.class.getClassLoader());
}
});
this.ioExecutorPool = new ThreadPoolExecutor(0, Integer.MAX_VALUE,
60L, TimeUnit.SECONDS, new SynchronousQueue<Runnable>(), tFactory);
this.ioExecutorFactory = new OrderedExecutorFactory(ioExecutorPool);
}
{code}
And it's using a {{SynchronousQueue}} to submit/take new wakeup tasks, it
worths investigate if using a different thread pool, executor or a different
"sleeping" strategy could reduce such cost under heavy load and improve
response time with/without replication.
Most of the problems of the existing implementation seems related to how
ThreadPoolExecutor + SynchrnoousQueue works in tandem with ArtemisExecutor.
This small program print, on my machine:
{code:java}
public static void main(String[] args) throws InterruptedException {
ThreadPoolExecutor executor = new ThreadPoolExecutor(0,
Integer.MAX_VALUE, 60L, TimeUnit.SECONDS, new SynchronousQueue(), new
ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
Thread t = new Thread(r);
System.err.println("created new thread: " + t);
return t;
}
});
ExecutorFactory factory = new OrderedExecutorFactory(executor);
ArtemisExecutor artemisExecutor = factory.getExecutor();
ConcurrentSet<Thread> executingT = new ConcurrentHashSet<>();
for (int j = 0; j< 100;j++) {
for (int i = 0; i < 10; i++) {
artemisExecutor.execute(() -> {
executingT.add(Thread.currentThread());
try {
TimeUnit.MILLISECONDS.sleep(10);
} catch (InterruptedException e) {
e.printStackTrace();
}
});
}
Thread.sleep(100*10);
}
executor.shutdown();
executor.awaitTermination(70, TimeUnit.SECONDS);
System.out.println("Executing threads: " + executingT);
}
{code}
{code:bash}
created new thread: Thread[Thread-1,5,main]
created new thread: Thread[Thread-2,5,main]
created new thread: Thread[Thread-3,5,main]
created new thread: Thread[Thread-4,5,main]
created new thread: Thread[Thread-5,5,main]
created new thread: Thread[Thread-6,5,main]
created new thread: Thread[Thread-7,5,main]
created new thread: Thread[Thread-8,5,main]
created new thread: Thread[Thread-9,5,main]
created new thread: Thread[Thread-10,5,main]
Executing threads: [Thread[Thread-5,5,], Thread[Thread-10,5,],
Thread[Thread-8,5,], Thread[Thread-3,5,], Thread[Thread-2,5,],
Thread[Thread-1,5,], Thread[Thread-9,5,], Thread[Thread-6,5,],
Thread[Thread-4,5,]]
{code}
Meaning that bursts of fast tasks can still create many different threads (and
they won't use'em all too - see {{Thread[Thread-7,5,main]}})
introducing the cost to create and awake them.
There are cases (like the message journal) where keep reusing the same thread
is highly beneficial and the "special" exclusive executor seems a good choice
to express the single-threaded affinity of the appending tasks.
> Reduce journal appender executor Thread wakeup cost
> ---------------------------------------------------
>
> Key: ARTEMIS-3289
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3289
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Reporter: Francesco Nigro
> Assignee: Francesco Nigro
> Priority: Major
> Attachments: 3289_backup.html, image-2021-05-11-09-32-15-538.png,
> main_backup.html
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> As mentioned in https://issues.apache.org/jira/browse/ARTEMIS-2877 one of the
> major factors that contribute to reduce the scalability of a shared-nothing
> replication setup is the thread wake-up cost of the {{JournalImpl}}'s
> {{appendExecutor}} I/O threads.
> See the flamegraph below for a busy replica while appending replicated
> journal record:
> !image-2021-05-11-09-32-15-538.png|width=966,height=313!
> The violet bars represent the CPU cycles spent to awake the Journal appender
> thread(s): despite https://issues.apache.org/jira/browse/ARTEMIS-2877 allow
> backup to batch append tasks as much as possible, it seems the signaling cost
> is still too high, if compared with the rest of replica packet processing.
> Given that the append executor is an ordered executor built on top of I/O
> thread pool, see {{ActiveMQServerImpl}}:
> {code:java}
> if (serviceRegistry.getIOExecutorService() != null) {
> this.ioExecutorFactory = new
> OrderedExecutorFactory(serviceRegistry.getIOExecutorService());
> } else {
> ThreadFactory tFactory = AccessController.doPrivileged(new
> PrivilegedAction<ThreadFactory>() {
> @Override
> public ThreadFactory run() {
> return new ActiveMQThreadFactory("ActiveMQ-IO-server-" +
> this.toString(), false, ClientSessionFactoryImpl.class.getClassLoader());
> }
> });
> this.ioExecutorPool = new ThreadPoolExecutor(0, Integer.MAX_VALUE,
> 60L, TimeUnit.SECONDS, new SynchronousQueue<Runnable>(), tFactory);
> this.ioExecutorFactory = new OrderedExecutorFactory(ioExecutorPool);
> }
> {code}
> And it's using a {{SynchronousQueue}} to submit/take new wakeup tasks, it
> worths investigate if using a different thread pool, executor or a different
> "sleeping" strategy could reduce such cost under heavy load and improve
> response time with/without replication.
> Most of the problems of the existing implementation seems related to how
> ThreadPoolExecutor + SynchrnoousQueue works in tandem with ArtemisExecutor.
> This small program print, on my machine:
> {code:java}
> public static void main(String[] args) throws InterruptedException {
> ThreadPoolExecutor executor = new ThreadPoolExecutor(0,
> Integer.MAX_VALUE, 60L, TimeUnit.SECONDS, new SynchronousQueue(), new
> ThreadFactory() {
> @Override
> public Thread newThread(Runnable r) {
> Thread t = new Thread(r);
> System.err.println("created new thread: " + t);
> return t;
> }
> });
> ExecutorFactory factory = new OrderedExecutorFactory(executor);
> ArtemisExecutor artemisExecutor = factory.getExecutor();
> ConcurrentSet<Thread> executingT = new ConcurrentHashSet<>();
> for (int j = 0; j< 100;j++) {
> for (int i = 0; i < 10; i++) {
> artemisExecutor.execute(() -> {
> executingT.add(Thread.currentThread());
> try {
> TimeUnit.MILLISECONDS.sleep(10);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> });
> }
> Thread.sleep(100*10);
> }
> executor.shutdown();
> executor.awaitTermination(70, TimeUnit.SECONDS);
> System.out.println("Executing threads: " + executingT);
> }
> {code}
> {code:bash}
> created new thread: Thread[Thread-1,5,main]
> created new thread: Thread[Thread-2,5,main]
> created new thread: Thread[Thread-3,5,main]
> created new thread: Thread[Thread-4,5,main]
> created new thread: Thread[Thread-5,5,main]
> created new thread: Thread[Thread-6,5,main]
> created new thread: Thread[Thread-7,5,main]
> created new thread: Thread[Thread-8,5,main]
> created new thread: Thread[Thread-9,5,main]
> created new thread: Thread[Thread-10,5,main]
> Executing threads: [Thread[Thread-5,5,], Thread[Thread-10,5,],
> Thread[Thread-8,5,], Thread[Thread-3,5,], Thread[Thread-2,5,],
> Thread[Thread-1,5,], Thread[Thread-9,5,], Thread[Thread-6,5,],
> Thread[Thread-4,5,]]
> {code}
>
> Meaning that bursts of short running tasks can still create many different
> threads (and they won't use'em all too - see {{Thread[Thread-7,5,main]}})
> introducing the cost to create and awake them.
> There are cases (like the message journal) where keep reusing the same thread
> is highly beneficial and the "special" exclusive executor seems a good choice
> to express the single-threaded affinity of the appending tasks.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
