[
https://issues.apache.org/jira/browse/ARTEMIS-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730915#comment-16730915
]
ASF GitHub Bot commented on ARTEMIS-2216:
-----------------------------------------
Github user qihongxu commented on the issue:
https://github.com/apache/activemq-artemis/pull/2484
@michaelandrepearce As presented in test result adding a specific executor
do improve producer rates, and also means page() method in PagingStoreImpl will
acquire writelock more often. At the same time QueueImpl::deliver will call
isPaging() to check whether queue is paging or not, which acquire readlock and
make it a race condition. This may explain why the increase of producer rates
is accompanied by the decrease of consumer rates.
According to this we have an internal branch that removes isPaging()
method’s readlock in PagingStoreImpl, along with adding pageSyncTimer’s
specific executor. After that we performed a same test and get a result of
21k/25k Send&Receive–in total 46k TPS. This version runs smoothly in our use
case but we are still exploring potential effects.
The reason why we use transactional receive is to make sure every message
indeed delivered and consumed. Say that if we do non-tx receive and consumer
system fail in the following processing chain, this message may be delivered
but actually “lost”. So we choose to let consumer decide when should they ack
and commit, instead of auto-commit acks.
> Use a specific executor for pageSyncTimer
> -----------------------------------------
>
> Key: ARTEMIS-2216
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2216
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Affects Versions: 2.6.3
> Reporter: Qihong Xu
> Priority: Major
>
> Improving throughput on paging mode is one of our concerns since our cluster
> uses paging a lot.
> We found that pageSyncTimer in PagingStoreImpl shared the same executor with
> pageCursorProvider from thread pool. In heavy load scenario like hundreds of
> consumers receiving messages simultaneously, it became difficult for
> pageSyncTimer to get the executor due to race condition. Therefore page sync
> was delayed and producers suffered low throughput.
>
> To achieve higher performance we assign a specific executor to pageSyncTimer
> to avoid racing. And we run a small-scale test on a single modified broker.
>
> Broker: 4C/8G/500G SSD
> Producer: 200 threads, non-transactional send
> Consumer 200 threads, transactional receive
> Message text size: 100-200 bytes randomly
> AddressFullPolicy: PAGE
>
> Test result：
> | |Only Send TPS|Only Receive TPS|Send&Receive TPS|
> |Original ver|38k|33k|3k/30k|
> |Modified ver|38k|34k|30k/12.5k|
>
> The chart above shows that on modified broker send TPS improves from “poor”
> to “extremely fast”, while receive TPS drops from “extremely fast” to
> “not-bad” under heavy load. Considering consumer systems usually have a long
> processing chain after receiving messages, we don’t need too fast receive
> TPS. Instead, we want to guarantee send TPS to cope with traffic peak and
> lower producer’s delay time. Moreover, send and receive TPS in total raises
> from 33k to about 43k. From all above this trade-off seems beneficial and
> acceptable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
