[
https://issues.apache.org/jira/browse/AMQ-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16565976#comment-16565976
]
ALAN DIEGO DANTAS PROTASIO commented on AMQ-7028:
-------------------------------------------------
I created a pull with the proposed change:
https://github.com/apache/activemq/pull/295
> Poor performance when concurrentStoreAndDispatchQueues + slow FS + Slow
> Consumers
> ---------------------------------------------------------------------------------
>
> Key: AMQ-7028
> URL: https://issues.apache.org/jira/browse/AMQ-7028
> Project: ActiveMQ
> Issue Type: Improvement
> Components: KahaDB
> Affects Versions: 5.15.4
> Reporter: ALAN DIEGO DANTAS PROTASIO
> Priority: Major
>
> Using high latency FS (as NFS) to store kahadb files and setting
> concurrentStoreAndDispatchQueues=true may cause poor performance for slow
> consumer. This happens because using this option makes activemq write the
> produced messages one by one to the underlying file system (this is
> implemented by using a SingleThread ExecutorService).
> Lets say that for each write to the FS takes 10ms and the queue has slow
> consumers. In this case, does not matter the number of concurrent messages
> the producers try to send to the queue, the maximum performance we can
> achieve is 100 TPS. Tuning this flag off, we can see a really better
> performance for sending messages in parallel as those messages can be batched
> to the FS in a single write (the performance increases with the number of
> concurrent messages being sent in parallel).
> Looking at Activemq code we found that there is an flag used on levelDb to
> detect if the queue has fast or slow consumers, and decide if it will use
> concurrentStoreAndDispach or not.
> https://issues.apache.org/jira/browse/AMQ-3750
> but this flag is not used on the KahaDb implementation.
> We made a code change to receive the flag in the KahaDbStore and use it to
> decide if the message will be stored async or not.
> We think that there is no reason to try to "StoreAndDispatch" if the
> destination has slow consumers. This only brings overhead and in case of high
> latency FS, really poor performance when the queue has slow consumer.
> For fast consumers, this change will have no effect giving the better of the
> 2 options.
> Some Results:
> Original Version:
> Fast Consumers:
> Producer
> mean rate = 8248.50 calls/second
> min = 0.42 milliseconds
> max = 756.61 milliseconds
> mean = 11.30 milliseconds
> stddev = 44.05 milliseconds
> median = 6.02 milliseconds
> 75% <= 9.79 milliseconds
> 95% <= 18.15 milliseconds
> 98% <= 27.71 milliseconds
> 99% <= 123.51 milliseconds
> 99.9% <= 756.61 milliseconds
> Slow consumers:
> Producer
> mean rate = 84.29 calls/second
> min = 86.27 milliseconds
> max = 1467.53 milliseconds
> mean = 1082.55 milliseconds
> stddev = 154.04 milliseconds
> median = 1075.94 milliseconds
> 75% <= 1169.10 milliseconds
> 95% <= 1308.90 milliseconds
> 98% <= 1350.85 milliseconds
> 99% <= 1363.61 milliseconds
> 99.9% <= 1466.67 milliseconds
> Patched Version:
> Fast Consumers:
> Producer
> count = 890783
> mean rate = 8099.33 calls/second
> min = 0.47 milliseconds
> max = 2259.10 milliseconds
> mean = 13.90 milliseconds
> stddev = 84.84 milliseconds
> median = 5.00 milliseconds
> 75% <= 9.08 milliseconds
> 95% <= 15.66 milliseconds
> 98% <= 32.94 milliseconds
> 99% <= 355.52 milliseconds
> 99.9% <= 731.69 milliseconds
> Slow consumers:
> Producer
> mean rate = 1732.25 calls/second
> 1-minute rate = 1811.80 calls/second
> min = 17.52 milliseconds
> max = 1249.54 milliseconds
> mean = 50.95 milliseconds
> stddev = 130.68 milliseconds
> median = 28.73 milliseconds
> 75% <= 32.51 milliseconds
> 95% <= 57.04 milliseconds
> 98% <= 461.46 milliseconds
> 99% <= 937.87 milliseconds
> 99.9% <= 1249.48 milliseconds
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
