[
https://issues.apache.org/jira/browse/AMQ-8247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rob Ambalu updated AMQ-8247:
----------------------------
Description:
activemq-cpp has a performance issue when subscribing to many topics. When I
try to subscribe to market data by ticker, with 1 topic per ticker, it starts
to bog down as I add more Consumers. Digging into it, I see the issue is in
how consumers are managed in ActiveMQSessionKernel.cpp. It maintains all
consumers in a linked list, and then for *every message* that is received, it
does a linear O(n) scan to find the matching consumer to call back into.
My application has close to 10k topics subscribed, every message does up to 10k
scan. This is completely unusable.
The fix is simple, maintain consumers by HashMap not a LinkedList. I dont have
access to create a pull-request in github, but I already have the diffs and
will attach it to this PR.
Some notes:
* adding an identical consumer will silently fail, but this is the same
behavior as it is today ( duplicates will add to the LinkedList of consumers,
but upon lookup only the first one will be returned... )
* Added slight optim to ConsumerId::getHashCode() because it was too
unnecessarily inefficient to bare
* I removed the ActiveMQSessionKernel::deliverAcks() because its not used
anywhere, and it does a full iteration so I want to make sure it never gets used
* I tested this on my use case and my application can easily keep up without
issue now
was:
activemq-cpp has a performance issue when subscribing to many topics. When I
try to subscribe to market data by ticker, with 1 topic per ticker, it starts
to bog down as I add more Consumers. Digging into it, I see the issue is in
how consumers are managed in ActiveMQSessionKernel.cpp. It maintains all
consumers in a linked list, and then for *every message* that is received, it
does a linear O(n) scan to find the matching consumer to call back into.
My application has close to 10k topics subscribed, every message does up to 10k
scan. This is completely unusable.
The fix is simple, maintain consumers by HashMap not a LinkedList. I dont have
access to create a pull-request in github, but I already have the diffs and
will attach it to this PR.
Some notes:
* adding an identical consumer will silently fail, but this is the same
behavior as it is today ( duplicates will add to the LinkedList of consumers,
but upon lookup only the first one will be returned... )
* Added slight optim to ConsumerId::getHashCode() because it was too
unnecessarily inefficient to bare
* I tested this on my use case and my application can easily keep up without
issue now
> Consumer performance issues with many topics
> --------------------------------------------
>
> Key: AMQ-8247
> URL: https://issues.apache.org/jira/browse/AMQ-8247
> Project: ActiveMQ
> Issue Type: Improvement
> Reporter: Rob Ambalu
> Priority: Major
> Attachments: diff.txt
>
>
> activemq-cpp has a performance issue when subscribing to many topics. When I
> try to subscribe to market data by ticker, with 1 topic per ticker, it starts
> to bog down as I add more Consumers. Digging into it, I see the issue is in
> how consumers are managed in ActiveMQSessionKernel.cpp. It maintains all
> consumers in a linked list, and then for *every message* that is received, it
> does a linear O(n) scan to find the matching consumer to call back into.
> My application has close to 10k topics subscribed, every message does up to
> 10k scan. This is completely unusable.
> The fix is simple, maintain consumers by HashMap not a LinkedList. I dont
> have access to create a pull-request in github, but I already have the diffs
> and will attach it to this PR.
>
> Some notes:
> * adding an identical consumer will silently fail, but this is the same
> behavior as it is today ( duplicates will add to the LinkedList of consumers,
> but upon lookup only the first one will be returned... )
> * Added slight optim to ConsumerId::getHashCode() because it was too
> unnecessarily inefficient to bare
> * I removed the ActiveMQSessionKernel::deliverAcks() because its not used
> anywhere, and it does a full iteration so I want to make sure it never gets
> used
> * I tested this on my use case and my application can easily keep up without
> issue now
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
