Re: JMS pub/sub (using Camel) performance issue with MQ in XA transactions
Hi, We do need XA as the client application is doing 2 phase commit in (1) the subscription and (2) a database. On Tue, Sep 3, 2019 at 14:49 Jean-Baptiste Onofré wrote: > Hi, > > What about using pure ACK mode ? > > You can see an example with queue (same can be applied to topic) here: > http://blog.nanthrax.net/?p=820 > > My point is: do you need XA as AFAIU you are using only JMS as resource > (XA is required when you want to use the same transaction with different > backends like JMS and database for instance) ? > > Regards > JB > > On 03/09/2019 19:03, Benoit Fortin wrote: > > We have a java application doing JMS subscriptions that is using Camel as > > its JMS provider. > > > > The application is subscribing to a topic using XA. It consumes 1 message > > in the queue, and then closes the XA transaction (each message is part of > > an XA transaction). Then the application re-attaches itself to the topic > to > > start over the same process for each message. > > > > When there are no messages to process, the client waits for 45 seconds > > (which is the XA transaction timeout) before closing the request and the > XA > > transaction and starting a new iteration. > > > > I have analyzed how this process is actually being done using a tcpdump, > > and here is what I found: > > > > —— > > > > MQ Client --[XA_START]--> MQ Server > > > > MQ Client <--[XA_START_REPLY]-- MQ Server > > > > > > MQ Client --[SPI (SUBSCRIBE)]--> MQ Server > > > > MQ Client <--[SPI REPLY]-- MQ Server > > > > > > MQ Client --[REQUEST_MSGS]--> MQ Server > > > > MQ Client --[REQUEST_MSGS]--> MQ Server > > > > MQ Client <--[NOTIFICATION]-- MQ Server > > > > > > MQ Client --[MQCLOSE]--> MQ Server > > > > MQ Client <--[MQCLOSE_REPLY]-- MQ Server > > > > > > MQ Client --[XA_END]--> MQ Server > > > > MQ Client <--[XA_END_REPLY]-- MQ Server > > > > > > MQ Client --[XA_COMMIT]--> MQ Server > > > > MQ Client <--[XA_COMMIT_REPLY]-- MQ Server > > > > —— > > > > This is process is on a durable subscription, so each of the iterations > > (even when there is no message to consume) ends up generating I/O on the > MQ > > server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, > where > > MQ keeps trace of its subscribers). > > > > Having a somewhat high number of subscribers doing this process, the MQ > > server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even > > outside of business hours, when there are no messages to consume). This > > process also consumes ~3 CPU outside of business hours. > > > > I am a bit puzzled that using XA on pub/sub scales so bad, and I am > > wondering if there is any way to implement this solution without doing so > > much subscribe operations. > > > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com >
Re: JMS pub/sub (using Camel) performance issue with MQ in XA transactions
Hi, What about using pure ACK mode ? You can see an example with queue (same can be applied to topic) here: http://blog.nanthrax.net/?p=820 My point is: do you need XA as AFAIU you are using only JMS as resource (XA is required when you want to use the same transaction with different backends like JMS and database for instance) ? Regards JB On 03/09/2019 19:03, Benoit Fortin wrote: > We have a java application doing JMS subscriptions that is using Camel as > its JMS provider. > > The application is subscribing to a topic using XA. It consumes 1 message > in the queue, and then closes the XA transaction (each message is part of > an XA transaction). Then the application re-attaches itself to the topic to > start over the same process for each message. > > When there are no messages to process, the client waits for 45 seconds > (which is the XA transaction timeout) before closing the request and the XA > transaction and starting a new iteration. > > I have analyzed how this process is actually being done using a tcpdump, > and here is what I found: > > —— > > MQ Client --[XA_START]--> MQ Server > > MQ Client <--[XA_START_REPLY]-- MQ Server > > > MQ Client --[SPI (SUBSCRIBE)]--> MQ Server > > MQ Client <--[SPI REPLY]-- MQ Server > > > MQ Client --[REQUEST_MSGS]--> MQ Server > > MQ Client --[REQUEST_MSGS]--> MQ Server > > MQ Client <--[NOTIFICATION]-- MQ Server > > > MQ Client --[MQCLOSE]--> MQ Server > > MQ Client <--[MQCLOSE_REPLY]-- MQ Server > > > MQ Client --[XA_END]--> MQ Server > > MQ Client <--[XA_END_REPLY]-- MQ Server > > > MQ Client --[XA_COMMIT]--> MQ Server > > MQ Client <--[XA_COMMIT_REPLY]-- MQ Server > > —— > > This is process is on a durable subscription, so each of the iterations > (even when there is no message to consume) ends up generating I/O on the MQ > server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, where > MQ keeps trace of its subscribers). > > Having a somewhat high number of subscribers doing this process, the MQ > server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even > outside of business hours, when there are no messages to consume). This > process also consumes ~3 CPU outside of business hours. > > I am a bit puzzled that using XA on pub/sub scales so bad, and I am > wondering if there is any way to implement this solution without doing so > much subscribe operations. > -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com
JMS pub/sub (using Camel) performance issue with MQ in XA transactions
We have a java application doing JMS subscriptions that is using Camel as its JMS provider. The application is subscribing to a topic using XA. It consumes 1 message in the queue, and then closes the XA transaction (each message is part of an XA transaction). Then the application re-attaches itself to the topic to start over the same process for each message. When there are no messages to process, the client waits for 45 seconds (which is the XA transaction timeout) before closing the request and the XA transaction and starting a new iteration. I have analyzed how this process is actually being done using a tcpdump, and here is what I found: —— MQ Client --[XA_START]--> MQ Server MQ Client <--[XA_START_REPLY]-- MQ Server MQ Client --[SPI (SUBSCRIBE)]--> MQ Server MQ Client <--[SPI REPLY]-- MQ Server MQ Client --[REQUEST_MSGS]--> MQ Server MQ Client --[REQUEST_MSGS]--> MQ Server MQ Client <--[NOTIFICATION]-- MQ Server MQ Client --[MQCLOSE]--> MQ Server MQ Client <--[MQCLOSE_REPLY]-- MQ Server MQ Client --[XA_END]--> MQ Server MQ Client <--[XA_END_REPLY]-- MQ Server MQ Client --[XA_COMMIT]--> MQ Server MQ Client <--[XA_COMMIT_REPLY]-- MQ Server —— This is process is on a durable subscription, so each of the iterations (even when there is no message to consume) ends up generating I/O on the MQ server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, where MQ keeps trace of its subscribers). Having a somewhat high number of subscribers doing this process, the MQ server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even outside of business hours, when there are no messages to consume). This process also consumes ~3 CPU outside of business hours. I am a bit puzzled that using XA on pub/sub scales so bad, and I am wondering if there is any way to implement this solution without doing so much subscribe operations.
R: Re: [ANNOUNCEMENT] Apache Camel 3.0.0-RC1 (Release Candidate 1) Released
Yes, se are working on that too. Thanks Inviato da Yahoo Mail su Android Il mar, 3 set, 2019 alle 10:41, Knut-Håvard Aksnes ha scritto: There seems to be a bug in the way the release notes are listed on the Apache Camel News page, the date is wrong. (In the future) I suspect month number is shown instead of day of month. man. 2. sep. 2019, 19.51 skrev Gregor Zurowski : > The Camel community announces the immediate availability of Camel > 3.0.0-RC1, the first release candidate towards a new 3.0.0 major > release with 148 improvements and fixes. > > The artifacts are published and ready for you to download from the > Central Maven repository. For more details please take a look at the > release notes [1]. > > Many thanks to all who made this release possible. > > On behalf of the Camel PMC, > Gregor Zurowski > > [1] > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12345723=12311211 >
Re: [ANNOUNCEMENT] Apache Camel 3.0.0-RC1 (Release Candidate 1) Released
There seems to be a bug in the way the release notes are listed on the Apache Camel News page, the date is wrong. (In the future) I suspect month number is shown instead of day of month. man. 2. sep. 2019, 19.51 skrev Gregor Zurowski : > The Camel community announces the immediate availability of Camel > 3.0.0-RC1, the first release candidate towards a new 3.0.0 major > release with 148 improvements and fixes. > > The artifacts are published and ready for you to download from the > Central Maven repository. For more details please take a look at the > release notes [1]. > > Many thanks to all who made this release possible. > > On behalf of the Camel PMC, > Gregor Zurowski > > [1] > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12345723=12311211 >
