Re: Question about rollbackOnlyOnAsyncException (AMQ-3166)
I'd echo Gary's suggestion that you capture thread dumps or perform sampling during the slow events. That would make it clear where the time is being spent. Tim On Wed, Feb 14, 2018 at 11:16 PM, alprausch77 wrote: > Yes, we use XA transactions. > > I don´t think that the NIO mixes this up somehow. We can run our system > with > a standalone ActiveMQ using tcp or nio but also in an embedded mode inside > Wildfly using the VM protocol. > > I just run our tests again with the enableAckCompaction set to false for > the > kahaDB - it looked a lot better as we now only have a few message sends > taking ~10ms or ~20ms (which is also a long time...) but no send took > longer > than that. > > I´ll have further tests in that direction and will keep this post updated. > > Thanks > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
Re: Question about rollbackOnlyOnAsyncException (AMQ-3166)
Yes, we use XA transactions. I don´t think that the NIO mixes this up somehow. We can run our system with a standalone ActiveMQ using tcp or nio but also in an embedded mode inside Wildfly using the VM protocol. I just run our tests again with the enableAckCompaction set to false for the kahaDB - it looked a lot better as we now only have a few message sends taking ~10ms or ~20ms (which is also a long time...) but no send took longer than that. I´ll have further tests in that direction and will keep this post updated. Thanks -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Question about rollbackOnlyOnAsyncException (AMQ-3166)
XA transactions? normally the transaction is on the session and connection... so any async call will complete before the commit.. unless maybe nio is mixing this up some. do you have some code that will show the rollbackOnlyOnAsyncException case where the tx is null. There is a test case that can provide the basis for some experiments: AMQ3166Test On Wed, 14 Feb 2018 at 14:38 alprausch77 wrote: > Hello Gary. > Thanks for your reply. > I already tried the rollbackOnlyOnAsyncException. But this one only works > if > the transaction still exists when the error in the async send happens. > But if the TX is already finished, than there is no such action because the > code which looks up the transaction returns null (naturally) and therfore > can´t do a rollback. > > That´s why I asked if there is some way to add a synchronization on the TX > commit on the asynch outcome... > > Some time ago I also tested out the preallocation strategies but found no > real performance differences between those. I think this has to do with the > Windows file system which we are using; on Unix based systems this seems to > have more of an effect. > > I haven´t tried the ackCompaction yet. But if I understand it correctly I > don´t think this will have much influence either because we aren´t doing > much bulk processing. In each transaction there are only a few messages > received / send. > > But I will have a look at this and also to Artemis. > > > > > -- > Sent from: > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html >
Re: Question about rollbackOnlyOnAsyncException (AMQ-3166)
Hello Gary. Thanks for your reply. I already tried the rollbackOnlyOnAsyncException. But this one only works if the transaction still exists when the error in the async send happens. But if the TX is already finished, than there is no such action because the code which looks up the transaction returns null (naturally) and therfore can´t do a rollback. That´s why I asked if there is some way to add a synchronization on the TX commit on the asynch outcome... Some time ago I also tested out the preallocation strategies but found no real performance differences between those. I think this has to do with the Windows file system which we are using; on Unix based systems this seems to have more of an effect. I haven´t tried the ackCompaction yet. But if I understand it correctly I don´t think this will have much influence either because we aren´t doing much bulk processing. In each transaction there are only a few messages received / send. But I will have a look at this and also to Artemis. -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Question about rollbackOnlyOnAsyncException (AMQ-3166)
in addition, try the latest Artemis release, there has been a bunch of work on latency w.r.t gc on exactly that flat send latency (reliable enqueue) use case. On Wed, 14 Feb 2018 at 14:29 Gary Tully wrote: > transaction commit is a sync operation... if an async send in a > transaction fails, then the commit will rollback via: > rollbackOnlyOnAsyncException > > w.r.t spikes in send, have you enabled the preallocation strategy on > kahadb, it may be worth toggling to see the effect in your env. Also, > ackCompaction may be in the way. To get to the bottom of it would need some > thread dumps or profiling... but those are the big events that effect io. > > On Mon, 12 Feb 2018 at 05:51 alprausch77 > wrote: > >> Hello Tim. >> Thank you for your suggestion but I already checked the GC (forgot to >> mention it). >> The times of the slow JMS sends doesn´t correlate with the garbage >> collector >> events. >> Btw: we use CMS as collector. >> >> Any other ideas? >> >> Thanks. >> Joachim >> >> >> >> -- >> Sent from: >> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html >> >
Re: Question about rollbackOnlyOnAsyncException (AMQ-3166)
transaction commit is a sync operation... if an async send in a transaction fails, then the commit will rollback via: rollbackOnlyOnAsyncException w.r.t spikes in send, have you enabled the preallocation strategy on kahadb, it may be worth toggling to see the effect in your env. Also, ackCompaction may be in the way. To get to the bottom of it would need some thread dumps or profiling... but those are the big events that effect io. On Mon, 12 Feb 2018 at 05:51 alprausch77 wrote: > Hello Tim. > Thank you for your suggestion but I already checked the GC (forgot to > mention it). > The times of the slow JMS sends doesn´t correlate with the garbage > collector > events. > Btw: we use CMS as collector. > > Any other ideas? > > Thanks. > Joachim > > > > -- > Sent from: > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html >
Re: Question about rollbackOnlyOnAsyncException (AMQ-3166)
Hello Tim. Thank you for your suggestion but I already checked the GC (forgot to mention it). The times of the slow JMS sends doesn´t correlate with the garbage collector events. Btw: we use CMS as collector. Any other ideas? Thanks. Joachim -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Question about rollbackOnlyOnAsyncException (AMQ-3166)
One thing that could cause pauses like that is the JVM's garbage collector. What GC strategy are you using, and can you see if the times at which the higher latency occurs correlate with the times at which full GCs are occurring? Tim On Feb 9, 2018 1:53 AM, "alprausch77" wrote: > Hello. > I have a question about the AMQ-3166 issue and the introduced > 'rollbackOnlyOnAsyncException' flag. > In the comments of the AMQ-3166 task Gary Tully says > /async exceptions on transactional ops - message send and message ack > will result in the transaction being marked rollback-only. Commit will fail > with an exception./ > > > But if I send the message in async mode than it´s likely that the client > side transaction is already committed before(!) an exception occurs on > > the AMQ side. In that case the TransportConnection class will no longer > find > a transaction for the message (as it´s already committed) and of > > course it can´t be rolled-back anymore. > Or do I misunderstand the behaviour? > > > My current problem is that we switched from async to sync sends because > we´ve seen some messages getting lost without "seeing" it on the client > > side. For our use-case it´s not acceptable to have a message loss. So we > switched to the sync send. > But this comes with a high performance impact. > We are seeing a lot of messages to be send in 0-3ms but we see occassional > spikes in the send times (in the ActiveMQMessageProducer) of 30-500ms. > > The 'producerFlowControl' is already disabled in the configuration; the > storage is mKahaDB. > And we already tested a lot of configuration combinations with no real > (positive) effect on the performance. > > Is there a way to avoid such spikes in the sync send? > (Tests has shown that the spikes are most likely caused due to I/O issues. > Because when using a RAM disk as location for the kahaDB I don´t see > > any such spikes.) > > Or in async mode: is there a way to have a synchronization on transaction > commit which verifies that the message is processed? > This way the message processing could be triggered in async mode and the > client could do further work and only synchronizes on the transaction > > commit. > > > Thanks. > > btw: we are running AMQ 5.14.5 (unfortunatelly on Windows) > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
Question about rollbackOnlyOnAsyncException (AMQ-3166)
Hello. I have a question about the AMQ-3166 issue and the introduced 'rollbackOnlyOnAsyncException' flag. In the comments of the AMQ-3166 task Gary Tully says async exceptions on transactional ops - message send and message ack will result in the transaction being marked rollback-only. Commit will fail with an exception. But if I send the message in async mode than it´s likely that the client side transaction is already committed before(!) an exception occurs on the AMQ side. In that case the TransportConnection class will no longer find a transaction for the message (as it´s already committed) and of course it can´t be rolled-back anymore. Or do I misunderstand the behaviour? My current problem is that we switched from async to sync sends because we´ve seen some messages getting lost without "seeing" it on the client side. For our use-case it´s not acceptable to have a message loss. So we switched to the sync send. But this comes with a high performance impact. We are seeing a lot of messages to be send in 0-3ms but we see occassional spikes in the send times (in the ActiveMQMessageProducer) of 30-500ms. The 'producerFlowControl' is already disabled in the configuration; the storage is mKahaDB. And we already tested a lot of configuration combinations with no real (positive) effect on the performance. Is there a way to avoid such spikes in the sync send? (Tests has shown that the spikes are most likely caused due to I/O issues. Because when using a RAM disk as location for the kahaDB I don´t see any such spikes.) Or in async mode: is there a way to have a synchronization on transaction commit which verifies that the message is processed? This way the message processing could be triggered in async mode and the client could do further work and only synchronizes on the transaction commit. Thanks. Joachim btw: we are running AMQ 5.14.5 (unfortunatelly on Windows) -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Question about rollbackOnlyOnAsyncException (AMQ-3166)
Hello. I have a question about the AMQ-3166 issue and the introduced 'rollbackOnlyOnAsyncException' flag. In the comments of the AMQ-3166 task Gary Tully says /async exceptions on transactional ops - message send and message ack will result in the transaction being marked rollback-only. Commit will fail with an exception./ But if I send the message in async mode than it´s likely that the client side transaction is already committed before(!) an exception occurs on the AMQ side. In that case the TransportConnection class will no longer find a transaction for the message (as it´s already committed) and of course it can´t be rolled-back anymore. Or do I misunderstand the behaviour? My current problem is that we switched from async to sync sends because we´ve seen some messages getting lost without "seeing" it on the client side. For our use-case it´s not acceptable to have a message loss. So we switched to the sync send. But this comes with a high performance impact. We are seeing a lot of messages to be send in 0-3ms but we see occassional spikes in the send times (in the ActiveMQMessageProducer) of 30-500ms. The 'producerFlowControl' is already disabled in the configuration; the storage is mKahaDB. And we already tested a lot of configuration combinations with no real (positive) effect on the performance. Is there a way to avoid such spikes in the sync send? (Tests has shown that the spikes are most likely caused due to I/O issues. Because when using a RAM disk as location for the kahaDB I don´t see any such spikes.) Or in async mode: is there a way to have a synchronization on transaction commit which verifies that the message is processed? This way the message processing could be triggered in async mode and the client could do further work and only synchronizes on the transaction commit. Thanks. btw: we are running AMQ 5.14.5 (unfortunatelly on Windows) -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html