Re: Transactional batching
thanks Scott, I noticed that SJMS supports batched Tx, but given that it hasn't been released yet, I assume others have had to work around this issue using camle-jms in the past...sounds like I'll need to roll my own or rely on aggregator/hawtdb EIP until 2.11 can be used... sully6768 wrote Hi Ben, Currently only SJMS (2.11.0) has support for batching of JMS transactions (local only, no XA). The consumer has natural batch support meaning that you set the number of messages you would like read before issuing a commit. You can also specify a timeout, default is 5000ms, to keep the consumer from holding the transaction open if you have a bursty message flow. On the producer side SJMS is able to support using a List of Objects or SJMS BatchMessage objects where the producer will iterate through the list and if configured for transactions, commit or rollback the transaction. This is necessary at this time since using a batch on the producer would potentially cause the initial consumer to loose visibility of what messages have been lost in the event of a rollback. This would work well when using a ProducerTemplate in an application that collects a given number of messages and then publishes the collection. There currently isn't support for XA but that can be mitigated by having applications monitor for the JMSRedelivery=true header. Let me know if this answers your question. Best Regards, Scott ES On Tue, Apr 2, 2013 at 11:02 PM, boday lt; ben.oday@ gt;wrote: this is a really old thread, but I'm wondering if anyone has any updated ideas on this...I have a similar requirement to read from an AMQ queue, batch X messages together and then process. I'm currently doing this with the aggregator/hawtdb repo combination but performance is proving too slow... I'd think it would be faster/cleaner to just be able to read multiple from the queue in a transaction, aggregate together, process, then end the transaction... any ideas? David J. M. Karlsen wrote Looking at this thread I wonder the same. During a resource local jmstransaction you are able to fetch several messages - but there is no way in camel to control the commit size. You can do the same during a XA tx. I see no support for batching here: http://camel.apache.org/batch-consumer.html. Either transaction mode would benefit a lot if you could process several messages in one go. Typical usecase would be tx( N messages, process them in paralell, aggregate and send out ) - Ben O'Day IT Consultant -http://consulting-notes.com -- View this message in context: http://camel.465427.n5.nabble.com/Transactional-batching-tp474002p5730245.html Sent from the Camel - Users mailing list archive at Nabble.com. -- -- Scott England-Sullivan Apache Camel Committer Principal Consultant / Sr. Architect | Red Hat, Inc. FuseSource is now part of Red Hat Web: fusesource.com lt;http://www.fusesource.comgt; | redhat.comlt;http://www.redhat.comgt; Blog: sully6768.blogspot.com Twitter: sully6768 - Ben O'Day IT Consultant -http://consulting-notes.com -- View this message in context: http://camel.465427.n5.nabble.com/Transactional-batching-tp474002p5730351.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transactional batching
Hi Ben, Currently only SJMS (2.11.0) has support for batching of JMS transactions (local only, no XA). The consumer has natural batch support meaning that you set the number of messages you would like read before issuing a commit. You can also specify a timeout, default is 5000ms, to keep the consumer from holding the transaction open if you have a bursty message flow. On the producer side SJMS is able to support using a List of Objects or SJMS BatchMessage objects where the producer will iterate through the list and if configured for transactions, commit or rollback the transaction. This is necessary at this time since using a batch on the producer would potentially cause the initial consumer to loose visibility of what messages have been lost in the event of a rollback. This would work well when using a ProducerTemplate in an application that collects a given number of messages and then publishes the collection. There currently isn't support for XA but that can be mitigated by having applications monitor for the JMSRedelivery=true header. Let me know if this answers your question. Best Regards, Scott ES On Tue, Apr 2, 2013 at 11:02 PM, boday ben.o...@initekconsulting.comwrote: this is a really old thread, but I'm wondering if anyone has any updated ideas on this...I have a similar requirement to read from an AMQ queue, batch X messages together and then process. I'm currently doing this with the aggregator/hawtdb repo combination but performance is proving too slow... I'd think it would be faster/cleaner to just be able to read multiple from the queue in a transaction, aggregate together, process, then end the transaction... any ideas? David J. M. Karlsen wrote Looking at this thread I wonder the same. During a resource local jmstransaction you are able to fetch several messages - but there is no way in camel to control the commit size. You can do the same during a XA tx. I see no support for batching here: http://camel.apache.org/batch-consumer.html. Either transaction mode would benefit a lot if you could process several messages in one go. Typical usecase would be tx( N messages, process them in paralell, aggregate and send out ) - Ben O'Day IT Consultant -http://consulting-notes.com -- View this message in context: http://camel.465427.n5.nabble.com/Transactional-batching-tp474002p5730245.html Sent from the Camel - Users mailing list archive at Nabble.com. -- -- Scott England-Sullivan Apache Camel Committer Principal Consultant / Sr. Architect | Red Hat, Inc. FuseSource is now part of Red Hat Web: fusesource.com http://www.fusesource.com | redhat.comhttp://www.redhat.com Blog: sully6768.blogspot.com Twitter: sully6768
Re: Transactional batching
this is a really old thread, but I'm wondering if anyone has any updated ideas on this...I have a similar requirement to read from an AMQ queue, batch X messages together and then process. I'm currently doing this with the aggregator/hawtdb repo combination but performance is proving too slow... I'd think it would be faster/cleaner to just be able to read multiple from the queue in a transaction, aggregate together, process, then end the transaction... any ideas? David J. M. Karlsen wrote Looking at this thread I wonder the same. During a resource local jmstransaction you are able to fetch several messages - but there is no way in camel to control the commit size. You can do the same during a XA tx. I see no support for batching here: http://camel.apache.org/batch-consumer.html. Either transaction mode would benefit a lot if you could process several messages in one go. Typical usecase would be tx( N messages, process them in paralell, aggregate and send out ) - Ben O'Day IT Consultant -http://consulting-notes.com -- View this message in context: http://camel.465427.n5.nabble.com/Transactional-batching-tp474002p5730245.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transactional batching
Looking at this thread I wonder the same. During a resource local jmstransaction you are able to fetch several messages - but there is no way in camel to control the commit size. You can do the same during a XA tx. I see no support for batching here: http://camel.apache.org/batch-consumer.html. Either transaction mode would benefit a lot if you could process several messages in one go. Typical usecase would be tx( N messages, process them in paralell, aggregate and send out ) -- View this message in context: http://camel.465427.n5.nabble.com/Transactional-batching-tp474002p4721633.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transactional batching
On Tue, Mar 10, 2009 at 8:23 PM, Thomas Beckmann thomas.beckm...@freiheit.com wrote: Hi, Am Dienstag 10 März 2009 schrieb Claus Ibsen: On Tue, Mar 10, 2009 at 3:23 PM, Thomas Beckmann thomas.beckm...@freiheit.com wrote: Hi, we are using camel 1.6 and we have a simple route that reads from an activemq queue and writes to another activemq queue based on a recipient list. The read and write is done in a transaction. Is it possible to somehow configure a batchsize and timeout so that a couple of messages are processed in one transaction? Could you also try the ActiveMQ user forum? Could be AMQ has such support. However in Camel I am not aware of such a support. But how could activemq help. Isn't it camel that is responsible for transaction demarcation? No Camel is just routing and mediation. The TX manager is AMQ, or whatever you configure with Spring TX. So if you need to be able to do batch consuming of messages in a single transaction then its something that AMQ should help support. Camel JMS is build on top of Spring JMS, so if its something that Spring JMS support as well, then it could very well be something Camel can do also. What is the reason for this? The FUSE tuning guide says: Even if you have to pay the performance cost of XA transactions - using batching (processing and sending many messages in a single transaction (JMS or XA) will boost performance considerably. This is because you share the synchrony cost (the waiting for things to be written and sync'd to disk) across many many messages. That's exactly what we want to achieve. Well this applies for XA only. Are you sure you want to use XA and not just regular TX? XA is painful, expensive and can be hard to get working accross different vendor products/drivers etc. If yes, then you should consider once more to be really really sure you want to use XA. And the rule of optmization, dont do it unless there has been established a fact that there is a performance problem Thomas -- Thomas Beckmann Dipl.-Math. freiheit.com technologies gmbh Straßenbahnring 22 / 20251 Hamburg, Germany fon +49 (0)40 / 890584-0 fax +49 (0)40 / 890584-20 HRB Hamburg 70814 DB48 4B15 11BA 9268 B74B 64A0 48AE 80AB C86A 46A3 Geschäftsführer: Claudia Dietze, Stefan Richter, Jörg Kirchhof -- Claus Ibsen Apache Camel Committer Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/
Re: Transactional batching
On Tue, Mar 10, 2009 at 3:23 PM, Thomas Beckmann thomas.beckm...@freiheit.com wrote: Hi, we are using camel 1.6 and we have a simple route that reads from an activemq queue and writes to another activemq queue based on a recipient list. The read and write is done in a transaction. Is it possible to somehow configure a batchsize and timeout so that a couple of messages are processed in one transaction? Could you also try the ActiveMQ user forum? Could be AMQ has such support. However in Camel I am not aware of such a support. What is the reason for this? Thanks in advance Thomas -- Claus Ibsen Apache Camel Committer Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/
