Hi Dirk, sorry for the late reply. I opened a JIRA ticket for reliable resequencing (https://issues.apache.org/activemq/browse/CAMEL-949). I'll try to address your questions (how to handle delivery failures ... etc.) when working on a patch. Did you make any progress in the meantime (ideas, patches ...)?
Cheers, Martin Dirk Mahler wrote: > > Hi Martin, > > sorry for the late response - lot's of work is waiting ;-) But > nevertheless thanx for your answer even if it contains bad news for us... > > I'd like to give you some feedback on your ideas - please note that I do > not know Camel and it's internal structures very well. > > > Martin Krasser wrote: >> >> - Batch resequencer: read n messages within a single transaction (instead >> of >> n transactions as done currently in 1.4), reorder them, send them to a >> destination and commit the transaction. This can easily done within a >> single >> thread. >> > > Might work but I doubt that the batch sequencer makes sense in a real > world because IMHO it fails on the borders between two batches. A short > example: Imagine a batch size of 3 and an incoming message order > 1|2|4|3|5|6. This would result in sequential processing of two batches. > There is no need to do any re-ordering in any of these batches but > messages 3 and 4 will finally be delivered in wrong order to the consumer. > > > Martin Krasser wrote: >> >> - Stream resequencer: the situation is more tricky here because there is >> a >> continuous stream of messages without having candidates for transaction >> boundaries (in contrast to the batch resequencer where the batch size can >> be >> used to determine a transaction boundary). I'm still not 100% sure how to >> solve this problem but a possible solution might roughly look like this: >> Read messages from a source endpoint with n threads, suspend them until >> they >> have been re-ordered based on the message they have read and >> resume/execute >> them sequentially after their message becomes eligible for delivery. >> After >> delivery of the message each thread commits the transaction it started. >> Consequently, each message is delivered to a destination within a >> separate >> transaction (in contrast to the batch resequencer where several messages >> can >> be delivered within a single transaction) but the resequencing is >> reliable >> too. >> > > What I understand seems to be similar to the thoughts I have in mind about > how it could work. I'll try to explain in my own words: > > MessageExchanges are consumed by the resequencer in n threads. Each thread > is stopped until the predecessor of this message exchange has been > delivered to the final consumer or the configured timeout is over. In the > first case the MessageExchange is sent in the same thread it came from to > the consumer. After receiving the response from there this will be sent > back to the originating message producer. > > Or shorter: this is just synchronous communication which is stopped (-> > messageExchange.wait(timeout)) - until a required precondition is met -> > messageExchange.notify(). This way there should be no problem of > propagating a transaction context or an error state. > > One further question: what should happen, if delivery of a MessageExchange > to the final consumer fails - should all pending MessageExchanges be > rejected or delivered? Maybe several strategies should be provided. > -- View this message in context: http://www.nabble.com/Reliable-resequencing-tp19331003s22882p19737595.html Sent from the Camel - Users mailing list archive at Nabble.com.
