I have added a TODO in the camel code to take a look at when time permits later about the handled property.
On Tue, Sep 8, 2009 at 9:07 AM, Claus Ibsen<[email protected]> wrote: > On Mon, Sep 7, 2009 at 9:56 PM, thabach<[email protected]> wrote: >> >> Hi Claus >> >> I guess I found the root of the problem now. >> >> It is in my first segment's recovery logic. All my producer (the >> CustomDirectProducer) does, in recovering from an IllegalStateException >> thrown by a seda queue running full, is the following: >> >> exchange.setException(null); >> >> As I understand now, this seems not being sufficient, as the recovered >> exchanges on consumption from the seda queues in the second segment, to my >> surprise, do have the Exchange.FAILURE_HANDLED property set to true. The >> property happens to be originally set by the DefaultErrorHandler in the >> first segment on the 'queue full' IllegalStateException propagation. >> > > Yeah you need to do a bit more work to clear traces of exceptions. The > code below does clear all traces. > > exchange.setException(null); > > exchange.setProperty(Exchange.ERRORHANDLER_HANDLED, null); > exchange.setProperty(Exchange.FAILURE_HANDLED, null); > exchange.setProperty(Exchange.EXCEPTION_CAUGHT, null); > exchange.getIn().removeHeader(Exchange.REDELIVERED); > exchange.getIn().removeHeader(Exchange.REDELIVERY_COUNTER); > > I think Camel should have a helper method for that you can invoke, such as: > ExchangeHelper.clearException(exchange); > > > >> This becomes fatal in the second segment of my routes, as the >> DeadLetterChannel in its isDone()-method considers the failed exchange as >> being handled already and does not forward the Exchange to its deadletter >> queue and the messages are 'lost'. >> >> I am left with two questions then :) , these are: >> >> - Shouldn't the DefaultErrorHandler as described >> http://camel.apache.org/defaulterrorhandler.html here refrain from marking >> the exchange as being handled, i.e. leave it as unhandled ? >> >> - How do I properly recover an exchange from failure without creating a new >> Exchange ? Is exchange.setException(null); and an additional >> exchange.getProperties().clear(); sufficient ? >> >> Thanks for all your help and for supporting me digging deep, Christian. >> >> >> >> Claus Ibsen-2 wrote: >>> >>> Hi >>> >>> You test case is kinda bogus as the route is triggered by a file >>> consumer, which splits the file and route from there. >>> >>> So if the caller in this case is the file consumer, how do you suppose >>> the default error handler should "handle" this? >>> It does this by rolling back the file and try the file again on next poll. >>> >>> So try with another kind of caller, for example try the dataset >>> endpoint in camel that can be used for sending in loads of messages >>> and verify they are routed as expected. >>> >>> >>> <route> >>> <from uri="file://inputdir?noop=true" /> >>> <unmarshal> >>> <string /> >>> </unmarshal> >>> <split> >>> <tokenize token="\n" /> >>> <to uri="customDirect:messageRouter" /> >>> </split> >>> </route> >>> >>> On Mon, Sep 7, 2009 at 1:15 PM, thabach<[email protected]> wrote: >>>> >>>> I think we might suffer from some misunderstanding. >>>> >>>> - For the first segment, I do have a custom error handler configured to >>>> redeliver failed messages to the seda queues, we are on the same page. >>>> >>>> - In the second segment, in the DeadLetterChannelTest, I do use a >>>> DeadLetterChannel which does not, for some reason, deliver all messages. >>>> Whereas in the ExceptionBarrierTest, I have another custom error handler >>>> installed right after the seda queue, an ExceptionBarrier endpoint, which >>>> simply forwards erroneous exchanges to the deadletter queue and leaves no >>>> message behind. This second test does not use the DeadLetterChannel, but >>>> rather shall illustrate that the general setup is not flawed. >>>> >>>> In both test cases the DeadLetterChannel and the ExceptionBarrier >>>> respectively, are sitting right before the ExceptionThrower, which in >>>> both >>>> test cases deterministically throws 25 exceptional messages. Somehow in >>>> the >>>> DeadLetterChannelTest case, depending on the run only some 38 to 42 >>>> messages >>>> out of 45 make it to the deadletter queue. >>>> >>>> Christian. >>>> >>>> >>>> >>>> Claus Ibsen-2 wrote: >>>>> >>>>> On Mon, Sep 7, 2009 at 10:28 AM, thabach<[email protected]> wrote: >>>>>> >>>>>> Hi Claus >>>>>> >>>>>> thanks for having a look, I appreciate. >>>>>> >>>>>> I don't think it is a misconfiguration though. On the first segment to >>>>>> the >>>>>> seda queues I do want the default error handler to return the >>>>>> 'exception'al-exchanges (i.e. the exchanges suffering from a 'Queue >>>>>> full' >>>>>> exception), to the producer. Theses messages are not supposed to end up >>>>>> in >>>>>> the deadletter queue, but to be recovered by the producer >>>>>> (CustomDirectProducer in the original unit tests, posted) and resent to >>>>>> the >>>>>> seda queues. >>>>>> >>>>> >>>>> Ah okay that is fine. Then we are on same page now. If you do *not* >>>>> use the DeadLetterChannel then its your own responsible for handling >>>>> errors. >>>>> And whether you build you own custom component to do redelivery is your >>>>> choice. >>>>> >>>>> >>>>> >>>>> >>>>>> With these 'blocking' producer semantics installed, all messages make >>>>>> it >>>>>> to >>>>>> the second segment (i.e. the routes consuming from the seda queues). In >>>>>> the >>>>>> original unit tests the ExceptionThrower always throws on the expected >>>>>> 25 >>>>>> messages, but not all of these 'exceptional' messages make it to the >>>>>> deadletter queue when using a DeadLetterChannel >>>>>> (DeadLetterChannelTest), >>>>>> whereas in the ExceptionBarrierTest - using our custom ExceptionBarrier >>>>>> component - they do. >>>>>> >>>>>> I am still under the impression of losing messages and don't get me >>>>>> wrong, I >>>>>> am happy to be told that it was just me (and my configuration) :) . >>>>>> >>>>> >>>>> No Camel does not lose messages. Its your own choice to do redelivery >>>>> and error handling in your own custom component. >>>>> >>>>> >>>>>> Cheers, Christian. >>>>>> >>>>>> >>>>>> Claus Ibsen-2 wrote: >>>>>>> >>>>>>> Hi >>>>>>> >>>>>>> I had a look at it. >>>>>>> There are *no* issues in Camel. DeadLetterChannel does *not* loose >>>>>>> messages. >>>>>>> >>>>>>> The problem Christian has is that he has not properly configured error >>>>>>> handling in Camel. >>>>>>> He have mixed error handlers >>>>>>> - default error handler >>>>>>> - dead letter channel error handler >>>>>>> >>>>>>> And since when he sends to the SEDA queue from a producer that are >>>>>>> handled by the default error handler, >>>>>>> then his message is *not* moved to a dead letter queue, but returned >>>>>>> back to the caller as an exception. (as it used the default error >>>>>>> handler) >>>>>>> >>>>>>> Configuring Camel to use dead letter channel error handler for the >>>>>>> entire routes remedy Christians problem. >>>>>>> >>>>>>> I have added an unit test to Camel at: >>>>>>> http://svn.apache.org/viewvc?rev=811997&view=rev >>>>>>> >>>>>>> >>>>>>> On Fri, Sep 4, 2009 at 9:04 AM, Bach >>>>>>> Christian<[email protected]> wrote: >>>>>>>> Heya Camel Riders >>>>>>>> >>>>>>>> I stumbled upon a problem on having DeadLetterChannel doing the error >>>>>>>> handling in an staged architecture (all in-VM, non-distributed). Our >>>>>>>> batch processing application (uses Camel 2.0 and) is losing messages. >>>>>>>> >>>>>>>> It seems to only happen under peak loads, when SEDA queues become >>>>>>>> full >>>>>>>> and start to throw IllegalStateExceptions on performing the add()s to >>>>>>>> their internal BlockingQueues. Our producer recovers those exchanges >>>>>>>> and >>>>>>>> yield()s, before re-sending the Exchanges to the SEDA endpoint. >>>>>>>> >>>>>>>> I have put together two unit tests that reproducibly illustrate the >>>>>>>> problem and the absence thereof. Both unit tests are similar expect >>>>>>>> for >>>>>>>> the dead letter queue exception handling implementation used. Both >>>>>>>> have >>>>>>>> the SEDA queues, facing stages, configured to size=1 in order to >>>>>>>> emulate >>>>>>>> peak load semantics. >>>>>>>> >>>>>>>> The first test case (DeadLetterChannelTest) is configured to use >>>>>>>> Camel's >>>>>>>> DeadLetterChannel and reproducibly loses messages, depending on the >>>>>>>> run >>>>>>>> between 5 to 15 (out of 45) on my machine. >>>>>>>> >>>>>>>> The second test case (ExceptionBarrierTest) is configured to use a >>>>>>>> custom ExceptionBarrier endpoint in place of DeadLetterChannel. The >>>>>>>> ExceptionBarrierComponent is derived from a Camel DirectComponent >>>>>>>> with >>>>>>>> some custom code to handle Exchanges suffering from exceptions and >>>>>>>> forwarding the erroneous exchanges to the dead letter queue. With the >>>>>>>> ExceptionBarrier, which takes in its simplistic implementation >>>>>>>> advantage >>>>>>>> of our specific routes to some extent, no messages are lost. >>>>>>>> >>>>>>>> The unit test are attached and I'd be most interested in learning >>>>>>>> what >>>>>>>> is going wrong. >>>>>>>> >>>>>>>> Cheers, Christian. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Claus Ibsen >>>>>>> Apache Camel Committer >>>>>>> >>>>>>> Open Source Integration: http://fusesource.com >>>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>>> Twitter: http://twitter.com/davsclaus >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238p25326799.html >>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Claus Ibsen >>>>> Apache Camel Committer >>>>> >>>>> Open Source Integration: http://fusesource.com >>>>> Blog: http://davsclaus.blogspot.com/ >>>>> Twitter: http://twitter.com/davsclaus >>>>> >>>>> >>>> >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238p25328848.html >>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>> >>>> >>> >>> >>> >>> -- >>> Claus Ibsen >>> Apache Camel Committer >>> >>> Open Source Integration: http://fusesource.com >>> Blog: http://davsclaus.blogspot.com/ >>> Twitter: http://twitter.com/davsclaus >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238p25335523.html >> Sent from the Camel - Users mailing list archive at Nabble.com. >> >> > > > > -- > Claus Ibsen > Apache Camel Committer > > Open Source Integration: http://fusesource.com > Blog: http://davsclaus.blogspot.com/ > Twitter: http://twitter.com/davsclaus > -- Claus Ibsen Apache Camel Committer Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/ Twitter: http://twitter.com/davsclaus
