On Mon, Sep 7, 2009 at 9:56 PM, thabach<[email protected]> wrote:
>
> Hi Claus
>
> I guess I found the root of the problem now.
>
> It is in my first segment's recovery logic. All my producer (the
> CustomDirectProducer) does, in recovering from an IllegalStateException
> thrown by a seda queue running full, is the following:
>
> exchange.setException(null);
>
> As I understand now, this seems not being sufficient, as the recovered
> exchanges on consumption from the seda queues in the second segment, to my
> surprise, do have the Exchange.FAILURE_HANDLED property set to true. The
> property happens to be originally set by the DefaultErrorHandler in the
> first segment on the 'queue full' IllegalStateException propagation.
>
Yeah you need to do a bit more work to clear traces of exceptions. The
code below does clear all traces.
exchange.setException(null);
exchange.setProperty(Exchange.ERRORHANDLER_HANDLED, null);
exchange.setProperty(Exchange.FAILURE_HANDLED, null);
exchange.setProperty(Exchange.EXCEPTION_CAUGHT, null);
exchange.getIn().removeHeader(Exchange.REDELIVERED);
exchange.getIn().removeHeader(Exchange.REDELIVERY_COUNTER);
I think Camel should have a helper method for that you can invoke, such as:
ExchangeHelper.clearException(exchange);
> This becomes fatal in the second segment of my routes, as the
> DeadLetterChannel in its isDone()-method considers the failed exchange as
> being handled already and does not forward the Exchange to its deadletter
> queue and the messages are 'lost'.
>
> I am left with two questions then :) , these are:
>
> - Shouldn't the DefaultErrorHandler as described
> http://camel.apache.org/defaulterrorhandler.html here refrain from marking
> the exchange as being handled, i.e. leave it as unhandled ?
>
> - How do I properly recover an exchange from failure without creating a new
> Exchange ? Is exchange.setException(null); and an additional
> exchange.getProperties().clear(); sufficient ?
>
> Thanks for all your help and for supporting me digging deep, Christian.
>
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> You test case is kinda bogus as the route is triggered by a file
>> consumer, which splits the file and route from there.
>>
>> So if the caller in this case is the file consumer, how do you suppose
>> the default error handler should "handle" this?
>> It does this by rolling back the file and try the file again on next poll.
>>
>> So try with another kind of caller, for example try the dataset
>> endpoint in camel that can be used for sending in loads of messages
>> and verify they are routed as expected.
>>
>>
>> <route>
>> <from uri="file://inputdir?noop=true" />
>> <unmarshal>
>> <string />
>> </unmarshal>
>> <split>
>> <tokenize token="\n" />
>> <to uri="customDirect:messageRouter" />
>> </split>
>> </route>
>>
>> On Mon, Sep 7, 2009 at 1:15 PM, thabach<[email protected]> wrote:
>>>
>>> I think we might suffer from some misunderstanding.
>>>
>>> - For the first segment, I do have a custom error handler configured to
>>> redeliver failed messages to the seda queues, we are on the same page.
>>>
>>> - In the second segment, in the DeadLetterChannelTest, I do use a
>>> DeadLetterChannel which does not, for some reason, deliver all messages.
>>> Whereas in the ExceptionBarrierTest, I have another custom error handler
>>> installed right after the seda queue, an ExceptionBarrier endpoint, which
>>> simply forwards erroneous exchanges to the deadletter queue and leaves no
>>> message behind. This second test does not use the DeadLetterChannel, but
>>> rather shall illustrate that the general setup is not flawed.
>>>
>>> In both test cases the DeadLetterChannel and the ExceptionBarrier
>>> respectively, are sitting right before the ExceptionThrower, which in
>>> both
>>> test cases deterministically throws 25 exceptional messages. Somehow in
>>> the
>>> DeadLetterChannelTest case, depending on the run only some 38 to 42
>>> messages
>>> out of 45 make it to the deadletter queue.
>>>
>>> Christian.
>>>
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Mon, Sep 7, 2009 at 10:28 AM, thabach<[email protected]> wrote:
>>>>>
>>>>> Hi Claus
>>>>>
>>>>> thanks for having a look, I appreciate.
>>>>>
>>>>> I don't think it is a misconfiguration though. On the first segment to
>>>>> the
>>>>> seda queues I do want the default error handler to return the
>>>>> 'exception'al-exchanges (i.e. the exchanges suffering from a 'Queue
>>>>> full'
>>>>> exception), to the producer. Theses messages are not supposed to end up
>>>>> in
>>>>> the deadletter queue, but to be recovered by the producer
>>>>> (CustomDirectProducer in the original unit tests, posted) and resent to
>>>>> the
>>>>> seda queues.
>>>>>
>>>>
>>>> Ah okay that is fine. Then we are on same page now. If you do *not*
>>>> use the DeadLetterChannel then its your own responsible for handling
>>>> errors.
>>>> And whether you build you own custom component to do redelivery is your
>>>> choice.
>>>>
>>>>
>>>>
>>>>
>>>>> With these 'blocking' producer semantics installed, all messages make
>>>>> it
>>>>> to
>>>>> the second segment (i.e. the routes consuming from the seda queues). In
>>>>> the
>>>>> original unit tests the ExceptionThrower always throws on the expected
>>>>> 25
>>>>> messages, but not all of these 'exceptional' messages make it to the
>>>>> deadletter queue when using a DeadLetterChannel
>>>>> (DeadLetterChannelTest),
>>>>> whereas in the ExceptionBarrierTest - using our custom ExceptionBarrier
>>>>> component - they do.
>>>>>
>>>>> I am still under the impression of losing messages and don't get me
>>>>> wrong, I
>>>>> am happy to be told that it was just me (and my configuration) :) .
>>>>>
>>>>
>>>> No Camel does not lose messages. Its your own choice to do redelivery
>>>> and error handling in your own custom component.
>>>>
>>>>
>>>>> Cheers, Christian.
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I had a look at it.
>>>>>> There are *no* issues in Camel. DeadLetterChannel does *not* loose
>>>>>> messages.
>>>>>>
>>>>>> The problem Christian has is that he has not properly configured error
>>>>>> handling in Camel.
>>>>>> He have mixed error handlers
>>>>>> - default error handler
>>>>>> - dead letter channel error handler
>>>>>>
>>>>>> And since when he sends to the SEDA queue from a producer that are
>>>>>> handled by the default error handler,
>>>>>> then his message is *not* moved to a dead letter queue, but returned
>>>>>> back to the caller as an exception. (as it used the default error
>>>>>> handler)
>>>>>>
>>>>>> Configuring Camel to use dead letter channel error handler for the
>>>>>> entire routes remedy Christians problem.
>>>>>>
>>>>>> I have added an unit test to Camel at:
>>>>>> http://svn.apache.org/viewvc?rev=811997&view=rev
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 4, 2009 at 9:04 AM, Bach
>>>>>> Christian<[email protected]> wrote:
>>>>>>> Heya Camel Riders
>>>>>>>
>>>>>>> I stumbled upon a problem on having DeadLetterChannel doing the error
>>>>>>> handling in an staged architecture (all in-VM, non-distributed). Our
>>>>>>> batch processing application (uses Camel 2.0 and) is losing messages.
>>>>>>>
>>>>>>> It seems to only happen under peak loads, when SEDA queues become
>>>>>>> full
>>>>>>> and start to throw IllegalStateExceptions on performing the add()s to
>>>>>>> their internal BlockingQueues. Our producer recovers those exchanges
>>>>>>> and
>>>>>>> yield()s, before re-sending the Exchanges to the SEDA endpoint.
>>>>>>>
>>>>>>> I have put together two unit tests that reproducibly illustrate the
>>>>>>> problem and the absence thereof. Both unit tests are similar expect
>>>>>>> for
>>>>>>> the dead letter queue exception handling implementation used. Both
>>>>>>> have
>>>>>>> the SEDA queues, facing stages, configured to size=1 in order to
>>>>>>> emulate
>>>>>>> peak load semantics.
>>>>>>>
>>>>>>> The first test case (DeadLetterChannelTest) is configured to use
>>>>>>> Camel's
>>>>>>> DeadLetterChannel and reproducibly loses messages, depending on the
>>>>>>> run
>>>>>>> between 5 to 15 (out of 45) on my machine.
>>>>>>>
>>>>>>> The second test case (ExceptionBarrierTest) is configured to use a
>>>>>>> custom ExceptionBarrier endpoint in place of DeadLetterChannel. The
>>>>>>> ExceptionBarrierComponent is derived from a Camel DirectComponent
>>>>>>> with
>>>>>>> some custom code to handle Exchanges suffering from exceptions and
>>>>>>> forwarding the erroneous exchanges to the dead letter queue. With the
>>>>>>> ExceptionBarrier, which takes in its simplistic implementation
>>>>>>> advantage
>>>>>>> of our specific routes to some extent, no messages are lost.
>>>>>>>
>>>>>>> The unit test are attached and I'd be most interested in learning
>>>>>>> what
>>>>>>> is going wrong.
>>>>>>>
>>>>>>> Cheers, Christian.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238p25326799.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238p25328848.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238p25335523.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
