Hello David, I'll try to create a repro with standard module, thanks for
checking back
On Mar 12, 2016 3:46 PM, "David Lang" <[email protected]> wrote:
> now that 8.17 is out and the high profile bugs found in the last couple of
> weeks have been dealt with, let's look at this again.
>
> Rainer, can you explain what happens when a batch fails on an action?
>
> Kane, can you try and duplicate your problem using standard modules? I'm
> thinking that if you define a omrelp action that doesn't go anywhere, it
> should return errors similar to what you are trying to do, so you should be
> able to duplicate the problem that way.
>
> David Lang
>
>
>
> On Tue, 23 Feb 2016, Kane Kim wrote:
>
> Looking at omkafka module source code it seems that it relies on rsyslog
>> retries in DoAction, returning RS_RET_SUSPENDED:
>>
>> DBGPRINTF("omkafka: writeKafka returned %d\n", iRet);
>> if(iRet != RS_RET_OK) {
>> iRet = RS_RET_SUSPENDED;
>> }
>>
>> I've tried similar code in DoAction, it seems that action processing
>> is suspended at that point and messages are not retried to module.
>>
>>
>> On Tue, Feb 23, 2016 at 9:06 AM, Kane Kim <[email protected]> wrote:
>>
>> Thanks for your help Rainer, I'll try to debug what's going on, so far it
>>> seems rsyslog doesn't retry even with batch size 1. It doesn't retry if I
>>> return RS_RET_SUSPENDED from DoAction as well.
>>>
>>> On Tue, Feb 23, 2016 at 9:05 AM, Rainer Gerhards <
>>> [email protected]
>>>
>>>> wrote:
>>>>
>>>
>>> 2016-02-23 18:03 GMT+01:00 David Lang <[email protected]>:
>>>>
>>>>> On Tue, 23 Feb 2016, Rainer Gerhards wrote:
>>>>>
>>>>> 2016-02-23 17:38 GMT+01:00 Kane Kim <[email protected]>:
>>>>>>
>>>>>>>
>>>>>>> Hello Rainer, thanks for the prompt reply! To give you some context:
>>>>>>> I
>>>>>>> want
>>>>>>> to write module that both using batching and also can't loose
>>>>>>>
>>>>>> messages in
>>>>
>>>>> any circumstances. Are you saying it is by design that rsyslog can't
>>>>>>>
>>>>>> do
>>>>
>>>>> that together? According to documentation rsyslog will retry if module
>>>>>>> returns any error. Do you plan to fix this in rsyslog or update
>>>>>>> documentation to say batching and retries don't work?
>>>>>>>
>>>>>>
>>>>>>
>>>>>> It depends on many things. In almost all cases, the retry should work
>>>>>> well (and does so in practice). Unfortunately, I am pretty swamped. I
>>>>>> need to go to a conference tomorrow and have had quite some unexpected
>>>>>> work today. It would probably be good if you could ping me next week
>>>>>> to see if we can look into more details what is causing you pain. But
>>>>>> I can't guarantee that I will be available early next week.
>>>>>>
>>>>>> In general, we cannot handle a fatal error here from an engine PoV,
>>>>>> because everything is already processed and we do no longer have the
>>>>>> original messages. This is simply needed if you want to process
>>>>>> messages one after another through the full config (a goal for v8 that
>>>>>> was muuuuch requested). As I said, the solution is to use batches of
>>>>>> one, because otherwise we would really need to turn back time and undo
>>>>>> everything that was already done on the messages in question by other
>>>>>> modules (including state advances).
>>>>>>
>>>>>
>>>>>
>>>>> I thought that if a batch failed, it pushed all the messages back on
>>>>> the
>>>>> queue and retried with a half size batch until it got to the one
>>>>> message
>>>>> that could not be processed and only did a fatal fail on that message.
>>>>>
>>>>> Now, there is a big difference between a module giving a hard error
>>>>>
>>>> "this
>>>>
>>>>> message is never going to be able to be processed no matter how many
>>>>>
>>>> times
>>>>
>>>>> it's retried" vs a soft error "there is a problem delivering things to
>>>>>
>>>> this
>>>>
>>>>> destination right now, retry later". I thought the batch processing
>>>>>
>>>> handled
>>>>
>>>>> these differently.
>>>>>
>>>>
>>>> That's no longer possible with v8, at least generically. As I said
>>>> above, we would need to turn back time.
>>>>
>>>> But I really run out of time now...
>>>>
>>>> Rainer
>>>>
>>>>>
>>>>> David Lang
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>
>>>> myriad of
>>>>
>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>>
>>>> DON'T
>>>>
>>>>> LIKE THAT.
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>>>
>>>
>>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
