I've added: queue.dequeuebatchsize: 1 to action confuguration. Still it seems I can't make rsyslog to retry messages. Btw, may be the problem is that I return RS_RET_DEFER_COMMIT from DoAction and RS_RET_SUSPENDED from EndTransaction. It seems that RS_RET_SUSPENDED is handled the same as RS_RET_OK if returned in EndTransaction?
On Tue, Feb 23, 2016 at 8:49 AM, Rainer Gerhards <[email protected]> wrote: > 2016-02-23 17:38 GMT+01:00 Kane Kim <[email protected]>: > > Hello Rainer, thanks for the prompt reply! To give you some context: I > want > > to write module that both using batching and also can't loose messages in > > any circumstances. Are you saying it is by design that rsyslog can't do > > that together? According to documentation rsyslog will retry if module > > returns any error. Do you plan to fix this in rsyslog or update > > documentation to say batching and retries don't work? > > It depends on many things. In almost all cases, the retry should work > well (and does so in practice). Unfortunately, I am pretty swamped. I > need to go to a conference tomorrow and have had quite some unexpected > work today. It would probably be good if you could ping me next week > to see if we can look into more details what is causing you pain. But > I can't guarantee that I will be available early next week. > > In general, we cannot handle a fatal error here from an engine PoV, > because everything is already processed and we do no longer have the > original messages. This is simply needed if you want to process > messages one after another through the full config (a goal for v8 that > was muuuuch requested). As I said, the solution is to use batches of > one, because otherwise we would really need to turn back time and undo > everything that was already done on the messages in question by other > modules (including state advances). > > Rainer > > > > > What would be your recommendation for my case? Is there any workaround to > > have batches and retries? > > > > Thanks! > > On Feb 22, 2016 11:03 PM, "Rainer Gerhards" <[email protected]> > > wrote: > > > >> 2016-02-23 0:08 GMT+01:00 Kane Kim <[email protected]>: > >> > >> > I return RS_RET_SUSPENDED from EndTransaction and expect rsyslog to > retry > >> > failed messages. My configuration looks like following: > >> > > >> > queue.spoolDirectory="/var/spool/rsyslog" > queue.filename="some_filename" > >> > queue.type="Disk" queue.maxdiskspace="1G" queue.saveonshutdown="on" > >> > action.resumeRetryCount="-1" action.reportSuspensionContinuation="on" > >> > action.resumeInterval="1" > >> > > >> > After logging a line: > >> > > >> > 6130.843876518:action 0 queue:Reg/w0: Action 0 transitioned to state: > rdy > >> > 6130.844112940:action 0 queue:Reg/w0: Action 0 transitioned to state: > itx > >> > 6130.844131796:action 0 queue:Reg/w0: Action 0 transitioned to state: > >> rtry > >> > 6130.844137106:action 0 queue:Reg/w0: actionCommit, in retry loop, > iRet > >> > -2007 > >> > 6130.844142731:action 0 queue:Reg/w0: regular consumer finished, > >> > iret=-2007, szlog 0 sz phys 1 > >> > 6130.844149164:action 0 queue:Reg/w0: DeleteProcessedBatch: we deleted > >> > 1 objects and enqueued 0 objects > >> > 6130.844154698:action 0 queue:Reg/w0: doDeleteBatch: delete batch from > >> > store, new sizes: log 0, phys 0 > >> > 6130.844160252:action 0 queue:Reg/w0: regular consumer finished, > >> > iret=4, szlog 0 sz phys 0 > >> > 6130.844165414:action 0 queue:Reg/w0: action 0 queue:Reg/w0: worker > >> > IDLE, waiting for work. > >> > > >> > That message is not resent by rsyslog after failure. Is there special > >> > configuration? > >> > > >> > I've found TODO here: > >> > https://github.com/rsyslog/rsyslog/blob/master/action.c#L1236 > >> > That says "For the time being, I do NOT implement all of this (not > even > >> > retry!)". I Wonder if retry functionality is not finished in rsyslog? > >> > > >> > > >> That comment is no longer really true, as many things have been changed > on > >> the upper level. However, what is true is that we should write an error > >> file. This actually slipped through. It is right that we cannot do a > retry > >> if we used the batch commit feature, because we do no longer have all > the > >> data in place. This was a tradeoff we had to make when moving to the v8 > >> engine. You can configure a batch size of 1, though, if you need to have > >> retry under all cases. > >> > >> Rainer > >> > >> > Thanks! > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> > DON'T LIKE THAT. > >> > > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
