2017-08-23 1:13 GMT+02:00 Thomas Deutschmann via rsyslog <rsyslog@lists.adiscon.com>: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On 2017-08-21 13:21, Andreas Wehrmann via rsyslog wrote: >>> actionProcessMessage() always returns RS_RET_ACTION_FAILED (which is >>> -2123), >>> meaning i will be decremented for re-submit and the loop starts all >>> over again. > > But this is not the problem itself. It *should* retry. See > action.resumeRetryCount and action.resumeInterval > documentation [1]. > > >> So I ultimately traced it down to this change: >> https://github.com/rsyslog/rsyslog/commit/128214fffac7dcec69b5c8dffdb8222bbd29af27 >> >> >> Reverting this makes everything seem to work as expected, >> though it probably introduces the bug it was supposed to fix... > > This should only trigger the problem, not causing the problem. > > > My current theory: _No_ msg can be delivered. Even internal messages > nowadays [2]. This will eat up your memory until rsyslogd segfaults:
Note that the change done in [2] did not modify that. Before, messages always went to the local queue. With the change, this is not necessarily the case, because they go to the system logger and bypass the queue. They only get to the queue if the system logger is bound to rsyslog. > > For example, when rsyslogd launches it will already create a message > like > >> rsyslogd: [origin software="rsyslogd" swVersion="8.29.0" x-pid="31213" >> x-info="http://www.rsyslog.com";] start > > Now this message has to go through your queue. Once it reaches > your omrelp action which is failing, rsyslogd will generate a > >> rsyslogd: action 'action 1' suspended, next retry is <date> > > message. This message will also go through your queue and be > processed by the still failing omrelp action (which will be skipped > like you can see in your logs: next retry (if applicable): > 966135869 [now 966135839]). At this point, the action should be suspended. By default, this means the message will not be submitted to the failing action, and so it will be processed and discarded. No queue buildup. Of course, if the configuration says that messages shall be queued until delivery is possible, then it goes to the queue. But the "suspended" message will not be rememitted in this case. > > Due to the small amount of memory you just need a small bunch of > message to run out of memory. > > And don't forget about the main queue [3]. I agree on the main point: if the system has very little memory, even default queue settings might be too large. So they would need to be decreased. > > > Do you get some core dumps when rsyslogd segfaults? Maybe you have to > enable this first on your system. > > Or could you rsyslogd through gdb/valgrind? > > What's about your dmesg? Any messages indicating OOM killer activity? > All very good suggestions! Rainer > > See also: > ========= > [1] http://www.rsyslog.com/doc/v8-stable/configuration/actions.html > > [2] http://www.rsyslog.com/rsyslog-error-reporting-improved/ > > [3] http://www.rsyslog.com/doc/v8-stable/concepts/queues.html > > > - -- > Regards, > Thomas > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0 > > iQJ8BAEBCgBmBQJZnLq0XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w > ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQzM0M1ODQ4MkM0MDIyOTJEMkUzQzVDMDY5 > NzA5RjkwQzNDOTZGRkM4AAoJEJcJ+Qw8lv/IGCQP/3rqvkkXxT2RL++kKawicJWc > DanVorfO+wl94CQScTQZIf+p+DPb0H7QaPh6GvzHN5VO13546+eQgQRn7ndO0ZBR > 29qXBf3l2jozaa0xyg2FDmGKrm8d1chzrYutbpWDQCpTlCdXWJYfZOT7SYvGj6LO > Kvuj/pifz3r6DoV6luZIBAde9IcCyGe/JSvzbyEUHWB2jcVdRQShOGt7mFUYlBB6 > YFW+CxaEXsC9Kzu7gHWxf6XtB1duNP0l9m1zL2xu4KtU8R1DVxKQeIvIk34JCNsS > 9ng5A2e52/5vBeAHw4lgCXUbuNOxtJHtGEwqyE3Re0dgHqd347CqtKY7vx/mM76I > +aXJjzPt4/qgj0t0mrLb/7YVr5tNSoK91aZaSvPLyb4nHMwAjUsjuYMjvfjgXxic > 6GQ5m6y+bGLKDDXLi14DVMO7zO2Jv2WQNEvv7NQVTSg0LMv1NIUpCNmORIlLbpyJ > H6LQJtv70e0jNNOOdWrAI6hNkArEKkUiT5fEpUGUUWywY9spKHr3m2iNC4zs8cNp > Bm0uDTV9tVybb58+6paVUsAM+sMrxPBQ+rWvi1C2eLmb6VUaC97OPl0LC0MLPJh5 > Eo5V6fI2llsVyjBc+tO/1H/HgusBaynNcwxVy1df9O1Az1ELFOyZFiHxYaO8EEG0 > Y/Hx7AtzsBD0yDFjhZjc > =hon9 > -----END PGP SIGNATURE----- > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
