2015-11-12 9:15 GMT+01:00 Radu Gheorghe <[email protected]>: > Hi Jörgen, > > This thread comes up on all searches for "perseverance" :D, that's really > nice. > > But this finding is really interesting. I remember David saying that > having just one config is safer and I also prefer that for simplicity, > but I didn't expect bugs to appear with multiple confs. Maybe the > order of statements is mixed up with includes? I would expect it to > work as if you inserted the included config to the main rsyslog.conf > at the location of the include statement.
yup, you can think of it of a copy and paste of the included file into the position at the main file. Ages ago, there was a version that did include wildcards in the wrong order, but that's long history. This substitution happens very early in config parsing, so the main config engine doesn't even know if things are coming from includes or the main file. > Though it sounds more like a > bug, I don't see a clear required order (other than loading modules > first, then defining inputs/actions). > Depends on the config. If legacy is used, order is very important. Rainer > Best regards, > Radu > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > > On Thu, Nov 12, 2015 at 9:14 AM, Jörgen Maas <[email protected]> wrote: >> More progress on this: >> >> When i move all rsyslog configuration into a single file all problems >> (mentioned in this thread) are gone! It seems that splitting functionality >> in seperate files is not working 100% as expected. >> >> Is this a known limitation/issue? >> >> Regards, >> Jörgen >> >> On Wed, Nov 11, 2015 at 2:00 PM, Jörgen Maas <[email protected]> wrote: >> >>> Hi all, >>> >>> I've verified the situation with the latest 8.14 release and the official >>> packages, and i can confirm that the issue is still present. Including the >>> segmentationfault when using rsyslogd -N3. I'll file a bug report on GitHub. >>> >>> Cheers, >>> Jörgen >>> >>> >>> >>> On Fri, Nov 6, 2015 at 3:51 PM, Radu Gheorghe <[email protected]> >>> wrote: >>> >>>> Hi Jörgen, >>>> >>>> Yes, this is really weird. Can you come up with a complete >>>> reproduction that you'd paste in a GitHub issue? >>>> >>>> Best regards, >>>> Radu >>>> -- >>>> Performance Monitoring * Log Analytics * Search Analytics >>>> Solr & Elasticsearch Support * http://sematext.com/ >>>> >>>> >>>> On Fri, Nov 6, 2015 at 1:43 PM, Jörgen Maas <[email protected]> >>>> wrote: >>>> > Hi Radu, >>>> > >>>> > First with only the new syntax: >>>> > >>>> > [root@logmanagement-client:/etc/rsyslog.d]# cat forward_syslog_tls.conf >>>> > # >>>> > # forward - syslog / tcp+tls >>>> > # >>>> > >>>> > >>>> > # testing shows that we need both the legacy and new style options >>>> > # when deleting options errors messages and even rsyslogd crashes occur >>>> > >>>> > #$ActionSendStreamDriver gtls >>>> > #$ActionSendStreamDriverMode 1 >>>> > #$ActionSendStreamDriverAuthMode x509/name >>>> > #$ActionSendStreamDriverPermittedPeers "logmanagement.xxx.yy" >>>> > >>>> > action( >>>> > type="omfwd" >>>> > target="192.168.124.100" >>>> > port="6514" >>>> > protocol="tcp" >>>> > template="RSYSLOG_SyslogProtocol23Format" >>>> > StreamDriver="gtls" >>>> > StreamDriverMode="1" >>>> > StreamDriverAuthMode="x509/name" >>>> > StreamDriverPermittedPeers="logmanagement.xx.yy" >>>> > ) >>>> > >>>> > # EOF >>>> > >>>> > [root@logmanagement-client:/etc/rsyslog.d]# rsyslogd -N3 >>>> > rsyslogd: version 7.4.7, config validation run (level 3), master config >>>> > /etc/rsyslog.conf >>>> > Segmentation fault >>>> > >>>> > [root@logmanagement-client:/etc/rsyslog.d]# systemctl restart rsyslog >>>> > Job for rsyslog.service failed. See 'systemctl status rsyslog.service' >>>> and >>>> > 'journalctl -xn' for details. >>>> > >>>> > [root@logmanagement-client:/etc/rsyslog.d]# tail /var/log/syslog |grep >>>> > rsyslogd >>>> > Nov 6 12:30:52 logmanagement-client kernel: traps: rsyslogd[2192] >>>> general >>>> > protection ip:7fdab462c4bd sp:7ffd52d30a50 error:0 in libc-2.17.so >>>> > [7fdab45b0000+1b6000] >>>> > Nov 6 12:31:11 logmanagement-client rsyslogd: [origin >>>> software="rsyslogd" >>>> > swVersion="7.4.7" x-pid="577" x-info="http://www.rsyslog.com";] exiting >>>> on >>>> > signal 15. >>>> > >>>> > >>>> > >>>> > Now with only the old syntax: >>>> > >>>> > [root@logmanagement-client:/etc/rsyslog.d]# cat forward_syslog_tls.conf >>>> > # >>>> > # forward - syslog / tcp+tls >>>> > # >>>> > >>>> > >>>> > # testing shows that we need both the legacy and new style options >>>> > # when deleting options errors messages and even rsyslogd crashes occur >>>> > >>>> > $ActionSendStreamDriver gtls >>>> > $ActionSendStreamDriverMode 1 >>>> > $ActionSendStreamDriverAuthMode x509/name >>>> > $ActionSendStreamDriverPermittedPeers "logmanagement.xxx.yy" >>>> > >>>> > action( >>>> > type="omfwd" >>>> > target="192.168.124.100" >>>> > port="6514" >>>> > protocol="tcp" >>>> > template="RSYSLOG_SyslogProtocol23Format" >>>> > #StreamDriver="gtls" >>>> > #StreamDriverMode="1" >>>> > #StreamDriverAuthMode="x509/name" >>>> > #StreamDriverPermittedPeers="logmanagement.xxx.yy" >>>> > ) >>>> > >>>> > # EOF >>>> > [ >>>> > root@logmanagement-client:/etc/rsyslog.d]# rsyslogd -N3 >>>> > rsyslogd: version 7.4.7, config validation run (level 3), master config >>>> > /etc/rsyslog.conf >>>> > rsyslogd: invalid or yet-unknown config file command >>>> > 'ActionSendStreamDriverPermittedPeers' - have you forgotten to load a >>>> > module? [try http://www.rsyslog.com/e/3003 ] >>>> > rsyslogd: End of config validation run. Bye. >>>> > >>>> > [root@logmanagement-client:/etc/rsyslog.d]# systemctl restart rsyslog >>>> > >>>> > [root@logmanagement-client:/etc/rsyslog.d]# tail /var/log/syslog |grep >>>> > rsyslogd >>>> > Nov 6 12:36:30 logmanagement-client rsyslogd: [origin >>>> software="rsyslogd" >>>> > swVersion="7.4.7" x-pid="2306" x-info="http://www.rsyslog.com";] start >>>> > Nov 6 12:36:30 logmanagement-client rsyslogd-3003: invalid or >>>> yet-unknown >>>> > config file command 'ActionSendStreamDriverPermittedPeers' - have you >>>> > forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] >>>> > Nov 6 12:36:30 logmanagement-client rsyslogd-2088: error: peer name not >>>> > authorized - not permitted to talk to it. Names: CN: >>>> logmanagement.xxx.yy >>>> > [try http://www.rsyslog.com/e/2088 ] >>>> > Nov 6 12:36:30 logmanagement-client rsyslogd-2088: error: peer name not >>>> > authorized - not permitted to talk to it. Names: CN: >>>> > logmanagement.xxx.yy; [try http://www.rsyslog.com/e/2088 ] >>>> > >>>> > >>>> > With both old and new: >>>> > >>>> > [root@logmanagement-client:/etc/rsyslog.d]# cat forward_syslog_tls.conf >>>> > # >>>> > # forward - syslog / tcp+tls >>>> > # >>>> > >>>> > >>>> > # testing shows that we need both the legacy and new style options >>>> > # when deleting options errors messages and even rsyslogd crashes occur >>>> > >>>> > $ActionSendStreamDriver gtls >>>> > $ActionSendStreamDriverMode 1 >>>> > $ActionSendStreamDriverAuthMode x509/name >>>> > $ActionSendStreamDriverPermittedPeers "logmanagement.xxx.yy" >>>> > >>>> > action( >>>> > type="omfwd" >>>> > target="192.168.124.100" >>>> > port="6514" >>>> > protocol="tcp" >>>> > template="RSYSLOG_SyslogProtocol23Format" >>>> > StreamDriver="gtls" >>>> > StreamDriverMode="1" >>>> > StreamDriverAuthMode="x509/name" >>>> > StreamDriverPermittedPeers="logmanagement.xxx.yy" >>>> > ) >>>> > >>>> > # EOF >>>> > >>>> > [root@logmanagement-client:/etc/rsyslog.d]# rsyslogd -N3 >>>> > rsyslogd: version 7.4.7, config validation run (level 3), master config >>>> > /etc/rsyslog.conf >>>> > rsyslogd: invalid or yet-unknown config file command >>>> > 'ActionSendStreamDriverPermittedPeers' - have you forgotten to load a >>>> > module? [try http://www.rsyslog.com/e/3003 ] >>>> > rsyslogd: End of config validation run. Bye. >>>> > >>>> > [root@logmanagement-client:/etc/rsyslog.d]# systemctl restart rsyslog >>>> > >>>> > [root@logmanagement-client:/etc/rsyslog.d]# tail /var/log/syslog >>>> > Nov 6 12:39:04 logmanagement-client rsyslogd: [origin >>>> software="rsyslogd" >>>> > swVersion="7.4.7" x-pid="2328" x-info="http://www.rsyslog.com";] start >>>> > Nov 6 12:39:04 logmanagement-client rsyslogd-3003: invalid or >>>> yet-unknown >>>> > config file command 'ActionSendStreamDriverPermittedPeers' - have you >>>> > forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] >>>> > >>>> > And this configuration actually does work. >>>> > >>>> > For sure i'm hitting some bugs here ;) >>>> > >>>> > Cheers, >>>> > Jörgen >>>> > >>>> > >>>> > On Fri, Nov 6, 2015 at 9:37 AM, Radu Gheorghe < >>>> [email protected]> >>>> > wrote: >>>> > >>>> >> Hello Jörgen, >>>> >> >>>> >> So if you "translate" the $Action... directives into RainerScript it >>>> >> doesn't work at all? And you also don't get any configuration errors? >>>> >> Then it would be a bug. >>>> >> >>>> >> Best regards, >>>> >> Radu >>>> >> -- >>>> >> Performance Monitoring * Log Analytics * Search Analytics >>>> >> Solr & Elasticsearch Support * http://sematext.com/ >>>> >> >>>> >> >>>> >> On Thu, Nov 5, 2015 at 2:22 PM, Jörgen Maas <[email protected]> >>>> wrote: >>>> >> > Hi, thanks Radu for your feedback! >>>> >> > >>>> >> > On the client and the server all rsyslog and gnutls versions are the >>>> >> same. >>>> >> > I did recreate the certs with openssl, instead of the certutil as >>>> >> described >>>> >> > in the docs. >>>> >> > I now have this working, it's just like yesterdays issue a case of >>>> using >>>> >> > new and old configuration *together* to make it work... >>>> >> > >>>> >> > But doing this seems to also cause some strange issues... i can >>>> imagine >>>> >> > that this isn't really a well tested configuration (mixing old and >>>> new). >>>> >> > Current config on the sender: >>>> >> > >>>> >> > $ActionSendStreamDriver gtls >>>> >> > $ActionSendStreamDriverMode 1 >>>> >> > $ActionSendStreamDriverAuthMode x509/name >>>> >> > $ActionSendStreamDriverPermittedPeers "logmanagement.xxx.yy" >>>> >> > >>>> >> > action( >>>> >> > type="omfwd" >>>> >> > target="192.168.124.100" >>>> >> > port="6514" >>>> >> > protocol="tcp" >>>> >> > template="RSYSLOG_SyslogProtocol23Format" >>>> >> > StreamDriver="gtls" >>>> >> > StreamDriverMode="1" >>>> >> > StreamDriverAuthMode="x509/name" >>>> >> > StreamDriverPermittedPeers="logmanagement.xxx.yy" >>>> >> > ) >>>> >> > >>>> >> > Without the legacy options (including >>>> >> > $ActionSendStreamDriverPermittedPeers) rsyslogd wont even start. >>>> >> > And with these options my log looks like this: >>>> >> > >>>> >> > Nov 5 10:59:49 logmanagement-client rsyslogd-3003: invalid or >>>> >> yet-unknown >>>> >> > config file command 'ActionSendStreamDriverPermittedPeers' - have you >>>> >> > forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] >>>> >> > Nov 5 10:59:26 logmanagement-client systemd: Stopping System Logging >>>> >> > Service... >>>> >> > Nov 5 10:59:26 logmanagement-client systemd: rsyslog.service: main >>>> >> process >>>> >> > exited, code=killed, status=6/ABRT >>>> >> > Nov 5 10:59:26 logmanagement-client systemd: Unit rsyslog.service >>>> >> entered >>>> >> > failed state. >>>> >> > Nov 5 10:59:26 logmanagement-client systemd: Starting System Logging >>>> >> > Service... >>>> >> > yikes -> Nov 5 10:59:26 logmanagement-client kernel: traps: >>>> >> rsyslogd[4698] >>>> >> > general protection ip:7fd55a8584bd sp:7ffe547d5f90 error:0 in >>>> >> libc-2.17.so >>>> >> > [7fd55a7dc000+1b6000] >>>> >> > >>>> >> > When i comment out the StreamDriverPermittedPeers, i get: >>>> >> > >>>> >> > Nov 5 11:02:58 logmanagement-client rsyslogd-2088: error: peer name >>>> not >>>> >> > authorized - not permitted to talk to it. Names: CN: >>>> >> > logmanagement.xxx.yy; [try http://www.rsyslog.com/e/2088 ] >>>> >> > >>>> >> > So with this "hybrid" config i can transfer logs over the tls >>>> channel. >>>> >> But >>>> >> > unfortunately the system is not really stable, i have seen some >>>> segfaults >>>> >> > and the general protection errors in the above log make me a bit wary >>>> >> using >>>> >> > this in a production setting. Any suggestions/hints on this specific >>>> >> error >>>> >> > and/or the segfaults? >>>> >> > >>>> >> > Thanks again! >>>> >> > >>>> >> > Regards, >>>> >> > Jörgen >>>> >> > >>>> >> > On Thu, Nov 5, 2015 at 8:07 AM, Radu Gheorghe < >>>> >> [email protected]> >>>> >> > wrote: >>>> >> > >>>> >> >> Hello, >>>> >> >> >>>> >> >> We had this problem at one point when having different versions of >>>> >> >> rsyslog (and/or gnutls) acting as client and server. Another time >>>> when >>>> >> >> I encountered this was when I didn't set up certificates properly. >>>> >> >> >>>> >> >> I hope this helps. >>>> >> >> >>>> >> >> Best regards, >>>> >> >> Radu >>>> >> >> -- >>>> >> >> Performance Monitoring * Log Analytics * Search Analytics >>>> >> >> Solr & Elasticsearch Support * http://sematext.com/ >>>> >> >> >>>> >> >> >>>> >> >> On Thu, Nov 5, 2015 at 7:27 AM, Jörgen Maas <[email protected]> >>>> >> wrote: >>>> >> >> > Hi all, >>>> >> >> > >>>> >> >> > With yesterdays help i've succeeded in setting up a TLS listener. >>>> I >>>> >> also >>>> >> >> > setup a forwarder as desribed in: >>>> >> >> > >>>> >> >> >>>> >> >>>> http://blog.sematext.com/2014/03/25/encrypting-logs-on-their-way-to-elasticsearch-part-2-tls-syslog/ >>>> >> >> > >>>> >> >> > On the server side i see this in my logs: >>>> >> >> > Nov 5 06:10:50 logmanagement rsyslogd-2083: gnutls returned >>>> error on >>>> >> >> > handshake: An unexpected TLS packet was received. >>>> >> >> > >>>> >> >> > I captured the network sessions and the messages are sent with >>>> plain >>>> >> tcp >>>> >> >> > (readable), so that explains the server side log entry. >>>> >> >> > >>>> >> >> > This is my client side config: >>>> >> >> > >>>> >> >> > action( >>>> >> >> > type="omfwd" >>>> >> >> > target="192.168.124.100" >>>> >> >> > port="6514" >>>> >> >> > protocol="tcp" >>>> >> >> > template="RSYSLOG_SyslogProtocol23Format" >>>> >> >> > StreamDriver="gtls" >>>> >> >> > StreamDriverMode="1" >>>> >> >> > StreamDriverAuthMode="x509/name" >>>> >> >> > StreamDriverPermittedPeers="logmanagement.xxx.yyy" >>>> >> >> > ) >>>> >> >> > >>>> >> >> > The "gtls" default settings are set in the global() section, as >>>> >> discussed >>>> >> >> > yesterday. >>>> >> >> > >>>> >> >> > Software version: >>>> >> >> > rsyslog-7.4.7-7.el7_1.1.x86_64 >>>> >> >> > >>>> >> >> > >>>> >> >> > What am I missing here? >>>> >> >> > >>>> >> >> > Thanks! >>>> >> >> > >>>> >> >> > >>>> >> >> > Regards, >>>> >> >> > Jörgen >>>> >> >> > _______________________________________________ >>>> >> >> > rsyslog mailing list >>>> >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> >> >> > http://www.rsyslog.com/professional-services/ >>>> >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> >> myriad >>>> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> you >>>> >> >> DON'T LIKE THAT. >>>> >> >> _______________________________________________ >>>> >> >> rsyslog mailing list >>>> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> >> >> http://www.rsyslog.com/professional-services/ >>>> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> myriad >>>> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> you >>>> >> >> DON'T LIKE THAT. >>>> >> > _______________________________________________ >>>> >> > rsyslog mailing list >>>> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> >> > http://www.rsyslog.com/professional-services/ >>>> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> myriad >>>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> >> DON'T LIKE THAT. >>>> >> _______________________________________________ >>>> >> rsyslog mailing list >>>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> >> http://www.rsyslog.com/professional-services/ >>>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> myriad >>>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> >> DON'T LIKE THAT. >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > Grtz, >>>> > Jörgen Maas >>>> > _______________________________________________ >>>> > rsyslog mailing list >>>> > http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> > http://www.rsyslog.com/professional-services/ >>>> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> you DON'T LIKE THAT. >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>> >>> >>> >> >> >> -- >> Grtz, >> Jörgen Maas >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
