Hi Jörgen, This thread comes up on all searches for "perseverance" :D, that's really nice.
But this finding is really interesting. I remember David saying that having just one config is safer and I also prefer that for simplicity, but I didn't expect bugs to appear with multiple confs. Maybe the order of statements is mixed up with includes? I would expect it to work as if you inserted the included config to the main rsyslog.conf at the location of the include statement. Though it sounds more like a bug, I don't see a clear required order (other than loading modules first, then defining inputs/actions). Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Thu, Nov 12, 2015 at 9:14 AM, Jörgen Maas <[email protected]> wrote: > More progress on this: > > When i move all rsyslog configuration into a single file all problems > (mentioned in this thread) are gone! It seems that splitting functionality > in seperate files is not working 100% as expected. > > Is this a known limitation/issue? > > Regards, > Jörgen > > On Wed, Nov 11, 2015 at 2:00 PM, Jörgen Maas <[email protected]> wrote: > >> Hi all, >> >> I've verified the situation with the latest 8.14 release and the official >> packages, and i can confirm that the issue is still present. Including the >> segmentationfault when using rsyslogd -N3. I'll file a bug report on GitHub. >> >> Cheers, >> Jörgen >> >> >> >> On Fri, Nov 6, 2015 at 3:51 PM, Radu Gheorghe <[email protected]> >> wrote: >> >>> Hi Jörgen, >>> >>> Yes, this is really weird. Can you come up with a complete >>> reproduction that you'd paste in a GitHub issue? >>> >>> Best regards, >>> Radu >>> -- >>> Performance Monitoring * Log Analytics * Search Analytics >>> Solr & Elasticsearch Support * http://sematext.com/ >>> >>> >>> On Fri, Nov 6, 2015 at 1:43 PM, Jörgen Maas <[email protected]> >>> wrote: >>> > Hi Radu, >>> > >>> > First with only the new syntax: >>> > >>> > [root@logmanagement-client:/etc/rsyslog.d]# cat forward_syslog_tls.conf >>> > # >>> > # forward - syslog / tcp+tls >>> > # >>> > >>> > >>> > # testing shows that we need both the legacy and new style options >>> > # when deleting options errors messages and even rsyslogd crashes occur >>> > >>> > #$ActionSendStreamDriver gtls >>> > #$ActionSendStreamDriverMode 1 >>> > #$ActionSendStreamDriverAuthMode x509/name >>> > #$ActionSendStreamDriverPermittedPeers "logmanagement.xxx.yy" >>> > >>> > action( >>> > type="omfwd" >>> > target="192.168.124.100" >>> > port="6514" >>> > protocol="tcp" >>> > template="RSYSLOG_SyslogProtocol23Format" >>> > StreamDriver="gtls" >>> > StreamDriverMode="1" >>> > StreamDriverAuthMode="x509/name" >>> > StreamDriverPermittedPeers="logmanagement.xx.yy" >>> > ) >>> > >>> > # EOF >>> > >>> > [root@logmanagement-client:/etc/rsyslog.d]# rsyslogd -N3 >>> > rsyslogd: version 7.4.7, config validation run (level 3), master config >>> > /etc/rsyslog.conf >>> > Segmentation fault >>> > >>> > [root@logmanagement-client:/etc/rsyslog.d]# systemctl restart rsyslog >>> > Job for rsyslog.service failed. See 'systemctl status rsyslog.service' >>> and >>> > 'journalctl -xn' for details. >>> > >>> > [root@logmanagement-client:/etc/rsyslog.d]# tail /var/log/syslog |grep >>> > rsyslogd >>> > Nov 6 12:30:52 logmanagement-client kernel: traps: rsyslogd[2192] >>> general >>> > protection ip:7fdab462c4bd sp:7ffd52d30a50 error:0 in libc-2.17.so >>> > [7fdab45b0000+1b6000] >>> > Nov 6 12:31:11 logmanagement-client rsyslogd: [origin >>> software="rsyslogd" >>> > swVersion="7.4.7" x-pid="577" x-info="http://www.rsyslog.com";] exiting >>> on >>> > signal 15. >>> > >>> > >>> > >>> > Now with only the old syntax: >>> > >>> > [root@logmanagement-client:/etc/rsyslog.d]# cat forward_syslog_tls.conf >>> > # >>> > # forward - syslog / tcp+tls >>> > # >>> > >>> > >>> > # testing shows that we need both the legacy and new style options >>> > # when deleting options errors messages and even rsyslogd crashes occur >>> > >>> > $ActionSendStreamDriver gtls >>> > $ActionSendStreamDriverMode 1 >>> > $ActionSendStreamDriverAuthMode x509/name >>> > $ActionSendStreamDriverPermittedPeers "logmanagement.xxx.yy" >>> > >>> > action( >>> > type="omfwd" >>> > target="192.168.124.100" >>> > port="6514" >>> > protocol="tcp" >>> > template="RSYSLOG_SyslogProtocol23Format" >>> > #StreamDriver="gtls" >>> > #StreamDriverMode="1" >>> > #StreamDriverAuthMode="x509/name" >>> > #StreamDriverPermittedPeers="logmanagement.xxx.yy" >>> > ) >>> > >>> > # EOF >>> > [ >>> > root@logmanagement-client:/etc/rsyslog.d]# rsyslogd -N3 >>> > rsyslogd: version 7.4.7, config validation run (level 3), master config >>> > /etc/rsyslog.conf >>> > rsyslogd: invalid or yet-unknown config file command >>> > 'ActionSendStreamDriverPermittedPeers' - have you forgotten to load a >>> > module? [try http://www.rsyslog.com/e/3003 ] >>> > rsyslogd: End of config validation run. Bye. >>> > >>> > [root@logmanagement-client:/etc/rsyslog.d]# systemctl restart rsyslog >>> > >>> > [root@logmanagement-client:/etc/rsyslog.d]# tail /var/log/syslog |grep >>> > rsyslogd >>> > Nov 6 12:36:30 logmanagement-client rsyslogd: [origin >>> software="rsyslogd" >>> > swVersion="7.4.7" x-pid="2306" x-info="http://www.rsyslog.com";] start >>> > Nov 6 12:36:30 logmanagement-client rsyslogd-3003: invalid or >>> yet-unknown >>> > config file command 'ActionSendStreamDriverPermittedPeers' - have you >>> > forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] >>> > Nov 6 12:36:30 logmanagement-client rsyslogd-2088: error: peer name not >>> > authorized - not permitted to talk to it. Names: CN: >>> logmanagement.xxx.yy >>> > [try http://www.rsyslog.com/e/2088 ] >>> > Nov 6 12:36:30 logmanagement-client rsyslogd-2088: error: peer name not >>> > authorized - not permitted to talk to it. Names: CN: >>> > logmanagement.xxx.yy; [try http://www.rsyslog.com/e/2088 ] >>> > >>> > >>> > With both old and new: >>> > >>> > [root@logmanagement-client:/etc/rsyslog.d]# cat forward_syslog_tls.conf >>> > # >>> > # forward - syslog / tcp+tls >>> > # >>> > >>> > >>> > # testing shows that we need both the legacy and new style options >>> > # when deleting options errors messages and even rsyslogd crashes occur >>> > >>> > $ActionSendStreamDriver gtls >>> > $ActionSendStreamDriverMode 1 >>> > $ActionSendStreamDriverAuthMode x509/name >>> > $ActionSendStreamDriverPermittedPeers "logmanagement.xxx.yy" >>> > >>> > action( >>> > type="omfwd" >>> > target="192.168.124.100" >>> > port="6514" >>> > protocol="tcp" >>> > template="RSYSLOG_SyslogProtocol23Format" >>> > StreamDriver="gtls" >>> > StreamDriverMode="1" >>> > StreamDriverAuthMode="x509/name" >>> > StreamDriverPermittedPeers="logmanagement.xxx.yy" >>> > ) >>> > >>> > # EOF >>> > >>> > [root@logmanagement-client:/etc/rsyslog.d]# rsyslogd -N3 >>> > rsyslogd: version 7.4.7, config validation run (level 3), master config >>> > /etc/rsyslog.conf >>> > rsyslogd: invalid or yet-unknown config file command >>> > 'ActionSendStreamDriverPermittedPeers' - have you forgotten to load a >>> > module? [try http://www.rsyslog.com/e/3003 ] >>> > rsyslogd: End of config validation run. Bye. >>> > >>> > [root@logmanagement-client:/etc/rsyslog.d]# systemctl restart rsyslog >>> > >>> > [root@logmanagement-client:/etc/rsyslog.d]# tail /var/log/syslog >>> > Nov 6 12:39:04 logmanagement-client rsyslogd: [origin >>> software="rsyslogd" >>> > swVersion="7.4.7" x-pid="2328" x-info="http://www.rsyslog.com";] start >>> > Nov 6 12:39:04 logmanagement-client rsyslogd-3003: invalid or >>> yet-unknown >>> > config file command 'ActionSendStreamDriverPermittedPeers' - have you >>> > forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] >>> > >>> > And this configuration actually does work. >>> > >>> > For sure i'm hitting some bugs here ;) >>> > >>> > Cheers, >>> > Jörgen >>> > >>> > >>> > On Fri, Nov 6, 2015 at 9:37 AM, Radu Gheorghe < >>> [email protected]> >>> > wrote: >>> > >>> >> Hello Jörgen, >>> >> >>> >> So if you "translate" the $Action... directives into RainerScript it >>> >> doesn't work at all? And you also don't get any configuration errors? >>> >> Then it would be a bug. >>> >> >>> >> Best regards, >>> >> Radu >>> >> -- >>> >> Performance Monitoring * Log Analytics * Search Analytics >>> >> Solr & Elasticsearch Support * http://sematext.com/ >>> >> >>> >> >>> >> On Thu, Nov 5, 2015 at 2:22 PM, Jörgen Maas <[email protected]> >>> wrote: >>> >> > Hi, thanks Radu for your feedback! >>> >> > >>> >> > On the client and the server all rsyslog and gnutls versions are the >>> >> same. >>> >> > I did recreate the certs with openssl, instead of the certutil as >>> >> described >>> >> > in the docs. >>> >> > I now have this working, it's just like yesterdays issue a case of >>> using >>> >> > new and old configuration *together* to make it work... >>> >> > >>> >> > But doing this seems to also cause some strange issues... i can >>> imagine >>> >> > that this isn't really a well tested configuration (mixing old and >>> new). >>> >> > Current config on the sender: >>> >> > >>> >> > $ActionSendStreamDriver gtls >>> >> > $ActionSendStreamDriverMode 1 >>> >> > $ActionSendStreamDriverAuthMode x509/name >>> >> > $ActionSendStreamDriverPermittedPeers "logmanagement.xxx.yy" >>> >> > >>> >> > action( >>> >> > type="omfwd" >>> >> > target="192.168.124.100" >>> >> > port="6514" >>> >> > protocol="tcp" >>> >> > template="RSYSLOG_SyslogProtocol23Format" >>> >> > StreamDriver="gtls" >>> >> > StreamDriverMode="1" >>> >> > StreamDriverAuthMode="x509/name" >>> >> > StreamDriverPermittedPeers="logmanagement.xxx.yy" >>> >> > ) >>> >> > >>> >> > Without the legacy options (including >>> >> > $ActionSendStreamDriverPermittedPeers) rsyslogd wont even start. >>> >> > And with these options my log looks like this: >>> >> > >>> >> > Nov 5 10:59:49 logmanagement-client rsyslogd-3003: invalid or >>> >> yet-unknown >>> >> > config file command 'ActionSendStreamDriverPermittedPeers' - have you >>> >> > forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] >>> >> > Nov 5 10:59:26 logmanagement-client systemd: Stopping System Logging >>> >> > Service... >>> >> > Nov 5 10:59:26 logmanagement-client systemd: rsyslog.service: main >>> >> process >>> >> > exited, code=killed, status=6/ABRT >>> >> > Nov 5 10:59:26 logmanagement-client systemd: Unit rsyslog.service >>> >> entered >>> >> > failed state. >>> >> > Nov 5 10:59:26 logmanagement-client systemd: Starting System Logging >>> >> > Service... >>> >> > yikes -> Nov 5 10:59:26 logmanagement-client kernel: traps: >>> >> rsyslogd[4698] >>> >> > general protection ip:7fd55a8584bd sp:7ffe547d5f90 error:0 in >>> >> libc-2.17.so >>> >> > [7fd55a7dc000+1b6000] >>> >> > >>> >> > When i comment out the StreamDriverPermittedPeers, i get: >>> >> > >>> >> > Nov 5 11:02:58 logmanagement-client rsyslogd-2088: error: peer name >>> not >>> >> > authorized - not permitted to talk to it. Names: CN: >>> >> > logmanagement.xxx.yy; [try http://www.rsyslog.com/e/2088 ] >>> >> > >>> >> > So with this "hybrid" config i can transfer logs over the tls >>> channel. >>> >> But >>> >> > unfortunately the system is not really stable, i have seen some >>> segfaults >>> >> > and the general protection errors in the above log make me a bit wary >>> >> using >>> >> > this in a production setting. Any suggestions/hints on this specific >>> >> error >>> >> > and/or the segfaults? >>> >> > >>> >> > Thanks again! >>> >> > >>> >> > Regards, >>> >> > Jörgen >>> >> > >>> >> > On Thu, Nov 5, 2015 at 8:07 AM, Radu Gheorghe < >>> >> [email protected]> >>> >> > wrote: >>> >> > >>> >> >> Hello, >>> >> >> >>> >> >> We had this problem at one point when having different versions of >>> >> >> rsyslog (and/or gnutls) acting as client and server. Another time >>> when >>> >> >> I encountered this was when I didn't set up certificates properly. >>> >> >> >>> >> >> I hope this helps. >>> >> >> >>> >> >> Best regards, >>> >> >> Radu >>> >> >> -- >>> >> >> Performance Monitoring * Log Analytics * Search Analytics >>> >> >> Solr & Elasticsearch Support * http://sematext.com/ >>> >> >> >>> >> >> >>> >> >> On Thu, Nov 5, 2015 at 7:27 AM, Jörgen Maas <[email protected]> >>> >> wrote: >>> >> >> > Hi all, >>> >> >> > >>> >> >> > With yesterdays help i've succeeded in setting up a TLS listener. >>> I >>> >> also >>> >> >> > setup a forwarder as desribed in: >>> >> >> > >>> >> >> >>> >> >>> http://blog.sematext.com/2014/03/25/encrypting-logs-on-their-way-to-elasticsearch-part-2-tls-syslog/ >>> >> >> > >>> >> >> > On the server side i see this in my logs: >>> >> >> > Nov 5 06:10:50 logmanagement rsyslogd-2083: gnutls returned >>> error on >>> >> >> > handshake: An unexpected TLS packet was received. >>> >> >> > >>> >> >> > I captured the network sessions and the messages are sent with >>> plain >>> >> tcp >>> >> >> > (readable), so that explains the server side log entry. >>> >> >> > >>> >> >> > This is my client side config: >>> >> >> > >>> >> >> > action( >>> >> >> > type="omfwd" >>> >> >> > target="192.168.124.100" >>> >> >> > port="6514" >>> >> >> > protocol="tcp" >>> >> >> > template="RSYSLOG_SyslogProtocol23Format" >>> >> >> > StreamDriver="gtls" >>> >> >> > StreamDriverMode="1" >>> >> >> > StreamDriverAuthMode="x509/name" >>> >> >> > StreamDriverPermittedPeers="logmanagement.xxx.yyy" >>> >> >> > ) >>> >> >> > >>> >> >> > The "gtls" default settings are set in the global() section, as >>> >> discussed >>> >> >> > yesterday. >>> >> >> > >>> >> >> > Software version: >>> >> >> > rsyslog-7.4.7-7.el7_1.1.x86_64 >>> >> >> > >>> >> >> > >>> >> >> > What am I missing here? >>> >> >> > >>> >> >> > Thanks! >>> >> >> > >>> >> >> > >>> >> >> > Regards, >>> >> >> > Jörgen >>> >> >> > _______________________________________________ >>> >> >> > rsyslog mailing list >>> >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> >> >> > http://www.rsyslog.com/professional-services/ >>> >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> >> myriad >>> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you >>> >> >> DON'T LIKE THAT. >>> >> >> _______________________________________________ >>> >> >> rsyslog mailing list >>> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> >> >> http://www.rsyslog.com/professional-services/ >>> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad >>> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you >>> >> >> DON'T LIKE THAT. >>> >> > _______________________________________________ >>> >> > rsyslog mailing list >>> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> >> > http://www.rsyslog.com/professional-services/ >>> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> >> DON'T LIKE THAT. >>> >> _______________________________________________ >>> >> rsyslog mailing list >>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> >> http://www.rsyslog.com/professional-services/ >>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> >> DON'T LIKE THAT. >>> >> >>> > >>> > >>> > >>> > -- >>> > Grtz, >>> > Jörgen Maas >>> > _______________________________________________ >>> > rsyslog mailing list >>> > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > http://www.rsyslog.com/professional-services/ >>> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you DON'T LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> >> >> > > > -- > Grtz, > Jörgen Maas > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
