Hi Kendall, I need a repro for this problem, it has remained open far too long for my comfort, but I simply haven't been able to get to the bottom of this over mail and issue-tracker.
Does the image (rhel version) you use exist as an AMI on EC2? Or if you can spare some time, it'll help me immensely if you can pass me a virtual box image. Or if possible, can you move a box reproducing this to DMZ, strip the config down to the least necessary, remove all sensitive material out of it and let me debug it in place? I need this issue to reproduce consistently while I debug it. Any other ideas on how we may do this? On 04-Jun-2017 7:26 AM, "Kendall Green via rsyslog" < [email protected]> wrote: > Updated following details on Issue: > https://github.com/rsyslog/rsyslog/issues/1071 > > I've recently setup lookup tables to control which ruleset to call with > $fromhost-ip. The call appears to work properly, however there's a segfault > in /var/log/messages on service rsyslog start, and the reload lookup tables > does not function on HUP nor with message trigger. The segfault only > happened after config of lookup tables, with and without the load and > reload as part of ruleset. The ruleset does call to another ruleset by set > $.var and if $.var == $fromhost-ip, using similar syntax as example configs > from lookup in rsyslog docs: > > lookup_table(name="test_lu" file="/etc/rsyslog.d/lu/test_lu.json" > reloadOnHUP="off") > > if ($rawmsg contains "test_lu") > then { > reload_lookup_table("test_lu, "unk") > stop > } > > set $.test = lookup("test_lu", $fromhost-ip); > > Tried other config tests, with same result, the match lookup table value > works to call other ruleset, but reload lookup table does not take effect, > as reload only happens when restarting rsyslog, and not with message > trigger or HUP. All tests with lookup table loaded has segfault in > /var/log/message on start. I had commented out the lookup() and > reload_lookup_table() and started with just loading the lookup_table in > config and still got segfault. Plus, tried with the reloadOnHUP on or off, > but segfault either way. The segfault does not appear after removing / > commenting out the lookup table from config, and these configs have not > produced segfaults in the past. > > kernel: rsyslogd[5434]: segfault at 7faab145f9d0 ip 00007faab44e5213 sp > 00007fff94cf9940 error 4 in libpthread-2.12.so[7faab44dd000+17000] > > Confirmed issue on rhel-6.9 with rsyslog v8.24, v8.26.x, and v8.27.0. > Verified libjson-c is not installed, and the issue was not resolved with > upgrade of libfastjson4-0.99.4-1.el6 to latest libfastjson4-0.99.5-1.el6. > Is there another version of libfastjson should use? > > On Thu, Jan 12, 2017 at 6:48 AM, David Lang <[email protected]> wrote: > > > is the version you are using linked to json-c or libfastjson? > > > > we know there were thread-safe problems as a result of json-c > > > > David Lang > > > > On Thu, 12 Jan 2017, Christopher Racky via rsyslog wrote: > > > > Date: Thu, 12 Jan 2017 12:43:31 +0100 > >> From: Christopher Racky via rsyslog <[email protected]> > >> To: rsyslog-users <[email protected]> > >> Cc: Christopher Racky <[email protected]> > >> Subject: [rsyslog] Problems with rsyslog Versions > 8.16 and > >> Thread-Handling > >> > >> > >> Hello, > >> > >> I still have big issues with different Servers running RHEL 6.8 (incl. > >> latest updates) and rsyslog > 8.16. > >> > >> While 8.16 works fine, all following versions, including 8.24 which > >> was released a few days ago leads to problems that results in > >> core-dumps: > >> rsyslogd[1408]: segfault at 7f9f2910b9d0 ip 00007f9f2e18e213 sp > >> 00007ffd3d9fe080 error 4 in libpthread-2.12.so[7f9f2e186000+17000] > >> rsyslogd[4201]: segfault at 7f991c0789d0 ip 00007f99210fb213 sp > >> 00007ffec5155240 error 4 in libpthread-2.12.so[7f99210f3000+17000] > >> rsyslogd[2783]: segfault at 7f5f695399d0 ip 00007f5f6e5bd213 sp > >> 00007ffe4b4da5c0 error 4 in libpthread-2.12.so[7f5f6e5b5000+17000] > >> rsyslogd[6816]: segfault at 7f519ad269d0 ip 00007f519fdaa213 sp > >> 00007ffd22bf2470 error 4 in libpthread-2.12.so[7f519fda2000+17000] > >> > >> > >> My hope was that this topic was solved by > >> https://github.com/rsyslog/rsyslog/pull/1274 > >> > >> But it was not. > >> As problems seems not (directly) related to lookup-table, I guess it > >> also has a problem with thread handling. > >> https://github.com/rsyslog/rsyslog/issues/1071 > >> > >> Do you have any further idea? > >> > >> With single-thread mode problem does not appear and also in lab, I was > >> not able to reproduce it. > >> But on several production servers the issue occures quite often. > >> > >> > >> regards > >> Chris > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > >> _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
