> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Jason Antman > Sent: Friday, November 05, 2010 6:13 PM > To: rsyslog-users > Subject: Re: [rsyslog] segfault with dynamic filenames > > Ok... > > in output from configure script: > checking whether the compiler provides atomic builtins... yes > checking whether the compiler provides atomic builtins for 64 bit data > types... yes > > in omfile.c, lines 89-102: > #if HAVE_ATOMIC_BUILTINS_64BIT > static uint64 clockFileAccess = 0; > #else > static unsigned clockFileAccess = 0; > #endif > /* and the "tick" function */ > #ifndef HAVE_ATOMIC_BUILTINS > static pthread_mutex_t mutClock; > #endif > static inline uint64 > getClockFileAccess(void) > { > return ATOMIC_INC_AND_FETCH(&clockFileAccess, &mutClock); > } > > Once again, my C skills are pretty poor... but if HAVE_ATOMIC_BUILTINS > is defined.... what is mutClock??
A mutex name for a mutex that is not even created in this case ;) > > Output, strace log, and valgrind output are being sent to rainer. I'll probably not manage to have more than a peek at it today, but will look ASAP (it's getting a bit late over here ;)). Rainer > > Jason Antman wrote: > > I haven't run it with valgrind yet... honestly I don't think I've > used > > valgrind more than once or twice, I'd need to do a bit of reading up > to > > remember how to use it. > > > > This isn't a matter of *if* the issue reappears. Every time rsyslogd > > tries to write out a message matching a dynafile output, it dies > (within > > a few ms of starting, or whenever it actually tries to write the > first > > line out to disk). > > > > At the moment, I have the issue localized to the call to > > getClockFileAccess() on line 517 of tools/omfile.c, and am continuing > to > > trace it through the code. I'll see if one of the more c-skilled guys > > can give me a hand with valgrind. > > > > -Jason > > > > Rainer Gerhards wrote: > > > >> An you run it under valgrind control? If the issue reappears, it > should give > >> us *very* good hints of where to look at... > >> > >> RAiner > >> > >> > >> > >>> -----Original Message----- > >>> From: [email protected] [mailto:rsyslog- > >>> [email protected]] On Behalf Of Jason Antman > >>> Sent: Friday, November 05, 2010 5:26 PM > >>> To: rsyslog-users > >>> Subject: Re: [rsyslog] segfault with dynamic filenames > >>> > >>> Thanks for the info! > >>> > >>> Just sent this to Tom, forgot to CC the list.... > >>> > >>> I just rebuilt the RPM with all debugging features enabled and am > >>> experiencing the same issue. > >>> > >>> I have: > >>> - debug log output (rsyslogd -c 5 -dn, with RSYSLOG_DEBUG env > variable > >>> set to "LogFuncFlow FileTrace=omfile.c Debug") > >>> - strace output > >>> - RPM spec file including configure options, etc. > >>> - the Fedora patch that is being applied > >>> - the actual RPMs, if they're of interest, and a source rpm > >>> > >>> Which of this is of use? Should I email it? Open a ticket and > attach? > >>> > >>> Here are the last few lines before the crash: > >>> 2954.318300688:b7f06b90: Processing next action > >>> 2954.318308532:b7f06b90: Called action(NotAllMark), logging to > builtin- > >>> file > >>> 2954.318315864:b7f06b90: Called action(Batch), logging to builtin- > file > >>> 2954.318382400:b7f06b90: omfile.c:682: beginTransaction: enter > >>> 2954.318390551:b7f06b90: omfile.c:682: beginTransaction: exit: 0 > >>> 2954.318397068:b7f06b90: Action 0x9fbb3a0 transitioned to state: > itx > >>> 2954.318404578:b7f06b90: entering actionCalldoAction(), state: itx > >>> 2954.318411997:b7f06b90: omfile.c:697: doAction: enter > >>> 2954.318418085:b7f06b90: file to log to: LocalHost > >>> 2954.318424740:b7f06b90: omfile.c:634: writeFile: enter > >>> 2954.318431685:b7f06b90: omfile.c:493: prepareDynFile: enter > >>> (and then it dies, in the middle of a select() call) > >>> > >>> I'm going to start looking through omfile.c and perhaps adding some > of > >>> my own instrumentation just so I can localize it a bit more. > >>> > >>> Thanks for any advice you can provide, > >>> Jason Antman > >>> > >>> Tom Bergfeld wrote: > >>> > >>> > >>>> Hi Jason, > >>>> > >>>> unfortunately we were not able to reproduce your issue in Fedora > 13. > >>>> We will create an environment with centos and try it again. > >>>> > >>>> Tom > >>>> > >>>> > >>>> -----Ursprüngliche Nachricht----- > >>>> Von: [email protected] > >>>> [mailto:[email protected]] Im Auftrag von Jason > >>>> > >>>> > >>> Antman > >>> > >>> > >>>> Gesendet: Donnerstag, 4. November 2010 22:47 > >>>> An: rsyslog-users > >>>> Betreff: [rsyslog] segfault with dynamic filenames (was: Rsyslog > >>>> evaluationquestions) > >>>> > >>>> Thanks to both of you guys for your answers. FYI, our days have 24 > >>>> > >>>> > >>> hours > >>> > >>> > >>>> in them :) > >>>> > >>>> At this point I just got a config file worked up. I got the > regexes > >>>> working to do all of the matching I need into separate fields (the > >>>> online regex tester was a big help), and upgraded to 5.6.0 (CentOS > >>>> > >>>> > >>> 5.5, > >>> > >>> > >>>> in-house RPM build) in order to of bind imudp to a ruleset (so we > can > >>>> have one ruleset for localhost/unix socket, and one ruleset for > all > >>>> remote hosts). > >>>> > >>>> Unfortunately, I'm getting a segfault almost immediately on > startup > >>>> > >>>> > >>> due > >>> > >>> > >>>> to the following lines: > >>>> $template > >>>> > >>>> > >>>> > >>> > RemoteHost,"/var/log/HOSTS/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%syslogfa > >>> cility > >>> > >>> > >>>> -text%.log" > >>>> *.* ?RemoteHost > >>>> > >>>> I believe I've narrowed the problem down to any templated > (dynamic) > >>>> filename... even if I dump everything external directly to > >>>> /var/log/remote and add the following in my local ruleset: > >>>> $template LocalHost,"/var/log/LOCAL/%syslogfacility-text%.log" > >>>> *.* ?LocalHost > >>>> > >>>> I still get a segfault. The last line in the output running with - > dn > >>>> > >>>> > >>> is > >>> > >>> > >>>> always: > >>>> "file to log to: RemoteHost" (or LocalHost, when I used that > config). > >>>> > >>>> Has anyone seen anything like this? I have rsyslog 2.0.6 running > at > >>>> another site and dynamic filenames work fine... sort of troubling > >>>> > >>>> > >>> that > >>> > >>> > >>>> 5.6.0 is having issues with them... > >>>> > >>>> Thanks, > >>>> Jason Antman > >>>> > >>>> Aaron Wiebe wrote: > >>>> > >>>> > >>>> > >>>>>> 3) Assuming an even distribution over time (not quite accurate), > >>>>>> > >>>>>> > >>> any > >>> > >>> > >>>>>> thoughts on how dumping ~2M lines/day of syslog to MySQL on a VM > >>>>>> > >>>>>> > >>> (Xen) > >>> > >>> > >>>>>> with a single 2.8GHz CPU and 512MB RAM would go? > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> 2 million lines a day is 70 lines per second assuming an 8 hour > day. > >>>>> MySQL, properly configured, will eat that. Rsyslog won't even > >>>>> > >>>>> > >>> notice > >>> > >>> > >>>>> that log level. > >>>>> > >>>>> Your table sizes may be problematic over time, and your ability > to > >>>>> query may impact things. Regardless, your concern should be > mysql, > >>>>> not rsyslog. > >>>>> > >>>>> -Aaron > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>>> > >>>> > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > >> > >> > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
