Awesome! Thanks Eric! From: Eric <[email protected]> Reply-To: <[email protected]> Date: Tuesday, October 4, 2016 at 10:49 AM To: <[email protected]> Subject: Re: [qmailtoaster] vchkpw segfaults and spamdyke errors
Hi Jamie and everyone else, I know its been a while, but I've found the reason for this segfault in qmailmrtg, on my server at least. I saw that you were interested in turning off qmailmrtg as you don't use it. To turn off edit /etc/crontab and change 0-59/5 * * * * root env LANG=C /usr/bin/mrtg /usr/share/toaster/mrtg/qmailmrtg.cfg > /dev/null 2>&1 to #0-59/5 * * * * root env LANG=C /usr/bin/mrtg /usr/share/toaster/mrtg/qmailmrtg.cfg > /dev/null 2>&1 or delete the line altogether. If you're interested in where I found the segfault (at least on my toaster) read on. It was really quite simple and I'm not sure why this hasn't been noticed before. I found that the error occurred during the concurrency check in qmailmrtg.cfg, specifically in the call `/usr/bin/qmailmrtg t /var/log/qmail/smtp`. Here qmailmrtg is finding concurrent connections in the smtp log by looking for lines with the following: 'status: [1-???]/100' from which it derives the max number of connections over a time interval (present-300 to present programmatically). The program (qmailmrtg), when searching for the substring 'status:' within single log entries, is NOT checking for the most important part: whether this is a 'tcpserver:' connection or not. If the program finds any occurrence of 'status:' the log entry in which it resides will be checked for concurrency ([1-???]/100) using POINTERS. If the program finds an occurrence of 'status:' within a log entry not of the form 'status: [1-???]/100' the pointer will be pointing to an undefined or unauthorized section of memory causing the segfault. The concurrency entries take the following from: @4000000057f3b9b42799c77c tcpserver: status: 3/100 I created a patch (soon to be out) so the program checks for 'tcpserver: status: ' (whether it's a tcp server connection) before it looks for concurrency numbers. I found the offending 'status:' entries in the smtp log by running the following command: # cat /var/log/qmail/smtp/* | tai64nlocal | grep status:[0-9] The resulting 'cat' of the smtp log provided the following log entries: @4000000057eb9d2f1c792d4c simscan:[21808]:CLEAN (2.90/12.00):0.5372s:John Q. Public updated her status:192.168.0.1:[email protected]:soandso@mydomains. tld on which qmailmrtg was segfault(ing). It ended up being email from facebook with subject lines indicating an update in status. It's a harmless problem but annoying, so, I'm going to patch the code and put out an updated version. I'm still surprised that this wasn't noticed years ago. Eric On 6/10/2016 10:10 AM, Jaime Lerner wrote: > Thanks for looking into this, Eric. The interesting thing is, the > qmailmrtg segfaults didn't happen until I cleared up the vchkpw faults. > I'm thinking they both might be related to memory and when I raised the > limit for vchkpw it didn't leave enough memory for qmailmrtg to run > sometimes. I'm thinking I could drop the softlimit down and see if that > solves it (i.e. Drop it until the qmailmrgt segfaults stop, but not too > much so as to cause the vchkpw segfaults to start up again). And yes, > they are in the messages log. I don't have any of the "abrt-server" > messages, nor anything about it not being signed. The only thing in my > log for qmailmrtg is the segfaults (and the initial install I did with yum). > > From: Eric <[email protected] <mailto:[email protected]>> > Reply-To: <[email protected] > <mailto:[email protected]>> > Date: Friday, June 10, 2016 at 12:02 PM > To: <[email protected] > <mailto:[email protected]>> > Subject: Re: [qmailtoaster] vchkpw segfaults and spamdyke errors > > Hi Jamie, > > I had these as well on a client server about a month ago for a few days > and they went away. They showed up in the postmaster logwatch email. I > traced them to the messages log. > > I ran the following command (with output): > > # cat messages* | grep -C 4 segfault > > May 18 09:25:02 mail kernel: qmailmrtg[20759]: segfault at 604000 ip > 0000000000400b17 sp 00007fff8f462560 error 4 in qmailmrtg[400000+2000] > May 18 09:25:02 mail abrt-server: Package 'qmailmrtg' isn't signed with > proper key > May 18 09:25:02 mail abrt-server: 'post-create' on > '/var/spool/abrt/ccpp-2016-05-18-09:25:02-20759' exited with 1 > May 18 09:25:02 mail abrt-server: Deleting problem directory > '/var/spool/abrt/ccpp-2016-05-18-09:25:02-20759' > > And, it looks like it has something to do with signing. > > I got distracted and will have to investigate this further. > > Eric > > > On 6/10/2016 9:15 AM, Jaime Lerner wrote: > > I don't get segfaults from vchkpw anymore (not since raising my > softlimit), but I get from 1-3 segfaults from qmailmrtg daily. I don't > really need or want qmailmrtg, so if anyone can tell me how to turn it > off, that would be great. :) Otherwise, it's not causing any problems > for me since I don't use it. > > [root@mail qmail]# grep segfault /var/log/messages > > Jun 5 16:20:01 mail kernel: qmailmrtg[26761]: *segfault* at 604000 ip > 0000000000400b17 sp 00007ffcbfdbc4a0 error 4 in qmailmrtg[400000+2000] > > Jun 5 16:40:01 mail kernel: qmailmrtg[28163]: *segfault* at 604000 ip > 0000000000400b17 sp 00007fffbac92110 error 4 in qmailmrtg[400000+2000] > > Jun 5 16:55:01 mail kernel: qmailmrtg[29324]: *segfault* at 604000 ip > 0000000000400b17 sp 00007fff8ca08810 error 4 in qmailmrtg[400000+2000] > > Jun 6 12:20:01 mail kernel: qmailmrtg[30300]: *segfault* at 604000 ip > 0000000000400b17 sp 00007ffe382ce270 error 4 in qmailmrtg[400000+2000] > > Jun 7 11:25:01 mail kernel: qmailmrtg[10676]: *segfault* at 604000 ip > 0000000000400b17 sp 00007fff0e24eff0 error 4 in qmailmrtg[400000+2000] > > Jun 7 15:00:02 mail kernel: qmailmrtg[20856]: *segfault* at 604000 ip > 0000000000400b17 sp 00007ffec310ee90 error 4 in qmailmrtg[400000+2000] > > Jun 7 15:05:01 mail kernel: qmailmrtg[21134]: *segfault* at 604000 ip > 0000000000400b17 sp 00007fff02dd4660 error 4 in qmailmrtg[400000+2000] > > Jun 8 07:20:01 mail kernel: qmailmrtg[31909]: *segfault* at 604000 ip > 0000000000400b17 sp 00007ffc923737e0 error 4 in qmailmrtg[400000+2000] > > Jun 8 12:15:01 mail kernel: qmailmrtg[12908]: *segfault* at 604000 ip > 0000000000400b17 sp 00007ffc75843060 error 4 in qmailmrtg[400000+2000] > > Jun 9 12:15:01 mail kernel: qmailmrtg[11826]: *segfault* at 604000 ip > 0000000000400b17 sp 00007ffd93dcda20 error 4 in qmailmrtg[400000+2000] > > Jun 10 10:15:01 mail kernel: qmailmrtg[6510]: *segfault* at 604000 ip > 0000000000400b17 sp 00007fff8e676c30 error 4 in qmailmrtg[400000+2000] > > Jun 10 10:25:01 mail kernel: qmailmrtg[6931]: *segfault* at 604000 ip > 0000000000400b17 sp 00007ffc884b8570 error 4 in qmailmrtg[400000+2000] > > Jun 10 10:40:01 mail kernel: qmailmrtg[7683]: *segfault* at 604000 ip > 0000000000400b17 sp 00007ffd97d86ab0 error 4 in qmailmrtg[400000+2000] > > > From: Steve Linberg <[email protected] > <mailto:[email protected]> > <mailto:[email protected]>> > Reply-To: <[email protected] > <mailto:[email protected]> > <mailto:[email protected]>> > Date: Friday, June 10, 2016 at 10:48 AM > To: <[email protected] > <mailto:[email protected]> > <mailto:[email protected]>> > Subject: Re: [qmailtoaster] vchkpw segfaults and spamdyke errors > > Still working on the segfault problem; got dozens of them overnight when > I definitely wasn’t using any services. softlimit > in /var/qmail/supervise/submission/run is at 500 megs and rising, but > I’m still having trouble believing it needs to be that high or higher. > > While I continue to push that: is there any way to know what process > chain is invoking the vchkpw process that’s segfaulting? Don’t a number > of different processes use it? I don’t know for a fact that submission > is the one that’s causing it. I can’t find any other clues in my logs, > like events happening at the same time as the segfaults, but are there > any other possible culprits that might invoke vchkpw without enough RAM > to do whatever it’s trying to do? > > > -- > Steve Linberg, Chief Goblin > Silicon Goblin Technologies > http://silicongoblin.com > Be kind. Remember, everyone you meet is fighting a hard battle. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > <mailto:[email protected]> > For additional commands, e-mail: [email protected] > <mailto:[email protected]> > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
