Hi Jamie and everyone else,
I know its been a while, but I've found the reason for this segfault in
qmailmrtg, on my server at least. I saw that you were interested in
turning off qmailmrtg as you don't use it.
To turn off edit /etc/crontab and change
0-59/5 * * * * root env LANG=C /usr/bin/mrtg
/usr/share/toaster/mrtg/qmailmrtg.cfg > /dev/null 2>&1
to
#0-59/5 * * * * root env LANG=C /usr/bin/mrtg
/usr/share/toaster/mrtg/qmailmrtg.cfg > /dev/null 2>&1
or delete the line altogether.
If you're interested in where I found the segfault (at least on my
toaster) read on. It was really quite simple and I'm not sure why this
hasn't been noticed before.
I found that the error occurred during the concurrency check in
qmailmrtg.cfg, specifically in the call `/usr/bin/qmailmrtg t
/var/log/qmail/smtp`. Here qmailmrtg is finding concurrent connections
in the smtp log by looking for lines with the following: 'status:
[1-???]/100' from which it derives the max number of connections over a
time interval (present-300 to present programmatically).
The program (qmailmrtg), when searching for the substring 'status:'
within single log entries, is NOT checking for the most important part:
whether this is a 'tcpserver:' connection or not.
If the program finds any occurrence of 'status:' the log entry in which
it resides will be checked for concurrency ([1-???]/100) using POINTERS.
If the program finds an occurrence of 'status:' within a log entry not
of the form 'status: [1-???]/100' the pointer will be pointing to an
undefined or unauthorized section of memory causing the segfault.
The concurrency entries take the following from:
@4000000057f3b9b42799c77c tcpserver: status: 3/100
I created a patch (soon to be out) so the program checks for 'tcpserver:
status: ' (whether it's a tcp server connection) before it looks for
concurrency numbers.
I found the offending 'status:' entries in the smtp log by running the
following command:
# cat /var/log/qmail/smtp/* | tai64nlocal | grep status:[0-9]
The resulting 'cat' of the smtp log provided the following log entries:
@4000000057eb9d2f1c792d4c simscan:[21808]:CLEAN
(2.90/12.00):0.5372s:John Q. Public updated her
status:192.168.0.1:[email protected]:[email protected]
on which qmailmrtg was segfault(ing). It ended up being email from
facebook with subject lines indicating an update in status.
It's a harmless problem but annoying, so, I'm going to patch the code
and put out an updated version.
I'm still surprised that this wasn't noticed years ago.
Eric
On 6/10/2016 10:10 AM, Jaime Lerner wrote:
Thanks for looking into this, Eric. The interesting thing is, the
qmailmrtg segfaults didn't happen until I cleared up the vchkpw faults.
I'm thinking they both might be related to memory and when I raised the
limit for vchkpw it didn't leave enough memory for qmailmrtg to run
sometimes. I'm thinking I could drop the softlimit down and see if that
solves it (i.e. Drop it until the qmailmrgt segfaults stop, but not too
much so as to cause the vchkpw segfaults to start up again). And yes,
they are in the messages log. I don't have any of the "abrt-server"
messages, nor anything about it not being signed. The only thing in my
log for qmailmrtg is the segfaults (and the initial install I did with yum).
From: Eric <[email protected] <mailto:[email protected]>>
Reply-To: <[email protected]
<mailto:[email protected]>>
Date: Friday, June 10, 2016 at 12:02 PM
To: <[email protected]
<mailto:[email protected]>>
Subject: Re: [qmailtoaster] vchkpw segfaults and spamdyke errors
Hi Jamie,
I had these as well on a client server about a month ago for a few days
and they went away. They showed up in the postmaster logwatch email. I
traced them to the messages log.
I ran the following command (with output):
# cat messages* | grep -C 4 segfault
May 18 09:25:02 mail kernel: qmailmrtg[20759]: segfault at 604000 ip
0000000000400b17 sp 00007fff8f462560 error 4 in qmailmrtg[400000+2000]
May 18 09:25:02 mail abrt-server: Package 'qmailmrtg' isn't signed with
proper key
May 18 09:25:02 mail abrt-server: 'post-create' on
'/var/spool/abrt/ccpp-2016-05-18-09:25:02-20759' exited with 1
May 18 09:25:02 mail abrt-server: Deleting problem directory
'/var/spool/abrt/ccpp-2016-05-18-09:25:02-20759'
And, it looks like it has something to do with signing.
I got distracted and will have to investigate this further.
Eric
On 6/10/2016 9:15 AM, Jaime Lerner wrote:
I don't get segfaults from vchkpw anymore (not since raising my
softlimit), but I get from 1-3 segfaults from qmailmrtg daily. I don't
really need or want qmailmrtg, so if anyone can tell me how to turn it
off, that would be great. :) Otherwise, it's not causing any problems
for me since I don't use it.
[root@mail qmail]# grep segfault /var/log/messages
Jun 5 16:20:01 mail kernel: qmailmrtg[26761]: *segfault* at 604000 ip
0000000000400b17 sp 00007ffcbfdbc4a0 error 4 in qmailmrtg[400000+2000]
Jun 5 16:40:01 mail kernel: qmailmrtg[28163]: *segfault* at 604000 ip
0000000000400b17 sp 00007fffbac92110 error 4 in qmailmrtg[400000+2000]
Jun 5 16:55:01 mail kernel: qmailmrtg[29324]: *segfault* at 604000 ip
0000000000400b17 sp 00007fff8ca08810 error 4 in qmailmrtg[400000+2000]
Jun 6 12:20:01 mail kernel: qmailmrtg[30300]: *segfault* at 604000 ip
0000000000400b17 sp 00007ffe382ce270 error 4 in qmailmrtg[400000+2000]
Jun 7 11:25:01 mail kernel: qmailmrtg[10676]: *segfault* at 604000 ip
0000000000400b17 sp 00007fff0e24eff0 error 4 in qmailmrtg[400000+2000]
Jun 7 15:00:02 mail kernel: qmailmrtg[20856]: *segfault* at 604000 ip
0000000000400b17 sp 00007ffec310ee90 error 4 in qmailmrtg[400000+2000]
Jun 7 15:05:01 mail kernel: qmailmrtg[21134]: *segfault* at 604000 ip
0000000000400b17 sp 00007fff02dd4660 error 4 in qmailmrtg[400000+2000]
Jun 8 07:20:01 mail kernel: qmailmrtg[31909]: *segfault* at 604000 ip
0000000000400b17 sp 00007ffc923737e0 error 4 in qmailmrtg[400000+2000]
Jun 8 12:15:01 mail kernel: qmailmrtg[12908]: *segfault* at 604000 ip
0000000000400b17 sp 00007ffc75843060 error 4 in qmailmrtg[400000+2000]
Jun 9 12:15:01 mail kernel: qmailmrtg[11826]: *segfault* at 604000 ip
0000000000400b17 sp 00007ffd93dcda20 error 4 in qmailmrtg[400000+2000]
Jun 10 10:15:01 mail kernel: qmailmrtg[6510]: *segfault* at 604000 ip
0000000000400b17 sp 00007fff8e676c30 error 4 in qmailmrtg[400000+2000]
Jun 10 10:25:01 mail kernel: qmailmrtg[6931]: *segfault* at 604000 ip
0000000000400b17 sp 00007ffc884b8570 error 4 in qmailmrtg[400000+2000]
Jun 10 10:40:01 mail kernel: qmailmrtg[7683]: *segfault* at 604000 ip
0000000000400b17 sp 00007ffd97d86ab0 error 4 in qmailmrtg[400000+2000]
From: Steve Linberg <[email protected]
<mailto:[email protected]>
<mailto:[email protected]>>
Reply-To: <[email protected]
<mailto:[email protected]>
<mailto:[email protected]>>
Date: Friday, June 10, 2016 at 10:48 AM
To: <[email protected]
<mailto:[email protected]>
<mailto:[email protected]>>
Subject: Re: [qmailtoaster] vchkpw segfaults and spamdyke errors
Still working on the segfault problem; got dozens of them overnight when
I definitely wasn’t using any services. softlimit
in /var/qmail/supervise/submission/run is at 500 megs and rising, but
I’m still having trouble believing it needs to be that high or higher.
While I continue to push that: is there any way to know what process
chain is invoking the vchkpw process that’s segfaulting? Don’t a number
of different processes use it? I don’t know for a fact that submission
is the one that’s causing it. I can’t find any other clues in my logs,
like events happening at the same time as the segfaults, but are there
any other possible culprits that might invoke vchkpw without enough RAM
to do whatever it’s trying to do?
--
Steve Linberg, Chief Goblin
Silicon Goblin Technologies
http://silicongoblin.com
Be kind. Remember, everyone you meet is fighting a hard battle.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
<mailto:[email protected]>
For additional commands, e-mail: [email protected]
<mailto:[email protected]>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
