Thank you for this comprehensive write up, Andy.
On 7/23/2020 12:14 PM, Andrew Swartz wrote:
My machine only had 2GB because I run a VERY bare bones server with no
web capabilities.
It has been about 8 months since I went down this rabbit hole, so
forgive me if I get a couple minor points wrong.
* The 'qq soft reject' is generated by qmail-queue (thus the 'qq'
part) when something in the path between qmail and the queue failed
for unclear reasons. Therefore the log entry is not super helpful. But
if you search the web on 'qmail' and 'qq soft reject' all the
meaningful results are from back in the 2010 era, but they all seemed
to have determined that clamd and insufficient RAM were the issue.
* For most stable toasters, the clam signature file is the only thing
between qmail and the the queue file which changes (i.e. grows)
frequently. But if you upgrade to a different version of clamd, this
change could easily be enough change in RAM usage to hit your RAM limit.
* I found that the best way to monitor clamd is by installing and
running "clamdtop" which is a top-like utility which connects to clamd
and displays its state in real time. It looks like this:
Notice that it is using 972 megabytes of RAM; that is almost entirely
the signature file residing in RAM (969 MB). I just sent a test
message with a one megabyte attachment, and the total clamd memory
went up to 977 MB for 2-3 seconds while scanning that message (which
you can see in real time with clamdtop). When clamd crashes, you'll
immediately see it because clamdtop immediately exits complaining that
it lost its connection. When I first ran clamdtop, I was aghast that
clamd was using an order of magnitude more memory than anything else
on the machine, and that this is normal behavior. But note that this
memory usage will slowly but relentlessly increase as the size of the
signature necessarily creeps upward.
As best I can remember, I think that clamd crashes when a message
arrives to be scanned, but it may be when freshclam runs and clamd
attempts to load the new signature file (its been 8 months and I did
not keep notes of the debugging). You can sort it out by running
clamdtop in one console and either manually running freshclam or
sending a message in another console, while also monitoring the log
file in another console (using 'tailf' command). You can see the
memory usage, the crash, and the 'qq soft reject' log entry all in
real time.
But the easiest test is to just give the VM another 1-2GB of RAM and
see if the problem goes away. If it does, this is almost certainly
the problem.
Hope this helps,
-Andy
On 7/23/2020 7:27 AM, Remo Mattei wrote:
well I have 6GB and it still happens at times to get the qq. I have changed the
ran on the run shall see. This only happens after I convert from Eric original
Clamv to the new rpm.
Remo
On Jul 23, 2020, at 12:40 AM, Andrew Swartz<[email protected]> wrote:
I had this problem about 8 months ago. It it was extremely difficult to
troubleshoot, but I eventually figured it out.
It is a problem which has been around for a decade or more. The clamav deamon
signature file, which is updated frequently, continuously grows as the amount
of malware it needs to recognize grows. Eventually, the signature file gets so
big that clamav daemon crashes when it tries to load it due to insufficient
RAM. But it was hard to diagnose because the deamon does not crash at startup
or when it updates the signature file, but rather when it is passed an email to
scan. You can confirm this by restarting clamav and noting that it will run
fine until a mail comes in, at which point it crashes and qmail starts
reporting the 'qq soft reject' to the log.
I was running on CentOS-7 VM with 2GB of RAM. I increased the RAM up to 4GB
and it fixed the problem.
Unfortunately, the signature file will always continue to grow as more malware
accrues, so in another couple years I'll surely need to increase the RAM again.
Hope this helps.
-Andy
On 7/20/2020 10:01 AM, Angus McIntyre wrote:
My qmailtoaster running on CentOS 7 was behaving fine, but now seems to soft
reject everything, and I'm having a hard time working out why.
It doesn't seem to be a ClamAV issue: I set 'clam=no' in
'/var/qmail/control/simcontrol' and restarted qmail, but I still get the
rejections.
I added 'SIMSCAN_DEBUG="5"' to the list of env vars in
'/etc/tcprules.d/tcp.smtp', but that doesn't seem to generate any actionable debugging
output anywhere that I can see.
Does anyone have any suggestions for debugging this issue? I know there's been
some talk of bad signatures for ClamAV recently, but I _thought_ I'd eliminated
that as a possibility by turning off clam in simcontrol. If that's not the
case, how would I identify (and suppress) a bad signature?
Thanks,
Angus
---------------------------------------------------------------------
To unsubscribe, e-mail:[email protected]
For additional commands, e-mail:[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail:[email protected]
For additional commands, e-mail:[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail:[email protected]
For additional commands, e-mail:[email protected]
