JME,
As you've pointed out, it appears that some signatures containing a PCRE regex
components are responsible for slow scan times on larger email files.
I did a bunch of profiling similar to what Maarten did earlier in order to
narrow it down. I found that Email.Phishing.VOF2 signatures are performing
slower with the eml sample you sent me. Email.Phishing.VOF2 signatures contain
a PCRE regex component to alert on email attachments with specific names. Now
that we've determined which signatures are performing slowly in these cases, I
am hopeful that we will be able to optimize the Email.Phishing.VOF2 signatures
to improve performance.
I will note that your idea to lower the PCRERecMatchLimit setting to 1 will
effectively neuter all signatures that rely on regexes and so I can't recommend
this.
Regards,
Micah
On 4/10/19, 12:36 PM, "clamav-users on behalf of JME via clamav-users"
<clamav-users-boun...@lists.clamav.net on behalf of
clamav-users@lists.clamav.net> wrote:
Helo,
I managed to significantly reduce the problems of very long analysis, more
than 400sec on some emails. Not by disabling PhishingSignatures that did not
work. But putting: PCRERecMatchLimit to 1.
The PCRE analyzes are thus bypassed, but SafeBrawsing and the other scans
continue to work. Is it a mistake to precede as well?
Regards,
JME
-----Message d'origine-----
De : clamav-users <clamav-users-boun...@lists.clamav.net> De la part de
Brent Clark via clamav-users
Envoyé : mercredi 10 avril 2019 12:33
À : ClamAV users ML <clamav-users@lists.clamav.net>
Cc : Brent Clark <brentgclarkl...@gmail.com>
Objet : Re: [clamav-users] [External] Re: Scan very slow
Thanks for doing this.
What Im getting out of your feedback is that maybe you guys need to look to
implementing or relooking at your CI process(es).
Before pushing a commit, your CI can run the same test(s) and alert on slow
or long running scans.
All this can be automated and report on issues.
I highly recommend to doing this, I dont think you guys realise how many
systems are running and dependent on Clamav. Might be a good time to too remind
the community and ask to support and donate for the project.
HTH
Regards
Brent
On 2019/04/09 17:58, Maarten Broekman via clamav-users wrote:
> Clearly the latest daily.cvd is performing better, but the remaining
> "Phishtank" sigs are _not_ a majority of the slowness.
>
> I unpacked the current (?) cvd (ClamAV-VDB:09 Apr 2019 03-53
> -0400:25414:1548262:63:X:X:raynman:1554796413) and then ran a test
> scan with each part to see what the load times looked like:
>
> daily.cdb ==== Time: 0.007 sec (0 m 0 s)
> daily.cfg ==== Time: 0.004 sec (0 m 0 s)
> daily.crb ==== Time: 0.006 sec (0 m 0 s)
> *daily.cvd ==== Time: 11.384 sec (0 m 11 s)*
> daily.fp ==== Time: 0.009 sec (0 m 0 s)
> daily.ftm ==== Time: 0.005 sec (0 m 0 s)
> daily.hdb ==== Time: 0.303 sec (0 m 0 s)
> daily.hdu ==== Time: 0.006 sec (0 m 0 s)
> daily.hsb ==== Time: 1.093 sec (0 m 1 s)
> daily.hsu ==== Time: 0.005 sec (0 m 0 s)
> daily.idb ==== Time: 0.006 sec (0 m 0 s)
> *daily.ldb ==== Time: 5.563 sec (0 m 5 s)
> *
> daily.ldu ==== Time: 0.005 sec (0 m 0 s)
> daily.mdb ==== Time: 0.061 sec (0 m 0 s)
> daily.mdu ==== Time: 0.007 sec (0 m 0 s)
> daily.msb ==== Time: 0.005 sec (0 m 0 s)
> daily.msu ==== Time: 0.005 sec (0 m 0 s)
> daily.ndb ==== Time: 0.017 sec (0 m 0 s)
> daily.ndu ==== Time: 0.005 sec (0 m 0 s)
> daily.pdb ==== Time: 0.010 sec (0 m 0 s)
> daily.sfp ==== Time: 0.006 sec (0 m 0 s)
> daily.wdb ==== Time: 0.014 sec (0 m 0 s)
>
> So, half the run time of a clamscan is from the daily.ldb. To break it
> down farther, I split the daily.ldb into "daily_<virus>.ldb" where
> <virus> is the first part of the dot-separated signature name.
>
> daily_Andr.ldb ==== Time: 0.008 sec (0 m 0 s)
> daily_Archive.ldb ==== Time: 0.009 sec (0 m 0 s)
> daily_Asp.ldb ==== Time: 0.004 sec (0 m 0 s)
> daily_Doc.ldb ==== Time: 0.116 sec (0 m 0 s)
> daily_Eicar-Test-Signature.ldb ==== Time: 0.009 sec (0 m 0 s)
> daily_Email.ldb ==== Time: 0.014 sec (0 m 0 s)
> daily_Emf.ldb ==== Time: 0.007 sec (0 m 0 s)
> daily_Heuristics.ldb ==== Time: 0.006 sec (0 m 0 s)
> daily_Html.ldb ==== Time: 0.010 sec (0 m 0 s)
> daily_Hwp.ldb ==== Time: 0.005 sec (0 m 0 s)
> daily_Img.ldb ==== Time: 0.006 sec (0 m 0 s)
> daily_Ios.ldb ==== Time: 0.006 sec (0 m 0 s)
> daily_Java.ldb ==== Time: 0.005 sec (0 m 0 s)
> daily_Js.ldb ==== Time: 0.007 sec (0 m 0 s)
> daily_Legacy.ldb ==== Time: 0.006 sec (0 m 0 s)
> daily_Lnk.ldb ==== Time: 0.005 sec (0 m 0 s)
> daily_Mp4.ldb ==== Time: 0.005 sec (0 m 0 s)
> daily_Multios.ldb ==== Time: 0.005 sec (0 m 0 s)
> daily_Osx.ldb ==== Time: 0.008 sec (0 m 0 s)
> daily_Pdf.ldb ==== Time: 0.007 sec (0 m 0 s)
> *daily_Phish.ldb ==== Time: 1.612 sec (0 m 1 s)*
> daily_Phishtank.ldb ==== Time: 0.146 sec (0 m 0 s)
> daily_Php.ldb ==== Time: 0.006 sec (0 m 0 s)
> daily_Ppt.ldb ==== Time: 0.007 sec (0 m 0 s)
> daily_Py.ldb ==== Time: 0.006 sec (0 m 0 s)
> daily_Rtf.ldb ==== Time: 0.006 sec (0 m 0 s)
> daily_Svg.ldb ==== Time: 0.005 sec (0 m 0 s)
> daily_Swf.ldb ==== Time: 0.007 sec (0 m 0 s)
> daily_Ttf.ldb ==== Time: 0.005 sec (0 m 0 s)
> daily_Txt.ldb ==== Time: 0.009 sec (0 m 0 s)
> daily_Unix.ldb ==== Time: 0.008 sec (0 m 0 s)
> daily_Vbs.ldb ==== Time: 0.009 sec (0 m 0 s)
> *daily_Win.ldb ==== Time: 3.391 sec (0 m 3 s)*
> daily_Xls.ldb ==== Time: 0.009 sec (0 m 0 s)
> daily_Xml.ldb ==== Time: 0.007 sec (0 m 0 s)
>
>
> "Phish.", not "Phishtank.", and "Win." are the longest run times.
> Looking at the /number/ of signatures in each, the 'Phish.' signatures
> are taking a disproportionate amount of time to load compared to the
> other signatures:
>
> 216 daily_Andr.ldb
> 3 daily_Archive.ldb
> 1 daily_Asp.ldb
> 2096 daily_Doc.ldb
> 1 daily_Eicar-Test-Signature.ldb
> 1017 daily_Email.ldb
> 2 daily_Emf.ldb
> 5 daily_Heuristics.ldb
> 250 daily_Html.ldb
> 1 daily_Hwp.ldb
> 15 daily_Img.ldb
> 6 daily_Ios.ldb
> 16 daily_Java.ldb
> 69 daily_Js.ldb
> 27 daily_Legacy.ldb
> 9 daily_Lnk.ldb
> 1 daily_Mp4.ldb
> 9 daily_Multios.ldb
> 175 daily_Osx.ldb
> 132 daily_Pdf.ldb
> 2515 daily_Phish.ldb
> 3516 daily_Phishtank.ldb
> 18 daily_Php.ldb
> 5 daily_Ppt.ldb
> 3 daily_Py.ldb
> 28 daily_Rtf.ldb
> 1 daily_Svg.ldb
> 103 daily_Swf.ldb
> 2 daily_Ttf.ldb
> 140 daily_Txt.ldb
> 222 daily_Unix.ldb
> 21 daily_Vbs.ldb
> 43928 daily_Win.ldb
> 165 daily_Xls.ldb
> 8 daily_Xml.ldb
>
>
> From the look of it, "Phish." has those REPHISH signatures. Those
> signatures seem to be looking at any file (Target 0) and have
> subsignatures that are combined to match depending on which filetype
> they are 'looking' for (so, href for HTML files, %PDF, Subtype, and
> URI objects for PDFs, etc) as opposed to the remaining Phishtank sigs
> which seem to have a separate signature depending on the target type.
>
> Breaking up daily_Win into it's constituent sub-parts doesn't reveal
> any particular culprit from just a simple scan timing though...
>
> daily_Win.Adware.ldb ==== Time: 0.013 sec (0 m 0 s)
> daily_Win.Coinminer.ldb ==== Time: 0.009 sec (0 m 0 s)
> daily_Win.Downloader.ldb ==== Time: 0.035 sec (0 m 0 s)
> daily_Win.Dropper.ldb ==== Time: 0.240 sec (0 m 0 s)
> daily_Win.Exploit.ldb ==== Time: 0.016 sec (0 m 0 s)
> daily_Win.Ircbot.ldb ==== Time: 0.006 sec (0 m 0 s)
> daily_Win.Keylogger.ldb ==== Time: 0.010 sec (0 m 0 s)
> daily_Win.ldb ==== Time: 3.418 sec (0 m 3 s)
> daily_Win.Macro.ldb ==== Time: 0.009 sec (0 m 0 s)
> *daily_Win.Malware.ldb ==== Time: 0.731 sec (0 m 0 s)*
> daily_Win.Packed.ldb ==== Time: 0.131 sec (0 m 0 s)
> daily_Win.Packer.ldb ==== Time: 0.008 sec (0 m 0 s)
> daily_Win.Phishing.ldb ==== Time: 0.005 sec (0 m 0 s)
> daily_Win.Proxy.ldb ==== Time: 0.005 sec (0 m 0 s)
> daily_Win.Ransomware.ldb ==== Time: 0.019 sec (0 m 0 s)
> daily_Win.Spyware.ldb ==== Time: 0.006 sec (0 m 0 s)
> daily_Win.Tool.ldb ==== Time: 0.008 sec (0 m 0 s)
> *daily_Win.Trojan.ldb ==== Time: 0.582 sec (0 m 0 s)*
> daily_Win.Virus.ldb ==== Time: 0.059 sec (0 m 0 s)
> daily_Win.Worm.ldb ==== Time: 0.030 sec (0 m 0 s)
>
> 158 daily_Win.Adware.ldb
> 14 daily_Win.Coinminer.ldb
> 561 daily_Win.Downloader.ldb
> 8084 daily_Win.Dropper.ldb
> 216 daily_Win.Exploit.ldb
> 9 daily_Win.Ircbot.ldb
> 193 daily_Win.Keylogger.ldb
> 43928 daily_Win.ldb
> 1 daily_Win.Macro.ldb
> * 14820 daily_Win.Malware.ldb*
> 4209 daily_Win.Packed.ldb
> 20 daily_Win.Packer.ldb
> 2 daily_Win.Phishing.ldb
> 4 daily_Win.Proxy.ldb
> 500 daily_Win.Ransomware.ldb
> 32 daily_Win.Spyware.ldb
> 121 daily_Win.Tool.ldb
> * 12051 daily_Win.Trojan.ldb*
> 1967 daily_Win.Virus.ldb
> 966 daily_Win.Worm.ldb
>
>
> Malware and Trojan take the longest, but they also have a majority of
> the signatures.
>
> On Tue, Apr 9, 2019 at 11:19 AM Steve Basford
> <steveb_cla...@sanesecurity.com
> <mailto:steveb_cla...@sanesecurity.com>>
> wrote:
>
> On 2019-04-09 12:02, Brent Clark via clamav-users wrote:
> > Cant those be adopted / managed by Sanesecurity?
> >
> > For all you know, those are already in Sanesecurity.
>
> They are... and have been for quite some time:
>
>
> "The following databases are distributed by Sanesecurity, but produced
> by Porcupine Signatures"
>
> phishtank.ndb.
>
> Briefly...
>
> Number of sigs in phishtank.ndb: 9,309
>
> eg:
>
> PhishTank.Phishing.6002281, matches:
>
> https://www.phishtank.com/phish_detail.php?phish_id=6002281
>
> So, there is going to be some possible cross over now that
> Phish.Phishing.REPHISH_ID_20190404_67-6931549-0
> type signatures names from PhishTank feed are in daily.ldb and
> daily.ndb.
>
> I'll check back on the thread later.
>
> --
> Cheers,
>
> Steve
> Twitter: @sanesecurity
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
