I do still have an interest in dropping the Phish.Phishing.REPHISH signatures. I will look into dropping these as well.
We identified the issue causing the PhishTank and Phish signatures to run slowly and how to make them run quickly, but have yet to decide whether or not to reintroduce the signatures as a new database, or perhaps under Email.Phishing or HTML.Phishing so that they can be enabled/disabled with clamd and clamscan configuration options. I’ll try to see that this is addressed sooner rather than later. Regards, Micah From: clamav-users <clamav-users-boun...@lists.clamav.net> on behalf of Al Varnell via clamav-users <clamav-users@lists.clamav.net> Reply-To: ClamAV users ML <clamav-users@lists.clamav.net> Date: Wednesday, April 17, 2019 at 7:03 AM To: "clamav-users@lists.clamav.net" <clamav-users@lists.clamav.net> Cc: Al Varnell <alvarn...@mac.com> Subject: Re: [clamav-users] [External] Re: Scan very slow There are still 2515 "Phish.Phishing.REPHISH_ID_...." signatures in daily.ldb -Al- On Apr 17, 2019, at 03:36, Maarten Broekman <maarten.broek...@gmail.com<mailto:maarten.broek...@gmail.com>> wrote: Are the "Phish" REPHISH signatures still in the daily or were they removed as well? Those were causing part of the issue. --Maarten On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote: An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were dropped by daily-25417 on 12 April, and I can't seem to locate any more. -Al- On Apr 17, 2019, at 02:01, Mark Allan via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote: Hi Micah, Sorry to pester you, but have you any update on when the remaining Phishtank signatures will be getting removed? It would be really great to get scan times properly back to normal. Best regards Mark On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micas...@cisco.com<mailto:micas...@cisco.com>> wrote: Mark, Yes, the plan is still to remove the rest of the Phishtank signatures. We wanted to get things back to relative normal and resolve the immediate crisis. We’ll remove the rest of them soon. Best, Micah From: Mark Allan <markjal...@gmail.com<mailto:markjal...@gmail.com>> Date: Tuesday, April 9, 2019 at 6:26 AM To: "Micah Snyder (micasnyd)" <micas...@cisco.com<mailto:micas...@cisco.com>> Cc: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> Subject: Re: [External] Re: [clamav-users] Scan very slow The scan times are definitely better than they were - in fact, they're back to how they were before last week's inclusion of the Phishtank signatures. They're still almost double what they used to be though, and as far as I can see, there are still almost 4000 Phishtank signatures in the DB: $ sigtool --find Phishtank | wc -l 3968 Can I request that those ones also be removed please? Best regards Mark On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micas...@cisco.com<mailto:micas...@cisco.com>> wrote: Tim, There are a couple of ways for users to drop specific categories of signatures at this time. Sadly, they wouldn’t have helped this last week. These include bytecode signatures, PUA (potentially unwanted applications) signatures, Email.Phishing and HTML.Phishing signatures, and the Safebrowsing database. If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank or Email.Phishing.Phishtank then they could have been disabled with the clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`). Maybe a better option would be for us to create a new optional database for phishing signatures. However, the names for the databases are hardcoded into freshclam, so it is non-trivial to add a new database and would require a few changes to ClamAV’s code. We have talked about making the databases easier to add/remove in the future so users can have more categories to enable/disable. In this light, it ties in well with existing plans. Of note the Phishtank sigs from Friday’s daily were removed yesterday and scan times should be back to normal. Regards, Micah From: Tim Hawkins <tim.hawk...@redflaggroup.com<mailto:tim.hawk...@redflaggroup.com>> Date: Friday, April 5, 2019 at 6:06 PM To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>, Mark Allan <markjal...@gmail.com<mailto:markjal...@gmail.com>> Cc: "Micah Snyder (micasnyd)" <micas...@cisco.com<mailto:micas...@cisco.com>> Subject: Re: [External] Re: [clamav-users] Scan very slow Hi Micah Does clamav partition the database so that signatures that are mainly associated with email scanning can be dropped out for folks only needing filesystems scans, none of our systems use email, and we dont make use of the mailer extension. Having to load all the email focused signatures could as you have observed impact performance. Sent from Nine<http://www.9folders.com/> ________________________________ From: "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> Sent: Saturday, April 6, 2019 03:18 To: ClamAV users ML; Mark Allan Cc: Micah Snyder (micasnyd) Subject: [External] Re: [clamav-users] Scan very slow Regarding slow scan times today (and slow scan times in general), it appears that the signatures we generate based on PhishTank’s feed for phishing URLs are resulting in very slow load and scan times. Today’s daily update saw 7448 new Phishtank signatures (much higher than usual) coinciding with the immediate performance drop for load time and scan time. One user reported that the load time today on some of his slower machines was slow enough to exceed the timeout for service startup (https://bugzilla.clamav.net/show_bug.cgi?id=12317). In limited testing on my own machine I saw the following change after dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file: * Database load time on my laptop went from 75.43203997612 seconds down to 14.859203100204468 seconds * Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec. After some discussion between the teams that work on ClamAV and ClamAV signature content and deployment, we’ve agreed to drop PhishTank signatures from the database until we can determine a way to craft Phishtank signatures without incurring such a significant performance hit. The daily update tomorrow will have the change. -Micah Micah Snyder ClamAV Development Talos Cisco Systems, Inc. From: clamav-users <clamav-users-boun...@lists.clamav.net<mailto:clamav-users-boun...@lists.clamav.net>> on behalf of "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> Reply-To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> Date: Friday, April 5, 2019 at 1:08 PM To: Mark Allan <markjal...@gmail.com<mailto:markjal...@gmail.com>>, ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> Cc: "Micah Snyder (micasnyd)" <micas...@cisco.com<mailto:micas...@cisco.com>> Subject: Re: [clamav-users] Scan very slow Hi Mark, Sorry about the delay in responding. I hadn’t looked at my clamav-users filter this morning. Just investigating now. Will respond when I know more. -Micah From: Mark Allan <markjal...@gmail.com<mailto:markjal...@gmail.com>> Date: Friday, April 5, 2019 at 9:12 AM To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>, "Micah Snyder (micasnyd)" <micas...@cisco.com<mailto:micas...@cisco.com>> Subject: Re: [clamav-users] Scan very slow Also CC'ing Micah directly as the mailing list would appear to be offline (at least lists.clamav.net<http://lists.clamav.net/> isn't responding to http requests anyway) It looks like scan times have gone through the roof. As Oya said, they're still considerably higher than they were a couple of months ago, but today's scan time is insane. Yesterday's scan using 0.101.2:58:25409:1554370140:1:63:48554:328 took 7m 3s On the same hardware, scanning the same read-only disk image, with today's scan using 0.101.2:58:25410:1554452941:1:63:48557:328 the scan time has jumped to 26m 15s This is the longest it has ever taken to scan this volume (cf my previous email of 25th March) Is there anything that can be excluded? Best regards Mark On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote: Thanks Oya for the update. We will continue to investigate the signature performance issue. Regards, Micah On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <clamav-users-boun...@lists.clamav.net<mailto:clamav-users-boun...@lists.clamav.net> on behalf of oyam...@promark-inc.com<mailto:oyam...@promark-inc.com>> wrote: Hi Micah It seems that the scanning slow down issue of this time has been solved at some level with CVD Update of the other day. However, there is still big discrepancy in between the current condition and the last condition in one month ago. Date Files Scan time 2019/02/15 2550338 08:53:57 2019/03/15 2612792 19:22:54 2019/03/26 2634489 18:13:56 2019/03/27 2637201 18:10:05 We know the improvement of this time is due to the details of CVD, because we did not make any change on the user's system. We are going to try some tuning for scanning. We like to know if you still have some room to make further improvement for this slow down issue. Thank you for your help, in advance. Best regards, Oya On Mon, 25 Mar 2019 15:45:02 +0000 "Micah Snyder \(micasnyd\) via clamav-users" <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote: > Hi Mark, all: > > I’m disappointed to hear that it is still slow for you. > > We found that the target-type of signatures used for PhishTank.Phishing signatures were causing a significant slowdown. We have dropped them as of this past Saturday (https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates have been re-adding them with more specific scan target types. We’re now investigating some other optimizations we can make for the next major ClamAV release to improve scan times but at present we don’t have any other leads for signatures that may be slowing down scans. > > Regards, > Micah > > > From: clamav-users <clamav-users-boun...@lists.clamav.net<mailto:clamav-users-boun...@lists.clamav.net>> on behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> > Date: Monday, March 25, 2019 at 9:37 AM > To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> > Cc: Mark Allan <markjal...@gmail.com<mailto:markjal...@gmail.com>> > Subject: Re: [clamav-users] Scan very slow > > Cheers Steve, > > In the interest of completeness, here's the scan from today (TXT from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked improvement in scan time, although at 6m 7s it's still almost twice what it used to be. > > Mark > > On Mon, 25 Mar 2019 at 12:56, Steve Basford <steveb_cla...@sanesecurity.com<mailto:steveb_cla...@sanesecurity.com><mailto:steveb_cla...@sanesecurity.com<mailto:steveb_cla...@sanesecurity.com>>> wrote: > On 2019-03-25 10:52, Mark Allan via clamav-users wrote: > > Hi all, > > > te. > > > > Hopefully this helps someone to narrow things down a bit. > > > > Mark > > > > 18/3/19 10m 49s TXT from DNS: > 0.101.1:58:25392:1552904941:1:63:48507:328 *** > > Here's the changes for the above update: > > https://lists.gt.net/clamav/virusdb/75154 > > You can also check sigs quickly per update: > > https://lists.gt.net/clamav/virusdb/ > > > > -- > Cheers, > > Steve > Twitter: @sanesecurity > > _______________________________________________ > > clamav-users mailing list > clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net><mailto:clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml _______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net> https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml _______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net> https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml DISCLAIMER The information contained in this email and any attachments are confidential. It is intended solely for the individual or entity to whom they are addressed. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The Red Flag Group is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt. Any advice, recommendations or opinion contained within this email or its attachments are not to be construed as legal advice. _______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net> https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml _______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net> https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
_______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
