Re: [clamav-users] [External] Re: Scan very slow

JME > > -Message d'origine- > De : clamav-users De la part > de Brent Clark via clamav-users > Envoyé : mercredi 10 avril 2019 12:33 > À : ClamAV users ML > Cc : Brent Clark > Objet : Re: [clamav-users] [External] Re: Scan very slow > >

Re: [clamav-users] [External] Re: Scan very slow

Reply-To: ClamAV users ML Date: Thursday, April 18, 2019 at 6:09 AM To: ClamAV users ML Cc: Mark Allan Subject: Re: [clamav-users] [External] Re: Scan very slow Fantastic! I can also confirm that scan times are back to normal now - more-or-less back to what they were in early February. The time

Re: [clamav-users] [External] Re: Scan very slow

Fantastic! I can also confirm that scan times are back to normal now - more-or-less back to what they were in early February. The time for one of our FP test volumes which I've been referencing in this thread is back down to 3m 30s, and the total time for our *full* FP test is back down from

Re: [clamav-users] [External] Re: Scan very slow

Looks like all Phish.Phishing.REPHISH_ID_... signatures were dropped by daily-25423 today. -Al- > On Apr 17, 2019, at 04:02, Al Varnell wrote: > > There are still 2515 "Phish.Phishing.REPHISH_ID_" signatures in daily.ldb > > -Al- > >> On Apr 17, 2019, at 03:36, Maarten Broekman >

Re: [clamav-users] [External] Re: Scan very slow

clamav-users Reply-To: ClamAV users ML Date: Wednesday, April 17, 2019 at 7:03 AM To: "clamav-users@lists.clamav.net" Cc: Al Varnell Subject: Re: [clamav-users] [External] Re: Scan very slow There are still 2515 "Phish.Phishing.REPHISH_ID_" signatures in daily.ldb -Al- On

Re: [clamav-users] [External] Re: Scan very slow

Gotcha. Those were slowing the scans down more than the 3000-someodd PhishTank sigs the last time I tested (Apr 9th). daily_Phish.ldb Time: 1.612 sec (0 m 1 s) daily_Phishtank.ldb Time: 0.146 sec (0 m 0 s) 2515 daily_Phish.ldb 3516 daily_Phishtank.ldb On Wed, Apr 17, 2019 at

Re: [clamav-users] [External] Re: Scan very slow

There are still 2515 "Phish.Phishing.REPHISH_ID_" signatures in daily.ldb -Al- > On Apr 17, 2019, at 03:36, Maarten Broekman > wrote: > > Are the "Phish" REPHISH signatures still in the daily or were they removed as > well? Those were causing part of the

Re: [clamav-users] [External] Re: Scan very slow

Are the "Phish" REPHISH signatures still in the daily or were they removed as well? Those were causing part of the issue. --Maarten On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users < clamav-users@lists.clamav.net> wrote: > An additional 3968 Phishtank.Phishing.PHISH_ID_???

Re: [clamav-users] [External] Re: Scan very slow

An additional 3968 Phishtank.Phishing.PHISH_ID_??? signatures were dropped by daily-25417 on 12 April, and I can't seem to locate any more. -Al- > On Apr 17, 2019, at 02:01, Mark Allan via clamav-users > wrote: > > Hi Micah, > > Sorry to pester you, but have you any update on when the

Re: [clamav-users] [External] Re: Scan very slow

Hi Micah, Sorry to pester you, but have you any update on when the remaining Phishtank signatures will be getting removed? It would be really great to get scan times properly back to normal. Best regards Mark On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) wrote: > Mark, > > > Yes, the

Re: [clamav-users] [External] Re: Scan very slow

Regexes can be slow or even extremely slow to apply, depending on the implementation. Backtracking is the worst, perhaps taking exponential time, but often is cut off by artificial limits. Does ClamAV perchance precompute Deterministic Finite Automata for the regexes? These run fast, but take

Re: [clamav-users] [External] Re: Scan very slow

We don't use the word engine in quite that way with ClamAV, but I think I understand your question. With regards to the word "engine": Clamd builds a scanning engine based on the databases and configuration options. The engine is shared by scanning threads. With regards to clamd's

Re: [clamav-users] [External] Re: Scan very slow

Objet : Re: [clamav-users] [External] Re: Scan very slow Thanks for doing this. What Im getting out of your feedback is that maybe you guys need to look to implementing or relooking at your CI process(es). Before pushing a commit, your CI can run the same test(s)

Re: [clamav-users] [External] Re: Scan very slow

Thanks for doing this. What Im getting out of your feedback is that maybe you guys need to look to implementing or relooking at your CI process(es). Before pushing a commit, your CI can run the same test(s) and alert on slow or long running scans. All this can be automated and report on

Re: [clamav-users] [External] Re: Scan very slow

On 2019-04-09 22:29, Micah Snyder (micasnyd) via clamav-users wrote: Maarten, Looking at a few of the Phish.Phishing signatures, these appear to have the same issue (href="http:// prefix). In testing with scan of a PDF document, I was able to reduce the scan time from 31.987 sec down to 2.632

Re: [clamav-users] [External] Re: Scan very slow

into scan time optimization is definitely welcome and > appreciated. > > > > Regards, > > Micah > > > > > > *From: *clamav-users on behalf of > Maarten Broekman via clamav-users > *Reply-To: *ClamAV users ML > *Date: *Tuesday, April 9, 2019 at 12:00 PM

Re: [clamav-users] [External] Re: Scan very slow

ers on behalf of Maarten Broekman via clamav-users Reply-To: ClamAV users ML Date: Tuesday, April 9, 2019 at 12:00 PM To: ClamAV users ML Cc: Maarten Broekman Subject: Re: [clamav-users] [External] Re: Scan very slow Clearly the latest daily.cvd is performing better, but the re

Re: [clamav-users] [External] Re: Scan very slow

Clearly the latest daily.cvd is performing better, but the remaining "Phishtank" sigs are *not* a majority of the slowness. I unpacked the current (?) cvd (ClamAV-VDB:09 Apr 2019 03-53 -0400:25414:1548262:63:X:X:raynman:1554796413) and then ran a test scan with each part to see what the load

Re: [clamav-users] [External] Re: Scan very slow

Mark, Yes, the plan is still to remove the rest of the Phishtank signatures. We wanted to get things back to relative normal and resolve the immediate crisis. We’ll remove the rest of them soon. Best, Micah From: Mark Allan Date: Tuesday, April 9, 2019 at 6:26 AM To: "Micah Snyder

Re: [clamav-users] [External] Re: Scan very slow

On 2019-04-09 12:02, Brent Clark via clamav-users wrote: Cant those be adopted / managed by Sanesecurity? For all you know, those are already in Sanesecurity. They are... and have been for quite some time: "The following databases are distributed by Sanesecurity, but produced by Porcupine

Re: [clamav-users] [External] Re: Scan very slow

Cant those be adopted / managed by Sanesecurity? For all you know, those are already in Sanesecurity. Regards Brent Clark On 2019/04/09 12:25, Mark Allan via clamav-users wrote: The scan times are definitely better than they were - in fact, they're back to how they were before last week's

Re: [clamav-users] [External] Re: Scan very slow

The scan times are definitely better than they were - in fact, they're back to how they were before last week's inclusion of the Phishtank signatures. They're still almost double what they used to be though, and as far as I can see, there are still almost 4000 Phishtank signatures in the DB: $

Re: [clamav-users] [External] Re: Scan very slow

Having the Phishtank sigs as an additional optional database would be great and, from my perspective, well worth the effort since we don't use them. On Sun, Apr 7, 2019 at 9:44 AM Micah Snyder (micasnyd) via clamav-users < clamav-users@lists.clamav.net> wrote: > Tim, > > > > There are a couple

Re: [clamav-users] [External] Re: Scan very slow

Tim, There are a couple of ways for users to drop specific categories of signatures at this time. Sadly, they wouldn’t have helped this last week. These include bytecode signatures, PUA (potentially unwanted applications) signatures, Email.Phishing and HTML.Phishing signatures, and the

Re: [clamav-users] [External] Re: Scan very slow

Given that the PhishTank signatures, specifically, have been causing the performance issues, no. It's not unreasonable to want to pull them, and only them, out. Having them in a separate db file would be highly beneficial to those of us that don't want or need them at all. Barring that, having a

Re: [clamav-users] [External] Re: Scan very slow

Hi Micah Does clamav partition the database so that signatures that are mainly associated with email scanning can be dropped out for folks only needing filesystems scans, none of our systems use email, and we dont make use of the mailer extension. Having to load all the email focused