Gotcha. Those were slowing the scans down more than the 3000-someodd
PhishTank sigs the last time I tested (Apr 9th).
daily_Phish.ldb ==== Time: 1.612 sec (0 m 1 s)
daily_Phishtank.ldb ==== Time: 0.146 sec (0 m 0 s)
2515 daily_Phish.ldb
3516 daily_Phishtank.ldb
On Wed, Apr 17, 2019 at 7:03 AM Al Varnell via clamav-users <
clamav-users@lists.clamav.net> wrote:
> There are still 2515 "Phish.Phishing.REPHISH_ID_...." signatures in
> daily.ldb
>
> -Al-
>
> On Apr 17, 2019, at 03:36, Maarten Broekman <maarten.broek...@gmail.com>
> wrote:
>
> Are the "Phish" REPHISH signatures still in the daily or were they removed
> as well? Those were causing part of the issue.
>
>
> --Maarten
>
> On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users <
> clamav-users@lists.clamav.net> wrote:
>
>> An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were
>> dropped by daily-25417 on 12 April, and I can't seem to locate any more.
>>
>> -Al-
>>
>> On Apr 17, 2019, at 02:01, Mark Allan via clamav-users <
>> clamav-users@lists.clamav.net> wrote:
>>
>> Hi Micah,
>>
>> Sorry to pester you, but have you any update on when the remaining
>> Phishtank signatures will be getting removed? It would be really great to
>> get scan times properly back to normal.
>>
>> Best regards
>> Mark
>>
>> On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micas...@cisco.com>
>> wrote:
>>
>>> Mark,
>>>
>>>
>>> Yes, the plan is still to remove the rest of the Phishtank signatures.
>>> We wanted to get things back to relative normal and resolve the immediate
>>> crisis. We’ll remove the rest of them soon.
>>>
>>>
>>>
>>> Best,
>>>
>>> Micah
>>>
>>>
>>>
>>> *From: *Mark Allan <markjal...@gmail.com>
>>> *Date: *Tuesday, April 9, 2019 at 6:26 AM
>>> *To: *"Micah Snyder (micasnyd)" <micas...@cisco.com>
>>> *Cc: *ClamAV users ML <clamav-users@lists.clamav.net>
>>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> The scan times are definitely better than they were - in fact, they're
>>> back to how they were before last week's inclusion of the Phishtank
>>> signatures. They're still almost double what they used to be though, and as
>>> far as I can see, there are still almost 4000 Phishtank signatures in the
>>> DB:
>>>
>>> $ sigtool --find Phishtank | wc -l
>>>
>>> 3968
>>>
>>>
>>>
>>> Can I request that those ones also be removed please?
>>>
>>>
>>>
>>> Best regards
>>>
>>> Mark
>>>
>>>
>>>
>>> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micas...@cisco.com>
>>> wrote:
>>>
>>> Tim,
>>>
>>>
>>>
>>> There are a couple of ways for users to drop specific categories of
>>> signatures at this time. Sadly, they wouldn’t have helped this last week.
>>> These include bytecode signatures, PUA (potentially unwanted applications)
>>> signatures, Email.Phishing and HTML.Phishing signatures, and the
>>> Safebrowsing database.
>>>
>>>
>>>
>>> If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank
>>> or Email.Phishing.Phishtank then they could have been disabled with the
>>> clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`).
>>>
>>>
>>>
>>> Maybe a better option would be for us to create a new optional database
>>> for phishing signatures. However, the names for the databases are hardcoded
>>> into freshclam, so it is non-trivial to add a new database and would
>>> require a few changes to ClamAV’s code. We have talked about making the
>>> databases easier to add/remove in the future so users can have more
>>> categories to enable/disable. In this light, it ties in well with existing
>>> plans.
>>>
>>>
>>>
>>> Of note the Phishtank sigs from Friday’s daily were removed yesterday
>>> and scan times should be back to normal.
>>>
>>>
>>>
>>> Regards,
>>>
>>> Micah
>>>
>>>
>>>
>>> *From: *Tim Hawkins <tim.hawk...@redflaggroup.com>
>>> *Date: *Friday, April 5, 2019 at 6:06 PM
>>> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, Mark Allan <
>>> markjal...@gmail.com>
>>> *Cc: *"Micah Snyder (micasnyd)" <micas...@cisco.com>
>>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Hi Micah
>>>
>>>
>>> Does clamav partition the database so that signatures that are mainly
>>> associated with email scanning can be dropped out for folks only needing
>>> filesystems scans, none of our systems use email, and we dont make use of
>>> the mailer extension.
>>>
>>> Having to load all the email focused signatures could as you have
>>> observed impact performance.
>>>
>>> Sent from Nine <http://www.9folders.com/>
>>> ------------------------------
>>>
>>> *From:* "Micah Snyder (micasnyd) via clamav-users" <
>>> clamav-users@lists.clamav.net>
>>> *Sent:* Saturday, April 6, 2019 03:18
>>> *To:* ClamAV users ML; Mark Allan
>>> *Cc:* Micah Snyder (micasnyd)
>>> *Subject:* [External] Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Regarding slow scan times today (and slow scan times in general), it
>>> appears that the signatures we generate based on PhishTank’s feed for
>>> phishing URLs are resulting in very slow load and scan times.
>>>
>>>
>>>
>>> Today’s daily update saw 7448 new Phishtank signatures (much higher than
>>> usual) coinciding with the immediate performance drop for load time and
>>> scan time. One user reported that the load time today on some of his
>>> slower machines was slow enough to exceed the timeout for service startup (
>>> https://bugzilla.clamav.net/show_bug.cgi?id=12317).
>>>
>>>
>>>
>>> In limited testing on my own machine I saw the following change after
>>> dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:
>>>
>>> - Database load time on my laptop went from 75.43203997612 seconds
>>> down to 14.859203100204468 seconds
>>> - Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.
>>>
>>>
>>>
>>> After some discussion between the teams that work on ClamAV and ClamAV
>>> signature content and deployment, we’ve agreed to drop PhishTank signatures
>>> from the database until we can determine a way to craft Phishtank
>>> signatures without incurring such a significant performance hit.
>>>
>>>
>>>
>>> The daily update tomorrow will have the change.
>>>
>>>
>>>
>>> -Micah
>>>
>>>
>>>
>>>
>>> Micah Snyder
>>> ClamAV Development
>>> Talos
>>> Cisco Systems, Inc.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From: *clamav-users <clamav-users-boun...@lists.clamav.net> on behalf
>>> of "Micah Snyder (micasnyd) via clamav-users" <
>>> clamav-users@lists.clamav.net>
>>> *Reply-To: *ClamAV users ML <clamav-users@lists.clamav.net>
>>> *Date: *Friday, April 5, 2019 at 1:08 PM
>>> *To: *Mark Allan <markjal...@gmail.com>, ClamAV users ML <
>>> clamav-users@lists.clamav.net>
>>> *Cc: *"Micah Snyder (micasnyd)" <micas...@cisco.com>
>>> *Subject: *Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Hi Mark,
>>>
>>>
>>>
>>> Sorry about the delay in responding. I hadn’t looked at my clamav-users
>>> filter this morning. Just investigating now. Will respond when I know
>>> more.
>>>
>>>
>>>
>>> -Micah
>>>
>>>
>>>
>>> *From: *Mark Allan <markjal...@gmail.com>
>>> *Date: *Friday, April 5, 2019 at 9:12 AM
>>> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, "Micah Snyder
>>> (micasnyd)" <micas...@cisco.com>
>>> *Subject: *Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Also CC'ing Micah directly as the mailing list would appear to be
>>> offline (at least lists.clamav.net isn't responding to http requests
>>> anyway)
>>>
>>>
>>>
>>> It looks like scan times have gone through the roof. As Oya said,
>>> they're still considerably higher than they were a couple of months ago,
>>> but today's scan time is insane.
>>>
>>>
>>>
>>> Yesterday's scan using
>>>
>>> 0.101.2:58:25409:1554370140:1:63:48554:328
>>>
>>> took 7m 3s
>>>
>>>
>>>
>>> On the same hardware, scanning the same read-only disk image, with
>>> today's scan using
>>>
>>> 0.101.2:58:25410:1554452941:1:63:48557:328
>>>
>>> the scan time has jumped to 26m 15s
>>>
>>>
>>>
>>> This is the longest it has ever taken to scan this volume (cf my
>>> previous email of 25th March)
>>>
>>>
>>>
>>> Is there anything that can be excluded?
>>>
>>>
>>>
>>> Best regards
>>>
>>> Mark
>>>
>>>
>>>
>>> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <
>>> clamav-users@lists.clamav.net> wrote:
>>>
>>> Thanks Oya for the update. We will continue to investigate the
>>> signature performance issue.
>>>
>>> Regards,
>>> Micah
>>>
>>> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <
>>> clamav-users-boun...@lists.clamav.net on behalf of
>>> oyam...@promark-inc.com> wrote:
>>>
>>> Hi Micah
>>>
>>> It seems that the scanning slow down issue of this time has been
>>> solved
>>> at some level with CVD Update of the other day.
>>> However, there is still big discrepancy in between the current
>>> condition and
>>> the last condition in one month ago.
>>>
>>> Date Files Scan time
>>> 2019/02/15 2550338 08:53:57
>>> 2019/03/15 2612792 19:22:54
>>> 2019/03/26 2634489 18:13:56
>>> 2019/03/27 2637201 18:10:05
>>>
>>> We know the improvement of this time is due to the details of CVD,
>>> because
>>> we did not make any change on the user's system.
>>> We are going to try some tuning for scanning.
>>>
>>> We like to know if you still have some room to make further
>>> improvement
>>> for this slow down issue.
>>> Thank you for your help, in advance.
>>>
>>> Best regards,
>>> Oya
>>>
>>> On Mon, 25 Mar 2019 15:45:02 +0000
>>> "Micah Snyder \(micasnyd\) via clamav-users" <
>>> clamav-users@lists.clamav.net> wrote:
>>>
>>> > Hi Mark, all:
>>> >
>>> > I’m disappointed to hear that it is still slow for you.
>>> >
>>> > We found that the target-type of signatures used for
>>> PhishTank.Phishing signatures were causing a significant slowdown. We
>>> have dropped them as of this past Saturday (
>>> https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates
>>> have been re-adding them with more specific scan target types. We’re now
>>> investigating some other optimizations we can make for the next major
>>> ClamAV release to improve scan times but at present we don’t have any other
>>> leads for signatures that may be slowing down scans.
>>> >
>>> > Regards,
>>> > Micah
>>> >
>>> >
>>> > From: clamav-users <clamav-users-boun...@lists.clamav.net> on
>>> behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net>
>>> > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net>
>>> > Date: Monday, March 25, 2019 at 9:37 AM
>>> > To: ClamAV users ML <clamav-users@lists.clamav.net>
>>> > Cc: Mark Allan <markjal...@gmail.com>
>>> > Subject: Re: [clamav-users] Scan very slow
>>> >
>>> > Cheers Steve,
>>> >
>>> > In the interest of completeness, here's the scan from today (TXT
>>> from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked
>>> improvement in scan time, although at 6m 7s it's still almost twice what it
>>> used to be.
>>> >
>>> > Mark
>>> >
>>> > On Mon, 25 Mar 2019 at 12:56, Steve Basford <
>>> steveb_cla...@sanesecurity.com<mailto:steveb_cla...@sanesecurity.com>>
>>> wrote:
>>> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
>>> > > Hi all,
>>> > >
>>> > te.
>>> > >
>>> > > Hopefully this helps someone to narrow things down a bit.
>>> > >
>>> > > Mark
>>> > >
>>> >
>>> > 18/3/19 10m 49s TXT from DNS:
>>> > 0.101.1:58:25392:1552904941:1:63:48507:328 ***
>>> >
>>> > Here's the changes for the above update:
>>> >
>>> > https://lists.gt.net/clamav/virusdb/75154
>>> >
>>> > You can also check sigs quickly per update:
>>> >
>>> > https://lists.gt.net/clamav/virusdb/
>>> >
>>> >
>>> >
>>> > --
>>> > Cheers,
>>> >
>>> > Steve
>>> > Twitter: @sanesecurity
>>> >
>>> > _______________________________________________
>>> >
>>> > clamav-users mailing list
>>> > clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net
>>> >
>>> > https://lists.clamav.net/mailman/listinfo/clamav-users
>>> >
>>> >
>>> > Help us build a comprehensive ClamAV guide:
>>> > https://github.com/vrtadmin/clamav-faq
>>> >
>>> > http://www.clamav.net/contact.html#ml
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clamav-users@lists.clamav.net
>>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq
>>>
>>> http://www.clamav.net/contact.html#ml
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clamav-users@lists.clamav.net
>>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq
>>>
>>> http://www.clamav.net/contact.html#ml
>>>
>>>
>>>
>>> *DISCLAIMER*
>>>
>>> The information contained in this email and any attachments are
>>> confidential. It is intended solely for the individual or entity to whom
>>> they are addressed. Access to this email by anyone else is unauthorized.
>>>
>>> If you are not the intended recipient, any disclosure, copying,
>>> distribution or any action taken or omitted to be taken in reliance on it,
>>> is prohibited and may be unlawful. If you have received this communication
>>> in error, please notify us immediately by responding to this email and then
>>> delete it from your system.
>>>
>>> The Red Flag Group is neither liable for the proper and complete
>>> transmission of the information contained in this communication nor for any
>>> delay in its receipt.
>>>
>>> Any advice, recommendations or opinion contained within this email or
>>> its attachments are not to be construed as legal advice.
>>>
>>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>>
>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>>
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
