Hi Dave, I noticed that the safebrowsing CVD was updated (I'm seeing version 48474 now) but the cdiff takes a VERY long time to apply and the new gdb file takes about the same time to load.
Freshclam hangs at this point: Wed Mar 6 16:03:05 2019 -> *Retrieving http://db.US.clamav.net/safebrowsing-48474.cdiff Wed Mar 6 16:03:05 2019 -> *Trying to download http://db.US.clamav.net/safebrowsing-48474.cdiff (IP: 104.16.218.84) Wed Mar 6 16:03:06 2019 -> Downloading safebrowsing-48474.cdiff [100%] # /opt/clamav/clamav/bin/clamscan -d ./safebrowsing.gdb /opt/scripts/signatures/samples/clam_test.html /opt/scripts/signatures/samples/clam_test.html: OK ----------- SCAN SUMMARY ----------- Known viruses: 3229612 Engine version: 0.100.2 Scanned directories: 0 Scanned files: 1 Infected files: 0 Data scanned: 0.00 MB Data read: 0.00 MB (ratio 0.00:1) Time: 79.206 sec (1 m 19 s) Looking at the gdb contents, the file is still sorted in the same way as version 48473 (by the hash in the third field), rather than by the second field (P / F). When I re-sort the file by the second field, it loads in under 5 seconds. --Maarten On Wed, Mar 6, 2019 at 12:22 PM David Raynor <dray...@sourcefire.com> wrote: > Maarten, > > Thanks for reporting that. There is an ordering difference of the content > in the latest GDB file which is affecting the load time, and we will be > fixing that in the next safebrowsing CVD version. > > Dave R. > > On Wed, Mar 6, 2019 at 10:42 AM Maarten Broekman via clamav-users < > clamav-users@lists.clamav.net> wrote: > >> I'm not sure if the safebrowsing.cld is included in the daily cdiff, but >> the current safebrowsing.cld takes between 50 and 70 seconds to *load* into >> clamscan, where a copy from February loads in <5 seconds. >> >> safebrowsing data: >> Old (fast): ClamAV-VDB:13 Feb 2019 13-16 >> -0500:48472:3041760:63:X:X:google:1550081775 >> >> >> New (slow): ClamAV-VDB:05 Mar 2019 19-20 >> -0500:48473:3229612:63:X:X:google:1551831615 >> >> >> >> Anyone know what might have changed in there to so drastically increased >> the load time? >> >> This happened after freshclam ran last night. >> >> # /opt/clamav/clamav/bin/clamscan -d ~/safebrowsing.cld >> samples/clam_test.html >> samples/clam_test.html: OK >> >> ----------- SCAN SUMMARY ----------- >> Known viruses: 3041760 >> Engine version: 0.100.2 >> Scanned directories: 0 >> Scanned files: 1 >> Infected files: 0 >> Data scanned: 0.00 MB >> Data read: 0.00 MB (ratio 0.00:1) >> Time: 2.423 sec (0 m 2 s) >> >> # /opt/clamav/clamav/bin/clamscan -d >> /opt/clamav/var/lib/clamav/safebrowsing.cld samples/clam_test.html >> samples/clam_test.html: OK >> >> ----------- SCAN SUMMARY ----------- >> Known viruses: 3229612 >> Engine version: 0.100.2 >> Scanned directories: 0 >> Scanned files: 1 >> Infected files: 0 >> Data scanned: 0.00 MB >> Data read: 0.00 MB (ratio 0.00:1) >> Time: 64.429 sec (1 m 4 s) >> >> >> On Wed, Mar 6, 2019 at 10:17 AM Micah Snyder (micasnyd) via clamav-users < >> clamav-users@lists.clamav.net> wrote: >> >>> I confirmed with our signature management team that the extended time >>> processing daily-25380 is because this change is significantly larger than >>> a standard update. >>> This update drops 768053 hash-based signatures for malware that is >>> detected by other more efficient logical signatures. The net result will >>> be a leaner database that should load a little faster and take up less >>> memory. >>> >>> The validation stage when creating the daily had estimated less than 26 >>> minutes for the cdiff to apply. You may be correct that it's much faster >>> on x86 than on Sparc. 3h15m is definitely worse than expected, and I >>> apologize for the inconvenience. >>> >>> Regards, >>> Micah >>> >>> Micah Snyder >>> ClamAV Development >>> Talos >>> Cisco Systems, Inc. >>> >>> >>> On 3/6/19, 9:31 AM, "Pierre Dehaen" <deha...@drever.be> wrote: >>> >>> Yes Micah, it finished while I was checking the computer because of >>> the messages received >>> on the mailing list. >>> >>> $ tail -50 /var/log/freshclam.log >>> ... >>> -------------------------------------- >>> ClamAV update process started at Wed Mar 6 11:37:46 2019 >>> WARNING: Your ClamAV installation is OUTDATED! >>> WARNING: Local version: 0.100.0 Recommended version: 0.101.1 >>> DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav >>> securiteinfo.hdb is up to date (version: custom database) >>> securiteinfo.ign2 is up to date (version: custom database) >>> Downloading javascript.ndb [*] >>> javascript.ndb updated (version: custom database, sigs: 45008) >>> securiteinfohtml.hdb is up to date (version: custom database) >>> securiteinfoascii.hdb is up to date (version: custom database) >>> securiteinfopdf.hdb is up to date (version: custom database) >>> Downloading spam_marketing.ndb [*] >>> spam_marketing.ndb updated (version: custom database, sigs: 24199) >>> main.cld is up to date (version: 58, sigs: 4566249, f-level: 60, >>> builder: sigmgr) >>> Downloading daily-25380.cdiff [100%] >>> daily.cld updated (version: 25380, sigs: 1503528, f-level: 63, >>> builder: raynman) >>> bytecode.cld is up to date (version: 328, sigs: 94, f-level: 63, >>> builder: neo) >>> Database updated (6139078 signatures) from db.be.clamav.net (IP: >>> 104.16.219.84) >>> Clamd successfully notified about the update. >>> >>> $ ls -l /var/log/freshclam.log >>> -rw-r--r-- 1 clamav clamav 701634 Mar 6 14:51 >>> /var/log/freshclam.log >>> >>> It ran from 11:37 to 14:51. It might run faster on x86 computers >>> though. >>> >>> Pierre >>> >>> On 6 Mar 2019 at 14:20, Micah Snyder (micasnyd) via clamav-users >>> wrote: >>> >>> Pierre, >>> >>> So you're saying it actually did finish after 3 hours, 15 minutes on >>> its own? That is good news >>> for all of the automated systems, even if this is a potentially >>> terrible bug. >>> >>> I'm still investigating the cause, and asking our signature >>> management team if they have any >>> additional details. >>> >>> Micah >>> >>> Micah Snyder >>> ClamAV Development >>> Talos >>> Cisco Systems, Inc. >>> >>> >>> >>> On 3/6/19, 9:06 AM, "clamav-users on behalf of Pierre Dehaen" >>> <clamav-users- >>> boun...@lists.clamav.net on behalf of deha...@drever.be> wrote: >>> >>> Here too: it took about 3 hours and 15 minutes to calm down >>> (SPARC, Solaris 11, >>> v0.100.0)... without noticiable error in freshclam.log. >>> >>> On 6 Mar 2019 at 6:27, J.R. via clamav-users wrote: >>> >>> > When crontab execs freshclam >>> > CPU server goes to 100% >>> > Hanged finishing Downloading daily-25380.cdiff [100%] >>> >>> Just checked my server and it happened to me too! A little after >>> 5am >>> central time. :( >>> >>> _______________________________________________ >>> >>> clamav-users mailing list >>> clamav-users@lists.clamav.net >>> https://lists.clamav.net/mailman/listinfo/clamav-users >>> >>> >>> Help us build a comprehensive ClamAV guide: >>> https://github.com/vrtadmin/clamav-faq >>> >>> http://www.clamav.net/contact.html#ml >>> >>> >>> _______________________________________________ >>> >>> clamav-users mailing list >>> clamav-users@lists.clamav.net >>> https://lists.clamav.net/mailman/listinfo/clamav-users >>> >>> >>> Help us build a comprehensive ClamAV guide: >>> https://github.com/vrtadmin/clamav-faq >>> >>> http://www.clamav.net/contact.html#ml >>> >>> >>> >>> _______________________________________________ >>> >>> clamav-users mailing list >>> clamav-users@lists.clamav.net >>> https://lists.clamav.net/mailman/listinfo/clamav-users >>> >>> >>> Help us build a comprehensive ClamAV guide: >>> https://github.com/vrtadmin/clamav-faq >>> >>> http://www.clamav.net/contact.html#ml >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> clamav-users mailing list >>> clamav-users@lists.clamav.net >>> https://lists.clamav.net/mailman/listinfo/clamav-users >>> >>> >>> Help us build a comprehensive ClamAV guide: >>> https://github.com/vrtadmin/clamav-faq >>> >>> http://www.clamav.net/contact.html#ml >>> >> >> _______________________________________________ >> >> clamav-users mailing list >> clamav-users@lists.clamav.net >> https://lists.clamav.net/mailman/listinfo/clamav-users >> >> >> Help us build a comprehensive ClamAV guide: >> https://github.com/vrtadmin/clamav-faq >> >> http://www.clamav.net/contact.html#ml >> > > > -- > --- > Dave Raynor > Talos Security Intelligence and Research Group > dray...@sourcefire.com > > _______________________________________________ > > clamav-users mailing list > clamav-users@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml >
_______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
