That's strange, the 48474 I have should have the sorting changed and has the improved loading time we're talking about.
$ sigtool --info safebrowsing.cvd File: safebrowsing.cvd Build time: 06 Mar 2019 13:24 -0500 Version: 48474 Signatures: 3232286 Functionality level: 63 Builder: google MD5: 70c61f41e52b5a2134ff7e272f5a6df1 SHA256 (safebrowsing.gdb) = 7f6645b8d865de3992be1ad5de215afd848acee4c021eed4818fdb760f76b57e Something must be different. Dave R. On Wed, Mar 6, 2019 at 5:39 PM Maarten Broekman via clamav-users < clamav-users@lists.clamav.net> wrote: > The new safebrowsing cvd (starting with version 48473) seems to be sorted > in a way that increases the load time of that file by several orders of > magnitude. > > I have a previous version from February where the entries in the gdb > section are sorted like this: > > S2:F:0000917787cff7b0993917209809ff3d94bec7e1de7188b323d9b88e0273cb71 > S2:F:000149794d90dc5bce4f685deed6076d00c9209bd81cef4cbdf8a4e41f0a2153 > S2:F:00042c895c912fd567afa35450cfe5d321d0d68eb3833156925c4e27d2c29aa2 > S2:F:0006d4dcb0d939d725e676a9e68aaeb303e04478e6861d2a77469d1b6a0a0f7d > S2:F:0007bf7c1808d12177f0ae90d336d60c5a7a3d89703806955b75c56f898dd919 > ... > S2:P:00009177 > S2:P:00014979 > S2:P:00042c89 > S2:P:0006d4dc > S2:P:0007bf7c > ... > S:F:00000860493997b798861956e06d3d3606f82384259b971bb922f94f886a4b55 > S:F:00000bddafae162a7a2f1249b3b38c8e4b6d3cb8bf0c30c26cc354ebcba16b37 > S:F:000046cad35fbecbcc8dd4ebb244bd08aa6dbf1078279115c82f8e21b2cf8478 > S:F:0000684200da7b11f38a6f4719bda4ec6c6ae8b2be1f7e12a16605b2d3a5d490 > S:F:000072f3f33e47a2f97b8711d240267462aa3f0a5f8130845b119a2ad3798292 > ... > S:P:00000860 > S:P:00000bdd > S:P:000046ca > S:P:00006842 > S:P:000072f3 > > > That loads into clamd (and clamscan) in under 5 seconds for the 3041760 > entries in it. > > Version 48473 and 48474 are sorted like this: > > S2:P:00009177 > S2:F:0000917787cff7b0993917209809ff3d94bec7e1de7188b323d9b88e0273cb71 > S2:P:00014979 > S2:F:000149794d90dc5bce4f685deed6076d00c9209bd81cef4cbdf8a4e41f0a2153 > ... > > > That version loads in 50+ seconds for the 3229612 entries in it. > > If I flip the order of the entries so the :F: entries comes before the > corresponding :P: entry, it loads the same number of entries in 5 - 10 > seconds. > If I reorder the entire file so that _all_ the :F: entries for each > section (S or S2) come before the :P: entries for that section, it loads in > under 5 seconds again. > > Earlier today it was mentioned that 'the next version of the CVD' would > fix it (when 48473 was the current version). That seems to have not been > the case since 48474 didn't fix it. Is there a plan to fix it? Or will we > have to live with the enormous load times for this database? > > --Maarten > > > > _______________________________________________ > > clamav-users mailing list > clamav-users@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml > -- --- Dave Raynor Talos Security Intelligence and Research Group dray...@sourcefire.com
_______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
