On Mon, Sep 30, 2013 at 2:02 PM, Boszormenyi Zoltan <[email protected]> wrote:
> 2013-09-30 17:58 keltezéssel, David Raynor írta: > >> >> Zoltán, >> >> Your idea of breaking the signature set into chunks to do repeated scans >> is >> a workable idea. It would require a few moving parts outside of ClamAV. I >> cannot write the wrapper for you but can give you some tips. >> >> - sigtool is your best bet for unpacking CVD files. You can break the >> signature files up into chunks after that. >> - Some signature files will need to be loaded for all chunks. They load >> configuration values or prevent FP. >> > > What does "prevent FP" mean? > > > Based on current main.cvd and >> daily.cvd, the extensions of these files are: [ .ftm .cfg .fp .ign2 ] This >> list may change in the future, but works for now. >> > > What about the other files in daily.cvd? > .db .ftm .hdb .hdu .idb .ign (without the 2) > .info .ldb .ldu .mdu .ndb .ndu .pdb .wdb .zmd > > Also, what about the "missing" files from main.cvd? > .ftm .cfg .ign2 are not present but there are .db .fp .hdb .ndb and .zmd > > > - If you use bytecode.cvd, that should also be in all chunks. >> > > If I understand you correctly, the final broken-up database should > consist of several directories and bytecode.cvd present in every one of > them? > > Or the contents of bytecode.cvd (.cbc files) should be packed into each > cvd? > > > - Many of the existing signature types are 1 signature per line. >> > > I have discovered it already, the text file can be nicely broken up into > pieces. > > > You can >> fragment those files as you see fit without breaking signatures as long as >> you keep whole lines. The mdb signature files are the largest and will >> probably need to be divided into the most pieces. The memory use is not >> purely linear with number of signatures. You will have to experiment to >> find out what works for you. >> > > At first thought, 10MB chunks created from the unpacked .mdb would do > nicely. > > > - You will need to collate results. Files that alert on only 1 signature >> _will_ report clean from scans that do not include that signature! BE >> CAREFUL. >> > > Of course. > > > >> Disclaimer (because I have to): YMMV. I cannot guarantee your results or >> support your configuration. >> > > Of course. > > > >> That said, your idea should let you find a way to operate in your >> environment and keep scanning. >> >> Good luck, >> >> Dave R. >> > > Thank you very much for the detailed explanation. > > With "sigtool --unpack-current=main" I already found out the .mdb file > needs to be broken up. My question is whether there is any cross-dependency > between different files and signatures? If there is, can it be detected > somehow? > > Best regards, > Zoltán Böszörményi > > > ______________________________**_________________ > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/**clamav-faq<https://github.com/vrtadmin/clamav-faq> > http://www.clamav.net/support/**ml <http://www.clamav.net/support/ml> > I will address these in order: 1) What does "prevent FP" mean? The .fp and .ign2 files are for whitelisting files or signatures. More details are in the signatures document (signatures.pdf or signatures.tex) in Section 3.8. 2) What about the other files in daily.cvd? There are 3 groups: the files I already pointed out, the non-signature files you can ignore [COPYING & .info & .ign (old-style ignores, for older versions of ClamAV only)], and the rest are signature files. For more details on what's what, check the signatures document. 3) Also, what about the "missing" files from main.cvd? Intentionally omitted. Daily gets revised at least once a day, and we keep main as static as we can. They do not overlap content. If main doesn't have a kind of file, it just does not contain that kind of sigs. Cvd files are essentially just containers and are additive. 4) If I understand you correctly, the final broken-up database should consist of several directories and bytecode.cvd present in every one of them? Or the contents of bytecode.cvd (.cbc files) should be packed into each cvd? Put the bytecode.cvd in each. Don't unpack it, for simplicity. Longer explanation available if you really really want. 5) My question is whether there is any cross-dependency between different files and signatures? If there is, can it be detected somehow? Good question. There is one dependency. The icon signatures [in idb files] are tied to related logical sigs [within the ldb files], so the idb depends on the ldb. Easiest solution is to keep the idb and ldb intact and together. Total in current daily they would be under 200 KB. Icon signatures are described in Section 3.4 of the signatures document. Hope this helps, Dave R. -- --- Dave Raynor Sourcefire Vulnerability Research Team [email protected] _______________________________________________ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/support/ml
