Re: [Clamav-devel] [PATCH] Added a autogen.sh for buildstrap the build system
On 04/16/2012 07:18 PM, Elia Pinto wrote: 2012/4/16 Török Edwin ed...@clamav.net: On 04/16/2012 07:02 PM, Elia Pinto wrote: Please ignore this patch. It is incomplete, i am sorry. I will reroll it shortly. autogen.sh is not needed, 'autoreconf -fvi' does the job. Yes, but not check the minimum version of the gnu buildtool used by a project, giving strange error if not, and, in the clamav case, don't check and give bad error if libtool libtdl tool or the static library are missing. autoconf version is checked by AC_PREREQ, and automake version by AM_INIT_AUTOMAKE, so checking those in autogen.sh is redundant. All that remains to be checked is the libtool version. Or just run the build-system regeneration on an up-to-date Debian testing/unstable, like we do :) Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Plz help me!!
On 02/11/2012 06:16 PM, infant deepak wrote: Hi, I am doing project on clamAV . I have chosen from http://wiki.clamav.net/bin/view/Main/GoogleSummerOfCode2011 4. DOCX Add support for parsing docx based MS Office files. Main purpose is extracting embedded files. You will need to parse the XML, locate the embedded data, then decode(base64/OLE?) / and decompress (deflate?) it. So I did analysis of how clamAV currently scanning a .DOCX file . From my understanding it treats as a ZIP file and extracts to a temporary folder, and scanning each xml file and inserted media files such pictures,video etc.(If I am not correct, kindly explain me). After that, I tried embedding a EICAR test virus in a picture file by using Steghide tool. Then I scanned that picture file ,but clamav didnt recognize it. Reason may be steghide encrypts the virus file. So I like to know following things, 1. Why clamav didnt recognize encrypted virus? Because once you've hidden it inside an image with steghide it is no longer executable, and no longer capable of infecting. You should embed/insert the EICAR as is inside a .DOCX, not hide it inside a picture! i.e. when you double click on the EICAR inside the DOCX you should get the eicar executed. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Why the function ac_maketrans defined size of array is 256?
On 02/02/2012 05:53 AM, chatsiri wrote: Hello All, I debug code of clamav. Aho-Corasick( AC) Algorithms concepts for matching between virus and signature files. Step for AC is build trie ( keyword tree) for inserting signature from virus database files. I have question in step build tire before matching with input information. Why source code in static int ac_maketrans(struct cli_matcher *root) [1] define size of array is 256?. Because the trie matches byte-by-byte, so each node has 256 children, and that includes the root. In addition, Do you using the Depth First Search Algorithm( DFS) for building trie? ac_maketrans uses BFS. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] How to compile unit_test framework on win32 ?
On 01/10/2012 07:03 AM, chatsiri wrote: Hello All, I built code of clamav success on M$ Virtual You mean Visual. C++ 10. I have an idea to build the unit_test package of clamav for testing modules. libcheck doesn't work on Windows right now, due its use of fork(). Although I've seen some patches on libcheck mailing list working towards solving that, so eventually it'll get there (in the next few years). Source code in package specify the Makefile file only which built them on Linux OS. Could i try the Cygwin for compiling unit_test package? Building ClamAV with Cygwin is not supported. And building part of it with MSVC and part of it with Cygwin even less. To test ClamAV on Windows you can run freshclam to download the databases, and then scan contrib\test, and make sure all the clam* files there are detected. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] crash when checking firefox/iceweasel deb files
On 10/07/2011 04:33 PM, Stephane CHAZELAS wrote: Hiya, (that's on debian testing amd64 with libclamav6 0.97.2+dfsg-1) I had a few c-icap crashes on scanning some debian packages for firefox (for instance: http://ftp.se.debian.org/debian/pool/main/i/iceweasel/iceweasel_7.0.1-2_amd64.deb) It can be reproduced on amd64 with clamscan --max-recursion=5 iceweasel_7.0.1-2_amd64.deb That gives: *** glibc detected *** clamscan: double free or corruption [...] Here recursion not increased but magic_scandesc reentered somehow via cli_scanraw() - ??? - cli_bcapi_extract_new(). Thanks for the report. There were only 2 bytecodes using that API, so I dropped them until the API is fixed in libclamav (at which point I will readd them). Bytecode.cvd 146 and 147 should not crash anymore, can you confirm (once you get the update via freshclam, in ~15m)? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] What's function for check between virus and signature on database ( AC)?
On 09/29/2011 09:24 PM, Chatsiri Ratana wrote: Hello All, I debug clamscan programs by insert a input test to it. I have question on function for check match string between virus and signature on database. What a function for checking a virus and signature on database? My view, It's should as function below in file name matcher-ac.c. int cli_ac_chklsig(const char *expr, const char *end, uint32_t *lsigcnt, unsigned int *cnt, uint64_t *ids, unsigned int parse_o nly) This is only one of the functions, have a look at cli_fmap_scandesc: that is the function that calls most of the other matchers. If you are about AC algorithm only, then look at cli_ac_scanbuff. In last email in mail-list suggested we should start with cli_parse_add() function in readdb.c file for debuging with AC algorithms,but It's step for AC algorithms in order to check start string with signature base( stage 1 check with prefix of string but not stage 2 for checking all string match with signature base),right? cli_parse_add is used when loading the database to parse and convert the signatures into the internal representation. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] ClamAV Algorithms
On 07/12/2011 02:11 AM, Jerry 270 wrote:
Hi Edwin,
Thanks for your reply. I am doing a Masters degree for which the
research is analyzing investigating malware. I am interested in evaluating
algorithms used in anti-virus software, but just investigating whether this
is a possibility at the moment. The research projects goal is to define a
problem domain, a scenario in which the problem to be investigated exists.
Within this problem domain, a research question is posed. This is the
question that the project will seek to answer.
I enabled DevAVOnly and only the AC signatures appear to be loaded when
the config file is reread but when I do a scan of some files the debug
information appears to suggest that BM signatures are loaded for GENERIC and
PE.
If you are using clamscan then use --dev-ac-only. I get 0 BM sigs:
LibClamAV debug: Matcher[0]: GENERIC: AC sigs: 35862 (reloff: 21, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) maxpatlen 470 (ac_only mode)
LibClamAV debug: Using filter for trie 1
LibClamAV debug: Matcher[1]: PE: AC sigs: 59482 (reloff: 47699, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) maxpatlen 468 (ac_only mode)
LibClamAV debug: Matcher[2]: OLE2: AC sigs: 1726 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) maxpatlen 176 (ac_only mode)
LibClamAV debug: Matcher[3]: HTML: AC sigs: 5773 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) maxpatlen 799 (ac_only mode)
LibClamAV debug: Using filter for trie 4
LibClamAV debug: Matcher[4]: MAIL: AC sigs: 1146 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) maxpatlen 255 (ac_only mode)
LibClamAV debug: Matcher[5]: GRAPHICS: AC sigs: 23 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) maxpatlen 227 (ac_only mode)
LibClamAV debug: Matcher[6]: ELF: AC sigs: 47 (reloff: 29, absoff: 0) BM sigs:
0 (reloff: 0, absoff: 0) maxpatlen 400 (ac_only mode)
LibClamAV debug: Using filter for trie 7
LibClamAV debug: Matcher[7]: ASCII: AC sigs: 1568 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) maxpatlen 492 (ac_only mode)
LibClamAV debug: Matcher[8]: NOT USED: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[9]: MACH-O: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs:
0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
What should DevACDepth be set to?
If AC is used for signatures containing wildcards and BM is used for
signatures without wildcards is it possible to scan using just one type of
signature and test the performance of each algorithm that way?
Well you won't be able to load the AC signatures into BM (BM doesn't support
the wildcards), so as a first step you would probably be to remove
the signatures that require AC from the DB.
You can use 'sigtool --unpack-current main' and 'sigtool --unpack-current
daily' to unpack the databases.
And then load the DB as by default (into BM), and with --dev-ac-only (into AC),
and compare them that way.
Also note that the BM algo has an optimization when signatures are tied to a
specific offset (PE for example).
How is prefiltering disabled?
Comment out this 'if' in matcher-ac.c:
if (cli_mtargets[root-type].enable_prefiltering dconf_prefiltering) {
Best regards,
--Edwin
___
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] ClamAV Algorithms
On 2011-07-11 01:00, Jerry 270 wrote: Hi, I am a newbie to ClamAV so require help please. I am doing a research project and would like to evaluate the BM and AC algorithms used by ClamAV. Is there anyway to get ClamAV to use either BM or AC on their own so scanning speed tests can be conducted for each algorithm separately? I have read on this list that you can't get BM to run on its own. What is the best way to compare the two algorithms and can someone give me more information on how ClamAV uses or chooses between BM and AC please? There are some AC/BM discussion in the archive, see for example: http://lurker.clamav.net/message/20100426.103047.eb6fd9d0.en.html http://lurker.clamav.net/message/20100427.131931.b705e603.en.html http://lurker.clamav.net/message/20081204.212941.c9fa45c2.en.html You can use DevACOnly to use only the AC algorithm for everything (there is no equivalent for BM), Other than that you can use tools such as 'oprofile', or 'perf record / perf report' to see how much time is spent in functions from matcher-ac.c and how much time in those from matcher-bm.c. Also note that there is the prefiltering step too, if you want to measure just the AC/BM performance, you should disable that (although you'll loose performance by doing so). What are your research project's goals? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Is clamav lgpl compatible?
On 07/09/2011 12:26 AM, Andrey V. Martyanov wrote: Hi! I see that Immunet uses libclamav.dll. Doesn't this violate the clamav license? Immunet is part of Sourcefire. For example, I'd like to build my own closed source commercial antivirus kit with the clamav engine. May I do that? No. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Is clamav lgpl compatible?
On 07/09/2011 03:49 PM, Andrey V. Martyanov wrote: The documentation states rather clearly that LibClamAV is GPL, not LGPL. See http://www.clamav.net/doc/latest/html/node35.html. Yes, but if you look at the root of the ClamAV source tree you'll see the COPYING.LGPL file. It's unclear what is the purpose of that file. You'll also see a COPYING file, and various other COPYING.* files. Look at each individual file in libclamav/ to find out which one is GPL, LGPL, BSD, etc. ClamAV as a whole is GPLv2. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Error building clamav 0.97.1 with external llvm 2.9
On 07/05/2011 03:32 AM, Renato Botelho wrote: Hello, I can build 20110704 snapshot with external llvm-2.9 on FreeBSD 8.2 without problems, but on 0.97.1 i got the following error: gmake[3]: Leaving directory `/work/a/ports/security/clamav/work/clamav-0.97.1/libltdl' gmake[2]: Leaving directory `/work/a/ports/security/clamav/work/clamav-0.97.1/libltdl' Making all in libclamav gmake[2]: Entering directory `/work/a/ports/security/clamav/work/clamav-0.97.1/libclamav' gmake all-recursive gmake[3]: Entering directory `/work/a/ports/security/clamav/work/clamav-0.97.1/libclamav' Making all in c++ gmake[4]: Entering directory `/work/a/ports/security/clamav/work/clamav-0.97.1/libclamav/c++' gmake all-am gmake[5]: Entering directory `/work/a/ports/security/clamav/work/clamav-0.97.1/libclamav/c++' CXXlibclamavcxx_la-bytecode2llvm.lo /bin/sh ./libtool --silent --tag=CXX --mode=compile c++ -DHAVE_CONFIG_H -I. -I./../.. -I./.. -I./../../ -I/usr/local/include -I./../.. -I./.. -I./../../ -I/usr/local/include -DNDEBUG -D_GNU_SOURCE -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O2 -fno-exceptions -fno-rtti -fPIC -Woverloaded-virtual -Wcast-qual -fexceptions -DLLVM28 -DLLVM29 -O2 -pipe -march=i486 -fno-strict-aliasing -c -o libclamavcxx_la-bytecode2llvm.lo `test -f 'bytecode2llvm.cpp' || echo './'`bytecode2llvm.cpp CXXlibclamavcxx_la-ClamBCRTChecks.lo /bin/sh ./libtool --silent --tag=CXX --mode=compile c++ -DHAVE_CONFIG_H -I. -I./../.. -I./.. -I./../../ -I/usr/local/include -I./../.. -I./.. -I./../../ -I/usr/local/include -DNDEBUG -D_GNU_SOURCE -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O2 -fno-exceptions -fno-rtti -fPIC -Woverloaded-virtual -Wcast-qual -fexceptions -DLLVM28 -DLLVM29 -O2 -pipe -march=i486 -fno-strict-aliasing -c -o libclamavcxx_la-ClamBCRTChecks.lo `test -f 'ClamBCRTChecks.cpp' || echo './'`ClamBCRTChecks.cpp ClamBCRTChecks.cpp:37:29: error: PointerTracking.h: No such file or directory The file is in git in two places, but only one was in Makefile.am, and 'make dist' only included one of them, not both. When building with internal LLVM it uses llvm/include/llvm/Analysis/PointerTracking.h, which is included in the tarball, with external LLVM it simply wants to use PointerTracking.h, which was not. I fixed the makefile on master, so that on next release we don't make this mistake again. Here is the PointerTracking.h file that you can add to the ports collection: http://git.clamav.net/gitweb?p=clamav-devel.git;a=blob_plain;f=libclamav/c%2B%2B/PointerTracking.h;hb=ba1dc7ffcbbec724f61ba9ec5c3da7df25a8bc0b It has to go in libclamav/c++/PointerTracking.h Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Error after upgrade to 0.97.1 on FreeBSD 7.3 amd64
On 2011-06-29 17:24, Renato Botelho wrote: Hi people, A FreeBSD user sent a PR [1] reporting a problem started to happen after recently update it to 0.97.1. Can you please take a look and let me know if it's a known problem? He submitted the same report to the clamav-users mailing list. We haven't found out whats wrong yet, we'd need a corefile or a stacktrace when it is hung. --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Custom CVD Server question
On 06/21/2011 02:45 PM, Fritz Elfert wrote: detection but what I want to achieve with this is to exploit clamav's ability to quickly scan over data (recursively unpacking of archives etc.), then quarantining the desired fragments of data for later processing. I specifically *do* need signed CVD however in order to assure, that only the proper internal authority can change the sigs. (The data to be detected and quarantined are selected certificates and CSRs in various forms). Sounds like something more easily solved by traditional Unix permissions and SSH keys: - create a user that can write to the ClamAV database directory - allow it to do scp (or rsync -e ssh) with an SSH key, but not actually login - update the signatures on your central server and use scp, or rsync -e ssh to securely distribute the database to all machines running ClamAV Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Is it possible to build clamav with llvm 2.9?
On 05/19/2011 03:56 PM, Renato Botelho wrote: Hi there, We have an option on FreeBSD ports collection to build clamav using llvm that was already installed in the system. It reduces a lot the time to build clamav. A user reported me he is having problems to build it because llvm he has installed is version 2.9, like you can see at [1]. Is it intentional? LLVM isn't API compatible between versions and 2.9 wasn't released yet at the time of 0.97. If not, is it easy to be fixed? I can add a patch on the ports collection to fix it on current clamav version. [1] http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/157147 0.97.1 (and current master and 0.97 branches) have support for building with external LLVM 2.9, which I've tested to work at least on Linux. The patch is here: http://git.clamav.net/gitweb?p=clamav-devel.git;a=commitdiff;h=0c79cc552990ad3af1f864929d44286632c963ac;hp=0e34854a7bda91540105e3ffd93050745365f80f Can you try if it works on FreeBSD? Note that the patch removes the ability to build with external 2.8 as that one is buggy on latest (Sandy Bridge?) Intel i5/i7 chips. The internal 2.8 version used by ClamAV has 2 patches backported from 2.9. If you want to support building with both 2.8 and 2.9 its a matter of patching configure.ac/configure to allow both, the source code itself has ifdefs to support both 2.8 and 2.9. Let me know if you want that or just 2.9 is enough. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Clamav 0.97 doesn't recognize old database
On 2011-02-23 14:09, Renato Botelho wrote: Hi guys, A FreeBSD user reported a problem when he upgraded to 0.97 as you can see at [1]. Does it make any sense for you? This has been reported on -users already, one of the 3rdparty signatures is invalid: it uses a size of 0 for an MD5 signature. The fix is to remove the offending line from the 3rdparty DB. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] clamscan can't detect malware inside a debian package
On 2011-02-15 10:20, Julien Reveret wrote: Hello, I've made a few tests lately to embed malwares inside UNIX packages like RPM or DEB packages. Once done, I scanned the packages with many anti-virus products to check their efficiency. Concerning clamav, there was no problem finding malware embedded into a RPM package. Nevertheless clamscan was unable to detect a known malware (the C99 PHP Backdoor) added to a preinst or postinst file. Should I report this as a bug ? Just published bytecode.cvd version 138, is the .deb detected now? (Run freshclam, make sure you get bytecode.cvd 138, and that you run 0.96.4+) Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Typo or intended?
On 2011-01-17 10:41, 邓尧 wrote: Hi, I'm reading the clamav source code, but the following in libclamav/others.h confused me: #define CLI_MAX_ALLOCATION (182*1024*1024) The number 182. Should it be 128 or 182? Not a typo, 176 MB was too small already: http://git.clamav.net/gitweb?p=clamav-devel.git;a=commitdiff;h=a66201acd1f9846099154099ca4fe8dc14324f4d;hp=08b5aec3817b031ac4c1542d5085aa14c0415781 Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Some questions about HeuristicScan
On Wed, 1 Dec 2010 15:40:17 +0800 Tony Zhai tonyz...@gmail.com wrote: Actually,I want to test everyone. We have tested Heuristics.Encrypted.RAR, what about others? Phishing: send an HTML email where the href points to some site, and the contents of the link points to ebay (or another one listed in daily.pdb). Safebrowsing: enable it, and send a link that you know is in the safebrowsing DB (for example because firefox reports it as an attack/phishing site). StructuredData: enable it, and send some SSN/credit card info in an email For the exploits you might find some PoCs on full-disclosure. For the Heuristics.W32.* malware you'll just have to rely on your own collection. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Change of behavior from 0.96.4 to 0.96.5
On Wed, 1 Dec 2010 10:36:29 -0500 David F. Skoll d...@roaringpenguin.com wrote: Hi, It seems that ClamAV 0.96.5 appends some sort of hash to the virus name. Whereas before, our tests would return a name like Eicar-Test-Signature, now we get something like Eicar-Test-Signature(aec7ffd14a66d8b5aa9c398eedad:1309) Where do you see this? clamd logs, clamdscan output, clamscan output? Is this intentional? Documented? Can it be turned off? It should only do that if you have 'ExtendedDetectionInfo Yes' in clamd.conf, which is not the default. Do you have that in clamd.conf? Also make sure that both your clamd and libclamav are 0.96.5. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Change of behavior from 0.96.4 to 0.96.5
On Wed, 1 Dec 2010 11:09:08 -0500 David F. Skoll d...@roaringpenguin.com wrote: On Wed, 1 Dec 2010 17:53:14 +0200 Török Edwin ed...@clamav.net wrote: Where do you see this? clamd logs, clamdscan output, clamscan output? We connect to clamd directly on the UNIX-domain socket and send a SCAN command (not CONTSCAN). When I configure it to listen on a TCP socket and telnet by hand, I get this: $ telnet localhost 3310 Trying 127.0.0.1... Connected to localhost.localdomain. Escape character is '^]'. SCAN /tmp/eicar-test /tmp/eicar-test: Eicar-Test-Signature(8fd1f39c7e4cf58dfa8c0618364779c2:70) FOUND Connection closed by foreign host. Works here: $ telnet localhost 3310 Trying ::1... Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. SCAN /home/edwin/clam/git/builds/default/test/clam.exe /home/edwin/clam/git/builds/default/test/clam.exe: ClamAV-Test-File FOUND Connection closed by foreign host. Same for CONTSCAN. What does clamdscan --version say? When I use CONTSCAN instead of SCAN, I get the same result (with the hash info). When I trace clamdscan, I see it uses the new(ish) zCONTSCAN zero-terminated command. Perhaps that one behaves differently? It should only do that if you have 'ExtendedDetectionInfo Yes' in clamd.conf, which is not the default. Do you have that in clamd.conf? No, we do not. Can you paste the clamconf -n output? Also make sure that both your clamd and libclamav are 0.96.5. That's the case. Regards, David. ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Change of behavior from 0.96.4 to 0.96.5
On Wed, 1 Dec 2010 11:34:19 -0500 David F. Skoll d...@roaringpenguin.com wrote: On Wed, 1 Dec 2010 18:15:17 +0200 Török Edwin ed...@clamav.net wrote: Works here: [... no extended info ...] That's bizarre! $ clamdscan --version ClamAV 0.96.5/11413/Thu Jul 22 09:24:53 2010 $ /usr/local/sbin/clamd --version ClamAV 0.96.5/11413/Thu Jul 22 09:24:53 2010 (The database is old because it's a test box that doesn't get DB updates.) And this is clamd.conf with all comments and blank lines removed: PidFile /var/spool/MIMEDefang/clamd.pid LocalSocket /var/spool/MIMEDefang/clamd.sock TCPSocket 3310 MaxDirectoryRecursion 15 User defang AllowSupplementaryGroups Yes ScanMail Yes ScanArchive Yes PhishingScanURLs no Finally, clamd dynamically-links against /usr/local/lib/libclamav.so.6 which resolves to libclamav.so.6.1.7 -- I believe the correct one. And here's the weird thing: Right after I start clamd, I get the extended stats info. If I wait for a while and retry, it stops giving the extended info! (Could you try your tests immediately after stopping/restarting clamd?) Yes, I can reproduce the problem now (I don't know why I couldn't before). Looks like ExtendedDetectionInfo only controls what is written to the logs, but clamd still sends the hash as replies (which clamdscan filters). Please open a bug so we fix this for 0.97. For now just filter the hash if you don't want it. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Some questions about HeuristicScan
On Tue, 30 Nov 2010 17:46:49 +0800 Tony Zhai tonyz...@gmail.com wrote: I have three questions about Heuristic Scan in ClamAV . 1.What type of file will be scanned as a Heuristic scan? All engine detections (as opposed to signature-based) are prefixed with Heuristics. 2.How can I configure the Heuristic function with enable or disable? Depends on category, you can enable/disable these in clamd.conf: ArchiveBlockEncrypted: Heuristics.Encrypted.RAR Heuristics.Encrypted.Zip OLE2BlockMacros: Heuristics.OLE2.ContainsMacros PhishingScanURLs: Heuristics.Phishing.Email Heuristics.Phishing.Email.Cloaked.Null Heuristics.Phishing.Email.Cloaked.NumericIP Heuristics.Phishing.Email.Cloaked.Username Heuristics.Phishing.Email.SpoofedDomain Heuristics.Phishing.Email.SSL-Spoof Heuristics.Phishing.URL.Blacklisted SafeBrowsing (freshclam.conf): Heuristics.Safebrowsing.Suspected-malware_safebrowsing.clamav.net Heuristics.Safebrowsing.Suspected-phishing_safebrowsing.clamav.net StructuredDataDetection: Heuristics.Structured.CreditCardNumber Heuristics.Structured.SSN AlgorithmicDetection: Heuristics.Exploit.W32.MS04-028 Heuristics.Exploit.W32.MS05-002 Heuristics.PDF.ObfuscatedNameObject Heuristics.Trojan.Swizzor.Gen Heuristics.W32.Kriz Heuristics.W32.Magistr.A Heuristics.W32.Magistr.A.dam Heuristics.W32.Magistr.B Heuristics.W32.Magistr.B.dam Heuristics.W32.Parite.B Heuristics.W32.Polipos.A Heuristics.Worm.Mydoom.M.log 3.How can I get some files that can test the Heuristic function? Depends on category again. For Heuristics.Encrypted.RAR you can create an encrypted file yourself. Which one do you want to test? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] clamav-devel Digest, Vol 73, Issue 3
On Fri, 5 Nov 2010 13:56:55 +0200
Amr Thabet amr.tha...@student.alx.edu.eg wrote:
Why do you need to generate assembly code to compare Thread.ecx with
something? Is it that much faster?
You could simply put a function pointer in your structure, a pointer
to the value you want to compare, and the constant to compare to.
Then compare using C code, not assembly.
.func = compare_values
.lhs = (char*)Thread.ecx - (char*)Thread
.rhs = 0x5678
Then call -func(bp-lhs, bp-rhs), and compare_values would
do *(uint32_t*)((char*)Thread + bp-lhs) == bp-rhs.
why??
because I don't need to decrease the performance . if you have a
breakpoint like :
eip =0x00401000 eip =0x00405000 __isdirty(Eip)
(__read(Eip) 0xFF) != 0xC3
if I create a parser parses these condition every time you emulate an
instruction that's will decrease the performance surely.
No, you don't have to parse the condition each time. You parse it once
and create a tree. When you need to evaluate, you evaluate the tree.
also if you
try to do something like that :
process-emulatecommand();
if (thread-Eip = Thread- Eip = xxx ){
break;
}
you will lose many of the features of the emulator and could not
Using assembly like this is not portable. It will only work on x86
(and with some work on x86-64). It won't work on Sparc.
Currently it doesn't seem to work on x86 either. The reason is you
allocate memory with malloc(), and then you execute it. That doesn't
work due to NX protection.
You will have to allocate using mmap and allow execution, but then
SELinux won't allow your code to run (W^X protection).
You're better off not emitting assembly on the fly.
On the other hand, why can't you use the emulator to execute the
assembly instructions you emit? (sure it'll be slower than tree approach
I suggested above).
emulate the SEH perfectly and will surely decrease the performance
What features does SEH need from the debugger to work?
surely but I only demand from you is to read the Manual of The
emulator in x86emu-docs.zip it's not so big maybe 5 pages to 10
maximum I think
OK, I'll read it.
it will make you easy to detect the bug and maybe for small bugs you
fix it by yourself :)
libemu will take more time for you to add a PE Loader
I already wrote one, one month ago.
Best regards,
--Edwin
___
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] clamav-devel Digest, Vol 73, Issue 3
On Tue, 2 Nov 2010 23:06:05 +0200 Amr Thabet amr.tha...@student.alx.edu.eg wrote: Hello Mr. Edwin, The real ones, or will wine's version do? Do you need the actual code in them, or just the export tables? it's really a linux version and it only needs the export table not the real code. we could create a modified kernel32.dll and ntdll and so on with inly the export table to decrease its size OK, so in that sense it is like libemu (they have the exports table included in their code). I don't know yet. Looks like I'm hitting some portability issues now: - missing cstdio and cstring include (otherwise fails to build with gcc 4.4) - dbg/dbg.cpp has x86 assembly code, so it won't work on x86-64: :71: Error: operand type mismatch for `push' Is there a way to disable that part of the code? it simply do the following mov eax,XXX mov ebx,XXX call ecx it could be converted into x64 but that's not the problem the problem is the parser in the dbg as it creates a 32-bit Assembly Code that test the breakpoint like: addbp(ecx==0x5678); the parser converts this into mov eax,0x5678 mov ecx,eax mov eax,dword ptr [Thread.ecx - Thread] test eax,ecx jz Lable1 mov eax,0 ret Label1: mov eax,0 ret this call is executed every time it emulates an instruction Why do you need to generate assembly code to compare Thread.ecx with something? Is it that much faster? You could simply put a function pointer in your structure, a pointer to the value you want to compare, and the constant to compare to. Then compare using C code, not assembly. .func = compare_values .lhs = (char*)Thread.ecx - (char*)Thread .rhs = 0x5678 Then call -func(bp-lhs, bp-rhs), and compare_values would do *(uint32_t*)((char*)Thread + bp-lhs) == bp-rhs. I think this code couldn't be executed on x64 so it's hard to convert the emulator to work on x64 I compiled with -m32, so it is executed as 32-bit code. Hint: you can run your program under valgrind on Linux. It is an excellent memory debugger. It shows errors like this in your code, looks like you are consistently off by 4 bytes. ==30022== Invalid read of size 4 ==30022==at 0x48E2299: op_pushad(Thread, ins_disasm*) (jmps.cpp:80) ==30022==by 0x48E8412: Process::emulate() (process.cpp:193) ==30022==by 0x80491E7: main (in /home/edwin/X86 Emulator/examples/02/a.out) ==30022== Address 0x70a072c is 0 bytes after a block of size 68 alloc'd ==30022==at 0x48AF72E: operator new(unsigned int) (vg_replace_malloc.c:255) ==30022==by 0x48E7E91: Process::CreateThread(unsigned long) (process.cpp:114) ==30022==by 0x48E7E07: Process::Process(System*, std::string) (process.cpp:100) ==30022==by 0x8048F82: main (in /home/edwin/X86 Emulator/examples/02/a.out) ==30022== Invalid write of size 4 ==30022==at 0x48EE787: op_stos(Thread, ins_disasm*) (strings.cpp:65) ==30022==by 0x48E8412: Process::emulate() (process.cpp:193) ==30022==by 0x80491E7: main (in /home/edwin/X86 Emulator/examples/02/a.out) ==30022== Address 0x70a072c is 0 bytes after a block of size 68 alloc'd ==30022==at 0x48AF72E: operator new(unsigned int) (vg_replace_malloc.c:255) ==30022==by 0x48E7E91: Process::CreateThread(unsigned long) (process.cpp:114) ==30022==by 0x48E7E07: Process::Process(System*, std::string) (process.cpp:100) ==30022==by 0x8048F82: main (in /home/edwin/X86 Emulator/examples/02/a.out) ... ==30119== ERROR SUMMARY: 1695990 errors from 324 contexts (suppressed: 15 from 6) The code doesn't crash when run under valgrind (because it prints the error, and continues). Once you fix the valgrind warnings I'm sure it'll work better without it too. Another hint: valid indexes for Thread::dword Exx[7] are from 0 to 6, you have for loops that go from 0 to 7 (inclusive). You should review your code and make sure you declare and use appropriate bounds. - with -m32 the first example works, but the 2nd example tries to execute some code in TestBP. What code is it trying to execute? It wouldn't be wise to execute the malware code on the real CPU ... thats the point of having an emulator in the first place Program received signal SIGSEGV, Segmentation fault. 0x08078e98 in ?? () (gdb) bt #0 0x08078e98 in ?? () #1 0xf7fedd39 in Debugger::TestBp (this=0x806b8c8, num=0, thread=..., ins=0x8078de8) at dbg/dbg.cpp:39 #2 0xf7fedda9 in Debugger::TestBp (this=0x806b8c8, thread=..., ins=0x8078de8) at dbg/dbg.cpp:48 #3 0xf7fe82f9 in Process::emulate (this=0x805fe68) at process.cpp:179 #4 0x08049239 in main (argc=1, argv=0xd354) at main.cpp:58 sometimes it returns error. but it's not execute the malware in the real CPU but the breakpoint testing procedure only OK, that makes sense. is your code supposed to work on Linux, or would I be better of testing in on Windows first? it works on Windows mainly but it should run on linux with no problems . but sometimes it returns Segmentation Fault if there's something like SEH
Re: [Clamav-devel] clamav-devel Digest, Vol 73, Issue 3
On Fri, 29 Oct 2010 14:21:40 +0200 Amr Thabet amr.tha...@student.alx.edu.eg wrote: Hello Mr. Edwin, - port it to ClamAV bytecode, i.e. make it compilable by the ClamAV bytecode compiler see: (not a trivial task) http://git.clamav.net/gitweb?p=clamav-bytecode-compiler.git;a=blob_plain;f=docs/user/clambc-user.pdf;hb=HEAD Yeah I see that it's not a trivial task but at least it could done - write a tool for sigmakers, to help them find malicious code sequences faster (or even automatically) I could help in supporting that also Pokas Dbg could help see this link: https://sourceforge.net/projects/pokasdbg Will have a look this weekend. There are many ways you can use an emulator in AV, and I think it is important to choose one: - detect trash code not so good :) - detect anti-emulator, VM detection code Need to make the emulator bypass these types of code Or we could just say its malicious when we find these. Unless it leads to too many FPs. - unpack some simple packed malware I write a detection code for win32.Virut.A with pokas emu at the examples\04 in x86emu-src.zip at https://sourceforge.net/projects/x86emu Thanks, will test it. - unpack both malware, and legit software - unpack/emulate enough to reach the actual malicious code (this can be quite hard) yeah it will take long time (in some malware) but you don't need to find the entry point so you will emulate until the Maximum Iterations reached and then scan the memory with the sig Right. Another decision that needs to be made is where the emulator is used: - is it an emulator run by the end-user? it's a small tool and could be run in any OS . it could run on windows users easily. and also linux with just a copy of (kernel32.dll,user32.dll,ntdll.dll) The real ones, or will wine's version do? Do you need the actual code in them, or just the export tables? - is it an emulator run by malware analysists / sigmakers? it's not a stand alone application so it's not so flexible to be used with malware analysists / sigmakers but Pokasdbg help a lot in this point - is it an emulator used by an automated signature creation process? surely (the benefit of this emulator is that it's a dll file (or .so) with many functions make you control the whole application you emulate and the emulator behavior easily and do many modifications in it ) I hope To see ClamAV more better and I hope you could support an Emulator . I think I'll write an app that uses libclamav, multiple emulators, and compares their execution. I think contrib/ would be good place for such an app. Then I'll scan part of our zoo with it, and see how much it can emulate. I also hope you find Pokas Emulator helpful on that . if you decide to choose it you will find me online with you for any questions. I don't know yet. Looks like I'm hitting some portability issues now: - missing cstdio and cstring include (otherwise fails to build with gcc 4.4) - dbg/dbg.cpp has x86 assembly code, so it won't work on x86-64: :71: Error: operand type mismatch for `push' Is there a way to disable that part of the code? - with -m32 the first example works, but the 2nd example tries to execute some code in TestBP. What code is it trying to execute? It wouldn't be wise to execute the malware code on the real CPU ... thats the point of having an emulator in the first place Program received signal SIGSEGV, Segmentation fault. 0x08078e98 in ?? () (gdb) bt #0 0x08078e98 in ?? () #1 0xf7fedd39 in Debugger::TestBp (this=0x806b8c8, num=0, thread=..., ins=0x8078de8) at dbg/dbg.cpp:39 #2 0xf7fedda9 in Debugger::TestBp (this=0x806b8c8, thread=..., ins=0x8078de8) at dbg/dbg.cpp:48 #3 0xf7fe82f9 in Process::emulate (this=0x805fe68) at process.cpp:179 #4 0x08049239 in main (argc=1, argv=0xd354) at main.cpp:58 Is your code supposed to work on Linux, or would I be better of testing in on Windows first? also it has a good reference in x86emu-docs.zip in https://sourceforge.net/projects/x86emu/ and good examples specially example 04 that has the detection and the disinfection of virut.A also you could find it in : http://www.woodmann.com/collaborative/knowledge/images/Bin_Virut.A_Malware_Analysis_Paper_2010-9-3_15.53_Virut.A.rar Will take a look. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] how to measure memory consumption of clamav database?
On Thu, 28 Oct 2010 12:03:38 -0400 Liu Yang yangliu...@gmail.com wrote: Is there an accurate way to measure the memory consumption of clamav database? The methods I used are top and /proc/pid/status, but I feel there might exist a better way to do that. clamdtop. Look at the 'Pool total'. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Upcoming release of ClamAV
On Tue, 19 Oct 2010 12:32:06 -0200 Renato Botelho rbga...@gmail.com wrote: On Tue, Oct 19, 2010 at 12:24 PM, Tomasz Kojm tk...@clamav.net wrote: Dear Users, we're going to release a new version of ClamAV on Monday, October 25. ClamAV 0.96.4 will fix some issues with the PDF parser, logical signatures and other problems reported for 0.96.3: https://wwws.clamav.net/bugzilla/buglist.cgi?resolution=FIXEDquery_format=advancedbug_status=RESOLVEDproduct=ClamAVtarget_milestone=0.96.4 You can help by testing (or just running ./configure make check) the latest code available in our Git repository - the latest snapshot tarball can be grabbed here: http://git.clamav.net/gitweb?p=clamav-devel.git;a=snapshot;h=refs/heads/master;sf=tgz Does update clamav-devel FreeBSD port to 20101019 help? Rather -20101020 to include today's commits. If the current version in the ports is clamav-devel-20101015, then definetely yes. The -20101015 version has a bug (see https://wwws.clamav.net/bugzilla/show_bug.cgi?id=2332) Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] freshclam and getaddrinfo()
On Mon, 4 Oct 2010 15:14:06 +0200 Jacek Zapala ja...@it.pl wrote: * Török Edwin (ed...@clamav.net) [101004 14:45] wrote: On Mon, 4 Oct 2010 14:31:21 +0200 Jacek Zapala ja...@it.pl wrote: Hi. Starting from the last upgrade (0.96.3 on debian) freshclam randomly chooses between ipv6/ipv4 connecting to the name that has addresses in both protocols. freshclam uses getaddrinfo() for resolving mirror dns name (wwwconnect in manager.c), but then it randomizes the list returned as a result. This way it breaks the algorithm of ipv6 address selection from rfc3484 implemented in getaddrinfo(). It is not intended to mix IPv4 and IPv6. It should be possible to randomize only IPv4 and IPv6 separately. Is the problem that your IPv6 link is slower than IPv4, or that you don't have an IPv6 connection at all? No, I have good IPv6 connection to my local mirror clamavmirror.ipartners.pl, but I found it strange to behave randomly. And sometimes it's better to choose IPv6 address over the IPv4 one, but this is not always true. I know we currently setup freshclam to use either IPv6 address set (db.ipv6.clamav.net) or the IPv4 one, but I think some day we will just add IPv6 addresses to db.country.clamav.net and leave making the decision up to the operating system (and getaddrinfo). See this: https://wwws.clamav.net/bugzilla/show_bug.cgi?id=2021#c1 Should randomizing only IPv4 addresses be the solution? Probably yes, please open a bug and mark it as enhancement. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] freshclam and getaddrinfo()
On Mon, 4 Oct 2010 14:31:21 +0200 Jacek Zapala ja...@it.pl wrote: Hi. Starting from the last upgrade (0.96.3 on debian) freshclam randomly chooses between ipv6/ipv4 connecting to the name that has addresses in both protocols. freshclam uses getaddrinfo() for resolving mirror dns name (wwwconnect in manager.c), but then it randomizes the list returned as a result. This way it breaks the algorithm of ipv6 address selection from rfc3484 implemented in getaddrinfo(). It is not intended to mix IPv4 and IPv6. It should be possible to randomize only IPv4 and IPv6 separately. Is the problem that your IPv6 link is slower than IPv4, or that you don't have an IPv6 connection at all? What is the reason for doing this? Because otherwise the list returned by getaddrinfo() is always in sorted order (on Linux at least), so it would only use the first mirror. See this: https://wwws.clamav.net/bugzilla/show_bug.cgi?id=2021#c1 Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Building ClamAV with external/system LLVM
On Tue, 28 Sep 2010 12:05:04 -0300 Renato Botelho rbga...@gmail.com wrote: 2010/9/28 Török Edwin ed...@clamav.net: Hi, I just committed some changes to configure and the bytecode2llvm.cpp to allow building with an external LLVM. This will allow you to build against LLVM 2.8 when it is released (right now you can test with LLVM 2.8 rc2). It won't work if you try linking against LLVM 2.7 for obvious reasons (upstream's 2.7 doesn't have our patches). How to use it: ./configure --enable-llvm --with-system-llvm=/path/to/llvm-config FreeBSD has llvm 2.8 imported on src tree (just on 9.0-CURRENT), Good. but llvm-config is not installed into the system, is there a way to test it without llvm-config installed? Not really, ClamAV must know how LLVM was built because a +Asserts build is not ABI compatible with a -Asserts build for example. Also some other flags, like -frtti or not must probably agree too. You can try writing a script that fakes llvm-config: llvm-config --cxxflags -I/path/to/llvmincludes -DNDEBUG -D_GNU_SOURCE -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -fno-exceptions -fno-rtti -fPIC llvm-config --ldflags /path/to/llvm.so llvm-config --libfiles nothing llvm-config --libs nothing Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] configuring to build 0.96.3
On Fri, 24 Sep 2010 17:33:39 +0200 Per Jessen p...@computer.org wrote: I'm trying to build 0.96.3, but configure keps getting stuck in a tight loop whilst checking for CVE-2010-0405... On SUSE Linux? Try this patch: https://wwws.clamav.net/bugzilla/show_bug.cgi?id=2287 You'll need to upgrade the system's bzip2 eventually though. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] About the size of daily.cvd
On Sat, 4 Sep 2010 12:44:35 +0800 Tony Zhai tonyz...@gmail.com wrote: Hi, I want to know the max size of the daily.cvd. I would like to just use daily.cvd instead of both of daily.cvd and mail.cvd I don't think that just skipping main.cvd is a good idea, you'll miss most signatures. What kind of files are you scanning? If you are scanning only your own email you might consider rejecting all emails that contain executable attachments. You can do that with a single ClamAV signature for example. Then you could remove the PE signatures from your local DB with a script. That will get you rid of 739024 signatures. because my system has a small memory. Have you tried 0.96.2? It has some memory usage optimizations. The daily.cvd contains about 70K signatures a month ago but more than 110K now. I am afraid it will be increased every day. Can you tell me the max size of daily.cvd? In theory there is no maximum size. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] question about the database in clamav
On Thu, 26 Aug 2010 19:33:44 -0700 (PDT) outstandingcandy outstandingca...@gmail.com wrote: Hi all! Does anybody know what is the following signature mean (especially the last two sections)? VBS.Redlof-1:3:*:666f73b2079706f735b695d3d79:0:26 See signatures.pdf, the last two are minimum and maximum requires functionality levels for the signature to be loaded. This signature won't be loaded at all because its max level is 26 (we are at 54 now). Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] what does ftonly in cli_fmap_scandesc mean?
On Fri, 27 Aug 2010 15:49:02 -0400 Liu Yang yangliu...@gmail.com wrote: I am a beginner of ClamAV and I am working on a malware detection project. Recently I was reading the ClamAV source code and have some questions. One question I have is: what does ftonly mean in scan file types only --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] [QUESTION] How does clamAV updates the signature database on-the-fly?
/**
* @file /magma/providers/external/clamav.c
*
* @brief Interface for the ClamAV library.
*
* $Author: Ladar Levison $
* $Date: 2010/08/13 10:32:38 $
* $Revision: ecaee526d4ba88a141c5b889dd023b13c05c2654 $
// Scan the message. The OLE code has a bug in it that causes
segfaults.
What bug ??
// We ignore email that ClamAV thinks is a phishing
based on scanner's internal heuristic checks. else if
(starts_ci_bl_bl(Phishing, 8, virname, ns_get_length(virname)) ||
starts_ci_bl_bl(Joke, 4, virname, ns_get_length(virname)))
{ pthread_rwlock_unlock(virus_lock);
stats_increment_by_name(provider.virus.scan.total);
stats_increment_by_name(provider.virus.scan.clean); close(fd);
return 0; }
This is incorrect, if you want to match the heuristic Phishing
detection use Heuristics.Phishing.
There are signatures which contain *Phishing*, and *Joke*. ClamAV stops
on first match.
So if you get a zip that contains something ClamAV detects as
Phishing/Joke as first element in zip followed a real malware, then it
will only report the first match (Phishing/Joke). Your code will mark
it as clean, when in fact it could be infected.
(Note that this is not the case for Heuristics.Phishing where ClamAV
keeps on scanning and only reports the heuristics if it didn't find
anything else).
The proper way to deal with this is to not load the Phishing signatures
at all, there is an option you can pass to cl_load() for that.
For *Joke* there is no flag that you can pass though.
Best regards,
--Edwin
___
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] 0.96.2 scan discrepency/reliability in Mac OS X 10.4, 10.5 and 10.6.
On Sat, 14 Aug 2010 08:35:42 -0400 Dale Walsh d...@daleenterprise.com wrote: all older versions and dependancies have been removed, config files updated per instructions. Tested on PowerPC and Intel based Macs with the same results. TEST SCAN USING CLAMSCAN: clam.ea06.exe not detected ea06 not detected is usually a sign that ClamAV was built with the wrong FPU endianness. Please attach your config.log. - GNU C: 3.3 20030304 (Apple Computer, Inc. build 1809) (3.3.0) CPPFLAGS: -I/usr/local/include CFLAGS: -arch ppc -arch i386 -g -Os -pipe -pipe -no-cpp-precomp - arch ppc -arch i386 CXXFLAGS: -arch ppc -arch i386 -g -Os -pipe LDFLAGS: -arch ppc -arch i386 Does it work if you build it for only one architecture at a time? Or if you change the order of -arch flags? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] win32 x64 build patch
On Sun, 01 Aug 2010 13:21:10 +0100 Mark Weaver mark-cl...@npsl.co.uk wrote: This patch adds an x64 target to the Windows build. (socket code in this build is not guaranteed to work due to casting the 64-bit SOCKET type to a 32-bit int, I will send a separate patch to address this). Hi Mark, There already is an x64 target for the Windows build. libclamav.vcxproj has this: PropertyGroup Condition='$(Configuration)|$(Platform)'=='Debug|Win32' Label=Configuration PropertyGroup Condition='$(Configuration)|$(Platform)'=='Debug|x64' Label=Configuration PropertyGroup Condition='$(Configuration)|$(Platform)'=='Release|Win32' Label=Configuration PropertyGroup Condition='$(Configuration)|$(Platform)'=='Release|x64' Label=Configuration Make sure you are using the git version, there are many changes there. (using VC10 instead of VC9 for example). Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Some Questions in the PE section scanning
On Wed, 30 Jun 2010 11:45:15 +0800 candy outstandingca...@gmail.com wrote: Hi all~ Thank you for solving my problem before. I still have some other questions when analysing the codes of ClamAV. Could anybody help me again? What does the member variable uint32_t *soff stands for? Have a look at readdb.c, and pe.c. Why sort the elements which the soff point to? Is the qsort function in the code implement the quick sort algorithm? See qsort.c. What are you trying to do with soff? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Some Questions in the PE section scanning
On Wed, 30 Jun 2010 15:37:07 +0800
candy outstandingca...@gmail.com wrote:
Thanks for your reply.
I am very interesting in the virus engien. I want to know the whole
algorithm of scaning and trying to optimize the int
cli_scanpe(cli_ctx *ctx, icon_groupset *iconset) function.
I have read the readdb.c and pe.c file, I think the soff variable
restore the length of the PE sections in the database. Am I right?
No, it is actually the section size, I don't know why it was called
soff.
And I want to know why not using the binary search instead of
following code? Maybe it can speed up scanning?
for(j = 0; j md5_sect-soff_len md5_sect-soff[j] =
exe_sections[i].rsz; j++) {
You could, but it would complicate the code. How much faster is it if
you are using binary search here?
--Edwin
___
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] what does the LLVM do?
On Mon, 28 Jun 2010 10:43:22 +0800 candy outstandingca...@gmail.com wrote: Hi all~ When the ClamAV scanning a PE file, does it use the LLVM and why use it? Yes, it can execute a bytecode program when scanning PE files. Bytecode can be executed either by the builtin interpreter (which doesn't need LLVM at all), or JITed using LLVM. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] what does the filter do?
On Fri, 25 Jun 2010 11:56:48 +0800 唐杰 outstandingca...@gmail.com wrote: Hi all~ I am a freshman of clamav and is reading the code of clamscan recently. I don't know why we need a filter? Does it filter some files which can be confirmed not a virus? Do you mean the --exclude command-line parameter? That is there for directories you'd never wish to scan like /sys, /dev and so on. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Sharing samples
On 06/05/2010 11:20 AM, Henri Salo wrote: Is there an email-address where I can send my samples automatically to ClamAV? I am collecting malware with automated scripts and checking those with several AV-engines. I could share everything, which isn't detected by the newest ClamAV-databases. Yes. Please contact Luca Gibelli, after June 15th. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] daily.cvd vs main.cvd
On 05/27/2010 06:51 PM, Mohammed Al-Saleh wrote: Hi, I first thought that the daily signatures are added to the main ones every while ( and thus removed from the daily.cvd). They are transferred, but main.cvd updates are rare: main.cvd is only at version 52, while daily is at version 11072 already. After checking, it does not seem to work as I thought, :). Not all signatures are transferred, those that may need updates in the near future are not. For example pdb, and wdb files are never transferred. The separation of main.cvd and daily.cvd helped a lot when ClamAV didn't know about incremental updates at all. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Boyer-Moore
On 05/23/2010 04:11 AM, Mohammed Al-Saleh wrote: I've read ClamAV's Boyer-Moore implementation. It does not seem to me that it uses Boyer-Moore algorithm at all. It is a multi-pattern version of Boyer Moore, I don't know its exact name. Boyer-Moore algorithm itself allows you to search for 1 pattern in 1 string. The one in ClamAV allows you to search for N patterns in 1 string, in a single scan, without significantly increasing the time needed when N is large. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Virus DB Repo
On 05/18/2010 03:13 PM, David F. Skoll wrote: From: Mohammed Al-Saleh moealsa...@gmail.com Is the virus database updated through a repository (for example svn or cvs)? I would like to see how virus database changes over time. Interesting. I guess there's nothing stopping you from putting the database in a git repo or similar. You could do a commit each time just before calling freshclam. Edwin's response: No. I found surprising, I must say... wouldn't it make sense to use git or similar We have a database of signatures and logs of what each update did. Using git (or another VCS) here would just complicate things, since we'd need to synchronize that with our internal DB both ways (in case of reverts). so you could easily revert bad signature additions? :) We can already drop the signature in the next update easily. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Virus DB Repo
On 05/18/2010 03:18 PM, Brandon Perry wrote: I agree that this would be neat as long as the current way is still available. I don't want to have to install git or svn on the servers just to be able to update my clam sigs. Distributing the virus DB via git/svn wouldn't scale, I don't think David was suggesting that. On Tue, May 18, 2010 at 7:13 AM, David F. Skoll d...@roaringpenguin.comwrote: From: Mohammed Al-Saleh moealsa...@gmail.com Is the virus database updated through a repository (for example svn or cvs)? I would like to see how virus database changes over time. Interesting. I guess there's nothing stopping you from putting the database in a git repo or similar. You could do a commit each time just before calling freshclam. Edwin's response: No. I found surprising, I must say... wouldn't it make sense to use git or similar so you could easily revert bad signature additions? :) Regards, David. ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Question
On 05/18/2010 09:09 PM, Mohammed Al-Saleh wrote: Hi Edwin, On Apr 27, 2010, at 7:19 AM, Török Edwin wrote: On 04/26/2010 10:20 PM, Mohammed Al-Saleh wrote: Hi Edwin, Thanks for your reply. I need to know the cases where ClamAV has performance bottlenecks or issues. The best way to do that is by measuring it. Read the last part of this reply: http://lurker.clamav.net/message/20081204.212941.c9fa45c2.en.html What kind of texts that could make ClamAV takes more time than usual. That question is hard to answer, since the signatures change each day, thus the AC trie changes, the prefiltering patterns change ... Aho-Corasick and Boyer-Moore might have some situations that cause performance issue. There is also a prefiltering step now. You can search bugzilla on why it was introduced. I might consider doing improvements or study performance impact. Don't expect it to be easy to make improvements. I spent quite a lot of time on the prefiltering step, and the problem is that some signatures falsely match a lot of times (like 'PE' from the PE signature), but the entire signature usually doesn't. So ClamAV has to stop the trie lookup, test the match, continue the trie lookup lots of times. My understanding (please correct me if I am wrong) is that the first step in matching (let's ignore the filetype recognition and such) is the prefiltering step. If the filter matches then further matching (using either AC or BM) is needed to make sure that it is not a false positive because the filter could contain more patterns than it should (and the filter matches at most 8 characters of the original signature so the other parts might not match). Yes. I am not sure if I understand your point here and I really want to understand it: So ClamAV has to stop the trie lookup, test the match, continue the trie lookup lots of times. Can you please explain this to me more? If the filter matches but AC or BM does not, would we return back to the filter to continue from the point it matches? No, I was refering to how AC works. After the AC trie detects a match it needs to check it, the AC trie contains only a tiny part of the entire signature (up to ac_max_depth), and the trie itself doesn't contain wildcards etc. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Adding targets for the bytecode interpreter
On 05/17/2010 09:26 PM, Mark Allan wrote: Hi there, I'm sending this to clamav-devel rather than clamav-users as I suspect I may have missed something during the configure/compile phase rather than at runtime. If I'm wrong, I'll happily repost to the -users list. Please open a bugreport, this situation should be detected and handled automatically (for 0.96.2 probably). I maintain a distribution of ClamAV for Mac OS X and have to generate a build which runs on both PPC and Intel architectures for the three most recent major releases of OS X. I achieve this by using the following settings for the configure phase (after applying the appropriate patch for 0.96 clamd on PPC): bash$ CC=/usr/bin/gcc-4.2 CXX=/usr/bin/g++-4.2 CFLAGS=-O2 -g -D_FILE_OFFSET_BITS=64 -mmacosx-version-min=10.4 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch i386 -arch ppc CXXFLAGS=-O2 -g -D_FILE_OFFSET_BITS=64 -mmacosx-version-min=10.4 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch i386 -arch ppc ./configure --disable-dependency-tracking --enable-llvm --enable-clamdtop --with-user=_clamav --with-group=_clamav When I run clamscan on the same PPC machine I use for building it, everything runs fine, however, when I copy the build over to a machine with an Intel processor, I get the following warning about there not being any available targets. libclamav JIT: error creating execution engine: No available targets are compatible with this triple, see -version for the available targets When I run clambc to see which targets are available, I get the following: bash$ ./clambc --version Clam AntiVirus Bytecode Testing Tool 0.96 Low Level Virtual Machine (http://llvm.org/): llvm version 2.7 Optimized build with assertions. Built May 9 2010 (01:15:14). Host: i386-apple-darwin10 Host CPU: penryn Registered Targets: ppc32 - PowerPC 32 ppc64 - PowerPC 64 If I do the configure and compile on the Intel machine, it gives the following targets Registered Targets: x86- 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 Predictably it then gives the same error message as earlier when I copy it over to the PowerPC machine. Could someone tell me how to register both PPC and x86 targets please? I completely forgot about this while writing configure. Upstream LLVM builds by default with *all* backends. For ClamAV that would be overkill (why compile the PPC backend on an x86 in general), but for bi-arch builds it needs to build both backends, yes. For now the easiest workaround is to pass --enable-all-jit-targets to configure (it'll build X86, PPC and ARM, although ARM is always disabled at runtime). A real fix would be to detect the Apple-style universal build (configure does this already), and build both ppc and x86 then. If you open a bugreport I'll try to do that for 0.96.2. On a related note, I'm intrigued to know how it managed to register 64 bit targets when I only specified 32bit architectures in the configure script. The X86 backend knows to generate code for both 32 and 64-bit targets, ditto for PPC backend. At runtime the correct target is chosen. In fact if you build all backends you get nice (cross)compilers for all targets. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Virus DB Repo
On 05/17/2010 10:21 PM, Mohammed Al-Saleh wrote: Hi, Is the virus database updated through a repository (for example svn or cvs)? No. I would like to see how virus database changes over time. You can look at the emails on clamav-virusdb@ (although signature removals don't show up there). Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Bug?
On 05/14/2010 08:59 PM, Mohammed Al-Saleh wrote: Is this a bug in ClamAV (filtering.c)? case CLI_MATCH_NIBBLE_LOW: spec-start = (p 0xf); spec-end = 0xf0 | spec-start; spec-step = 0x10; Should not the step be 1 here? NIBBLE_LOW means the low nibble is fixed, high nibble is anything, i.e. ?X, where X is a hexadecimal digit. Step 0x10 is correct. ?a = 0a,1a,2a,3a,... Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] MAXSOPATLEN and Signatures
On 2010-05-12 20:53, Mohammed Al-Saleh wrote: Hi, I have two questions. Thanks in advance for answers. 1- MAXSOPATLEN constant was commented in more than one location to contain the value 32 but in the actual implementation its value is 8. So, was it implemented first as 32 then changed to 8? if yes, can you please tell me why? Does this mean that we only have 8 states for the filtering step? Yes it was changed to 8 because it is sufficient, you'll find that most of the subsignatures are very short (2-4 bytes) which makes it that much harder to prefilter (since these 2-4 byte sequence generate a lot of false matches). commit 2c04b207e01eed2bdcdc5fca596476b25aa7b7cd Author: Török Edvin ed...@clamav.net Date: Sat Nov 29 10:07:33 2008 + 8 bits are sufficient, we only care if it is longer than 4 characters or not. TODO: maybe we can use the rest of 24 states for something else? git-svn-id: file:///var/lib/svn/clamav-devel/branches/prefilter...@4491 77e5149b-7576-45b1-b177-96237e5ba77b 2- if we count the number of signatures in the 10 roots (root[0] through root[9]) for both AC and BM, the total would around 100,000 signatures. So, would the remaining number of signatures (759261 - (~100,000) ) be the md5 and such signatures? Yeah .hdb and .mdb make up the majority of signatures. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] libclamav grew by 10Mb ?
On 04/27/2010 02:19 PM, Per Jessen wrote: Reini Urban wrote: 2010/4/26 nicolas dumont nicolas.dumont.pe...@gmail.com: Could you explain why it has so much increase between 0.95.3 and 0.96 ? Is there a way, by disable some new features ( bytecoder for ex ) to decrease the size of the lib ? Is it that difficult? --with-llvm links statically the whole llvm into libclamav. --without-llvm does not do this. Why is a user question sent to -devel? It's hardly a user question when it concerns how to build clamav. Besides, the configure help output doesn't exactly say much: --enable-llvm Enable 'llvm' JIT/verifier support [default=auto] (automatic selection based on what?) It is automatically enabled if you have a known-to-work GNU C++ compiler installed, and you have a CPU supported by the LLVM JIT. Btw, the README doesn't mention the 500% increase in the library size, nor does the Changelog. The wiki mentions something about the size: http://wiki.clamav.net/bin/view/Main/UpgradeNotes096 I don't see an easy way of reducing the size (except for --disable-llvm). Can you give some more details on why this size increase is not acceptable for you? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] libclamav grew by 10Mb ?
On 04/27/2010 09:58 PM, Per Jessen wrote: Török Edwin wrote: --enable-llvm Enable 'llvm' JIT/verifier support [default=auto] (automatic selection based on what?) It is automatically enabled if you have a known-to-work GNU C++ compiler installed, and you have a CPU supported by the LLVM JIT. Okay. Just out of curiosity - what happens if I build/configure on a machine where the JIT is supported, but distribute to a machine where it isn't? I.e. will it cause a problem? supported has 2 parts: - C++ compiler recent enough - CPU *architecture* supported Obviously you can't take the build from one CPU architecture (say X86) and run it on another (say PowerPC), unless you are cross compiling, so I don't think there should be any problems with that part. Regarding the C++ part: the machines you'll run ClamAV on don't need to have a C/C++ compiler installed, they only need a libstdc++. That libstdc++ has to be something that is compatible with the libstdc++ on the build machine (this usually means same version or newer). Although in some cases it might work with older versions, I don't know. The worse that can happen regarding libstdc++ mismatches AFAIK is that you get a missing symbol error, and the program doesn't start. In conclusion no, there shouldn't be any problems, and if you do get problems they should be obvious when you start clamscan/clamd. I don't see an easy way of reducing the size (except for --disable-llvm). Can you give some more details on why this size increase is not acceptable for you? Oh, I don't have an issue with the size as such, it was just a surprise. Ah ok. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] libclamav grew by 10Mb ?
On 04/26/2010 01:07 PM, Reini Urban wrote: 2010/4/26 nicolas dumont nicolas.dumont.pe...@gmail.com: Could you explain why it has so much increase between 0.95.3 and 0.96 ? Is there a way, by disable some new features ( bytecoder for ex ) to decrease the size of the lib ? Is it that difficult? --with-llvm links statically the whole llvm into libclamav. --without-llvm does not do this. Actually --disable-llvm does that, there is no --with/--without-llvm. This is similar to --disable-bzip2, again no --with/--without there. I think you are right that configure should be more consistent. Maybe something to improve for 0.97. Why is a user question sent to -devel? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Question
On 04/24/2010 11:39 PM, Mohammed Al-Saleh wrote: Does ClamAV use Aho-Corasick algorithm to match files against static signatures and Boyer-Moore against signatures that have *'s and ??'s ? No it is not as simple as that, and it is usually the other way around. read the cli_parse_add() function, it has all the logic of choosing between AC and BM. Why do you ask? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Error building clamav on FreeBSD 8.0 sparc64
On 04/22/2010 02:10 PM, Renato Botelho wrote: Hi, Since it's a platform i don't have access, i'll just show you the error and you tell me if you have any idea what could be happening. http://pointyhat.FreeBSD.org/errorlogs/sparc64-errorlogs/a.8.20100421055147/clamav-0.96_1.log LLVM doesn't support Sparc in JIT mode (only in static compilation mode), so don't add --enable-llvm for Sparc. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] clamav-0.96 is this properly working???
On 04/19/2010 09:21 AM, Dale Walsh wrote: Foreground = yes Debug = yes Since you have Foreground+Debug active please redirect the stderr output of clamd to a file. Then scan the files in test/ Then open a bugreport on bugs.clamav.net, and attach the stderr output (not the syslog output, since the debug info is not sent to syslog). Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] clamav-0.96 is this properly working???
On 2010-04-19 12:44, Dale Walsh wrote: On Apr 19, 2010, at 05:11 AM, Török Edwin wrote: You can remove daily.cvd/cld, and stop freshclam as temporary solution to get 0.92 working, until you get 0.95.3, or 0.96 working. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net I built everything as static so including the dynamic (shared) libclamav.dylib library is not required for clamd however libclamunrar.dylib is linked in the libclamunrar_iface module so it has been installed. The code looks like it expects to load the libclamunrar_iface module from disk and doesn't check to see if it's embedded (static linked). I submitted the clamd stderr information as requested,, is there anything else you need? Thanks, I'll let you know if I need additional information. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] clamav-0.96 is this properly working???
On 2010-04-19 15:40, Dale Walsh wrote: On Apr 19, 2010, at 07:05 AM, Török Edwin wrote: On 2010-04-19 12:44, Dale Walsh wrote: I submitted the clamd stderr information as requested,, is there anything else you need? I tested on a Mac OS X 10.5.8 ppc, with gcc 4.0.1 and all testfiles (except RAR) were detected in a static build. Tested 10.4.0 - 10.4.11 ppc, built with gcc 4.0 using a modified build process and all testfiles including rar now pass. The only problem is that clam.mail is missed, right? (well quite a big problem if you're scanning mail). Fixed some of the incompatible pointer type warnings and mail now works. When you have some time can you post a patch with the modifications you made? What version of Mac OS X do you have (sw_vers will tell you), what version of gcc (gcc -v will tell you)? Configured with gcc 3.3 but built with gcc 4.0. Maybe thats why it failed. And could you post the output from clamd's stderr when you run clamdscan test/clam.mail? (I assume it is missed in this case too) No longer missed. OK, glad to hear. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Emulation
On 2010-04-18 16:10, Mohammed Al-Saleh wrote: Hi, Does ClamAV do code emulation to detect viruses/worms? No. Unless you consider the trivial emulator in yc_poly_emulator() to be an emulator. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
[Clamav-devel] FreeBSD 6.2 ClamAV/zlib problems and solution
Hi, I just had a chat on #clamav with 2 people having problem with ClamAV 0.96 on FreeBSD 6.2. (everything was OK on FreeBSD 6.3) The symptom is that clamscan/clamd never starts, just loops infinitely trying to load the DB, --debug shows: Libclamav debug: in cli_tgzload() The system was using zlib 1.2.3, and gzseek() looked broken (it always returned 0 for gzseek(0, SEEK_CUR), in fact it moved the file position to 0 again). Solution was to use upstream zlib 1.2.3 instead of system one: $ wget http://www.zlib.net/fossils/zlib-1.2.3.tar.gz $ tar xzvf zlib-1.2.3.tar.gz $ cd zlib-1.2.3 $ ./configure --prefix=/usr -s $ make # make install Does FreeBSD 6.2 patch zlib in any way that could cause this error? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] VirusDB hashes and CVE
On 04/16/2010 07:15 PM, Henri Salo wrote: It would be very nice to see hashes and possible CVEs in submissions in virusdb-mailinglist. Every submission should have at least MD5-, SHA1-checksums Of the databases? The database itself contains the hash, and it is digitally signed. The public key for that is embedded in clamav, and freshclam checks those digital signatures. You can run sigtool --info on a CVD and find out the MD5 hash, the digital signature. Also the newer databases contain sha256 hashes (in daily/main.info), and their corresponding digital signatures. and possible CVE listed. I'm not sure what you mean by this. Do you mean CVE references for exploits we detect? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] VirusDB hashes and CVE
On 04/16/2010 07:31 PM, Henri Salo wrote: On Fri, 16 Apr 2010 19:18:23 +0300 Török Edwinedwinto...@gmail.com wrote: On 04/16/2010 07:15 PM, Henri Salo wrote: It would be very nice to see hashes and possible CVEs in submissions in virusdb-mailinglist. Every submission should have at least MD5-, SHA1-checksums Of the databases? Nope. Of the files submitted to virusdb and the hashes could be in for example: http://lurker.clamav.net/message/20090116.041716.f1c8d70e.en.html Ah for the samples. That might be doable. Please open an enhancement request on our bugzilla (component website and other services). I'm not sure what you mean by this. Do you mean CVE references for exploits we detect? CVE-ID's of the submitted files IF CVE-ID is available. I'm not sure if people mention CVE IDs, and it would be extra work for sigmakers to lookup if there is a CVE associated. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] One more FreeBSD user problem report
On 04/15/2010 02:58 PM, nicolas dumont wrote: Hello ! I'm trying to compile clamav 0.96 on FreeBSD 7.2 release gcc 4.2.1 and I have same kind of issue as Renato : This looks like a different issue, does GCC complain about an internal error for you? In function `(anonymous namespace)::LLVMCodegen::convertMDNode(unsigned int)': : undefined reference to `operator new[](unsigned int)' In function `(anonymous namespace)::LLVMCodegen::convertMDNode(unsigned int)': : undefined reference to `operator delete[](void*)' What is the output of gcc -v, and g++ -v? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] One more FreeBSD user problem report
On 04/15/2010 04:00 PM, nicolas dumont wrote: 2010/4/15 Török Edwined...@clamav.net On 04/15/2010 02:58 PM, nicolas dumont wrote: Hello ! I'm trying to compile clamav 0.96 on FreeBSD 7.2 release gcc 4.2.1 and I have same kind of issue as Renato : This looks like a different issue, does GCC complain about an internal error for you? It's not an internal error and the libclamav is correctly compiled. I've the errors when I try to link the libclamav with my own dameon ( same way as for 0.95.3 ) Are you using libtool to link? If not are you trying to link with libclamav.a, or libclamav.so? If you're trying to link with libclamav.a you should add -lstdc++ to your linker command-line, or use libtool to link against libclamav.la (which will automatically add that). Its recommended however that you link your daemon against libclamav.so, then you don't need to rebuild it everytime libclamav is upgraded. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
[Clamav-devel] EOL signature for = 0.94.2 is live
Hi, The EOL signature for ClamAV = 0.94.2 is now live (daily 10749). Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] One more FreeBSD user problem report
On 04/13/2010 07:11 PM, Renato Botelho wrote: 2010/4/13 Török Edwinedwinto...@gmail.com: On 04/13/2010 06:59 PM, Renato Botelho wrote: http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/145509 Any idea? Its probably one of the broken compilers listed here: http://llvm.org/docs/GettingStarted.html#brokengcc Which gcc version is it, and on what architecture? gcc 4.2.1 FreeBSD 7.2-RELEASE amd64 Then open a gcc bugreport. Doesn't 7.2 have a newer gcc though? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] One more FreeBSD user problem report
On 04/13/2010 07:26 PM, Renato Botelho wrote: 2010/4/13 Török Edwined...@clamav.net: On 04/13/2010 07:11 PM, Renato Botelho wrote: 2010/4/13 Török Edwinedwinto...@gmail.com: On 04/13/2010 06:59 PM, Renato Botelho wrote: http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/145509 Any idea? Its probably one of the broken compilers listed here: http://llvm.org/docs/GettingStarted.html#brokengcc Which gcc version is it, and on what architecture? gcc 4.2.1 FreeBSD 7.2-RELEASE amd64 Then open a gcc bugreport. Doesn't 7.2 have a newer gcc though? It can be installed by ports, this is the version of gcc contained in base system. The strange thing is just this user reported a problem, i asked for more people test now, depending of the results, i'll open the bug on gcc. Or he might have run out of memory, and gcc didn't handle that gracefully. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Python with thread support enabled is required
On 04/09/2010 07:06 PM, Renato Botelho wrote: After upgrade FreeBSD ports tree to 0.96, a user filled this PR: http://www.freebsd.org/cgi/query-pr.cgi?pr=145520 Would be nice if clamav checks installed python and detect this situation, what do you think? See the -users archives, yes I think I could add a test for that (Tiesi even suggested a way to test), and skip make check for the JIT. BTW its strange that the broken python has threading.py but not thread.py. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] The upcoming 15 April kill-switch
On 04/07/2010 09:21 PM, David F. Skoll wrote: Hi, I have a question about the CVD that will contain a special signature which disables all clamd installations older than 0.95. What exactly will this do? Will old versions always report No virus? Or will they always report Virus? Or will they always report an error? It will refuse to load the database and print an error message. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Two buglets on 0.96
On 04/07/2010 10:17 PM, Jose-Marcio Martins da Cruz wrote:
Sun Studio Compiler finds two buglets in 0.96.
**
CC bcrun.o
bcrun.c, line 240: warning: initializer does not fit or is out of
range: -1
Line 240 is :
unsigned int fd = -1;
An unsigned integer can't be negative. Should delete the word unsigned.
Right, it should be an int.
**
Many lines of :
../libclamav/bytecode_api.h, line 55: warning: enumerator value
overflows INT_MAX (2147483647)
enum { PE_INVALID_RVA = 0x };
Well, enum types aren't necessarily unsigned (as long as I know), but as
long as clamav shall be 64 bits compatible, IMHO, it should be better to
write something of the kind :
enum { PE_INVALID_RVA = UINT_MAX };
instead of hardcode this value, unless the exact value cares.
The exact value matters: it is used by the bytecode, and this value is
part of the bytecode - libclamav API.
However I could use an (unsigned) integer constant instead of the enum.
Best regards,
--Edwin
___
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] The upcoming 15 April kill-switch
On 04/08/2010 12:05 AM, David F. Skoll wrote: Török Edwin wrote: Would you prefer freshclam/ClamAV crash/corrupt memory when loading the new databases with 980 byte lines? No. I can think of ways around this to make things degrade gracefully: o The server You mean to do this on all the 122 mirrors here: http://www.clamav.net/mirrors.html could look at the Freshclam user-agent version number and not serve up the new database if it's too old. o The 0.96 Freshclam client could use a different request to get the newer longer-lined files. (I guess it's a bit late for that now...) How about 0.95? That version has been out for a while, and its not affected by this bug. The initial announcement about this was 6 month ago. If a 6 month window to upgrade is not enough, what would be? Nothing justifies a kill-switch. If the database is malformed .. ClamAV refuses to load. This is what prevents malformed databases to be published by accident in the first place. I understand your frustration with people who take a long time to upgrade. We have 754868 signatures right now, out of those 626061 are .mdb signatures. .mdb signatures are not supported, and not loaded by some older ClamAV versions. Is it better if they keep running the old version, thinking they have some anti-virus protection? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] 0.96 checks still fail on FreeBSD 6.x
On 04/05/2010 04:48 PM, Renato Botelho wrote: I used --threads=1 and that problem was fixed Ok. , now I got a core dumped: make check-TESTS Segmentation fault (core dumped) FAIL: check_clamav PASS: check_freshclam.sh PASS: check_sigtool.sh SKIP: check_unit_vg.sh PASS: check1_clamscan.sh PASS: check2_clamd.sh PASS: check3_clamd.sh PASS: check4_clamd.sh SKIP: check5_clamd_vg.sh SKIP: check6_clamd_vg.sh SKIP: check7_clamd_hg.sh SKIP: check8_clamd_hg.sh 1 of 7 tests failed (5 tests were not run) See unit_tests/test-suite.log Can you pastebin the test-suite.log? (or upload it somewhere compressed). Likely its the -lthr vs -pthread mismatch in ClamAV vs libcheck.so ... for which I think we haven't figured a solution yet. Maybe you could build libcheck as part of the ClamAV build, and use --with-libcheck-prefix in ClamAV to pick up that libcheck. You wouldn't install this libcheck, just use it during ClamAV's make check. Alternatively you can pass --disable-check on FreeBSD 6, and then check_clamav will be SKIPed. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Bytecode interpreter
On 03/12/2010 06:54 PM, G.W. Haywood wrote: Hi there, On Fri, 12 Mar 2010 Tomasz Kojm wrote: G.W. Haywood wrote: I'd like to add my voice to those who want an easy way to disable [the bytecode interpreter] - I can see nothing in the clamd.conf man page for 0.96-rc1 which offers any solace. As Edwin already described, you just set the Bytecode option to no in freshclam.conf. I'm starting to wonder if you guys shouldn't get out more. Simply giving the bytecode interpreter nothing to interpret is not acceptable. I don't want to just be able to give the interpreter nothing to do; I would want to be able to disable it, so that it can't do anything, even (especially!) if it is given something to do. How would you give it something to do if you didn't load any bytecodes? You'll understand why I didn't look in the freshclam.conf man page; I was thinking more along the lines of an option to the daemon at the time it is started, or perhaps - much better - a 'configure' option, so that the interpreter code isn't even built into the, er, daemon binary. I think a configure option is possible, it would work the same way as --enable-llvm/--disable-llvm builds/links either libclamav/c++ or libclamav/bytecode_nojit.c: there could be a libclamav/bytecode_disabled.c Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
[Clamav-devel] Announcing ClamAV bytecode compiler 0.10
Hi!
The ClamAV bytecode compiler version 0.10 is now available.
You can get it by using one of these commands:
$ git clone git://git.clamav.net/git/clamav-bytecode-compiler
$ git clone http://git.clamav.net/clamav-bytecode-compiler.git
The repository can be browsed online here:
http://git.clamav.net/gitweb?p=clamav-bytecode-compiler.git;a=summary
You can checkout the clambc-0.10 version using:
$ git checkout clambc-0.10
The README for the compiler, including build instructions can be found here:
http://git.clamav.net/gitweb?p=clamav-bytecode-compiler.git;a=blob_plain;f=README
The User manual can be found here:
http://git.clamav.net/gitweb?p=clamav-bytecode-compiler.git;a=blob_plain;f=docs/user/clambc-user.pdf
Bugs for the compiler should be filed using the clambc-compiler
component in bugzilla.
Here is an example of using the compiler (example source code available
in repository)
$ clambc-compiler examples/in/match_with_read.o1.c -o test.cbc
To load it into clamscan [1]
$ clamscan --debug --trust -dtest.cbc test/clam.exe
LibClamAV debug: bytecode debug: EP:
LibClamAV debug: bytecode debug: 64
LibClamAV debug: bytecode debug: VA of cyphertext is
LibClamAV debug: bytecode debug: 4198513
LibClamAV debug: bytecode debug: RVA of cyphertext is
LibClamAV debug: bytecode debug: 4209
LibClamAV debug: bytecode debug: Cyphertext starts at
LibClamAV debug: bytecode debug: 113
LibClamAV debug: bytecode debug: HELLO WORM
LibClamAV debug: Bytecode found virus:
ClamAV-Test-File-detected-via-bytecode
test/clam.exe: ClamAV-Test-File-detected-via-bytecode FOUND
To see information about the bytecode run:
$ clambc -i test.cbc
Bytecode format functionality level: 6
Bytecode metadata:
compiler version: clambc-0.10
compiled on: Fri Mar 12 23:59:52 2010
compiled by: edwin
target exclude: 0
bytecode type: PE hook
bytecode logical signature:
.{ClamAV-Test-File-detected-via-bytecode};Target:1;(210);0:4d5a520004000f00;EOF-544:4d5a520004000f00;S0+0:4d5a520004000f00
virusname prefix: (null)
virusnames: 0
bytecode triggered on: PE files matching logical signature
number of functions: 2
number of types: 51
number of global constants: 39
number of debug nodes: 0
bytecode APIs used:
read, seek, setvirusname, debug_print_str, debug_print_uint,
pe_rawaddr
To see the sourcecode of a bytecode run:
$ clambc -p test.cbc
[1] You will need to build the git version of clamscan with
--enable-debug, and use the --trust commandline parameter to load it.
This is just a temporary situation that will be solved before the final
0.96 release.
The RC release only loads signed bytecode from bytecode.cvd.
For 0.96 you will have the possibility to create your own bytecode using
this compiler (more on this later).
P.S.:
This version was tested on Linux/x86-64, if you encounter problems on
other systems please open a bugreport.
Note that regardless of what system you build the compiler on, the
compiler creates the same bytecode.
Best regards,
--Edwin
___
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Bytecode interpreter
On 2010-03-11 15:44, Renato Botelho wrote: IIRC, you can use --enable-llvm=no at ./configure to disable. That just disables the JIT, not the interpreter. On 2010-03-11 16:26, Tomasz Kojm wrote: On Thu, 11 Mar 2010 13:29:16 + (GMT) G.W. Haywood clamav-de...@jubileegroup.co.uk wrote: Hi there, On Thu, 11 Mar 2010 David F. Skoll wrote: I noticed the announcement of the bytecode interpreter in the 0.96-rc1 announcement. ... Why do we need the bytecode interpreter? Can we disable it if we decide the cons outweigh the pros? I was about to write something along these lines when Mr. Skoll's post arrived. The very idea of a bytecode interpreter in ClamAV gives me the creeps. It sounds like a whole bunch of vulnerabilities waiting to happen. Due to security reasons all bytecodes need to be digitally signed, so no 3rd parties will be able to inject any code into your installations. When it comes to vulnerabilities, they will not be that critical as vulnerabilities in the regular code since all bytecodes can be remotely fixed/removed. Yes, and let me explain some of the other security features: - bytecode can only call functions it defines itself, and a limited ClamAV API (see libclamav/bytecode_api.h), no syscalls - no direct access to the filesystem, it can only read the currently scanned file (via the API), and write to a temporary file via the API (when unpacking) - no arbitrary memory access, bounds of all accesses must be known, bounds checks inserted by the compiler, or libclamav itself (see BytecodeSecurity in clamd.conf) - although the above should be enough, there is also stack smashing protection in the JITed code (which simply aborts the bytecode, not clamd) I'd like to add my voice to those who want an easy way to disable it - I can see nothing in the clamd.conf man page for 0.96-rc1 which offers any solace. As Edwin already described, you just set the Bytecode option to no in freshclam.conf. In the same man page there are a couple of small formatting errors in the bold attributes for LocalSocketGroup and LocalSocketMode. Thanks, this will be fixed in the next release Regards, ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] One more problem on unit tests at freebsd 9
On 03/11/2010 09:57 PM, Renato Botelho wrote: On Thu, Mar 11, 2010 at 4:54 PM, Renato Botelho rbga...@gmail.com wrote: A FreeBSD user contacted me reporting a problem building clamav under FreeBSD 9. I reproduced it locally, using 20100308 snapshot: [--] 3 tests from JITEventListenerTest [ RUN ] JITEventListenerTest.Simple [ OK ] JITEventListenerTest.Simple [ RUN ] JITEventListenerTest.MultipleListenersDontInterfere [ OK ] JITEventListenerTest.MultipleListenersDontInterfere [ RUN ] JITEventListenerTest.MatchesMachineCodeInfo [ OK ] JITEventListenerTest.MatchesMachineCodeInfo [--] Global test environment tear-down [==] 17 tests from 4 test cases ran. [ PASSED ] 17 tests. PASS: llvmunittest_JIT ./llvmcheck.sh: not found FYI, file is there: ./work/clamav-devel-20100308/libclamav/c++/llvmcheck.sh Seems like this bug: https://wwws.clamav.net/bugzilla/show_bug.cgi?id=1876 Does my suggestion here fix it? https://wwws.clamav.net/bugzilla/show_bug.cgi?id=1876#c2 Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] One more problem on unit tests at freebsd 9
On 03/11/2010 10:31 PM, Renato Botelho wrote: I needed to stop and start again, now I got this: FAIL: LLVM :: CodeGen/X86/bswap-inline-asm.ll (1340 of 2135) TEST 'LLVM :: CodeGen/X86/bswap-inline-asm.ll' FAILED Script: -- llc /usr/ports/security/clamav-devel/work/clamav-devel-20100308/libclamav/c++/llvm/test/CodeGen/X86/bswap-inline-asm.ll -march=x86-64 /usr/ports/security/clamav-devel/work/clamav-devel-20100308/libclamav/c++/llvm/test/CodeGen/X86/Output/bswap-inline-asm.ll.tmp not grep APP /usr/ports/security/clamav-devel/work/clamav-devel-20100308/libclamav/c++/llvm/test/CodeGen/X86/Output/bswap-inline-asm.ll.tmp FileCheck /usr/ports/security/clamav-devel/work/clamav-devel-20100308/libclamav/c++/llvm/test/CodeGen/X86/bswap-inline-asm.ll /usr/ports/security/clamav-devel/work/clamav-devel-20100308/libclamav/c++/llvm/test/CodeGen/X86/Output/bswap-inline-asm.ll.tmp -- Exit Code: 1 Command Output (stdout): -- Command 0: llc -march=x86-64 Command 0 Result: 0 Command 0 Output: None Command 0 Stderr: Command 1: not grep APP /usr/ports/security/clamav-devel/work/clamav-devel-20100308/libclamav/c++/llvm/test/CodeGen/X86/Output/bswap-inline-asm.ll.tmp Command 1 Result: 1 Command 1 Output: #APP #NO_APP #APP #NO_APP #APP #NO_APP #APP #NO_APP Command 1 Stderr: -- Command Output (stderr): -- -- Testing Time: 95.31s Failing Tests (1): LLVM :: CodeGen/X86/bswap-inline-asm.ll Expected Passes: 1360 Expected Failures : 17 Unsupported Tests : 757 Unexpected Failures: 1 We're close, only 1 test failing. On FreeBSD 8 I can't reproduce this though: Testing Time: 75.70s Expected Passes: 1361 Expected Failures : 17 Unsupported Tests : 757 This is on: FreeBSD 8.0-RELEASE amd64 amd64 Please open a bugreport on bugs.clamav.net, and attach output from these commands: $ libclamav/c++/llc --version $ libclamav/c++/llc libclamav/c++/llvm/test/CodeGen/X86/bswap-inline-asm.ll $ which llc $ llc --version Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Tests failing on FreeBSD 6.x
On 03/04/2010 03:12 PM, Renato Botelho wrote: I got a real 6.4 amd64 machine, and i reproduced the problem, here is the data i collected after build with -g: Did you set CXXFLAGS to -g? Setting CFLAGS has no effect on libclamav/c++ testegw# gdb ./llvmunittest_ADT GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as amd64-marcel-freebsd... (gdb) run Starting program: /home/garga/clamav-devel/work/clamav-devel-20100303/libclamav/c++/llvmunittest_ADT [New LWP 100047] [New Thread 0x60f000 (LWP 100047)] [New LWP 100047] Program received signal SIGSEGV, Segmentation fault. [Switching to LWP 100047] 0x00080073931d in _pthread_mutex_trylock () from /usr/lib/libthr.so.2 (gdb) bt #0 0x00080073931d in _pthread_mutex_trylock () from /usr/lib/libthr.so.2 #1 0x000800739686 in pthread_mutex_lock () from /usr/lib/libthr.so.2 #2 0x000800b740af in __register_frame_info_bases (begin=Variable begin is not available. ) at gthr-posix.h:611 #3 0x000800b6c191 in frame_dummy () from /usr/local/lib/gcc42/libgcc_s.so.1 #4 0x in ?? () #5 0x000800b6bf01 in _init () from /usr/local/lib/gcc42/libgcc_s.so.1 #6 0x7fffeaf0 in ?? () #7 0x000800601e4d in _rtld_error () from /libexec/ld-elf.so.1 #8 0x000800604ac2 in _rtld () from /libexec/ld-elf.so.1 #9 0x000800601479 in .rtld_start () from /libexec/ld-elf.so.1 #10 0x in ?? () #11 0x in ?? () #12 0x in ?? () #13 0x in ?? () #14 0x0001 in ?? () #15 0x7fffeda0 in ?? () #16 0x in ?? () #17 0x7fffedf3 in ?? () #18 0x7fffedfd in ?? () #19 0x7fffee08 in ?? () #20 0x7fffee17 in ?? () #21 0x7fffee7d in ?? () #22 0x7fffee91 in ?? () #23 0x7fffee9d in ?? () #24 0x7fffeeb2 in ?? () #25 0x7fffeebd in ?? () #26 0x7fffeece in ?? () #27 0x7fffeedd in ?? () Let me know if you need something else. Unfortunately it looks like -g wasn't used during the build. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Tests failing on FreeBSD 6.x
On 03/04/2010 04:07 PM, Renato Botelho wrote: 2010/3/4 Török Edwin edwinto...@gmail.com: On 03/04/2010 03:12 PM, Renato Botelho wrote: I got a real 6.4 amd64 machine, and i reproduced the problem, here is the data i collected after build with -g: Did you set CXXFLAGS to -g? Setting CFLAGS has no effect on libclamav/c++ testegw# gdb ./llvmunittest_ADT GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as amd64-marcel-freebsd... (gdb) run Starting program: /home/garga/clamav-devel/work/clamav-devel-20100303/libclamav/c++/llvmunittest_ADT [New LWP 100047] [New Thread 0x60f000 (LWP 100047)] [New LWP 100047] Program received signal SIGSEGV, Segmentation fault. [Switching to LWP 100047] 0x00080073931d in _pthread_mutex_trylock () from /usr/lib/libthr.so.2 (gdb) bt #0 0x00080073931d in _pthread_mutex_trylock () from /usr/lib/libthr.so.2 #1 0x000800739686 in pthread_mutex_lock () from /usr/lib/libthr.so.2 #2 0x000800b740af in __register_frame_info_bases (begin=Variable begin is not available. ) at gthr-posix.h:611 #3 0x000800b6c191 in frame_dummy () from /usr/local/lib/gcc42/libgcc_s.so.1 #4 0x in ?? () #5 0x000800b6bf01 in _init () from /usr/local/lib/gcc42/libgcc_s.so.1 #6 0x7fffeaf0 in ?? () #7 0x000800601e4d in _rtld_error () from /libexec/ld-elf.so.1 #8 0x000800604ac2 in _rtld () from /libexec/ld-elf.so.1 #9 0x000800601479 in .rtld_start () from /libexec/ld-elf.so.1 #10 0x in ?? () #11 0x in ?? () #12 0x in ?? () #13 0x in ?? () #14 0x0001 in ?? () #15 0x7fffeda0 in ?? () #16 0x in ?? () #17 0x7fffedf3 in ?? () #18 0x7fffedfd in ?? () #19 0x7fffee08 in ?? () #20 0x7fffee17 in ?? () #21 0x7fffee7d in ?? () #22 0x7fffee91 in ?? () #23 0x7fffee9d in ?? () #24 0x7fffeeb2 in ?? () #25 0x7fffeebd in ?? () #26 0x7fffeece in ?? () #27 0x7fffeedd in ?? () Let me know if you need something else. Unfortunately it looks like -g wasn't used during the build. Added, as you can see here: g++42 -DHAVE_CONFIG_H -I. -I./llvm/include -I./llvm/include -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -D_DEBUG -D_GNU_SOURCE -I./llvm/utils/unittest/googletest/include -g -I/usr/local/include -Woverloaded-virtual -pedantic -Wno-long-long -Wall -W -Wno-unused-parameter -Wwrite-strings -Wno-missing-field-initializers -Wno-variadic-macros -fno-exceptions -O2 -fno-strict-aliasing -pipe -g -Wl,-rpath=/usr/local/lib/gcc42 -c -o Yeah but its still using -O2, try removing -O2 too. ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Tests failing on FreeBSD 6.x
On 03/04/2010 06:02 PM, Renato Botelho wrote: Without -O2, results changed #10 0x in ?? () #11 0x in ?? () #12 0x in ?? () #13 0x in ?? () #14 0x0001 in ?? () #15 0x in ?? () #16 0x7fffedf3 in ?? () #17 0x7fffedfd in ?? () #18 0x7fffee08 in ?? () Sorry that still doesn't tell me what the bug is. Can you show me config.log (from libclamav/c++), maybe its linking to a wrong library. Could you also post a strace output of the program? (so we see what it does prior to crashing) Also try doing this: (gdb) breakpoint main (gdb) run (gdb) next (gdb) next (gdb) quit Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Tests failing on FreeBSD 6.x
On 03/04/2010 07:01 PM, Renato Botelho wrote: 2010/3/4 Török Edwin edwinto...@gmail.com: On 03/04/2010 06:02 PM, Renato Botelho wrote: Without -O2, results changed #10 0x in ?? () #11 0x in ?? () #12 0x in ?? () #13 0x in ?? () #14 0x0001 in ?? () #15 0x in ?? () #16 0x7fffedf3 in ?? () #17 0x7fffedfd in ?? () #18 0x7fffee08 in ?? () Sorry that still doesn't tell me what the bug is. Can you show me config.log (from libclamav/c++), maybe its linking to a wrong library. Attached Could you also post a strace output of the program? (so we see what it does prior to crashing) Sorry, we don't have strace for freebsd amd64, just for i386 Also try doing this: (gdb) breakpoint main (gdb) run (gdb) next (gdb) next (gdb) quit It seems to segfault before main. testegw# gdb ./llvmunittest_ADT GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as amd64-marcel-freebsd... (gdb) break main Breakpoint 1 at 0x462c39: file llvm/utils/unittest/UnitTestMain/TestMain.cpp, line 13. (gdb) run Starting program: /home/garga/clamav-devel/work/clamav-devel-20100303/libclamav/c++/llvmunittest_ADT [New LWP 100101] [New Thread 0x5dd000 (LWP 100101)] [New LWP 100101] Program received signal SIGSEGV, Segmentation fault. [Switching to LWP 100101] 0x0008006f531d in _pthread_mutex_trylock () from /usr/lib/libthr.so.2 (gdb) next Single stepping until exit from function _pthread_mutex_trylock, which has no line number information. Program terminated with signal SIGSEGV, Segmentation fault. The program no longer exists. (gdb) next The program is not being run. (gdb) quit testegw# I'm thinking this problem could be related with thread library. I'll try to link it with libthr instead of libpthread and see what happens. It already links with lthr, maybe try libpthread then? LDFLAGS=' -L/usr/local/lib -lthr -Wl,-rpath=/usr/local/lib/gcc42' Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Tests failing on FreeBSD 6.x
On 03/04/2010 07:25 PM, Renato Botelho wrote: testegw# ldd llvmunittest_ADT llvmunittest_ADT: libthr.so.2 = /usr/lib/libthr.so.2 (0x8006e7000) libstdc++.so.6 = /usr/local/lib/gcc42/libstdc++.so.6 (0x8007fe000) libm.so.4 = /lib/libm.so.4 (0x800a0a000) libgcc_s.so.1 = /usr/local/lib/gcc42/libgcc_s.so.1 (0x800b26000) libpthread.so.2 = /lib/libpthread.so.2 (0x800c33000) libc.so.6 = /lib/libc.so.6 (0x800d5e000) Ouch, how did the linker even allow that?! Yeah, I think we found it: Ok that comes from here libclamav/c++/Makefile.am: libllvmsystem_la_LDFLAGS=-pthread I thought -pthread is the appropriate thing to use (the OpenBSD manpage says so at least), guess I should pass @THREAD_LIBS@ from ClamAV's main configure to the one in libclamav/c++. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Tests failing on FreeBSD 6.x
On 03/04/2010 08:51 PM, Renato Botelho wrote: 2010/3/4 Török Edwin edwinto...@gmail.com: On 03/04/2010 08:44 PM, Renato Botelho wrote: 2010/3/4 Török Edwin edwinto...@gmail.com: On 03/04/2010 08:37 PM, Renato Botelho wrote: 2010/3/4 Török Edwin edwinto...@gmail.com: On 03/04/2010 08:29 PM, Renato Botelho wrote: 2010/3/4 Török Edwin edwinto...@gmail.com: On 03/04/2010 08:23 PM, Renato Botelho wrote: 2010/3/4 Török Edwin edwinto...@gmail.com: On 03/04/2010 08:16 PM, Renato Botelho wrote: No -pthread around unit_tests Please look in all .la files for pthread. No results last chance, grep the Makefiles :) Result of this command: find ./ -type f | xargs fgrep -i -- -pthread here: http://pastebin.com/MHuScqae testegw# find ./ -type f -iname \*.la | xargs fgrep pthread from clamav's source/build root, right? Yes. Ok it seems that this: -lthr -L/usr/local/lib -lcheck -R/usr/local/lib got expanded to this: -lthr usr/local/lib/libcheck.so -pthread -Wl,-rpath -Wl,/usr/local/lib Is there a /usr/local/lib/libcheck.la, and does it have anything about pthread in it? Bingo, # Linker flags that can not go in dependency_libs. inherited_linker_flags=' -pthread' But why is that like so? Isn't -lthr FreeBSD's threading library that you build the ports with? The default thread library on FreeBSD ports is -pthread, I forced clamav to build using -lthr because we had problems in the past with -pthread. For freebsd 4 and 5 configure sets THREAD_LIBS to -pthread -lc_r and for later ones to -lthr (unless you patched configure to use PTHREAD_LIBS?). Exactly If you patched the main configure to use PTHREAD_LIBS, then please patch libclamav/c++ too. Ok, i'll do (after patch you sent is it still necessary?) Well it does the same as the main configure: use -lthr. So if you patched the main configure to use PTHREAD_LIBS instead of -lthr, you should do the same for libclamav/c++. I don't know what our version should do: - official freebsd port uses -lthr for clamav (and so do we) - libcheck uses -pthread I don't want to switch clamav to -pthread just because of libcheck, if -lthr works better (for clamav). FWIW on Debian there is no libcheck.la, there is just a libcheck.a, and libcheck_pic.a. Would it be possible to link to libcheck.a on FreeBSD? libclamav is already linked with the proper threading libs... Also I'm told latest FreeBSD only has 1 threading lib. Which is it? -lthr or -lpthread? (or does -pthread imply -lthr there?). libcheck port doesn't have this forced, so, it used the default when it was built. I can try to build clamav using the default thread and see what happens. On 03/04/2010 08:46 PM, Renato Botelho wrote: I cannot remember when it was done, but there is a comment about it on ports Makefile # This port has a problem with -pthread, # force to use -lthr until it's not fixed. PTHREAD_LIBS= -pthread I don't understand, the comment says force -lthr, but the Makefile does -pthread. I replaced it with -pthread right now to make a test, the official port uses -lthr Ah ok. And why did it work so far? I don't think I changed anything wrt libcheck. good question Did you build the unit tests for 0.95? Nope, i'll try it too Thanks. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Do I really have to upgrade to 0.95 ?...
On 03/02/2010 02:00 PM, Limor Tal wrote: 1. Can I keep using code that is older than 0.95 with the future CVD files Why? What prevents you from upgrading? What version are you running now? (those that will be distributed starting from May 2010) if I do not use sigtool and cdiff? If you somehow workaround the special signature (your question 4), then the CVD will load. It may, or may not work; it may, or may not crash. There is also bug #1331 (which got fixed in 0.95) affecting libclamav with logical signatures. All signatures can specify a functionality level to say what is the minimum engine version needed to load them. When we release a signature that makes use of these new features, we usually set the minimum functionality level (so old engines will skip the signature). However due to bug #1331, ClamAV 0.95 which tries to load a logical signature with a functionality level specified, it will either read uninitialized memory, or crash. So even if we wanted to add functionality level to the new ldbs, so that older engines (than 0.95) can load it, we can't since adding the functionality level would cause a crash for them. If we don't add the functionality level, libclamav won't crash, but will probably fail to load the signature with a syntax error. 2. Are those the only places in the code where the long signatures in the daily file cause a problem? cdiff is the only problem with long signatures, which affects freshclam. But as I've shown above there are other bugs with 0.95 that may cause problems. 3. Is the signature length the only incompatibility issue? No, see above for an example. 4. Can I choose to ignore the special signature which disables all clamd installations older than 0.95? Nothing prevents you from removing that signature with a script, or modifying the code to skip it. But if you go through all that trouble, you might as well just upgrade. You are: - spending time to implement something to workaround the special signature, possibly more time than what an upgrade would need - running a ClamAV installation that has known bugs (including security bugs) that got fixed in later versions - depending on how old your ClamAV engine is, you could be missing lots of signatures. Look at the number of Known viruses reported by clamscan, and compare it to the one on clamav.net - there is no support for bugs in clamav 0.94.x or older, you should run the latest stable to get all the security fixes [1] Considering all this, you could simply install clamav-0.95.3 using a package from your distro, or compile it from source. Then you would have something that you know that loads all signatures, and works. [1] distributions may backport security fixes to older fixes. They may or may not backport all the fixes that affect signature loading. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Tests failing on FreeBSD 6.x
On 03/01/2010 10:46 PM, Renato Botelho wrote: 2010/3/1 Török Edwin edwinto...@gmail.com: On 03/01/2010 09:22 PM, Renato Botelho wrote: Hello one more time, I was trying to update clamav-devel port to a more recent snapshot and I got some segfaults during make test, like you can see here: Which FreeBSD version, and which architecture? 6.4-RELEASE amd64 I have 6.4 i386 in a VM :( I just installed gcc42 on it (pkg_add needed some convincing to use packages-6-stable instead of packages-6.4-release), and llvmunittest_ADT worked. I did this: ./configure CXX=g++42 CC=gcc42 --disable-clamav --enable-llvm cd libclamav/c++ make llvmunittest_ADT ./llvmunittest_ADT It told me that 92 tests passed. My guess is that something goes wrong during startup, the ADT test is really simple and tehre isn't much that can segfault there. Can you get a gdb backtrace for one of these? (when built with debug symbols) Just run gdb ./llvmunittest_ADT (gdb) run ... (gdb) bt I don't have a real FreeBSD 6.4 machine around, I was testing it using tinderbox [1] Did you mean to include a link? , i'll read tinderbox documentation and see if there is a way to stop the building without clean environment, so I can do it. ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Error at unit tests, 20100127 snapshot on FreeBSD 8
On 01/28/2010 12:16 PM, Renato Botelho wrote: Hello again, I've tested this snapshot on my personal tinderboxes and got no problems, but after commit the changes i recieved an email from QAT, freebsd automated tests with an error on unit tests. Buildlog can be found here: http://qat.tecnik93.com/logs/8-STABLE-NPD/clamav-devel-20100127.log Reproduced on Debian too with g++-4.2. Any idea? I added -fno-rtti to CXXFLAGS, and everything worked fine here (gcc 4.4). Apparently with gcc 4.2 googletest needs RTTI due to some system header. Does the attached patch help (it disables -fno-rtti for now)? Best regards, --Edwin diff --git a/libclamav/c++/Makefile.am b/libclamav/c++/Makefile.am index 12d6034..ac7b4ca 100644 --- a/libclamav/c++/Makefile.am +++ b/libclamav/c++/Makefile.am @@ -19,7 +19,7 @@ LLVM_INCLUDES=-I$(top_srcdir)/llvm/include -I$(top_builddir)/llvm/include # TODO: HP-UX should have -D_REENTRANT -D_HPUX_SOURCE LLVM_DEFS=-D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -D_DEBUG -D_GNU_SOURCE AM_CPPFLAGS = -I$(top_srcdir)/../.. -I$(top_srcdir)/.. -I$(top_builddir)/../../ $(LLVM_INCLUDES) $(LLVM_DEFS) -AM_CXXFLAGS = $(LLVM_CXXFLAGS) -fno-exceptions -fno-rtti +AM_CXXFLAGS = $(LLVM_CXXFLAGS) -fno-exceptions ACLOCAL_AMFLAGS=-I m4 if DEBUG_BUILD LLVM_CONFIG=llvm/Debug/bin/llvm-config @@ -43,7 +43,7 @@ lli_LDADD=libllvmbitreader.la libllvmfullcodegen.la libllvmjit.la libclamavcxx_la_LIBADD=libllvmjit.la libclamavcxx_la_DEPENDENCIES=libllvmjit.la libllvmcodegen.la libllvmsystem.la libclamavcxx_la_LDFLAGS=-no-undefined -libclamavcxx_la_CXXFLAGS = $(LLVM_CXXFLAGS) -fno-rtti +libclamavcxx_la_CXXFLAGS = $(LLVM_CXXFLAGS) libclamavcxx_la_SOURCES = bytecode2llvm.cpp if BUILD_X86 libclamavcxx_la_LIBADD+=libllvmx86codegen.la @@ -818,7 +818,7 @@ lli_SOURCES=\ llvm/tools/lli/lli.cpp llc_CPPFLAGS=$(LLVM_INCLUDES) $(LLVM_DEFS) -llc_CXXFLAGS=$(LLVM_CXXFLAGS) -fno-rtti +llc_CXXFLAGS=$(LLVM_CXXFLAGS) llc_LDADD=libllvmbitreader.la libllvmasmprinter.la libllvmasmparser.la $(lli_LDADD) llc_SOURCES=\ llvm/lib/MC/MCInstPrinter.cpp\ diff --git a/libclamav/c++/Makefile.in b/libclamav/c++/Makefile.in index 165b619..135a797 100644 --- a/libclamav/c++/Makefile.in +++ b/libclamav/c++/Makefile.in @@ -901,7 +901,7 @@ LLVM_INCLUDES = -I$(top_srcdir)/llvm/include -I$(top_builddir)/llvm/include # TODO: HP-UX should have -D_REENTRANT -D_HPUX_SOURCE LLVM_DEFS = -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -D_DEBUG -D_GNU_SOURCE AM_CPPFLAGS = -I$(top_srcdir)/../.. -I$(top_srcdir)/.. -I$(top_builddir)/../../ $(LLVM_INCLUDES) $(LLVM_DEFS) -AM_CXXFLAGS = $(LLVM_CXXFLAGS) -fno-exceptions -fno-rtti +AM_CXXFLAGS = $(LLVM_CXXFLAGS) -fno-exceptions ACLOCAL_AMFLAGS = -I m4 @debug_build_fa...@llvm_config = llvm/Release/bin/llvm-config @debug_build_t...@llvm_config = llvm/Debug/bin/llvm-config @@ -928,7 +928,7 @@ libclamavcxx_la_DEPENDENCIES = libllvmjit.la libllvmcodegen.la \ libllvmsystem.la $(am__append_2) $(am__append_6) \ $(am__append_10) libclamavcxx_la_LDFLAGS = -no-undefined -libclamavcxx_la_CXXFLAGS = $(LLVM_CXXFLAGS) -fno-rtti +libclamavcxx_la_CXXFLAGS = $(LLVM_CXXFLAGS) libclamavcxx_la_SOURCES = bytecode2llvm.cpp LLVM_CXXFLAGS = -Woverloaded-virtual -pedantic -Wno-long-long -Wall -W -Wno-unused-parameter -Wwrite-strings -Wno-missing-field-initializers -Wno-variadic-macros TBLGENFILES = llvm/include/llvm/Intrinsics.gen X86GenRegisterInfo.h.inc X86GenRegisterNames.inc X86GenRegisterInfo.inc X86GenInstrNames.inc X86GenInstrInfo.inc\ @@ -1539,7 +1539,7 @@ lli_SOURCES = \ llvm/tools/lli/lli.cpp llc_CPPFLAGS = $(LLVM_INCLUDES) $(LLVM_DEFS) -llc_CXXFLAGS = $(LLVM_CXXFLAGS) -fno-rtti +llc_CXXFLAGS = $(LLVM_CXXFLAGS) llc_LDADD = libllvmbitreader.la libllvmasmprinter.la libllvmasmparser.la $(lli_LDADD) llc_SOURCES = \ llvm/lib/MC/MCInstPrinter.cpp\ ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Prblems building snapshots on FreeBSD [FIXED]
On 01/27/2010 12:48 PM, Renato Botelho wrote: 2010/1/26 Török Edwin edwinto...@gmail.com: On 12/22/2009 02:39 PM, Renato Botelho wrote: 2009/12/22 Török Edwin edwinto...@gmail.com: Yes, I opened a bug about that now: https://wwws.clamav.net/bugzilla/show_bug.cgi?id=1781 For now the workaround for the user is to set the CPU for the build to at least i486. OK, i'll mark clamav-devel port as BROKEN on FreeBSD i386 for now. Hi Renato, I have added the mutex workaround when building on i386, sorry for taking so long to fix. Also removed the Perl dependency from LLVM's configure, which would be needed only if we'd use LLVM's buildsystem. We don't use that, I've written a Makefile.am for LLVM. ClamAV should now build again on i386/FreeBSD, please test and report if you still have any build problems. The fix is in the git master repository (if you use git). If you are using a snapshot, then it will be in tomorrow's snapshot. I had started the tests, but i was thinking something, i think we just have a name problem here, FreeBSD call 32-bit arch as i386, but it doesn't mean that is i386, looking at FreeBSD kernel config file we have 3 possibilities of CPU: cpu I486_CPU cpu I586_CPU cpu I686_CPU As you can see, it's not really i386, so, i think we can build llvm thread safe. Would this solve it in your ports makefile (or is it too late to override?): CPUTYPE?=i586 That way the user can still set the CPUTYPE explicitly in /etc/make.conf to something better, and we'd get a good default (i586 instead of i386). LLVM JIT generates code for pentium and above only, so if you really run it on a i386/i486 ClamAV won't use LLVM's JIT (but fallback to an interpreter). Please add back this code to libclamav/c++/bytecode2llvm.cpp while testing if CPUTYPE override works: #if !defined(LLVM_MULTITHREADED) || !LLVM_MULTITHREADED #error Multithreading not available for llvm? #endif What do you think about it? I think the problem is how FreeBSD's gcc is built. If gcc is built with i386-* triple, it will generate i386 code by default. Debian's is built as i486-linux-gnu. Anyway I don't think we can do anything about how gcc is built for old versions of FreeBSD. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Prblems building snapshots on FreeBSD [FIXED]
On 01/27/2010 01:27 PM, Renato Botelho wrote: 2010/1/27 Török Edwin edwinto...@gmail.com: On 01/27/2010 12:48 PM, Renato Botelho wrote: 2010/1/26 Török Edwin edwinto...@gmail.com: On 12/22/2009 02:39 PM, Renato Botelho wrote: 2009/12/22 Török Edwin edwinto...@gmail.com: Yes, I opened a bug about that now: https://wwws.clamav.net/bugzilla/show_bug.cgi?id=1781 For now the workaround for the user is to set the CPU for the build to at least i486. OK, i'll mark clamav-devel port as BROKEN on FreeBSD i386 for now. Hi Renato, I have added the mutex workaround when building on i386, sorry for taking so long to fix. Also removed the Perl dependency from LLVM's configure, which would be needed only if we'd use LLVM's buildsystem. We don't use that, I've written a Makefile.am for LLVM. ClamAV should now build again on i386/FreeBSD, please test and report if you still have any build problems. The fix is in the git master repository (if you use git). If you are using a snapshot, then it will be in tomorrow's snapshot. I had started the tests, but i was thinking something, i think we just have a name problem here, FreeBSD call 32-bit arch as i386, but it doesn't mean that is i386, looking at FreeBSD kernel config file we have 3 possibilities of CPU: cpu I486_CPU cpu I586_CPU cpu I686_CPU As you can see, it's not really i386, so, i think we can build llvm thread safe. Would this solve it in your ports makefile (or is it too late to override?): CPUTYPE?=i586 The problem is I cannot set this var under ports Makefile Can you detect whether CPUTYPE was set or not? If variable was not set, then add -march=i486 to CFLAGS and CXXFLAGS. If variable is set then I assume CFLAGS/CXXFLAGS are already setup by the ports system? That way the user can still set the CPUTYPE explicitly in /etc/make.conf to something better, and we'd get a good default (i586 instead of i386). LLVM JIT generates code for pentium and above only, so if you really run it on a i386/i486 ClamAV won't use LLVM's JIT (but fallback to an interpreter). If user choose to set it on make.conf ports will respect. Please add back this code to libclamav/c++/bytecode2llvm.cpp while testing if CPUTYPE override works: #if !defined(LLVM_MULTITHREADED) || !LLVM_MULTITHREADED #error Multithreading not available for llvm? #endif What do you think about it? I think the problem is how FreeBSD's gcc is built. If gcc is built with i386-* triple, it will generate i386 code by default. Debian's is built as i486-linux-gnu. Anyway I don't think we can do anything about how gcc is built for old versions of FreeBSD. The oldest supported FreeBSD version is 6.x, and on this version we just support i486+ CPUs on kernel. Based on this, when FreeBSD say you i386-freebsd it just try to say it's running x86 arch, but cpu supported is 486+ Yes, I understand that. But for gcc i386-* means generate code that runs on i386, i486-* means generate code that runs on i486 by default (and gcc itself is compiled in such a way). IMHO it would make sense to do this: - for official FreeBSD packages build with CPUTYPE/-march set to the oldest CPU supported (i486 in this case) - for ports built by users, it should default to -march=native to get best performance But this is getting offtopic, I don't want to change how FreeBSD builds its packages by default. I'll try to contact someone directly from freebsd gcc team. Anyway, clamav built fine on FreeBSD 6, 7 and 8 i386 and amd64. If you have some time, take a look at build logs here: http://freebsd1.asmallorange.com/index.php?action=describe_portid=566 7.2 and 8.0 buildlogs look fine. 6.4 has this: checking for supported C++ compiler version... configure: error: C++ compiler too old (3.4.6) BTW you can add --enable-llvm to configure. That way it will fail at configure time if LLVM is not supported. http://freebsd2.asmallorange.com/index.php?action=describe_portid=365 phase 5: make test make: don't know how to make regression-test(continuing) You can run it: make check (needs the check package to be really useful). Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Prblems building snapshots on FreeBSD [FIXED]
On 12/22/2009 02:39 PM, Renato Botelho wrote: 2009/12/22 Török Edwin edwinto...@gmail.com: Yes, I opened a bug about that now: https://wwws.clamav.net/bugzilla/show_bug.cgi?id=1781 For now the workaround for the user is to set the CPU for the build to at least i486. OK, i'll mark clamav-devel port as BROKEN on FreeBSD i386 for now. Hi Renato, I have added the mutex workaround when building on i386, sorry for taking so long to fix. Also removed the Perl dependency from LLVM's configure, which would be needed only if we'd use LLVM's buildsystem. We don't use that, I've written a Makefile.am for LLVM. ClamAV should now build again on i386/FreeBSD, please test and report if you still have any build problems. The fix is in the git master repository (if you use git). If you are using a snapshot, then it will be in tomorrow's snapshot. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Prblems building snapshots on FreeBSD [FIXED]
On 2009-12-22 13:34, Renato Botelho wrote:
2009/12/22 Török Edwin edwinto...@gmail.com:
On 2009-12-22 12:40, Renato Botelho wrote:
2009/12/22 Török Edwin edwinto...@gmail.com:
On 2009-12-21 16:57, Renato Botelho wrote:
2009/12/21 Török Edwin edwinto...@gmail.com:
Hi Renato,
Thanks for the patch, what if you only wrap the include by #ifdef
HAVE_UNISTD_H?
Something like:
#ifdef HAVE_UNISTD_H
#include unistd.h
#endif
We have a similar bug for OS X
https://wwws.clamav.net/bugzilla/show_bug.cgi?id=1776, which I haven't
had time to look into yet.
FYI, I've attached this patch on bug 1776 asking someone to test it on OSX
Thanks, I applied the patch here:
http://git.clamav.net/gitweb?p=clamav-devel.git;a=commit;h=ce0c204a7ebcd6075cd58daeb5a5c2909343556c
Thank you.
But there is a user saying he still has problems on FreeBSD 7.2, i can't
reproduce, maybe you can help me to identify what's happening.
Attached is the user's buildlog.
The problems is around here:
checking build system type... i386-portbld-freebsd7.2
checking host system type... i386-portbld-freebsd7.2
checking target system type... i386-portbld-freebsd7.2
That should be i686-...
.
checking for supported C++ compiler version... ok (4.2.1)
checking for supported OS... ok (i386-freebsd7.2)
checking for GCC atomic builtins... no
configure: WARNING: LLVM will be built thread-unsafe because atomic
builtins are missing
...
bytecode2llvm.cpp:63:2: error: #error Multithreading support must be
available to LLVM!
I can work around the missing atomics (by adding a mutex around the
entire code), however I think its better if we find out why the atomics
are not available.
This is the test program:
int main() {
volatile unsigned long val = 1;
__sync_synchronize();
__sync_val_compare_and_swap(val, 1, 0);
__sync_add_and_fetch(val, 1);
__sync_sub_and_fetch(val, 1);
return 0;
}
It builds fine with default settings for me:
gcc test.c
However it fails with -march=i386:
gcc test.c -march=i386
/tmp/cc6dcRh7.o: In function `main':
test.c:(.text+0x29): undefined reference to `__sync_val_compare_and_swap_4'
collect2: ld returned 1 exit status
But even building for 486 works:
gcc test.c -march=i486
What does happen on user's system if he/she runs gcc test.c?
Question is why is ClamAV/LLVM built for i386? It should build for i686
I'll ask user to test it, but anyway, here you can see a buildlog on
my 7.2 tinderbox that went fine, and it was built for i386 too...
http://freebsd1.asmallorange.com//logs/7.2-i386-garga/clamav-devel-20091221.log
LLVM got disabled because you don't have Perl:
configure: error: perl is required but was not found, please install it
configure: error: ./configure failed for llvm
So ClamAV build nojit.c, instead of bytecode2llvm.cpp (where the user had the
problem).
Does FreeBSD build packages for i386? Debian for example builds for i486 AFAIK.
Best regards,
--Edwin
___
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Prblems building snapshots on FreeBSD [FIXED]
On 2009-12-22 13:36, Török Edwin wrote: I'll ask user to test it, but anyway, here you can see a buildlog on my 7.2 tinderbox that went fine, and it was built for i386 too... http://freebsd1.asmallorange.com//logs/7.2-i386-garga/clamav-devel-20091221.log LLVM got disabled because you don't have Perl: configure: error: perl is required but was not found, please install it configure: error: ./configure failed for llvm So ClamAV build nojit.c, instead of bytecode2llvm.cpp (where the user had the problem). Does FreeBSD build packages for i386? Debian for example builds for i486 AFAIK. Meanwhile I found out that minimum CPU for FreeBSD 9.x is i386, and i486 for 9.x. Is there a way to force ClamAV being built as i486 in the clamav port itself? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Prblems building snapshots on FreeBSD [FIXED]
On 2009-12-22 14:23, Renato Botelho wrote: 2009/12/22 Török Edwin edwinto...@gmail.com: On 2009-12-22 13:36, Török Edwin wrote: I'll ask user to test it, but anyway, here you can see a buildlog on my 7.2 tinderbox that went fine, and it was built for i386 too... http://freebsd1.asmallorange.com//logs/7.2-i386-garga/clamav-devel-20091221.log LLVM got disabled because you don't have Perl: configure: error: perl is required but was not found, please install it configure: error: ./configure failed for llvm So ClamAV build nojit.c, instead of bytecode2llvm.cpp (where the user had the problem). Does FreeBSD build packages for i386? Debian for example builds for i486 AFAIK. Meanwhile I found out that minimum CPU for FreeBSD 9.x is i386, and i486 for 9.x. Is there a way to force ClamAV being built as i486 in the clamav port itself? FreeBSD has just i386 distribution, and I cannot change that. Is there a way to fix it for i386? Yes, I opened a bug about that now: https://wwws.clamav.net/bugzilla/show_bug.cgi?id=1781 For now the workaround for the user is to set the CPU for the build to at least i486. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Prblems building snapshots on FreeBSD [FIXED]
On 2009-12-22 14:28, Renato Botelho wrote: 2009/12/22 Török Edwin edwinto...@gmail.com: On 2009-12-22 13:36, Török Edwin wrote: I'll ask user to test it, but anyway, here you can see a buildlog on my 7.2 tinderbox that went fine, and it was built for i386 too... http://freebsd1.asmallorange.com//logs/7.2-i386-garga/clamav-devel-20091221.log LLVM got disabled because you don't have Perl: configure: error: perl is required but was not found, please install it configure: error: ./configure failed for llvm So ClamAV build nojit.c, instead of bytecode2llvm.cpp (where the user had the problem). Does FreeBSD build packages for i386? Debian for example builds for i486 AFAIK. Meanwhile I found out that minimum CPU for FreeBSD 9.x is i386, and i486 for 9.x. Is there a way to force ClamAV being built as i486 in the clamav port itself? FYI, on FreeBSD 6.x it's building fast, probably without use llvm. http://freebsd1.asmallorange.com//logs/6.4-i386-garga/clamav-devel-20091221.log Because the compiler is too old: checking for supported C++ compiler version... configure: error: C++ compiler too old (3.4.6) ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Prblems building snapshots on FreeBSD [FIXED]
On 2009-12-22 14:38, Renato Botelho wrote: 2009/12/22 Török Edwin edwinto...@gmail.com: On 2009-12-22 13:36, Török Edwin wrote: I'll ask user to test it, but anyway, here you can see a buildlog on my 7.2 tinderbox that went fine, and it was built for i386 too... http://freebsd1.asmallorange.com//logs/7.2-i386-garga/clamav-devel-20091221.log LLVM got disabled because you don't have Perl: configure: error: perl is required but was not found, please install it configure: error: ./configure failed for llvm So ClamAV build nojit.c, instead of bytecode2llvm.cpp (where the user had the problem). Does FreeBSD build packages for i386? Debian for example builds for i486 AFAIK. Meanwhile I found out that minimum CPU for FreeBSD 9.x is i386, and i486 for 9.x. Is there a way to force ClamAV being built as i486 in the clamav port itself? Just one more information, we have llvm available under FreeBSD ports collection, is it possible to use it instead of embedded one? The LLVM in the FreeBSD ports has same problem (built w/o multithreading support). At this point its not possible to use an external LLVM, since I am not using LLVM 2.6, I am using their SVN code that will become LLVM 2.7. When ClamAV is released we'll probably have to settle with an LLVM release, and at that point it may be possible to use the system's LLVM, instead of the builtin one. But either way we have to ship the parts of LLVM we use, since we can't expect people to build LLVM in order to use ClamAV. Building the upstream LLVM takes far more time than building just the parts we use in ClamAV. Just for curious, why clamav is now using llvm? I meant, an embedded one? Before it it took less than 2 minutes to build clamav, now it takes almos 20. Before ClamAV didn't use LLVM at all, it is needed for the upcoming bytecode execution feature planned for 0.96. We can't use LLVM 2.6, because I needed some features not yet available there. Its an optional component, in the sense that if your system doesn't have a proper C++ compiler (or if you pass --enable-llvm=no) then ClamAV will fallback to an interpreter mode (that doesn't require LLVM), and LLVM is not even built. The interpreter will be slower than LLVM, but it will work. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Prblems building snapshots on FreeBSD [FIXED]
On 2009-12-22 14:43, Renato Botelho wrote: 2009/12/22 Török Edwin edwinto...@gmail.com: Because the compiler is too old: checking for supported C++ compiler version... configure: error: C++ compiler too old (3.4.6) By default, ports use the default base system compiler, of course i can force it to use other version of GCC, do you think it's the correct option? See my other mail on what happens, to sum up: no LLVM - fallback to the slower interpreter mode. It also defaults to auto-detecting at configure time whether LLVM will work or not. Now there is nothing wrong with 3.4.6, except that atomic builtins are not available, not even for i686. But since atomic builtins won't be available on any compiler for FreeBSD i386 anyway, I think you could force ClamAV to build LLVM: passing --enable-llvm=yes to configure should take care of that. It will then be your responsibility to check that your compiler is not broken, but for users the default is to enable it only if safe. See here for more details on requirements: http://git.clamav.net/gitweb?p=clamav-devel.git;a=blob;f=docs/clamdoc.pdf;h=ad82aa037d85d0d66c537916b223e308df37cf57;hb=HEAD If yes, which version do you think is necessary at least? We have from 4.2 to 4.5 available. I think 4.3 or 4.4 would be a good choice. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Prblems building snapshots on FreeBSD [FIXED]
On 2009-12-21 16:30, Renato Botelho wrote: On Mon, Dec 21, 2009 at 10:21 AM, Renato Botelho rbga...@gmail.com wrote: I'm still having problems building clamav recent snapshots on FreeBSD hosts, this time on FreeBSD 6.4 and 7.2 (x86 and x86_64) Here is the log: /bin/sh ../libtool --silent --tag=CC --mode=compile cc -DHAVE_CONFIG_H -I. -I.. -I.. -I./nsis -I../libltdl -DWARN_DLOPEN_FAIL -I/usr/local/include-O2 -fno-strict-aliasing -pipe -c -o bytecode_nojit.lo bytecode_nojit.c In file included from bytecode.h:27, from bytecode_nojit.c:26: fmap.h:45: error: syntax error before off_t gmake[4]: *** [bytecode_nojit.lo] Error 1 gmake[4]: *** Waiting for unfinished jobs gmake[4]: Leaving directory `/work/a/ports/security/clamav-devel/work/clamav-devel-20091221/libclamav' gmake[3]: *** [all-recursive] Error 1 gmake[3]: Leaving directory `/work/a/ports/security/clamav-devel/work/clamav-devel-20091221/libclamav' gmake[2]: *** [all] Error 2 gmake[2]: Leaving directory `/work/a/ports/security/clamav-devel/work/clamav-devel-20091221/libclamav' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/work/a/ports/security/clamav-devel/work/clamav-devel-20091221' gmake: *** [all] Error 2 *** Error code 1 Attached patch fixed the problem Hi Renato, Thanks for the patch, what if you only wrap the include by #ifdef HAVE_UNISTD_H? Something like: #ifdef HAVE_UNISTD_H #include unistd.h #endif We have a similar bug for OS X https://wwws.clamav.net/bugzilla/show_bug.cgi?id=1776, which I haven't had time to look into yet. Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Prblems building snapshots on FreeBSD [FIXED]
On 2009-12-21 16:57, Renato Botelho wrote: 2009/12/21 Török Edwin edwinto...@gmail.com: Hi Renato, Thanks for the patch, what if you only wrap the include by #ifdef HAVE_UNISTD_H? Something like: #ifdef HAVE_UNISTD_H #include unistd.h #endif We have a similar bug for OS X https://wwws.clamav.net/bugzilla/show_bug.cgi?id=1776, which I haven't had time to look into yet. FYI, I've attached this patch on bug 1776 asking someone to test it on OSX Thanks, I applied the patch here: http://git.clamav.net/gitweb?p=clamav-devel.git;a=commit;h=ce0c204a7ebcd6075cd58daeb5a5c2909343556c Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Error building 20091216 snapshot on FreeBSD
On 2009-12-16 13:51, Renato Botelho wrote: I got following error trying to build recent snapshot under FreeBSD gmake[4]: Entering directory `/home/garga/prs/clamav-devel/work/clamav-devel-20091216/libclamav/c++' CXXLD tblgen /usr/bin/ld: cannot find -ldl gmake[4]: *** [tblgen] Error 1 We don't have this library on FreeBSD. Any tips? What is the value of lt_cv_dlopen_libs in libclamav/c++/llvm/config.log? Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: [Clamav-devel] Error building 0.95.3 on FreeBSD 7.2
On 2009-11-16 13:54, Renato Botelho wrote: A user reported me he is having problems building 0.95.3 under FreeBSD 7.2, here is the error: libtool: link: ranlib .libs/libclamunrar_iface.a libtool: link: ( cd .libs rm -f libclamunrar_iface.la ln -s ../libclamunrar_iface.la libclamunrar_iface.la ) /bin/sh ../libtool --tag=CC --mode=compile cc -DHAVE_CONFIG_H -I. -I.. -I.. -I./nsis -I./lzma -I../libltdl -DWARN_DLOPEN_FAIL -I/usr/local/include -DSEARCH_LIBDIR=\/usr/local/lib\ -O2 -fno-strict-aliasing +-pipe -c -o libclamav_la-version.lo `test -f 'version.c' || echo './'`version.c libtool: compile: cc -DHAVE_CONFIG_H -I. -I.. -I.. -I./nsis -I./lzma -I../libltdl -DWARN_DLOPEN_FAIL -I/usr/local/include -DSEARCH_LIBDIR=\/usr/local/lib\ -O2 -fno-strict-aliasing -pipe -c version.c -fPIC +-DPIC -o .libs/libclamav_la-version.o +In file included from version.c:5: version.h:1:22: warning: missing terminating character +version.h:2:9: warning: missing terminating character In file included from version.c:5: version.h:2: error: missing terminating character version.c:12: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'const' version.c:14: error: missing terminating character *** Error code 1 1 error *** Error code 1 1 error *** Error code 2 1 error *** Error code 1 1 error *** Error code 2 *** Error code 1 Stop in /disk1/usr/ports/security/clamav. *** Error code 1 Stop in /disk1/usr/ports/security/clamav. Do you have idea? What is the contents of version.h? Its supposed to be empty for a release, or to contain a define of the version for a devel version (autogenerated by make). Best regards, --Edwin ___ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
