Re: [clamav-users] [SUSPECTED SPAM] Re: Calamav cannot scan tar file and gzip files?
On Tuesday, February 17, 2015 11:58:02 PM Manoj Ramakrishnan wrote: On 18/02/15 6:09 AM, Steven Morgan smor...@sourcefire.com wrote: On Tue, Feb 17, 2015 at 1:11 AM, Manoj Ramakrishnan manojramakrish...@nbnco.com.au wrote: Hi Al, Thanks for replying. It is exactly what I thought. But why is it different from ZIP file? I added extra characters in the beginning of the ZIP file but no issues in scanning that and finding eicar signature. It may be because of this file typing signature, which is not tied to a fixed offset (the '*' in second field is wildcard offset): 1:*:504b0304:ZIP-SFX:CL_TYPE_ANY:CL_TYPE_ZIPSFX There are no corresponding wildcard magics for GZIP. Could you please confirm by looking for a message containing ZIP/ZIP-SFX signature found at in your debug output. Also curious to see why is it not working in case #4 and #6? Using LeaveTemporaryFiles yes, you should be able to inspect files in the ClamAV temp directory as forwarded by your web proxy. This will show the files as seen by ClamAV. As already pointed out, if there are any additional characters (http headers, etc.), it will not be recognized as GZIP. Are there any settings in squidclamav to control how files are formed for forwarding to ClamAV? At the moment there is no settings in squidclamav to extract the multipart form data and send only the attachment to clamd. As Kevin mentioned, if clamd doesn't natively support parsing HTTP messages then we need to find a way to pass correct data to clamd. Is HTTP message parsing support on your feature roadmap for clamd? I haven't been following this thread very closely, so this may be off track, but would havp do what you need: http://www.server-side.de/ Scott K ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] [SUSPECTED SPAM] Re: Calamav cannot scan tar file and gzip files?
On 18/02/15 6:09 AM, Steven Morgan smor...@sourcefire.com wrote: On Tue, Feb 17, 2015 at 1:11 AM, Manoj Ramakrishnan manojramakrish...@nbnco.com.au wrote: Hi Al, Thanks for replying. It is exactly what I thought. But why is it different from ZIP file? I added extra characters in the beginning of the ZIP file but no issues in scanning that and finding eicar signature. It may be because of this file typing signature, which is not tied to a fixed offset (the '*' in second field is wildcard offset): 1:*:504b0304:ZIP-SFX:CL_TYPE_ANY:CL_TYPE_ZIPSFX There are no corresponding wildcard magics for GZIP. Could you please confirm by looking for a message containing ZIP/ZIP-SFX signature found at in your debug output. Also curious to see why is it not working in case #4 and #6? Using LeaveTemporaryFiles yes, you should be able to inspect files in the ClamAV temp directory as forwarded by your web proxy. This will show the files as seen by ClamAV. As already pointed out, if there are any additional characters (http headers, etc.), it will not be recognized as GZIP. Are there any settings in squidclamav to control how files are formed for forwarding to ClamAV? At the moment there is no settings in squidclamav to extract the multipart form data and send only the attachment to clamd. As Kevin mentioned, if clamd doesn't natively support parsing HTTP messages then we need to find a way to pass correct data to clamd. Is HTTP message parsing support on your feature roadmap for clamd? Regards Manoj Hope this helps, Steve ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] [SUSPECTED SPAM] Re: Calamav cannot scan tar file and gzip files?
UmmmŠ the text diagram is not rendered as intended. What I was trying to show is: Client --- Apache Reverse Proxy ---non scanning urlsbunch of application servers Client --- Apache Reverse Proxy ---Scan a list of urls for virus in client uploaded files -- Squid(act as a reverse proxy) + CICAP + Clamd --- Virus found -- HTTP 403 to Clent Client --- Apache Reverse Proxy ---Scan a list of urls for virus in client uploaded files -- Squid(act as a reverse proxy) + CICAP + Clamd --- No virus -- bunch of application servers Manoj On 18/02/15 12:42 PM, Manoj Ramakrishnan manojramakrish...@nbnco.com.au wrote: Hi Scott, I had a look at what havp does and am not sure it will fit with our current design. Will do a spike to find out. Our application stack has the following design Client == Apache Reverse Proxy (non scanning urls) Bunch of app servers || ^^ || || Scan a list of urls for virus || in client uploaded files || || || || || || || Squid(act as a reverse proxy) + CICAP + Clamd No virus ==Go to || || || Virus found(Go back to client with 403) I probably can replace CICAP with HAVP But I am not sure how can I use the HAVP to act as a reverse proxy without another Squid. Hope this explains. Manoj On 18/02/15 11:10 AM, Scott Kitterman ubu...@kitterman.com wrote: On Tuesday, February 17, 2015 11:58:02 PM Manoj Ramakrishnan wrote: On 18/02/15 6:09 AM, Steven Morgan smor...@sourcefire.com wrote: On Tue, Feb 17, 2015 at 1:11 AM, Manoj Ramakrishnan manojramakrish...@nbnco.com.au wrote: Hi Al, Thanks for replying. It is exactly what I thought. But why is it different from ZIP file? I added extra characters in the beginning of the ZIP file but no issues in scanning that and finding eicar signature. It may be because of this file typing signature, which is not tied to a fixed offset (the '*' in second field is wildcard offset): 1:*:504b0304:ZIP-SFX:CL_TYPE_ANY:CL_TYPE_ZIPSFX There are no corresponding wildcard magics for GZIP. Could you please confirm by looking for a message containing ZIP/ZIP-SFX signature found at in your debug output. Also curious to see why is it not working in case #4 and #6? Using LeaveTemporaryFiles yes, you should be able to inspect files in the ClamAV temp directory as forwarded by your web proxy. This will show the files as seen by ClamAV. As already pointed out, if there are any additional characters (http headers, etc.), it will not be recognized as GZIP. Are there any settings in squidclamav to control how files are formed for forwarding to ClamAV? At the moment there is no settings in squidclamav to extract the multipart form data and send only the attachment to clamd. As Kevin mentioned, if clamd doesn't natively support parsing HTTP messages then we need to find a way to pass correct data to clamd. Is HTTP message parsing support on your feature roadmap for clamd? I haven't been following this thread very closely, so this may be off track, but would havp do what you need: http://www.server-side.de/ Scott K ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] [SUSPECTED SPAM] Re: Calamav cannot scan tar file and gzip files?
On 2/17/15 3:58:02PM, Manoj Ramakrishnan wrote: At the moment there is no settings in squidclamav to extract the multipart form data and send only the attachment to clamd. As Kevin mentioned, if clamd doesn't natively support parsing HTTP messages then we need to find a way to pass correct data to clamd. Is HTTP message parsing support on your feature roadmap for clamd? Regards Manoj ClamAV is a back-end tool. Any parsing is done in the front-end tool such as a milter or proxy server. dp ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] [SUSPECTED SPAM] Re: Calamav cannot scan tar file and gzip files?
Hi Scott, I had a look at what havp does and am not sure it will fit with our current design. Will do a spike to find out. Our application stack has the following design Client == Apache Reverse Proxy (non scanning urls) Bunch of app servers || ^^ || || Scan a list of urls for virus || in client uploaded files || || || || || || || Squid(act as a reverse proxy) + CICAP + Clamd No virus ==Go to || || || Virus found(Go back to client with 403) I probably can replace CICAP with HAVP But I am not sure how can I use the HAVP to act as a reverse proxy without another Squid. Hope this explains. Manoj On 18/02/15 11:10 AM, Scott Kitterman ubu...@kitterman.com wrote: On Tuesday, February 17, 2015 11:58:02 PM Manoj Ramakrishnan wrote: On 18/02/15 6:09 AM, Steven Morgan smor...@sourcefire.com wrote: On Tue, Feb 17, 2015 at 1:11 AM, Manoj Ramakrishnan manojramakrish...@nbnco.com.au wrote: Hi Al, Thanks for replying. It is exactly what I thought. But why is it different from ZIP file? I added extra characters in the beginning of the ZIP file but no issues in scanning that and finding eicar signature. It may be because of this file typing signature, which is not tied to a fixed offset (the '*' in second field is wildcard offset): 1:*:504b0304:ZIP-SFX:CL_TYPE_ANY:CL_TYPE_ZIPSFX There are no corresponding wildcard magics for GZIP. Could you please confirm by looking for a message containing ZIP/ZIP-SFX signature found at in your debug output. Also curious to see why is it not working in case #4 and #6? Using LeaveTemporaryFiles yes, you should be able to inspect files in the ClamAV temp directory as forwarded by your web proxy. This will show the files as seen by ClamAV. As already pointed out, if there are any additional characters (http headers, etc.), it will not be recognized as GZIP. Are there any settings in squidclamav to control how files are formed for forwarding to ClamAV? At the moment there is no settings in squidclamav to extract the multipart form data and send only the attachment to clamd. As Kevin mentioned, if clamd doesn't natively support parsing HTTP messages then we need to find a way to pass correct data to clamd. Is HTTP message parsing support on your feature roadmap for clamd? I haven't been following this thread very closely, so this may be off track, but would havp do what you need: http://www.server-side.de/ Scott K ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] [SUSPECTED SPAM] Re: Calamav cannot scan tar file and gzip files?
On 2/17/2015 12:11 AM, Manoj Ramakrishnan wrote: Hi Al, Thanks for replying. It is exactly what I thought. But why is it different from ZIP file? I added extra characters in the beginning of the ZIP file but no issues in scanning that and finding eicar signature. zip and gzip are very different formats. I suppose you added your random character at a point where unzip ignored it. Also curious to see why is it not working in case #4 and #6? Either broke the eicar file with leading or trailing characters, or maybe the squid plugin didn't recognize the file as a gzip. Use the clam debug tools to examine the files extracted and scanned. The eicar signature is *very* specific, anchored at both the beginning and end allowing only for a few extra spaces at the end of the payload, no other extra characters. ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] [SUSPECTED SPAM] Re: Calamav cannot scan tar file and gzip files?
There are a number of reasons for the differences in the detection cases. The first of which is how ClamAV identifies the file type of file being scanned. ClamAV determines the file type of a scanned file using the 'ftm' signature files. The important signatures follow: type:offset:magic:rtype:type 0:0:504b0304:ZIP:CL_TYPE_ANY:CL_TYPE_ZIP 0:0:504b3030504b0304:ZIP:CL_TYPE_ANY:CL_TYPE_ZIP 1:*:504b0304:ZIP-SFX:CL_TYPE_ANY:CL_TYPE_ZIPSFX 0:0:1f8b:GZip:CL_TYPE_ANY:CL_TYPE_GZ There ZIP archive file type signatures, two of which look for a specific magic at offset 0. However, the last signature uses a '*' offset which indicates the magic can be located anywhere within the file. Do note that the signature is meant to detect the specific variant of ZIPSFX. The GZ file, on the other hand, only has one magic that only triggers if found at offset 0. While an argument could be to extend the GZ file type signature file to search the entire file, there are a number of important counter arguments: 1. The GZ file magic is only 2 bytes long, this means that the extension over the whole file would result in a large number of false positives 2. In theory, by modifying the original GZ file, the file may no longer be a valid GZ file. Thus it's likely that ClamAV would not be able to correctly parse the file. Argument 2 may also result in the lack of detection as the file may not be possible to parse with modifications. As for the reason for the curl POST issue in case #6, can I ask how you what response you get back from clamd when you upload the file using curl POST? clamd is designed to handle a specific of commands that are described in the clamdoc.pdf that comes with the ClamAV source distribution. From what I can see, clamd does not natively support parsing HTTP messages. When I send a file to scan to clamd using curl, clamd fails to understand the message and sends back the message: UNKNOWN COMMAND -Kevin On Tue, Feb 17, 2015 at 1:23 PM, Noel Jones njo...@megan.vbhcs.org wrote: On 2/17/2015 12:11 AM, Manoj Ramakrishnan wrote: Hi Al, Thanks for replying. It is exactly what I thought. But why is it different from ZIP file? I added extra characters in the beginning of the ZIP file but no issues in scanning that and finding eicar signature. zip and gzip are very different formats. I suppose you added your random character at a point where unzip ignored it. Also curious to see why is it not working in case #4 and #6? Either broke the eicar file with leading or trailing characters, or maybe the squid plugin didn't recognize the file as a gzip. Use the clam debug tools to examine the files extracted and scanned. The eicar signature is *very* specific, anchored at both the beginning and end allowing only for a few extra spaces at the end of the payload, no other extra characters. ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] [SUSPECTED SPAM] Re: Calamav cannot scan tar file and gzip files?
On Tue, Feb 17, 2015 at 1:11 AM, Manoj Ramakrishnan manojramakrish...@nbnco.com.au wrote: Hi Al, Thanks for replying. It is exactly what I thought. But why is it different from ZIP file? I added extra characters in the beginning of the ZIP file but no issues in scanning that and finding eicar signature. It may be because of this file typing signature, which is not tied to a fixed offset (the '*' in second field is wildcard offset): 1:*:504b0304:ZIP-SFX:CL_TYPE_ANY:CL_TYPE_ZIPSFX There are no corresponding wildcard magics for GZIP. Could you please confirm by looking for a message containing ZIP/ZIP-SFX signature found at in your debug output. Also curious to see why is it not working in case #4 and #6? Using LeaveTemporaryFiles yes, you should be able to inspect files in the ClamAV temp directory as forwarded by your web proxy. This will show the files as seen by ClamAV. As already pointed out, if there are any additional characters (http headers, etc.), it will not be recognized as GZIP. Are there any settings in squidclamav to control how files are formed for forwarding to ClamAV? Hope this helps, Steve ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
[clamav-users] [SUSPECTED SPAM] Re: Calamav cannot scan tar file and gzip files?
Hi Steve, Thanks for the reply. Really appreciated I tried your suggestion and it mostly works when we use the clamdscan command except some cases like modified gzip, other types like tar, bz2. Will explain below. Dowloaded these two files wget http://www.eicar.org/download/eicar.com wget http://www.eicar.org/download/eicarcom2.zip Case 1: clamdscan eicar.com -- WORKS Case 2: gzip the eicar.com file and scan it using clamdscan --- WORKS Case 3: clamdscan eicarcom2.zip -- WORKS Case 4: Opened the gz file(in Case #2) in vi editor and add a character say a at the beginning of the file and scan it using clamdscan. Not WORKING Case 5: Modified the zip files in Case #3 in vi editor, added some character in beginning and scan it using clamdscan -- WORKS. This always works if there is a PK\003\004 signature in the test file Case 6: use the eicar.com as it is in a curl command POST request(upload file) then it is NOT WORKING We modified these files assuming this is exactly what is happening in POST request. Request body may have additional form data at the begging of the byte stream or/and end of the byte stream. As an example here is the strace output for the POST request (curl -v -H Expect: -H host:www.srv1.com -F attachment=@/tmp/clamd/eicar.com.gz http://localhost:9091/form1/submit) pread(12, --d40d9eade79b\r\nContent-Disposition: form-data; name=\attachment\; filename=\eicar.com.gz\\r\nContent-Type: application/octet-stream\r\n\r\n\37\213\10\10t~\342T\0\3eicar.com\0\2130\36 5W\fPup\f\2106\211\t\210\21205\321\10\210\3234wv\3264\257Uq\365tv\f\322\r\1 6q\364sq\fr\321u\364\v\361\f\363\f\n\r\326\rq\r\16\321u\363\364qUT\361\320\ 366\320\2\0\317QhD\0\0\0\r\n--d40d9eade79b--\r \n, 318, 0) = 318 write(2, LibClamAV debug: Recognized binary data\n, 40) = 40 write(2, LibClamAV debug: cache_check: 6503faa52c4f86f6aa90119703c7f352 is negative\n, 75) = 75 write(2, LibClamAV debug: in cli_check_mydoom_log()\n, 43) = 43 write(2, LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0\n, 68) = 68 write(2, LibClamAV debug: cli_magic_scandesc: returning 0 at line 2470\n, 63) = 63 write(2, LibClamAV debug: cache_add: 6503faa52c4f86f6aa90119703c7f352 (level 0)\n, 71) = 71 munmap(0x2b7e6856d000, 8192)= 0 sendto(11, fd[12]: OK\0, 11, 0, NULL, 0) = 11 I am wondering why is this difference in test results for ZIP and GZIP? Is there a difference between handling magic sequence for ZIP (PK\003\004) and GZIP(\037\213)? Or are we missing any proper configuration settings? Please let us know if you want me to provide more information. Regards Manoj Ramakrishnan DevOps Engineer | POS | P +61 2 8918 5906 | M 0416 128 308 On 17/02/15 5:13 AM, Steven Morgan smor...@sourcefire.com wrote: Manoj, Seem like this should work. What happens if you scan your tar and tar.gz files just using clamscan? You can run your clamd in debug mode by setting Foreground yes and Debug yes in clamd.conf, then run clamd from a terminal window. This may give you an indication about why clamd does not see the inner file when using squid. Also, LeaveTemporaryFiles yes will keep the inner files from archives in the ClamAV temp directory for inspection. Hope this helps, Steve On Sun, Feb 15, 2015 at 11:30 PM, Manoj Ramakrishnan manojramakrish...@nbnco.com.au wrote: Hi, I tried to scan tar files and tar.gz files using clamav(through squid, squidclamav and c-icap) but it just pass through. Both these files contain the eicar.com test file. But if it is a zip file then it works!!! ScanArchive parameter is enabled in clamd.conf. Do I need any special setting to scan these files? I am using a RHEL5 server and clamd/clamav version 0.98.5 Regards Manoj Ramakrishnan DevOps Engineer | POS | P +61 2 8918 5906 | M 0416 128 308 ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] [SUSPECTED SPAM] Re: Calamav cannot scan tar file and gzip files?
Hi Al, Thanks for replying. It is exactly what I thought. But why is it different from ZIP file? I added extra characters in the beginning of the ZIP file but no issues in scanning that and finding eicar signature. Also curious to see why is it not working in case #4 and #6? Regards Manoj On 17/02/15 3:35 PM, Al Varnell alvarn...@mac.com wrote: On Mon, Feb 16, 2015 at 05:27 PM, Manoj Ramakrishnan wrote: Case 4: Opened the gz file(in Case #2) in vi editor and add a character say a at the beginning of the file and scan it using clamdscan. Not WORKING That would be correct. The signature specifies an offset of zero, so adding anything at the beginning ³disinfects² it. -Al- -- Al Varnell Mountain View, CA ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
