Re: [clamav-users] Streaming support in ClamD
I'm still using HAVP for HTTP scanning, and it seems to still work OK with the latest ClamAV (i.e., libclamav etc.). I hope that ClamAV doesn't become incompatible in a way that can't be accommodated. (I had to change HAVP's init temporarily during to the openssl hiccup). Paul Kosinski On Tue, 21 Jul 2015 12:00:01 -0400 clamav-users-requ...@lists.clamav.net wrote: -- Message: 2 Date: Thu, 2 Jul 2015 12:55:54 +0300 From: Henrik K h...@hege.li To: clamav-users@lists.clamav.net Subject: Re: [clamav-users] Streaming support in ClamD Message-ID: 20150702095554.ga32...@hege.li Content-Type: text/plain; charset=us-ascii Let's say you have a zip file. How do you expect ClamAV to scan it packet by packet? Or any other data really. I think there are very few wild signatures in database that are allowed to match any position anywhere in a file. Only reliable way is to scan a complete file, so it knows the length and can decode it properly etc. The now abandoned HAVP proxy scanner does many tricks (filesystem mandatory locking to pseudo-stream files into clamav, zip header prefetch etc) to achieve near realtime scanning for large files and reduce user hanging to a minimum. I guess this is what you are after, but ICAP can't achieve such trickery. ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Streaming support in ClamD
On 08/07/15 17:33, Rafael Ferreira wrote: Well, the progress you see is likely to be transfer, not processing, time since that’s where most time is going to be spent for a sizable file anyways (under normal circumstances) so I doubt clamd is your main latency source here. ? I said clam was the only one that performed well - not the other way around! :-) Can you elaborate on your setup a bit? Is the ICAP proxy in-line to your users or alongside another caching proxy like squid? You can't use ICAP inline - it's always used in conjunction with a proxy server. We use squid. We used to use client-squid-havp-(clam|sophie)-Internet with great success, but havp is dead and showing it's age (some of the newer HTTP options confuse it) and so we want to move to ICAP, primarily because it involves the least number of changes (ie it's either that or throw away squid entirely) c-icap using clam seems to be able to stream: a large download starts flowing to the client very quickly (which is what havp as an AV proxy did really well too) - whereas all the commercial ones I've tried seem to effectively block until the content is passed to ICAP, so it can run AV over the file in it's entirety and then throws it at the client. End result is wigged out users. (BTW: they don't totally block - but they trickle at such an absurd rate that they might as well have blocked) I must say all the commercial ICAP products are always part of a full proxy server - so I wonder if they actually work fine if you use their proprietary product instead of what I'm trying to do (ie maybe this is a marketing trick). I find it hard to believe anyone would want to buy these products as they stand. You know people: they want security with *no* overhead/inconvenience ;-) I'm also aware of the consequence of not scanning the full file in advance - it could miss something - but compromise is acceptable: a product that scans in streamed chunks, pushing each finished piece to the client, and then at the end is able to do the proper scan can still drop the last chunk - breaking the webpage (and therefore corrupting malware executables or zip files - which are 90% of the baddies) and saving the client. If the only proper solution is to block and scan the entire webpage (I keep using that phrase because 99.999% of ICAP queries are of webpages) before handing anything to the client, well that would explain why not enough organizations do AV content filtering of web traffic: their IT groups got lynched when they tried to implement it ;-) -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Streaming support in ClamD
Great timing for me on this topic. We are currently phasing out our use of Henrik's great havp proxy and are going to ICAP - and I have been majorly disappointed with the performance of the commercial ICAP services I've tried I had a 60M zip file I was testing with via Firefox: Kaspersky would say 9 hours to download, then go 9h, 9h, 9h, 9h, complete! - which is an AWFUL end-user experience. F-Secure would go unknown, unknown, unknown, complete! - which is even worse. Strangely enough, the one ICAP service that worked well was - c-icap with CLAMD! That seemed to give a much better feeling of 2m, 2m, 1m, complete! - which seems to contradict what's been said here Henrik - you said ICAP can't achieve such trickery - are you sure about that? If that's true (and my c-icap test is somehow mistaken), then I'm majorly disappointed - and a bit stuck as to how to do AV filtering without users screaming (PS: yes the AVs all took 2minutes to download and process the same file - but the *perception* of performance is the key attribute I want to see) Jason -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Streaming support in ClamD
Well, the progress you see is likely to be transfer, not processing, time since that’s where most time is going to be spent for a sizable file anyways (under normal circumstances) so I doubt clamd is your main latency source here. Can you elaborate on your setup a bit? Is the ICAP proxy in-line to your users or alongside another caching proxy like squid? - Rafael -- Scanii.com | the web friendly virus scanner On Jul 7, 2015, at 5:14 PM, Jason Haar jason_h...@trimble.com wrote: Great timing for me on this topic. We are currently phasing out our use of Henrik's great havp proxy and are going to ICAP - and I have been majorly disappointed with the performance of the commercial ICAP services I've tried I had a 60M zip file I was testing with via Firefox: Kaspersky would say 9 hours to download, then go 9h, 9h, 9h, 9h, complete! - which is an AWFUL end-user experience. F-Secure would go unknown, unknown, unknown, complete! - which is even worse. Strangely enough, the one ICAP service that worked well was - c-icap with CLAMD! That seemed to give a much better feeling of 2m, 2m, 1m, complete! - which seems to contradict what's been said here Henrik - you said ICAP can't achieve such trickery - are you sure about that? If that's true (and my c-icap test is somehow mistaken), then I'm majorly disappointed - and a bit stuck as to how to do AV filtering without users screaming (PS: yes the AVs all took 2minutes to download and process the same file - but the *perception* of performance is the key attribute I want to see) Jason -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Streaming support in ClamD
Hi guys, Waiting for your reply. It should be simpler answer. Does ClamAv support virus checking in stream mode for large files? If i have file size of 10Mb do i have to send all data to clamAv and clamAv will send status ok or it can scan data in each packet and return status for each segment? Thanks On Tue, Jun 30, 2015 at 12:28 PM, P K pkopen...@gmail.com wrote: Hi Guys, I am new to Clamd and was trying to use it for virus scanning. I used squid + icap + clamAv. But i seen once all data is recieved clamAv INSTREAM is called and data is passed to it. Is it issue with icap server or Clamd doesn't support streaming support? Any guidance will be helpful for me and how can we make ClamAv streaming support. Awaiting for reply. ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Streaming support in ClamD
Let's say you have a zip file. How do you expect ClamAV to scan it packet by packet? Or any other data really. I think there are very few wild signatures in database that are allowed to match any position anywhere in a file. Only reliable way is to scan a complete file, so it knows the length and can decode it properly etc. The now abandoned HAVP proxy scanner does many tricks (filesystem mandatory locking to pseudo-stream files into clamav, zip header prefetch etc) to achieve near realtime scanning for large files and reduce user hanging to a minimum. I guess this is what you are after, but ICAP can't achieve such trickery. On Thu, Jul 02, 2015 at 12:57:00PM +0530, P K wrote: Hi guys, Waiting for your reply. It should be simpler answer. Does ClamAv support virus checking in stream mode for large files? If i have file size of 10Mb do i have to send all data to clamAv and clamAv will send status ok or it can scan data in each packet and return status for each segment? Thanks On Tue, Jun 30, 2015 at 12:28 PM, P K pkopen...@gmail.com wrote: Hi Guys, I am new to Clamd and was trying to use it for virus scanning. I used squid + icap + clamAv. But i seen once all data is recieved clamAv INSTREAM is called and data is passed to it. Is it issue with icap server or Clamd doesn't support streaming support? Any guidance will be helpful for me and how can we make ClamAv streaming support. Awaiting for reply. ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
[clamav-users] Streaming support in ClamD
Hi Guys, I am new to Clamd and was trying to use it for virus scanning. I used squid + icap + clamAv. But i seen once all data is recieved clamAv INSTREAM is called and data is passed to it. Is it issue with icap server or Clamd doesn't support streaming support? Any guidance will be helpful for me and how can we make ClamAv streaming support. Awaiting for reply. ___ Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
