Re: [squid-users] transparent squid + clamav + https
ons 2010-03-17 klockan 03:53 + skrev Amos Jeffries: During the infected period imaginary-HAVP scans the documents and sends a large clean prefix to all visitors. BUT... aborts when the appended infection is detected. Browser is lucky enough to notice the file is incomplete and retires later with a range request for the missing bit. a) during the infected period the fetched ranges will never succeed. ok. b) after the infection is cleaned up the file will pass through imaginary-HAVP and client will get a truncated version. With complete-file being indicated. Only if the server is seriously broken and uses the same cache validator for this modified response. This is exacly why ETag SHOULD be used and not Last-Modified. Regards Henrik
Re: [squid-users] transparent squid + clamav + https
ons 2010-03-17 klockan 07:51 +0100 skrev Henrik Nordström: ons 2010-03-17 klockan 03:53 + skrev Amos Jeffries: During the infected period imaginary-HAVP scans the documents and sends a large clean prefix to all visitors. BUT... aborts when the appended infection is detected. Browser is lucky enough to notice the file is incomplete and retires later with a range request for the missing bit. a) during the infected period the fetched ranges will never succeed. ok. b) after the infection is cleaned up the file will pass through imaginary-HAVP and client will get a truncated version. With complete-file being indicated. Only if the server is seriously broken and uses the same cache validator for this modified response. This is exacly why ETag SHOULD be used and not Last-Modified. And I forgot to mention that clients accepting to merge such responses is also broken as the object signature differs (differnt advertised length) and MUST NOT be merged. Inplace alterations of files without change in size or last-modified is trickier to detect, especially as many servers are known to not detect that and still responds with same ETag (Apache is one of them). Regards Henrik
Re: [squid-users] transparent squid + clamav + https
Hi, I have configured the viralator. But I have some problems with the redirector. When I run /opt/squirm/bin/squirm as user squid by hand, I get the following text: squid1 logs # tail -f squirm.info Wed Mar 17 15:33:19 2010:processing configuration file [/opt/squirm/etc/squirm.conf] Wed Mar 17 15:33:19 2010:Reading patterns from file /opt/squirm/etc/squirm.patterns Wed Mar 17 15:33:19 2010:Squirm (PID 10474) started Then when I enter an url in the running squirm like: http://putty.very.rulez.org/latest/x86/putty.exe 192.9.200.123/- - GET or http://putty.very.rulez.org/latest/x86/putty.exe 127.0.0.1/- - GET I get no output (only a empty row). When I run squirm as root, it will be the same, but on std out: squid1 bin # ./squirm Squirm running as UID 0: writing logs to stderr Wed Mar 17 15:36:42 2010:processing configuration file [/opt/squirm/etc/squirm.conf] Wed Mar 17 15:36:42 2010:Reading patterns from file /opt/squirm/etc/squirm.patterns Wed Mar 17 15:36:42 2010:Squirm (PID 10477) started http://putty.very.rulez.org/latest/x86/putty.exe 192.9.200.123/- - GET http://putty.very.rulez.org/latest/x86/putty.exe 127.0.0.1/- - GET My squirm folder: squid1# ls -al /opt/squirm/etc/ total 40 drwxrwx--- 2 root squid 4096 Mar 17 15:29 . drwxr-xr-x 5 root bin4096 Mar 16 11:11 .. -rw-r--r-- 1 root root 12288 Mar 17 15:29 .squirm.conf.swp -rw-r--r-- 1 root root 1186 Mar 17 15:17 backup.tar.gz -rw-r--r-- 1 root root 1168 Mar 17 14:46 squirm.conf -rw-rw 1 root squid 1064 Mar 17 15:18 squirm.conf.dist -rw-r--r-- 1 root root 1139 Mar 17 15:13 squirm.patterns -rw-rw 1 root squid 682 Mar 17 15:18 squirm.patterns.dist cat squirm.conf: #squids ip ist 192.9.200.32/24 begin network 192.9.200.0/24 log logs/match.log abort-log logs/abort.log pattern squirm.patterns get end begin network 127.0.0.0/24 log logs/private-match.log abort-log logs/private-abort.log pattern squirm.patterns get end cat squirm.patterns: abortregexi (^http://192.9.200.32.*) abortregexi (^http://squid1.testingdomain.de.*) regexi (^.*\.zip$) http://192.9.200.32/cgi-bin/viralator.cgi?url=|\1 regexi (^.*\.doc$) http://192.9.200.32/cgi-bin/viralator.cgi?url=|\1 regexi (^.*\.exe$) http://192.9.200.32/cgi-bin/viralator.cgi?url=|\1 So there is only the squirm.info in the log folder, nothing else. The viralator seems working, but there is now entry about squirm in the logs. Can I test the viralator.cgi with parameters on bash or webbrowser? Thanks, Stefan
Re: [squid-users] transparent squid + clamav + https
On Wed, 17 Mar 2010 16:04:03 +0100, Stefan Reible m...@stefan-reible.de wrote: Hi, I have configured the viralator. But I have some problems with the redirector. When I run /opt/squirm/bin/squirm as user squid by hand, I get the following text: squid1 logs # tail -f squirm.info Wed Mar 17 15:33:19 2010:processing configuration file [/opt/squirm/etc/squirm.conf] Wed Mar 17 15:33:19 2010:Reading patterns from file /opt/squirm/etc/squirm.patterns Wed Mar 17 15:33:19 2010:Squirm (PID 10474) started Then when I enter an url in the running squirm like: http://putty.very.rulez.org/latest/x86/putty.exe 192.9.200.123/- - GET or http://putty.very.rulez.org/latest/x86/putty.exe 127.0.0.1/- - GET I get no output (only a empty row). Empty row is a working output. It means don't alter the URL to Squid. If you did want the URL altered, you have a problem inside your squirm patterns. Amos
Re: [squid-users] transparent squid + clamav + https
Tks for your info Amos. Amos Jeffries wrote: On Mon, 15 Mar 2010 14:50:54 -0300, Leonardo Carneiro - Veltrac lscarne...@veltrac.com.br wrote: I have always read that transparent proxy + https was not possible. It is now? There is a stable squid version with this feature? There aew any major drawbacks using this feature? Tks in advance. Sadly, yes it's now possible. No there is not yet a stable version of Squid to do it. Yes there are still some limits thankfully: 1) it is only useful for corporate environments which closely monitor their own staff. 1b) has some use catching viruses etc if thats whats monitored for. It is a slippery slope problem. 2) it does not work for ISP setups. 3) requires a CA certificate on all client machines, which authorizes the proxy fake certificates. 4) does not work for any hidden-mole attacks (they are still invisible and actually gain extra info about the network from the certificate challenges). Amos Henrik K wrote: On Mon, Mar 15, 2010 at 12:30:11PM +0100, Stefan Reible wrote: PS: I have an secound problem with downloading big files, is it possilbe to send any infos about the download progress to the webbrowser? Like opening an ajax script or something else. If you don't want this limitation, you can use HAVP. It scans the file while it's being transferred to client, while keeping small part of it buffered (in case of virus, it is not transferred so client can't open incomplete file). It's as close to transparent as you can get.
Re: Re: [squid-users] transparent squid + clamav + https
On 01/-10/-28163 12:59 PM, Henrik Nordström wrote: Yes. See the viralator mode of c-icap srv_clamav. The service supports 3 different modes of download management - Wait with response until scanning have completed - Send some data of the file while scanning is performed to keep the client patiently waiting. - viralator mode showing progress while scanning is done, and then redirecting to a download URL when complete The problem with viralator mode is that it may break some things as it responds with another response while scanning. Regards Henrik Using acls in Squid, this can be avoided. I am doing that here and allow things like Trend Micro, Microsoft, Mozilla, etc. update services (based on whatever criteria I can which is most restrictive) to us the some data mode, everything else is Viralator mode. Trever -- Anger is momentary madness. -- Horace signature.asc Description: OpenPGP digital signature
Re: [squid-users] transparent squid + clamav + https
mån 2010-03-15 klockan 18:47 +0200 skrev Henrik K: If you don't want this limitation, you can use HAVP. It scans the file while it's being transferred to client, while keeping small part of it buffered (in case of virus, it is not transferred so client can't open incomplete file). It's as close to transparent as you can get. That's also one of the three modes supported by c-icap clamav service. Regards Henrik
Re: [squid-users] transparent squid + clamav + https
On Tue, Mar 16, 2010 at 08:58:27PM +0100, Henrik Nordström wrote: mån 2010-03-15 klockan 18:47 +0200 skrev Henrik K: If you don't want this limitation, you can use HAVP. It scans the file while it's being transferred to client, while keeping small part of it buffered (in case of virus, it is not transferred so client can't open incomplete file). It's as close to transparent as you can get. That's also one of the three modes supported by c-icap clamav service. Such comment can only be made when one doesn't understand what HAVP does. It is NOT the same thing. http://www.server-side.de/documentation.htm While one can speculate about the usefulness of scanning huge files at HTTP level, HAVP with mandatory locking does it much more efficiently. C-icap will only call the scanner after a file is completely received, resulting in additional wait and a load spike. HAVP starts scanning the file immediately as it is received from the server and gradually unlocked. When c-icap has just started scanning the file, HAVP has already scanned most (if not all) of it and is sending final bytes to client. If a virus had happened to be found, HAVP would have already stopped the unnecessary download without wasting time on the whole file. This also works on ZIP files as it first tries to download the header at end of the file using Range request.
Re: [squid-users] transparent squid + clamav + https
On Wed, 17 Mar 2010 05:24:38 +0200, Henrik K h...@hege.li wrote: On Tue, Mar 16, 2010 at 08:58:27PM +0100, Henrik Nordström wrote: mån 2010-03-15 klockan 18:47 +0200 skrev Henrik K: If you don't want this limitation, you can use HAVP. It scans the file while it's being transferred to client, while keeping small part of it buffered (in case of virus, it is not transferred so client can't open incomplete file). It's as close to transparent as you can get. That's also one of the three modes supported by c-icap clamav service. Such comment can only be made when one doesn't understand what HAVP does. It is NOT the same thing. http://www.server-side.de/documentation.htm While one can speculate about the usefulness of scanning huge files at HTTP level, HAVP with mandatory locking does it much more efficiently. C-icap will only call the scanner after a file is completely received, resulting in additional wait and a load spike. HAVP starts scanning the file immediately as it is received from the server and gradually unlocked. When c-icap has just started scanning the file, HAVP has already scanned most (if not all) of it and is sending final bytes to client. If a virus had happened to be found, HAVP would have already stopped the unnecessary download without wasting time on the whole file. This also works on ZIP files as it first tries to download the header at end of the file using Range request. So HAVP is designed specifically to send client scanned parts of the file before the entire thing is checked? That explains something that I was wondering about... Consider this scenario which I have seen in the wild: Background: Clients visit website A and fetch a large document PDF file. Unknown to the website author the server has been infected and PDF is one of the files which get a macro virus appended. The CRM system records in a database the upload and change times for efficient if-modified responses. The server admin is alerted and runs a virus scan, cleaning the files some time later. The CRM database gets omitted from the update. Imagining that HAVP was in use in a proxy between this site and a user... During the infected period imaginary-HAVP scans the documents and sends a large clean prefix to all visitors. BUT... aborts when the appended infection is detected. Browser is lucky enough to notice the file is incomplete and retires later with a range request for the missing bit. a) during the infected period the fetched ranges will never succeed. b) after the infection is cleaned up the file will pass through imaginary-HAVP and client will get a truncated version. With complete-file being indicated. This is where the problem comes in. Being a macro infection one of the changes to the file was that the virus appended some undetectable jump code at the beginning to go with the virus at the end. We are left with the situation where intermediary proxies are holding corrupted files (first part being original infected with jump, followed by terminal bytes of teh file. the server is left with a pristine and working file. New visitors loading it will be fine, and sill analysts coming along later. However for clients visiting through one of the proxies which cached the file meanwhile ... One of two things will happen to depending on the file viewer used: 1) dumb viewer will try to run the random part of file (now text!) where virus inserted itself as binary code and crash. 2) smart viewer will notice the missing/corrupt macro (its past the end of file maybe) and display the file without running it. However, even then there is a discrepancy in file prefix and some of the content appears corrupted. This type of traffic is the #1 reason for buffering until fully processed. I do like the idea of incremental scanning as it arrives though. That will at least reduce the delays to very little more than the total receiving time. Amos
Re: [squid-users] transparent squid + clamav + https
On Wed, Mar 17, 2010 at 03:53:01AM +, Amos Jeffries wrote: So HAVP is designed specifically to send client scanned parts of the file before the entire thing is checked? Right. Of course all this is configurable. That explains something that I was wondering about... Consider this scenario which I have seen in the wild: Background: Clients visit website A and fetch a large document PDF file. Unknown to the website author the server has been infected and PDF is one of the files which get a macro virus appended. The CRM system records in a database the upload and change times for efficient if-modified responses. The server admin is alerted and runs a virus scan, cleaning the files some time later. The CRM database gets omitted from the update. Imagining that HAVP was in use in a proxy between this site and a user... During the infected period imaginary-HAVP scans the documents and sends a large clean prefix to all visitors. BUT... aborts when the appended infection is detected. Browser is lucky enough to notice the file is incomplete and retires later with a range request for the missing bit. a) during the infected period the fetched ranges will never succeed. HAVP doesn't allow Range requests by default anyway, always forcing a full download. b) after the infection is cleaned up the file will pass through imaginary-HAVP and client will get a truncated version. With complete-file being indicated. This is where the problem comes in. Being a macro infection one of the changes to the file was that the virus appended some undetectable jump code at the beginning to go with the virus at the end. We are left with the situation where intermediary proxies are holding corrupted files (first part being original infected with jump, followed by terminal bytes of teh file. the server is left with a pristine and working file. New visitors loading it will be fine, and sill analysts coming along later. However for clients visiting through one of the proxies which cached the file meanwhile ... One of two things will happen to depending on the file viewer used: 1) dumb viewer will try to run the random part of file (now text!) where virus inserted itself as binary code and crash. 2) smart viewer will notice the missing/corrupt macro (its past the end of file maybe) and display the file without running it. However, even then there is a discrepancy in file prefix and some of the content appears corrupted. This type of traffic is the #1 reason for buffering until fully processed. I do like the idea of incremental scanning as it arrives though. That will at least reduce the delays to very little more than the total receiving time. The recommended configuration is not to cache between client and HAVP. It comes at a small penalty of scanning files every time, but also has a bonus of detecting viruses that are in cache but hadn't signatures at that time. http://havp.hege.li/forum/viewtopic.php?f=2t=11 So I'm not sure your worries apply.. please clarify if I didn't understand.
Re: [squid-users] transparent squid + clamav + https
mån 2010-03-15 klockan 12:30 +0100 skrev Stefan Reible: The transparent http proxy with clamav ist working very nice, but now i have problems with the implementation of ssl. My first idea was, to break down the encryption at the squid, an then create a new one. http://wiki.squid-cache.org/Features/SslBump Is this possible? I think the problem is, that if someone opens an https encrypted website like https://google.de he gets the certificate from the proxy in his browser, not from the webserver. This wouldn`t be so fine.. Well, it's the only possibility, othewise the proxy (and clamav) won't be able to inspect the https traffic. PS: I have an secound problem with downloading big files, is it possilbe to send any infos about the download progress to the webbrowser? Like opening an ajax script or something else. Yes. See the viralator mode of c-icap srv_clamav. The service supports 3 different modes of download management - Wait with response until scanning have completed - Send some data of the file while scanning is performed to keep the client patiently waiting. - viralator mode showing progress while scanning is done, and then redirecting to a download URL when complete The problem with viralator mode is that it may break some things as it responds with another response while scanning. Regards Henrik
Re: [squid-users] transparent squid + clamav + https
Le Lundi 15 Mars 2010 05:30:11, Stefan Reible a écrit : Hi, for my exam I want to set up a transparent proxy with http and https under gentoo linux. The transparent http proxy with clamav ist working very nice, but now i have problems with the implementation of ssl. My first idea was, to break down the encryption at the squid, an then create a new one. http://wiki.squid-cache.org/Features/SslBump Is this possible? I think the problem is, that if someone opens an https encrypted website like https://google.de he gets the certificate from the proxy in his browser, not from the webserver. This wouldn`t be so fine.. Do you have any solutions, informations or ideas for this problem? Thanks, Stefan PS: I have an secound problem with downloading big files, is it possilbe to send any infos about the download progress to the webbrowser? Like opening an ajax script or something else. There are 2 ways you may do that. 1. Use 3.1's sslbump capabilities. However you need a CA already installed in your clientes to avoid the non-confidence windows of browsers about ssl cert. But this won work in transparent mode. Just explicit. 2. Use de DynamicSSLCert branch code. https://code.launchpad.net/~rousskov/squid/DynamicSslCert Not available at 3.1, but at 3.2 (can Ammos or Henrik confirm this?). However you still need the CA and this could work in transparent mode. LD
Re: [squid-users] transparent squid + clamav + https
On Mon, Mar 15, 2010 at 12:30:11PM +0100, Stefan Reible wrote: PS: I have an secound problem with downloading big files, is it possilbe to send any infos about the download progress to the webbrowser? Like opening an ajax script or something else. If you don't want this limitation, you can use HAVP. It scans the file while it's being transferred to client, while keeping small part of it buffered (in case of virus, it is not transferred so client can't open incomplete file). It's as close to transparent as you can get.
Re: [squid-users] transparent squid + clamav + https
I have always read that transparent proxy + https was not possible. It is now? There is a stable squid version with this feature? There aew any major drawbacks using this feature? Tks in advance. Henrik K wrote: On Mon, Mar 15, 2010 at 12:30:11PM +0100, Stefan Reible wrote: PS: I have an secound problem with downloading big files, is it possilbe to send any infos about the download progress to the webbrowser? Like opening an ajax script or something else. If you don't want this limitation, you can use HAVP. It scans the file while it's being transferred to client, while keeping small part of it buffered (in case of virus, it is not transferred so client can't open incomplete file). It's as close to transparent as you can get.
Re: [squid-users] transparent squid + clamav + https
On Mon, 15 Mar 2010 14:50:54 -0300, Leonardo Carneiro - Veltrac lscarne...@veltrac.com.br wrote: I have always read that transparent proxy + https was not possible. It is now? There is a stable squid version with this feature? There aew any major drawbacks using this feature? Tks in advance. Sadly, yes it's now possible. No there is not yet a stable version of Squid to do it. Yes there are still some limits thankfully: 1) it is only useful for corporate environments which closely monitor their own staff. 1b) has some use catching viruses etc if thats whats monitored for. It is a slippery slope problem. 2) it does not work for ISP setups. 3) requires a CA certificate on all client machines, which authorizes the proxy fake certificates. 4) does not work for any hidden-mole attacks (they are still invisible and actually gain extra info about the network from the certificate challenges). Amos Henrik K wrote: On Mon, Mar 15, 2010 at 12:30:11PM +0100, Stefan Reible wrote: PS: I have an secound problem with downloading big files, is it possilbe to send any infos about the download progress to the webbrowser? Like opening an ajax script or something else. If you don't want this limitation, you can use HAVP. It scans the file while it's being transferred to client, while keeping small part of it buffered (in case of virus, it is not transferred so client can't open incomplete file). It's as close to transparent as you can get.
