[squid-users] WL-Proxy-SSL - telling weblogic about proxied ssl
Hi people I'm trying to set up squid as an http and ssl accelerator for WebCT Vista, which is an application running on BEA WebLogic. I want all connections between squid and weblogic to be plain text, on port 80. User connections will be a combination of SSL (https port 443) and plain http. Aparently, if I can get squid to add the following header to HTTPS requests, then things will be good: WL-Proxy-SSL: true My hope is that this will allow Weblogic to generate https urls when a user is using https, and http when they're using http. This is a non-trivial excercise when you're using squid as an accelerator, so you need some extra header to pass this info on to the backend appserver. How would I go about achieving this? Is it possible? Note that I've already written a perl redirector script but AFAIK there's no way to add extra headers with the redirector interface? Thank you Jesse -- ::: Jesse Reynolds +61 (0)414 669 790 ::: AIM - jessedreynolds ::: ::: Virtual Artists Pty Ltd, Adelaide ::: http://www.va.com.au :::
Re: [squid-users] access.log redirection to mysql
At 23:16 +0200 17/9/04, Henrik Nordstrom wrote: On Fri, 17 Sep 2004, Muthukumar wrote: How can we directly write the access logs on my-sql database there.? There has been patches floating around. Try searching for it. But the preferred method by the Squid developers is to have Squid log to a flat file and then dump this file in real time into mysql. This gives you the best of both worlds as Squid is not encoumbered with the mysql logging, mysql gets the log data immediately and you do not need to save the logs for very long. What is the recommended way of doing this? I have tried doing a tail on the access log and piping it through a perl script which parses and inserts into mysql... is this the best approach? Cheers Jesse -- ::: Jesse Reynolds +61 (0)414 669 790 ::: AIM - jessedreynolds ::: ::: Virtual Artists Pty Ltd, Adelaide ::: http://www.va.com.au :::
Re: [squid-users] 000 status code being logged for redirects
At 8:23 +0100 18/12/03, Henrik Nordstrom wrote: On Thu, 18 Dec 2003, Jesse Reynolds wrote: We have an array of squid servers acting as reverse proxy servers (web accelerators). They also work as URL rewriters, via the redirector interface, eg bouncing http to https in some cases, and mapping certain paths to different backend web servers... Which Squid version? If this is 2.5.STABLE4 please file a bug report. Otherwise first try upgrading. Yes it's 2.5.STABLE4, sorry to forget to mention that! OK, I'll file a bug. :-) For now I'm filtering my log files to replace the 000's with 302's, which works for now, kludgey as it is. Cheers Jesse -- ::: Jesse Reynolds +61 (0)414 669 790 ::: AIM - jessedreynolds ::: ::: Virtual Artists Pty Ltd, Adelaide ::: http://www.va.com.au :::
[squid-users] 000 status code being logged for redirects
Hello We have an array of squid servers acting as reverse proxy servers (web accelerators). They also work as URL rewriters, via the redirector interface, eg bouncing http to https in some cases, and mapping certain paths to different backend web servers... I am getting lots of lines in my access.log files with 000 for the HTTP status code. It looks like it's logging this when it serves a redirect. Here are some example lines (as identified by analog which considers the lines corrupt on account of the 000 status code:) Wed Dec 17 23:29:05 2003161 203.45.116.35 TCP_MISS/000 0 GET http://mydomain.dom:443/act/ho/home.xml - DIRECT/163.241.210.3 - Wed Dec 17 23:29:08 2003 22 203.45.116.35 TCP_MISS/000 0 GET http://mydomain.dom:443/act/ho/home.xml - DIRECT/163.241.210.3 - Wed Dec 17 23:31:22 2003 6 203.12.97.112 TCP_MISS/000 220 GET http://www.mydomain.dom/ - NONE/- - Wed Dec 17 23:31:23 2003 30 202.158.104.38 TCP_MISS/000 245 GET http://mydomain.dom/stu/dummy.html - NONE/- - Wed Dec 17 23:32:29 2003 0 211.108.90.39 TCP_MISS/000 245 GET http://mydomain.dom/stu/dummy.html - NONE/- - the first and second lines are the web server issueing a redirect, the rest of them are the redirector script telling squid to issue a redirect to https. Is this supposed to happen? Any way around it so that I can get our log analysis software to read these lines? Thanks jesse -- ::: Jesse Reynolds +61 (0)414 669 790 ::: AIM - jessedreynolds ::: ::: Virtual Artists Pty Ltd, Adelaide ::: http://www.va.com.au :::
Re: [squid-users] SSL rev proxy, redirector, 302 problems
At 10:43 +0100 4/12/03, Henrik Nordstrom wrote: On Thu, 4 Dec 2003, Jesse Reynolds wrote: Right. So what is the best way of letting the web server know whether the client is using HTTP or HTTPS ? We are currently thinking of adding some lines to the redirector to add ssl=1 to the URLs if the user is coming in via HTTPS, so that the application can know to generate an https:// url rather than http:// - does this sound like the best solution? I would use the Front-End-Https: On header of Squid-3.. you should avoid the use of redirectors unless you absolutely need to as these only worsens the situation. Hi Henrik Why do redirectors worsen the situation? We are on 2.5 so can't use Front-End-Https: unfortuntaly, but that sounds more elegant that what we're doing. We have gone ahead and are tacking a SSL=1 param on the end of the URLs if they were accessed with HTTPS, this is working well for us, if a bit ugly. Regards Jesse -- ::: Jesse Reynolds +61 (0)414 669 790 ::: AIM - jessedreynolds ::: ::: Virtual Artists Pty Ltd, Adelaide ::: http://www.va.com.au :::
Re: [squid-users] SSL rev proxy, redirector, 302 problems
At 3:01 +0100 5/12/03, Henrik Nordstrom wrote: On Fri, 5 Dec 2003, Jesse Reynolds wrote: Why do redirectors worsen the situation? Depends on what the redirector does. Provided it only adds options to the URL and does not modify the URL there is no problem. But if the redirector modifies the host compontent of the URL or the URL-path then there is even less information to the web server/application on what the original URL was in the browser and a bigger risk for mismatches. We change the hostname and port of the URL in the redirector. We have to do this because we have different backend web servers for different paths (eg www.host.com/app1 is redirected to internalhost.host.com:8080/app1 ) Isn't this the purpose of a rediretor when squid is in accelerator mode? We are on 2.5 so can't use Front-End-Https: unfortuntaly, but that sounds more elegant that what we're doing. We have gone ahead and are tacking a SSL=1 param on the end of the URLs if they were accessed with HTTPS, this is working well for us, if a bit ugly. Another option which you might be able to try is to rewrite the URLs into https:// and configure the web server as a parent proxy (but remember to disable server-side persistent connections). This will make Squid send the full URL to the server including protocol, not only the URL-path + query. Ah, interesting. Can you do this in combination with a redirector to separate different path to host relationships? ... Wouldn't the web server try and encrypt the response if it gets a https? Or does it decide whether to encrypt or not based other headers? Jesse -- ::: Jesse Reynolds +61 (0)414 669 790 ::: AIM - jessedreynolds ::: ::: Virtual Artists Pty Ltd, Adelaide ::: http://www.va.com.au :::
[squid-users] SSL rev proxy, redirector, 302 problems
Hi I have successfully set up four Squid reverse proxies (Squid 2.5-STABLE4) listening on port 80 (HTTP) and port 443 (HTTPS). Using a simple perl redirector program the squids are calling a few different backend servers depending on the path, (/app1 goes to appserver1:8080/app1 etc). SSL is only enabled between the browser and the reverse proxy servers. Traffic between the reverse proxies and all the backend web and appservers is non-encrypted HTTP, on non-standard ports. The redirector script will bounce you from HTTP to HTTPS for some URLs, namely the URLs for the web applications. I'm having a problem where the backend appserver sends a 302 (moved temporarily) which is an absolute URL, and begins with http rather than https because it can't see that it was an https URL that it is servicing. This results in the browser receiving a redirect to a non-SSL page, then a redirect to an SSL page again (and over again). How can I get around this? Is it possible to have squid rewrite the URL in the Location: header of the 302 response? (s/http:/https:/) Or is there some other way of altering the HTTP headers that the backend appserver sees such that the appserver will create the correct URL... Or can you send a partial URL in the Location field, eg just /app1/welcome.xml ? By the way this is all on Solaris 8, and the backend appservers are Sun ONE Application Server 7 update 1, so the web apps themselves are servlets. Thankyou Jesse -- ::: Jesse Reynolds +61 (0)414 669 790 ::: AIM - jessedreynolds ::: ::: Virtual Artists Pty Ltd, Adelaide ::: http://www.va.com.au :::
Re: [squid-users] SSL rev proxy, redirector, 302 problems
Thanks Henrik, comments below... At 0:29 +0100 4/12/03, Henrik Nordstrom wrote: On Wed, 3 Dec 2003, Jesse Reynolds wrote: I'm having a problem where the backend appserver sends a 302 (moved temporarily) which is an absolute URL, and begins with http rather than https because it can't see that it was an https URL that it is servicing. This is one aspect of the general problem of the web server not knowing it's real URL. If you can it is absolutely best to address this in the web server to make both the server and applications know the correct URL of the server, allowing them to generate correct URLs whenever a full URL needs to be in a reply of any form. Right. So what is the best way of letting the web server know whether the client is using HTTP or HTTPS ? We are currently thinking of adding some lines to the redirector to add ssl=1 to the URLs if the user is coming in via HTTPS, so that the application can know to generate an https:// url rather than http:// - does this sound like the best solution? The only other thing I can see in the headers that may help is the Referrer which will work for pages that haven't been bookmarked by the user only... so I don't think this is good enough, better to tack ssl=1 on the end of the urls I think. Cheers Jesse -- ::: Jesse Reynolds +61 (0)414 669 790 ::: AIM - jessedreynolds ::: ::: Virtual Artists Pty Ltd, Adelaide ::: http://www.va.com.au :::
[squid-users] Slow to Stop - 2.5-STABLE on Solaris 8
Hi Most of the time squid is taking about 30 seconds to shut down for me. I have built 2.5-STABLE on Solaris 8. The same thing happens on all 7 boxes here. Occasionally it stops very quickly, but usally it just takes ages to stop. Any way to make it stop faster? Any way to find out what it's doing when it's stopping? We are using squid for an accelerator only, it handles SSL connections, and it starts up a simple redirector I've written in perl. Thankyou Jesse
Re: [squid-users] Slow to Stop - 2.5-STABLE on Solaris 8
At 11:58 +0100 26/11/03, Henrik Nordstrom wrote: On Wed, 26 Nov 2003, Jesse Reynolds wrote: Most of the time squid is taking about 30 seconds to shut down for me. I have built 2.5-STABLE on Solaris 8. The same thing happens on all 7 boxes here. Occasionally it stops very quickly, but usally it just takes ages to stop. What is said in cache.log? Maybe your squid is simply configured to wait up to 30 seconds for clients to finish their requests (this is the default). Ah right, thankyou Henrik! It's odd that it seems to do this most of the time, even when noone is accessing the cache. Perhaps it is also considering the file descriptors for the child redirector processes? From the squid.conf: # TAG: shutdown_lifetime time-units # When SIGTERM or SIGHUP is received, the cache is put into # shutdown pending mode until all active sockets are closed. # This value is the lifetime to set for all open descriptors # during shutdown mode. Any active clients after this many # seconds will receive a 'timeout' message. # #Default: # shutdown_lifetime 30 seconds Perhaps I'll set this to 5 seconds. ... There are four squids in accelerator mode behind a load balancer. We want to be able to take one out of service with minimal impact on user experience. Currently you randomly get a squid is shutting down error if you do this. Closer interaction with the Cisco CSM load balancer could also be a better way to do this! Cheers Jesse -- ::: Jesse Reynolds +61 (0)414 669 790 ::: AIM - jessedreynolds ::: ::: Virtual Artists Pty Ltd, Adelaide ::: http://www.va.com.au :::
Re: [squid-users] Compiling Squid against Sun's OpenSSL 0.96b forHardware Crypto Accelerator Support
At 10:57 +0200 5/8/2003, Henrik Nordstrom wrote: On Tuesday 05 August 2003 02.56, Jesse Reynolds wrote: But I can't figure out how to specify the rest of those linker flags. Any suggestions would be greatly appreciated! I assume I need to edit the Makefile but I'm lost. Search for -lssl in src/Makefile after running configure. Then modify the line to your liking. Thanks Henrik So, I've tried that but it doesn't seem to make any difference. I am a bit of a C dummy so I've probably done something daft. I have modified the SSLLIB line of src/Makefile to read: SSLLIB = -L/opt/SUNWconn/crypto/lib -lssl -lcrypto -I /opt/SUNWconn/crypto/include -R/opt/SUNWconn/crypto/lib -lcryptography -lnvpair does that make sense? ... Anyway, running 'make clean' and 'make all' yields the same Undefined Symbol errors as before. Perhaps I should try compiling against the latest version of OpenSSL 0.97 which aparently has support for hardware accelerators (instead of this Sun modified version of 0.96). Jesse ... gcc -g -O2 -Wall -g -o squid access_log.o acl.o asn.o authenticate.o cache_cf.o CacheDigest.o cache_manager.o carp.o cbdata.o client_db.o client_side.o comm.o comm_select.o debug.o disk.o dns_internal.o errorpage.o ETag.o event.o external_acl.o fd.o filemap.o forward.o fqdncache.o ftp.o gopher.o helper.o http.o HttpStatusLine.o HttpHdrCc.o HttpHdrRange.o HttpHdrContRange.o HttpHeader.o HttpHeaderTools.o HttpBody.o HttpMsg.o HttpReply.o HttpRequest.o icmp.o icp_v2.o icp_v3.o ident.o internal.o ipc.o ipcache.o logfile.o main.o mem.o MemPool.o MemBuf.o mime.o multicast.o neighbors.o net_db.o Packer.o pconn.o peer_digest.o peer_select.o redirect.o referer.o refresh.o send-announce.o ssl.o ssl_support.o stat.o StatHist.o String.o stmem.o store.o store_io.o store_client.o store_digest.o store_dir.o store_key_md5.o store_log.o store_rebuild.o store_swapin.o store_swapmeta.o store_swapout.o tools.o unlinkd.o url.o urn.o useragent.o wais.o wccp.o whois.o repl_modules.o auth_modules.o store_modules.o globals.o string_arrays.o -L../lib repl/liblru.a fs/libufs.a auth/libbasic.a -lcrypt -L/opt/SUNWconn/crypto/lib -lssl -lcrypto -lmiscutil -lm -lresolv -lsocket -lnsl Undefined first referenced symbol in file nvlist_alloc /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_key_fini /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) nvlist_add_byte_array /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_encrypt /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_sign /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_seed /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_key_init /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) nvlist_add_uint32 /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_fini /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_random /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_verify /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_decrypt /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_init /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) nvlist_free /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) ld: fatal: Symbol referencing errors. No output written to squid collect2: ld returned 1 exit status make[3]: *** [squid] Error 1 make[3]: Leaving directory `/usr/local/sources/inst/squid-2.5.STABLE3/src' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/usr/local/sources/inst/squid-2.5.STABLE3/src' make[1]: *** [all] Error 2 make[1]: Leaving directory `/usr/local/sources/inst/squid-2.5.STABLE3/src' make: *** [all-recursive] Error 1 -- Jesse Reynolds - [EMAIL PROTECTED] Division of Information Services, University of New South Wales Phone: +61 (0)2 9385 2893 AIM: jessedreynolds
Re: [squid-users] Compiling Squid against Sun's OpenSSL 0.96b forHardware Crypto Accelerator Support
Oh I forgot to mention that the point the compilation breaks is as follows... I believe this is after all the components have compiled and and we're trying to link it all together. jesse gcc -g -O2 -Wall -g -o squid access_log.o acl.o asn.o authenticate.o cache_cf.o CacheDigest.o cache_manager.o carp.o cbdata.o client_db.o client_side.o comm.o comm_select.o debug.o disk.o dns_internal.o errorpage.o ETag.o event.o external_acl.o fd.o filemap.o forward.o fqdncache.o ftp.o gopher.o helper.o http.o HttpStatusLine.o HttpHdrCc.o HttpHdrRange.o HttpHdrContRange.o HttpHeader.o HttpHeaderTools.o HttpBody.o HttpMsg.o HttpReply.o HttpRequest.o icmp.o icp_v2.o icp_v3.o ident.o internal.o ipc.o ipcache.o logfile.o main.o mem.o MemPool.o MemBuf.o mime.o multicast.o neighbors.o net_db.o Packer.o pconn.o peer_digest.o peer_select.o redirect.o referer.o refresh.o send-announce.o ssl.o ssl_support.o stat.o StatHist.o String.o stmem.o store.o store_io.o store_client.o store_digest.o store_dir.o store_key_md5.o store_log.o store_rebuild.o store_swapin.o store_swapmeta.o store_swapout.o tools.o unlinkd.o url.o urn.o useragent.o wais.o wccp.o whois.o repl_modules.o auth_modules.o store_modules.o globals.o string_arrays.o -L../lib repl/liblru.a fs/libufs.a auth/libbasic.a -lcrypt -L/opt/SUNWconn/crypto/lib -lssl -lcrypto -lmiscutil -lm -lresolv -lsocket -lnsl Undefined first referenced symbol in file nvlist_alloc /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_key_fini /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) nvlist_add_byte_array /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_encrypt /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_sign /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_seed /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_key_init /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) nvlist_add_uint32 /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_fini /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_random /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_verify /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_decrypt /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) crypto_init /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) nvlist_free /opt/SUNWconn/crypto/lib/libcrypto.a(hw_kcl.o) ld: fatal: Symbol referencing errors. No output written to squid collect2: ld returned 1 exit status make[3]: *** [squid] Error 1 make[3]: Leaving directory `/usr/local/sources/inst/squid-2.5.STABLE3/src' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/usr/local/sources/inst/squid-2.5.STABLE3/src' make[1]: *** [all] Error 2 make[1]: Leaving directory `/usr/local/sources/inst/squid-2.5.STABLE3/src' make: *** [all-recursive] Error 1 [EMAIL PROTECTED] /usr/local/sources/inst/squid-2.5.STABLE3 # At 10:56 +1000 5/8/2003, Jesse Reynolds wrote: Hi folx I'm trying to get Squid compiled so that it will make use of Sun's Crypto 500 daughter board. (The Crypto 500 appears to have a similar Broadcomm chip on it to their Crypto 1000 and Crypto 4000 cards). The Sun card ships with a CD containing, among other things, a modified version of OpenSSL 0.96b and a manual which states that you can compile against it if you do the following: a) Your application must be configured to include OpenSSL headers from /opt/SUNWconn/crypto/include, such as with the compiler flag: -I /opt/SUNWconn/crypto/include b) Additionally, the linker must be directed to include references to the appropriate libraries. Most OpenSSL-compatible applications will reference either or both of the libcrypto.a and libssl.a libraries. The Sun cryptographic libraries must be included as well. The following linker flags will accomplish this: -L/opt/SUNWconn/crypto/lib -R/opt/SUNWconn/crypto/lib \ -lcrypto -lssl -lcryptography -lnvpair So, I've given Squid 2.5-STABLE3 the following configure flags: # ./configure --prefix=/opt/squid --enable-useragent-log --enable-referer-log --enable-ssl --with-openssl=/opt/SUNWconn/crypto But I can't figure out how to specify the rest of those linker flags. Any suggestions would be greatly appreciated! I assume I need to edit the Makefile but I'm lost. The following versions are relevant: Solaris 8 GCC - SMCgcc version 3.3 GNU make - installed SMCmake version 3.80 GNU binutils - installed SMCbinut version 2.11.2 squid-2.5.STABLE3 Hardware: Sun V210 with Sun Crypto 500 daugher board Thanks very much Jesse -- Jesse Reynolds - [EMAIL PROTECTED] Division of Information Services, University of New South Wales Phone: +61 (0)2 9385 2893 -- Jesse Reynolds - [EMAIL PROTECTED] Division of Information Services, University of New South Wales Phone: +61 (0)2 9385 2893 AIM: jessedreynolds
