Squid3.0, ICAP server in python and respmod
Hey all, After some tests, I made my mind for squid3.0 and python based ICAP server. All works fine with reqmod precache but I cant' manage to get a response modification. On the side of the ICAP server, log messages display normally but on the squid side no error is raised and the browser displays nothing (the response body seems empty). Note : I got a parent cache. First of all my squidbox (and my ICAP server) is a debian etch with a version 2.6.8-2-386 linux core. Here is the squid.conf part about ICAP : icap_enable on icap_service s1 respmod_precache 0 icap://icap server IP:1344/respmod icap_class c1 s1 acl images urlpath_regex \.jpg$ acl images urlpath_regex \.png$ acl images urlpath_regex \.gif$ acl images urlpath_regex \.ico$ icap_access c1 deny images acl css urlpath_regex \.css$ icap_access c1 deny css acl HTTP proto HTTP acl GET method GET icap_access c1 allow HTTP GET access.log : 1175608936.978117 squidbox IP TCP_MISS/200 593 GET http://www.lemonde.fr/rss/sequence/0,2-3208,1-0,0.xml - DEFAULT_PARENT/parent cache address text/html 1175608938.340 43 squidbox IP TCP_MISS/200 522 GET http://linuxfr.org/backend/news/rss20.rss - DEFAULT_PARENT/parent cache address text/html cache.log : 2007/04/03 16:01:47| Starting Squid Cache version 3.0.PRE5-20070330 for i686-pc-linux-gnu... 2007/04/03 16:01:47| Process ID 14008 2007/04/03 16:01:47| With 1024 file descriptors available 2007/04/03 16:01:47| Performing DNS Tests... 2007/04/03 16:01:47| Successful DNS name lookup tests... 2007/04/03 16:01:47| DNS Socket created at 0.0.0.0, port 38546, FD 4 2007/04/03 16:01:47| Adding domain univ-st-etienne.fr from /etc/resolv.conf 2007/04/03 16:01:47| Adding nameserver dns server 1 address from /etc/resolv.conf 2007/04/03 16:01:47| Adding nameserver dns server 2 address from /etc/resolv.conf 2007/04/03 16:01:48| Unlinkd pipe opened on FD 9 2007/04/03 16:01:48| Swap maxSize 102400 KB, estimated 7876 objects 2007/04/03 16:01:48| Target number of buckets: 393 2007/04/03 16:01:48| Using 8192 Store buckets 2007/04/03 16:01:48| Max Mem size: 8192 KB 2007/04/03 16:01:48| Max Swap size: 102400 KB 2007/04/03 16:01:48| Rebuilding storage in /home/adminlardon/squid/var/cache (CLEAN) 2007/04/03 16:01:48| Using Least Load store dir selection 2007/04/03 16:01:48| Set Current Directory to /home/adminlardon/squid/var/cache 2007/04/03 16:01:48| Loaded Icons. 2007/04/03 16:01:48| Accepting HTTP connections at 0.0.0.0, port 3128, FD 11. 2007/04/03 16:01:48| Accepting ICP messages at 0.0.0.0, port 3130, FD 12. 2007/04/03 16:01:48| WCCP Disabled. 2007/04/03 16:01:48| Ready to serve requests. 2007/04/03 16:01:48| Configuring Parent parent cache address/3128/3130 2007/04/03 16:01:48| Done reading /home/adminlardon/squid/var/cache swaplog (184 entries) 2007/04/03 16:01:48| Finished rebuilding storage from disk. 2007/04/03 16:01:48| 184 Entries scanned 2007/04/03 16:01:48| 0 Invalid entries. 2007/04/03 16:01:48| 0 With invalid flags. 2007/04/03 16:01:48| 184 Objects loaded. 2007/04/03 16:01:48| 0 Objects expired. 2007/04/03 16:01:48| 0 Objects cancelled. 2007/04/03 16:01:48| 0 Duplicate URLs purged. 2007/04/03 16:01:48| 0 Swapfile clashes avoided. 2007/04/03 16:01:48| Took 0.1 seconds (3231.2 objects/sec). 2007/04/03 16:01:48| Beginning Validation Procedure 2007/04/03 16:01:48| Completed Validation Procedure 2007/04/03 16:01:48| Validated 393 Entries 2007/04/03 16:01:48| store_swap_size = 1052 2007/04/03 16:01:49| storeLateRelease: released 0 objects 2007/04/03 16:01:59| ICAP server address's clock is skewed by 3600 seconds! 2007/04/03 16:02:34| Preparing for shutdown after 7 requests 2007/04/03 16:02:34| Waiting 0 seconds for active connections to finish 2007/04/03 16:02:34| FD 11 Closing HTTP connection 2007/04/03 16:02:35| Shutting down... 2007/04/03 16:02:35| FD 12 Closing ICP connection 2007/04/03 16:02:35| Closing unlinkd pipe on FD 9 2007/04/03 16:02:35| storeDirWriteCleanLogs: Starting... 2007/04/03 16:02:35| Finished. Wrote 184 entries. 2007/04/03 16:02:35| Took 0.0 seconds (260992.9 entries/sec). CPU Usage: 1.099 seconds = 0.141 user + 0.958 sys Maximum Resident Size: 0 KB Page faults with physical i/o: 0 Memory usage for squid via mallinfo(): total space in arena:3556 KB Ordinary blocks: 3407 KB 9 blks Small blocks: 0 KB 6 blks Holding blocks: 3464 KB 16 blks Free Small blocks: 0 KB Free Ordinary blocks: 148 KB Total in use:6871 KB 193% Total free: 148 KB 4% 2007/04/03 16:02:35| Squid Cache (Version 3.0.PRE5-20070330): Exiting normally. Final note : all works perfectly without icap transformation. All help will be greatly appreciated. Jérémy Lardon
Re: direct responce from ICAP server in REQMOD
On Wed, 2007-03-28 at 21:03 +0300, Tsantilas Christos wrote: When an ICAP server responds with an http response in request modification, the web client waits more data from the proxy. Iam using the squid3-icap branch. An example of such ICAP response is: ICAP/1.0 200 OK Connection: keep-alive ISTag: CI0001-X Encapsulated: res-hdr=0, res-body=108 HTTP/1.0 403 Forbidden Content-Type: text/html Content-Language: en Connection: close 17 H1Permition deny!H1 0 The proxy server sends the correct data to the client, but does not close the connection after send all data. Adding the line storeEntry()-complete(); in ClientHttpRequest::endRequestSatisfaction method solves the problem but I think that the solution is not so simple, we need to know how to setup the StoreEntry... Nevertheless, I will commit your fix. Using the right sequences of entry-related calls has always eluded me, but calling complete() looks reasonable based on the existing code than handles real responses. If somebody knows whether any other entry-related things should be done to correctly finish an ICAP-made HTTP response, please chime in. Thank you, Alex.
Re: Squid3.0, ICAP server in python and respmod
On Tue, 2007-04-03 at 16:25 +0200, Jeremy Lardon wrote: Hey all, After some tests, I made my mind for squid3.0 and python based ICAP server. All works fine with reqmod precache but I cant' manage to get a response modification. There were many improvements in ICAP support on the squid3-icap CVS branch. Please consider running with that code or waiting for the next PRE release of Squid3 (which will include those ICAP improvements). If you find bugs in squid3-icap ICAP code, I will do my best to fix them. Thank you, Alex.
squid3.0.PRE6 and bugzilla
Hi, Which is the goal for squid3.0.PRE6? Which bugs waiting a fix before released? Can we assume that bug 1637 has a fix? If I am not wrong most of the bugs causing crashes to squid fixed... Regards, Christos
Re: squid3.0.PRE6 and bugzilla
On Tue, 2007-04-03 at 21:32 +0300, Tsantilas Christos wrote: Which is the goal for squid3.0.PRE6? Which bugs waiting a fix before released? Release recent improvements in a form convenient for testing and experimental deployment. I hate telling non-developers to use CVS. Can we assume that bug 1637 has a fix? http://www.squid-cache.org/bugs/show_bug.cgi?id=1637 I cannot tell. If I am not wrong most of the bugs causing crashes to squid fixed... I do not know of any reproducible crashes in Squid3 core or ICAP code, but there are a few known corner cases like squid -k shutdown. Alex.
Re: squid translation
fre 2007-03-30 klockan 21:19 +0300 skrev Петро Нек: Hello my name is Petro Nek I'm from Ukraine and I'm interesting that squid could speak with users in ukrainian so I translate squid error messages in ukrainian using 2 typical codepages cp-1251, koi8-u and also unicode utf-8. So I send to You these files in archive and I'll be happy if You include it in next squid release. Many thanks. Your translations have been imported, and is scheduled to appear in the next Squid-2.6.STABLE release. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: DEBUG Section for IPAddress
Alex Rousskov wrote: On Mon, 2007-03-26 at 23:59 +1200, Amos Jeffries wrote: I have so far not placed any permanent debug statements in the IPAddress object. It's come about time I should be making some more permanent ones and so need a section number. Is there some method I don't know yet for assigning a unique level or do I just pick an unused number and write it in? If you have not received better instructions yet, just use the next unused number for the debug section and add it to your file(s) prefix just like most other source files do. The stale doc/debug-sections.txt is (or should be) generated based on the source code. Do not update that file unless you are willing to freshen it up first. Cheers, Alex. Thank you Alex, No I have not received any further instructions and assumed that was the best action to take. I picked level 36, but that turns out to be cache dir processing and I expect its in use own another branch. If by 'freshen up' you mean rebuild the file from source using a grep of the source files headers for DEBUG section and a manual massage. I did that earlier before seeing the doc/ file. The update patch resulting is attached. FYI to all: IP Handling will join IP Cache in section 14, unless there are any objections by easter. Amos --- DEBUG_levels.txt 2007-04-04 12:56:03.0 +1200 +++ doc/debug-sections.txt 2007-03-27 20:12:49.0 +1200 @@ -3,14 +3,11 @@ section 0 Client Database section 0 Debug Routines section 0 DNS Resolver -section 0 Hash Tables -section 0 UFS Store Dump section 0 WWW Client section 1 Startup and Main Loop section 2 Unlink Daemon section 3 Configuration File Parsing section 4 Error Generation -section 5 Comms section 5 Socket Functions section 6 Disk I/O Routines section 7 Multicast @@ -21,35 +18,21 @@ section 12Internet Cache Protocol section 13High Level Memory Pool Management section 14IP Cache -section 14IP Storage and Handling section 15Neighbor Routines section 16Cache Manager Objects section 17Request Forwarding section 18Cache Manager Statistics section 19Store Memory Primitives section 20Storage Manager -section 20Storage Manager Heap-based replacement -section 20Storage Manager Logging Functions -section 20Storage Manager MD5 Cache Keys -section 20Storage Manager Swapfile Metadata -section 20Storage Manager Swapfile Unpacker -section 20Storage Manager Swapin Functions -section 20Storage Manager Swapout Functions -section 20Store Rebuild Routines -section 21Integer functions section 21Misc Functions -section 21Time Functions section 22Refresh Calculation section 23URL Parsing -section 23URL Scheme parsing - +section 24WAIS Relay section 25MIME Parsing section 26Secure Sockets Layer Proxy section 27Cache Announcer section 28Access Control section 29Authenticator -section 29Negotiate Authenticator -section 29NTLM Authenticator section 30Ident (RFC 931) section 31Hypertext Caching Protocol section 32Asynchronous Disk I/O @@ -61,26 +44,20 @@ section 38Network Measurement Database section 39Cache Array Routing Protocol section 40User-Agent and Referer logging -section 40User-Agent logging section 41Event Processing section 42ICMP Pinger program section 43AIOPS -section 43Windows AIOPS section 44Peer Selection Algorithm section 45Callback Data Registry section 46Access Log -section 47Store COSS Directory Routines section 47Store Directory Routines section 48Persistent Connections -section 49SNMP Interface -section 49SNMP support +section 49SNMP section 50Log file handling section 51Filedescriptor Functions section 52URN Parsing section 53AS Number handling -section 53 Radix tree data structure implementation section 54Interprocess Communication -section 54Windows Interprocess Communication section 55HTTP Header section 56HTTP Message Body section 57HTTP Status-line @@ -89,7 +66,7 @@ section 60Packer: A uniform interface to store-like modules section 61Redirector section 62Generic Histogram -section 63Low Level Memory Pool Management +section 63- section 64HTTP Range Header section 65HTTP Cache Control Header section 66HTTP Header Tools @@ -104,38 +81,18 @@ section 75WHOIS protocol section 76Internal Squid Object handling section 77Delay Pools -section 78DNS lookups; interacts with lib/rfc1035.c -section 79Disk IO Routines -section 79Squid-side DISKD I/O functions. -section 79Squid-side Disk I/O functions. -section 79Storage Manager COSS Interface -section 79Storage Manager UFS Interface -section 80WCCP Support -section 81aio_xxx() POSIX emulation on Windows -section 81 CPU Profiling
Re: DEBUG Section for IPAddress
Gah! I diffed the files backwards. Here is the actual patch for doc/debug-sections.txt Amos --- doc/debug-sections.txt 2007-03-27 20:12:49.0 +1200 +++ DEBUG_levels.txt 2007-04-04 12:56:03.0 +1200 @@ -3,11 +3,14 @@ section 0 Client Database section 0 Debug Routines section 0 DNS Resolver +section 0 Hash Tables +section 0 UFS Store Dump section 0 WWW Client section 1 Startup and Main Loop section 2 Unlink Daemon section 3 Configuration File Parsing section 4 Error Generation +section 5 Comms section 5 Socket Functions section 6 Disk I/O Routines section 7 Multicast @@ -18,21 +21,35 @@ section 12Internet Cache Protocol section 13High Level Memory Pool Management section 14IP Cache +section 14IP Storage and Handling section 15Neighbor Routines section 16Cache Manager Objects section 17Request Forwarding section 18Cache Manager Statistics section 19Store Memory Primitives section 20Storage Manager +section 20Storage Manager Heap-based replacement +section 20Storage Manager Logging Functions +section 20Storage Manager MD5 Cache Keys +section 20Storage Manager Swapfile Metadata +section 20Storage Manager Swapfile Unpacker +section 20Storage Manager Swapin Functions +section 20Storage Manager Swapout Functions +section 20Store Rebuild Routines +section 21Integer functions section 21Misc Functions +section 21Time Functions section 22Refresh Calculation section 23URL Parsing -section 24WAIS Relay +section 23URL Scheme parsing + section 25MIME Parsing section 26Secure Sockets Layer Proxy section 27Cache Announcer section 28Access Control section 29Authenticator +section 29Negotiate Authenticator +section 29NTLM Authenticator section 30Ident (RFC 931) section 31Hypertext Caching Protocol section 32Asynchronous Disk I/O @@ -44,20 +61,26 @@ section 38Network Measurement Database section 39Cache Array Routing Protocol section 40User-Agent and Referer logging +section 40User-Agent logging section 41Event Processing section 42ICMP Pinger program section 43AIOPS +section 43Windows AIOPS section 44Peer Selection Algorithm section 45Callback Data Registry section 46Access Log +section 47Store COSS Directory Routines section 47Store Directory Routines section 48Persistent Connections -section 49SNMP +section 49SNMP Interface +section 49SNMP support section 50Log file handling section 51Filedescriptor Functions section 52URN Parsing section 53AS Number handling +section 53 Radix tree data structure implementation section 54Interprocess Communication +section 54Windows Interprocess Communication section 55HTTP Header section 56HTTP Message Body section 57HTTP Status-line @@ -66,7 +89,7 @@ section 60Packer: A uniform interface to store-like modules section 61Redirector section 62Generic Histogram -section 63- +section 63Low Level Memory Pool Management section 64HTTP Range Header section 65HTTP Cache Control Header section 66HTTP Header Tools @@ -81,18 +104,38 @@ section 75WHOIS protocol section 76Internal Squid Object handling section 77Delay Pools -section 78DNS lookups -section 79Lowlevel store I/O -section 80WCCP -section 81Cache Store +section 78DNS lookups; interacts with lib/rfc1035.c +section 79Disk IO Routines +section 79Squid-side DISKD I/O functions. +section 79Squid-side Disk I/O functions. +section 79Storage Manager COSS Interface +section 79Storage Manager UFS Interface +section 80WCCP Support +section 81aio_xxx() POSIX emulation on Windows +section 81 CPU Profiling Routines section 82External ACL section 83SSL accelerator support section 84Helper process maintenance -section 85Client side request management - after parsing, before caching -section 86ESI processing -section 87client side stream management -section 88Client side reply management - from store to stream -section 89NAT / IP Interception -section 90Store Client -section 91Http Surrogate-Control Header -section 92Store File System +section 85Client-side Request Routines +section 86ESI Expressions +section 86ESI Processing +section 87Client-side Stream routines. +section 88Client-side Reply Routines +section 89NAT / IP Interception +section 90HTTP Cache Control Header +section 90Storage Manager Client-Side Interface + +section 92Storage File System +section 93ICAP (RFC 3507) Client + + +section ? Common Removal policy +section --External DISKD process implementation. +section ?? FDE +section ? HEAP based removal policies +section ??Linked list functions +section ? LRU Removal
