compliant proxies found, at last (fwd)
-- Here is something to light a fire under Robert Collins :-). I wish Squid3 would make it there first, but it is still good news. Alex. -- Forwarded message -- Date: Tue, 16 Sep 2003 12:33:52 -0600 (MDT) From: Alex Rousskov [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: compliant proxies found, at last I would like to share a piece of good news with folks who care about HTTP: Two HTTP proxies, an open-source RabbIT[1] and a commercial implementation from Datapower[2], have recently managed to pass all applicable RFC 2616 MUSTs that we could test. These are the first compliant HTTP intermediaries we know of. IIRC, it took both teams almost a year to fix all violations. If you look at RabbIT and Datapower product descriptions, you will see that these are real, full-featured content-aware proxies, not just some L4 tunnels that do not care about HTTP. From what I heard, a couple of core design alternations were required to pass some of the tough test cases. Now we have a living proof that it is possible to be feature-reach and HTTP/1.1 compliant! Disclaimer: I do realize that passing all Co-Advisor test cases does not imply or guarantee formal compliance, even on conditional level. I am sure that there are still bugs to be fixed. No testing can guarantee a bug-free implementation. We can never prove compliance by testing. When I say compliant, I mean passes all applicable MUST-level test cases. Call it a marketing trick/spin if you wish. I applaud Patrick McManus (Datapower), Robert Olofsson (RabbIT), and their development/QA teams on having the dedication and skills required to support all relevant HTTP/1.1 MUSTs. I believe that their success is significant, regardless of the marketing spin. I hope you do not perceive this message as an ad for our test suite. I tried hard to simply share the good news and praise the good guys :-). Thank you, Alex. [1] http://www.khelekore.org/rabbit/ [2] http://www.datapower.com/
Hot memory cache
Hello, AFAIK, the memory cache in Squid 2.5 keeps objects fetched from the network and in-transit objects. The memory cache does not keep objects fetched from disk. In-transit objects purge cached memory objects if there is not enough space and may even exceed the configured memory cache size (the cache_mem directive). Only objects under certain configurable size can be kept in the memory cache. Please correct me if any of the above is wrong. Is the above still true in Squid 3? Has any functionality related to the memory cache changed? We have a customer who may benefit from a true hot memory cache: a dedicated RAM space for storing recently requested objects (whether fetched from disk or from the network) and independent from in-transit objects. I am trying to understand the amount of work needed to implement a hot cache in Squid 2.5 and Squid 3.0. Any comment/advise is welcomed. Thank you, Alex.
Re: A policy question
On Sun, 2005/03/20 (MST), [EMAIL PROTECTED] wrote: drweb-icap server responce with Conection:keep-alive but then imediatelly closes the connection OK, drweb-icap breaks the ICAP protocol here FWIW, the above drweb behavior is compliant with ICAP. Conection: keep-alive does not (and cannot) guarantee that the connection will remain open after the ICAP response is sent. Not using persistent connections with a given ICAP server would also be compliant with ICAP. Persistent connections are optional. Have you tried asking drweb folks whether there is something Squid can do to prevent them from closing the connection? Alex.
Re: A policy question
On Mon, 2005/04/04 (MDT), [EMAIL PROTECTED] wrote: Have you tried asking drweb folks whether there is something Squid can do to prevent them from closing the connection? No I did not. But the DRweb team had post a patch to squid mailing list (http://www.mail-archive.com/squid-dev@squid-cache.org/msg02325.html) for squid-icap in which had modify the code to always closing the connection. So, I believe that drweb-icapd requires closing the connection. That seems likely, however the comments about the patch seem to imply that drweb ICAP server does send Connection: close. Here is a quote: if ICAP daemon does not support persistent connection and answers with 'Connection: Close' squid can stop receiving the document from a network and send the cut (without the ending) document to the client. That comment, combined with your observations, confuse me. Somethind does not add up here. I did not look at the patch code. Alex.
Re: Linux filesystem speed comparison
On Mon, 2005/04/11 (MDT), [EMAIL PROTECTED] wrote: We currently get arount 70 reqs/sec using 25% CPU (5 minute average for both values) on this hardware. I'm confident that I'll get a pretty high number of requests/second through these proxies becase of the epoll patch. Polygraph using PolyMix-4 workload does 500 req/sec per client-server pair. With a decent pair of PCs and Gbit links, you can 1000 req/sec or more per pair. Thus, you should be able to stress your Squid proxies with just a couple of PCs. Alex.
weak pointers, cbdata, refcounting
Hi there,
I need to implement what some call a weak pointer. A weak pointer
becomes null or otherwise invalid when the corresponding object goes
away or is invalidated. For an example, please see
http://www.boost.org/libs/smart_ptr/smart_ptr.htm
I cannot use existing RefCount.h for weak pointing because the interface
does not support invalidation of a pointer -- the RefCount pointer
stays valid until nobody uses it.
I seem to be unable to use cbdata because that interface does not seem
to work well with non-trivial classes(*). Please correct me if I am
wrong.
I can add an invalidate()/isInvalid() pair to RefCount so that folks
can invalidate a RefCount pointer. This is very simple, but does not
catch cases where folks forget to check for validity before
de-referencing the pointer. It also makes RefCount do things besides
refcounting.
Alternatively, I can add a new WeakPtr class to accompany RefCount,
similar to weak_ptr interface at the above Boost URL. This is a lot more
work (than adding an invalid bit to the existing RefCount). Is that
the best option? Is there a better way?
Thank you,
Alex.
(*) For example, multiple inheritance does not seem to be supported by
cbdata due to (void*) casting:
class B {...};
class A {...};
class C {...};
class ABC: public A, public B, public C { ...};
ABC *abc = new ABC;
cbdataReference(abc);
...
B *b = abc;
assert(abc != b);
...
if (cbdataReferenceValid(b)) // fails the cbdata cookie check
...
Destructors are probably not called either, for similar reasons.
However, I see classes that are declared with CBDATA_CLASS and that have
destructors. Are all those destructors empty? Am I missing something?
Re: weak pointers, cbdata, refcounting
On Tue, 23 Aug 2005, Robert Collins wrote: On Mon, 2005-08-22 at 16:38 -0600, Alex Rousskov wrote: Hi there, I need to implement what some call a weak pointer. A weak pointer becomes null or otherwise invalid when the corresponding object goes away or is invalidated. For an example, please see http://www.boost.org/libs/smart_ptr/smart_ptr.htm I cannot use existing RefCount.h for weak pointing because the interface does not support invalidation of a pointer -- the RefCount pointer stays valid until nobody uses it. CBData works fine with standard classes. Hi Robert, That's good to know! Apparently, I do not use cbdata correctly because it does not seem to work for me. Could you please point me to the updated (with respect to classes) cbdata docs or notes? Perhaps I just need to add more #define-driven glue to base (or all?) classes... If there are no docs or notes, what should I add to A, B, C, and ABC classes in my original question so that: (a) cbdata operations on A, B, C, or ABC pointer would be noticeable when working with pointers of the other three types, and (b) all four destructors are called when the last reference is gone via a cbdata operations on A, B, C, or ABC pointer? I'm curious though why you need a weak ref? Most weak ref uses I've seen are roughly a failure to use observers. I am not sure I follow. Is not a weak pointer an example of an observer? I agree that I can implement other kinds of observers that would be more explicitly notified when the original object is invalidated, but having a simple and generic weak pointer seems like a more straightforward and a standard solution than a custom subscribe/publish interface. I am not sure I can explain why I want a weak pointer in a short email, but I will try. I have a MsgPipe class that asynchronously (via Squid's events) forwards messages from the Source object to the Sink object. MsgPipe needs to ignore the pending message when the Sink is gone/invalidated. MsgPipe does not know the final type of the Sink, and several classes inherit from the Sink abstract interface (i.e., many types can be used as sinks). Ideally, a sink would not have to maintain a list of pipes that are using it (so that it can inform them that it is gone). Thanks, Alex. P.S. MsgPipe.h and friends are in the squid3-icap branch.
Re: weak pointers, cbdata, refcounting
On Mon, 2005-08-22 at 18:50 -0400, Nick Lewycky wrote: The nice thing about using Boost is that if the library is added to C++ in the (admittedly distant) future, our API would not need to change, On the other hand, if we do not use Boost, our API would not need to change either :-). I did not mean to start a let's migrate to Boost! discussion. While it could be worthwhile to revisit that topic (perhaps in a different thread), my needs are different and immediate: I am looking for either a way to make current cbdata work for non-trivial classes OR opinions on adding class-friendly weak pointer functionality to Squid. Thank you, Alex.
Re: [Subscription] Contribution
On Sat, 3 Sep 2005, Nicolas Harounen wrote: In fact, my testing plans are very poor for the moment, it's only a perl script, which works with the icap server from the i-cap forum (I only found this.), and with the 2.5 version of squid patched with the icap client. It makes some requests and just make a diff with a result. I've just finished the first version of the script, I have to write tests now. I will modify it in order to use the squid3 branch now, and next, write tests. I post on the mailing the work, when the modification for the squid3 branch is done. (I think in 2 or 3 days max). Thanks for the info! Your tests would be a good start. Hopefully, you can make them independent from Squid version and ICAP server software. In my experience, it helps to have some simple logic that replaces certain header values with placeholders to avoid version and similar dependencies (Server, Date, ISTag are likely candidates for such a replacement). Dealing with Encapsulated value differences is much harder, unfortunately: Any HTTP header modification would cause that value to change. If you only compare HTTP responses, then Encapsulated values do not matter, of course. Good luck, Alex.
pconns and WHEN_SQUID_IS_NOT_HTTP1_1
Hello, I have a couple of questions about WHEN_SQUID_IS_NOT_HTTP1_1 #define that was added to HttpMsg.cc while fixing now-closed bug #890: http://www.squid-cache.org/bugs/show_bug.cgi?id=890 http://www.squid-cache.org/cgi-bin/cvsweb.cgi/squid3/src/HttpMsg.cc.diff?r1=1.13r2=1.14f=h Should it be named WHEN_SQUID_IS_HTTP1_1 instead? Can the intended semantics of httpMsgIsPersistent() be summarized as The HTTP agent sending the supplied message wants to keep the connection persistent (AND we trust that agent can handle pconns). That is, Keep the connection with the agent persistent if possible Or is it something else? I am afraid it is something more convoluted because I see client_side.cc forcing the HTTP version to 1.0 before calling that method(*). The corresponding(?) bugzilla comment says ... uncovered another HTTP compliance bug in Squid where HTTP/1.1 messages was considered (partly) as implicitly keep-alive even if Squid is still HTTP/1.0. What is wrong with considering HTTP/1.1 messages keep-alive by default? Are we trying to catch cases where a (broken?) HTTP/1.1 agent expects Squid to default to non-persistent connections because Squid is HTTP/1.0? Are we always sending explicit Connection: close/keep-alive headers? Would that take care of any ambiguities when talking to HTTP clients, regardless of the client HTTP version? Same for servers? Thank you, Alex. (*) Although the version parameter of httpMsgIsPersistent() is currently unused.
Re: Squid-ICAP
On Thu, 2005-09-15 at 17:10 +0200, Ghislain Garçon wrote: does the ICAP team plan to implement load balancing for v3? If yes, will it only be round-robin or a different protocol?// The plan is to support basic ICAP first, without load balancing complications. The ICAP framework being developed should simplify adding load balancing features later. Do you expect to contribute ICAP load-balancing algorithms? Thank you, Alex. Duane Wessels a écrit : On Tue, 13 Sep 2005, Ghislain Garcon wrote: Hello, I'm interrested in the ICAP patch for SQUID-Cache and I would like to know if the developpement for squid 3.x will start. What kind of help will be the more needed? Hi Ghislain, I am working on ICAP for Squid-3. The code is in the sourceforge CVS repository with tag 'squid3-icap'. I don't think it is very useable yet, but in a while perhaps you can run some tests with Squid talking to an ICAP server. Duane W.
Re: pconns and WHEN_SQUID_IS_NOT_HTTP1_1
On Sat, 2005-09-10 at 23:54 +0200, Henrik Nordstrom wrote: The corresponding(?) bugzilla comment says ... uncovered another HTTP compliance bug in Squid where HTTP/1.1 messages was considered (partly) as implicitly keep-alive even if Squid is still HTTP/1.0. What is wrong with considering HTTP/1.1 messages keep-alive by default? Are we trying to catch cases where a (broken?) HTTP/1.1 agent expects Squid to default to non-persistent connections because Squid is HTTP/1.0? HTTP/1.0 keep-alive can not be enabled unless the request explicitly says this is understood by the requestor. I am not sure I agree. Persistent connection is an optional optimization without any guarantees (regardless of the protocol version). From protocol point of view, it should be safe to send keep-alive in every message with known length. What am I missing? Why do we need the requester to understand the keep-alive token? Are we concerned about who is going to close the connection first? Some other performance optimization like that? Or is it related to the known message length issues? HTTP/1.1 alone does not imply that the requestor implements HTTP/1.0 keep-alive Agreed, but the above reasoning about pconns being an optimization without guarantees should still apply. If the agent does not recognize the keep-alive token, it will just treat the connection as non-persistent, right? An agent may do that for many other reasons as well, even if it does recognize the keep-alive token! The HTTP/1.0 keep-alive was never accepted into any standard [ Side note: How about RFC 2068, Section 19.7.1? RFC 2616 cites it. ] Additionally a HTTP/1.1 response to a HTTP/1.0 request does not not automatically imply keep-alive even if the response is a HTTP/1.1 response. This is a difficult case since it is not clear whether the response version or the lack of the keep-alive token take precedence. We can assume Connection: close, but what would be the harm in assuming a persistent connection? [ An HTTP/1.1 server that does not send Connection: header to HTTP/1.0 client (that did send it!) should be fixed, IMO ] This is what the quote from the bug report is about. I understand now, but I am not sure why assuming persistent connection was a bug. Do you recall what was happening when connections were assumed to be persistent? The primary reason for these questions is the desire to make httpMsgIsPersistent() an HttpMsg method without breaking too many things, and with future HTTP/1.1 support in mind. At the first glance, we do not need any information external to the message (and the old interface kind of confirms that). Either message implies persistency or it does not. But then I noticed old code that supplies http version that does not come from the same message as the supplied http headers and paused... Now, it seems like we will need two things: bool HttpMsg::wantsPconn(void) const; and bool connIsPeristent(const HttpRequest , const HttpReply ); with all the above questions applying to the second function, not the first. Thank you, Alex.
Re: would like to fix OpenWave mobile browser issue
On Wed, 2005-10-19 at 13:05 -0500, Eric wrote: I need a way to force squid to include the correct Content-length header for every response, even if it's not included by the originating server. In Squid3, the simplest way of doing that may be using an ICAP server. The server will get every response preview (just the HTTP headers) and will request the whole response if there is no Content-Length. The server will then add the Content-Length header. Squid3 and its ICAP client is not ready for production use yet, but we are actively working on it. In Squid2, I am not sure whether the ICAP client implementation can handle such an abuse, but you can test. I am sure it is possible to do this in Squid directly, without ICAP, but cannot give you hints. There are at least two major advantages of using ICAP for this: - Avoid Squid code bloat. - Your solution will work with any popular proxy, not just Squid. $0.02, Alex.
Re: Dynamic filtering features
On Fri, 2005-10-21 at 00:47 -0600, Ahmed Obied wrote: Anyway, what I really want to do is to officially implement the dyanmic URL feature. Also, I can add a dynamic content filtering feature as well where such feature can be used to block malware at the proxy. Please let me know what you think. I really have a passion for writing code and would like to contribe to the open source community. How about writing an ICAP (RFC 3507) server plugin that does the above instead? This way, you will help not only those who are running Squid, but also those who are running other HTTP proxies. It will also help to avoid Squid code bloat. Alex.
Re: Summary of Squid-2.6 opinions
On Mon, 2005-11-07 at 00:35 +0100, Henrik Nordstrom wrote: If Squid-3.0.STABLE is realistic within a not too distant future then Squid-2.6 is not needed and would in fact hurt Squid in the long run, no matter how temting the release it may be to myself and others. If Squid-3.0.STABLE is not realistic in the near future (months, not years) then the proposed Squid-2.6 in my eyes is highly needed, and would in the long run probably allow Squid-3.0.STABLE sooner rather than later even if there is some dissapointment with current sponsors. I agree. Duane and I hope to see Squid3 stable in our environments by the end of 2005 or so. This does not include design fixes (an on-going effort) and features we can live without (e.g., NTLM authentication, LDAP support, or Windows port). If more folks start focusing on Squid3 stability problems (excluding new features), we can indeed see an official STABLE designation within months, I guess. On the other hand, we failed to find Squid developers with enough spare cycles last time we shopped around. Still looking... Alex.
Re: .cvsignore files
On Tue, 8 Nov 2005, Serassio Guido wrote: What should be the standard content of a .cvsignore file ? I doubt there is a standard. The .cvsignore file should probably [only] contain every file or subdirectory name that satisfies all of these three conditions: - Should never be under CVS control. - CVS marks with a '?' when doing cvs update. - May be produced when building Squid on some platform (make dist and such do not count). Wildcards seem to work. HTH, Alex.
Re: squid-3 icap related patch
On Thu, 2006-09-28 at 05:18 -0400, Tsantilas Christos wrote: Hi all, I am sending a small patch for icap client. With this patch squid-icap works for me (using 1-2 clients). I will test it more under load, using different scenarios. I am working on committing yours and other ICAP fixes to CVS. My goal is to make Squid3/ICAP pass Polygraph and real-world tests in a week or so. If you find other ICAP bugs, please post them here or add them to bugzilla. Thanks a lot for your patch, Alex.
Re: squid-3 icap related patch
On Tue, 2006-10-03 at 11:36 +0300, Tsantilas Christos wrote: Alex Rousskov wrote: I believe zero size content is valid. If writeMore() cannot handle it, writeMore() should be fixed and still called unconditionally in the above code. I believe that the problem here is not the zero sized content. Squid calls this function when read all headers. There are cases in which squids did not read any part of body yet (and there is not Content-Lenght header so preview is disabled). In this case squid sends to the icap server the 0\r\n\r\n string (0\r\n definition + \r\n after chunk) The icap server believes that this is the end of body and does not expect more data. I see! Writing last-chunk when the body is [still] expected is obviously wrong and should be fixed. I will try to reproduce and fix this. In general, I am worried about adding a guard for writeMore() outside of writeMore() because writeMore() is called from several places, on several events. Thank you, Alex.
Re: squid-3 icap related patch
On Tue, 2006-10-03 at 08:48 -0600, Alex Rousskov wrote: On Tue, 2006-10-03 at 11:36 +0300, Tsantilas Christos wrote: I believe that the problem here is not the zero sized content. Squid calls this function when read all headers. There are cases in which squids did not read any part of body yet (and there is not Content-Lenght header so preview is disabled). In this case squid sends to the icap server the 0\r\n\r\n string (0\r\n definition + \r\n after chunk) The icap server believes that this is the end of body and does not expect more data. I see! Writing last-chunk when the body is [still] expected is obviously wrong and should be fixed. I will try to reproduce and fix this. I cannot reproduce the above. I made a server that responds to an HTTP GET with HTTP headers and then waits before sending the message body. There is no Content-Length header. After Squid received the HTTP response headers and successfully wrote the ICAP message prefix to the ICAP server, I get: | ICAPModXact::noteCommWrote called [Comm(13wr)w(2)/] | ICAP/ICAPModXact.cc(164) Wrote 670 bytes | ICAP/ICAPModXact.cc(258) will write up to 0 bytes of prime virgin body | ICAPModXact has no writeable prime virgin body content | ICAP/ICAPModXact.cc(287) will write 0 raw bytes of prime virgin body | ICAPModXact::noteCommWrote ended [Comm(13r)w(5)/] As you can see, ICAPModXact refuses to write anything when there is no response body yet. Was your test case different? Thank you, Alex.
Re: protocol clarity
On Sun, 2006-09-10 at 21:40 +1000, Robert Collins wrote: So I've finished my analysis of the protocol stuff in squid - protocol_t etc. http://wiki.squid-cache.org/ForwardRework has some thoughts about how we can improve the current api, to make SSL fit in more cleanly, and make doing some nifty things, like forwarding via ssh tunnels, or peers that need a vpn etc, easy to do. Do you perceive any related significant changes to the ICAP code? Squid implements an ICAP client, but the ICAP implementation differs from other protocols because ICAP is not a forward-and-forget protocol or, at least, it cannot be implemented that way today (perhaps unfortunately). Today, the client- and server-side HTTP code pipes the HTTP message to ICAP and expects a new message to be piped back, possibly at the same time. Various ICAP aborts are rather difficult to recover from today. It would be nice if your scheme would help with that. Another issue is message data copying. Today, HTTP bodies are copied when they go to ICAP and are copied again when they go back to HTTP, even though the content is known to be identical (e.g., the ICAP server said 204 No Content). I am not sure whether this should be optimized on the MemBuf level (i.e., optimize copying itself via lazy copying and such) or on a protocol-handling level (i.e., remove the need for copying). The latter approach may seem better from the design point of view, but is far more difficult to implement with the existing code; we currently do not have sane means of sharing the same I/O buffer among three protocols (HTTP client, HTTP server, and ICAP client). Thank you, Alex.
Re: protocol clarity
On Wed, 2006-10-11 at 11:54 +1000, Robert Collins wrote: http://wiki.squid-cache.org/ForwardRework I haven't considered icap deeply. In terms of forwarding icap is mostly/nearly orthogonal. There are things we handle that are not icap interceptable - but perhaps they should be. (For instance, internal static urls). IMHO, an ideal implementation would allow to filter through ICAP everything that internally resembles an HTTP message and goes into Squid or out of Squid. I believe you have stated a similar ideal. The only way to do that with our more than limited resources is to provide a single way to hookup ICAP into an HTTP message processing stream, whatever that stream is. Today, adding hookups is too much work; too many similar but different places need to be changed. However, if you are rewriting how messages are represented and shoveled around, you have a fighting chance of creating necessary protocol-, source-, and destination-agnostic places to hookup ICAP. [lot of good stuff snipped] I agree that the canonization of HTTP-like message representation is the right direction, but am not qualified to comment on how to get there from what we have now or on the details of that interface. In terms of data copying, if the icap adaption says 'no change', you can just hand the body object over, or have a decorator object that uses the original one with no copying. FWIW, this approach does not work well in the current code because (a) There is no body object to hand over. Object body is represented by a few data structures _and_ by the complex stateful code that handles them (e.g., http.cc). The body state and metainfo is not encapsulated in one object/place. (b) The virgin message carries not just the headers and body state, but a bunch of protocol and vectoring-point specific attributes that will often become stale or inapplicable after passing through ICAP, even if ICAP performed no message modification. For example, the metadata about connection will be wrong if the HTTP connection was long gone while the ICAP was thinking whether to adapt the response. I do not know if your changes will help with the above, but it would be nice if they did :-) I have to stop here, but I think the shape is clear - what do you think,does it sound doable, are there holes ? It all sounds doable, but I can only comment on ICAP needs (I do not remember or know enough about the rest of the code). Here are few additional ICAP-related things that I hope you would keep in mind when designing your changes: 1) ICAP request satisfaction: ICAP should be able to take a virgin HTTP message and return an HTTP response to that message. The code should not assume that ICAP is only a pass through mechanism or that all HTTP responses originally came from an HTTP server/peer or store. 2) ICAP bypass: When ICAP fails to process a message (e.g., when ICAP service is down), it would be wonderful if ICAP-transaction-independent code would be able to recover and go on as if ICAP was not in the loop (if still possible and allowed by Squid configuration). I am guessing that would be somewhat similar to recovering from a peer failure. 3) ICAP buffers: Ideally, Squid client and server sides would pump data through ICAP without copying the data when ICAP is not modifying it. A logging-only ICAP server that wants to look at most message bodies is a good example to consider here. HTTP client, server, and ICAP speeds may all differ, making the above far from trivial. 4) ICAP chains: There may be more than one ICAP service interested in processing a given message. Ideally, the ICAP hookup interface will take care of multiple services instead of making an individual ICAP transaction concerned with ICAP transactions before or after it. This aspect complicates all of the above items! 5) ICAP encoding: Any HTTP transfer encodings and hop-by-hop headers must be removed before feeding messages to ICAP servers. Perhaps this removal should _not_ be done by ICAP because of #4 and, perhaps, HTTP caching reasons. Finally, there are other places in Squid that might have some needs similar to ICAP because they look at and modify messages on-the-fly. Proxy-side includes is one example of an internal ICAP server, I guess. HTH, Alex. P.S. I wonder if FreeBSD netgraph API is worth studying in this context!
Re: Squid-dev subscription
On Fri, 2006-10-20 at 21:53 +0200, Gernot Tenchio wrote: My name is Gernot. I'm a developer from Germany and I'm interested in getting the ICAP part of Squid to work. Currently I'm tried to get things working with Squid's 2.6er icap branch and with Squid3 as well. Both versions worked more or less for me, but I think it (c|sh)ould be improved. Hi Gernot, Glad you are interested in ICAP. FWIW, quite a few ICAP bugs have been recently fixed on squid3-icap branch of Squid CVS tree. You may want to use that branch as a base for your work. Currently, I am not aware of any serious ICAP-specific bugs except for OPTIONS response processing, but I am sure there are a few. If you know of any ICAP problems, please post here or, better, submit a bug report. I may be able to help. An avalanche of bug reports notwithstanding, I hope squid3-icap will be ready for HEAD integration by the end of this week. My longer-term goal is to make Squid3/ICAP stable for the Squid3 stable release. HTH, Alex.
Re: Squid-dev subscription
On Sun, 2006-10-22 at 20:50 +0200, Gernot Tenchio wrote: On Sun, 22 Oct 2006 11:22:20 +0200 Moshe Beeri [EMAIL PROTECTED] wrote: Hi All Squiders, In the following couple of weeks I will post some new features that are needed for parental control, as well as context connectivity from req_mod to resp_mod. But the major change I made is to add the ability to direct response to client in case that classification is already known at req_mod. The changes I made are on 2.5 Stable 10. I hope you will be able to use those features to Squid benefit. Thanks allot, Moshe Beeri. SW Engineer, PureSight Technologies. Hmmh, I had no luck to get error messages displayed as part of REQMOD responses (squid-icap 2.6). Does it meen this will work in the future? Does the patch appear in icap-2_6 too? FWIW, ICAP request satisfaction mode seems to be working in Squid3, using squid3-icap branch. Alex.
Re: crash when ICAPModXact::parseHead throws
On Tue, 2006-10-24 at 07:55 +0200, Gernot Tenchio wrote: I see random crashes because ICAPModXact::parseHead throws an exception. This always happens if there are no headers to parse at all (maybe because of a bug on the server side). #11 0x08113ce9 in __cxa_throw () #12 0x0813ca7f in Throw () #13 0x0813f435 in ICAPModXact::parseHead () #14 0x08141f5e in ICAPModXact::parseIcapHead () #15 0x0814229a in ICAPModXact::parseHeaders () #16 0x08142675 in ICAPModXact::parseMore () #17 0x081427ae in ICAPModXact::handleCommRead () #18 0x08146161 in ICAPXaction::noteCommRead () #19 0x081462cd in ICAPXaction_noteCommRead () #20 0x080ed2cc in commio_call_callback () #21 0x080ed318 in commio_call_callbacks () #22 0x080efa19 in CommDispatcher::dispatch () #23 0x0808b815 in EventLoop::runOnce () #24 0x0808b941 in EventLoop::run () #25 0x080bd877 in main () Throwing in exception is expected under exceptional conditions such as a parsing failure. The exception should be caught by noteCommRead, terminating the ICAP transaction. I have just tested it by throwing an exception unconditionally in parseHead, and it seems to work: The ICAP transaction terminates and the HTTP client gets an ICAP protocol error response from Squid. This is all with squid3-icap branch. I have not tested with HEAD and would not be surprised if terminated ICAP transactions are not handled well there. If you can reproduce with squid3-icap, please note that the above stack trace is missing the first 11 frames. You said that Squid crashes. How does it crash? What compiler are you using? Thank you, Alex.
Squid3 and multiple ICAP services
Hi there, Handling of multiple ICAP services in Squid3 does not match squid.conf expectations well. Here is a brief summary of the current state, with Squid2 comparison: load balancing (using similar services in a round-robin): Squid2: via multiple identically named icap_service options Squid3: not documented and not supported. backup (using another service when a similar service is down): Squid2: via multiple identically named icap_service options Squid3: not documented but supported via icap_class option chaining (applying one service after another): Squid2: via icap_class option Squid3: semi-documented but not supported I believe that using multiple identically named icap_service options is a bad interface. Each service should have its own name for configuration clarity, reporting, debugging, and such. The way chaining is explained in squid3.conf will confuse many users. The name of the option (icap_class) does not imply chaining either. Many users will think that icap_class is about backup, not chaining. In fact, the original Squid3 implementation was using icap_class for something resembling backup, not chaining. I would like to polish this for Squid3 STABLE release, but since my proposal requires changing squid.conf option names, I do not want to do it without your feedback. Here is what I would like to do: service naming: Each service must have a unique name. load balancing and backup: Add icap_service_set option. Services (or service chains) listed under this option are load balanced and down entries are skipped. chaining: Rename icap_class into icap_service_chain option. Services (or service sets) listed under this option are applied one after another. Any objections or better ideas? Thank you, Alex. P.S. I am not going to implement load balancing and chaining in Squid3 until somebody needs it enough, but the configuration will be cleaned up and ready for that implementation. The backup code is already done.
Re: ICAP: 403 Forbidden
On Fri, 2006-11-03 at 08:32 -0500, Jeremy Hall wrote: I'm currently running 2.6-branch and am considering upgrading to 3, but I need to know how mature the code is. I would be updating for icap--I know an icap patch exists for 2.6 and I have it working with two anomolies: 1: the entire file is pulled before the content filter responds to the user rather than a deferred scan taking place Can you describe the above in more detail? Some content filters do want to see the whole message before they can make a decision. I am not sure what you mean by a deferred scan; a monitoring (i.e., read-only) ICAP service that does not block any messages? What exactly is your ICAP server doing and what exactly do you want Squid to do? 2: if the icap server returns a 403 squid just hangs up the connection and doesn't send anything back to the browser at all. This produces page cannot be displayed errors. I have looked at the icap code a bit but was wondering if you had some pointers on where I might look to determine why the traffic is not rendering properly. I cannot help with Squid2. With Squid3, there are several possible scenarios. Please pick the one(s) relevant to you: If an ICAP 403 Forbidden response is received from an optional REQMOD service, Squid seems to be able to recover and proceed with the request in simple cases. I bet Squid will fail to recover if request has a [large] body. If an ICAP 403 Forbidden response is received from a RESPMOD service, Squid will return an ICAP error message to the client, even if the service is optional. ICAP RESPMOD bypass only works for connection establishment errors, I think. If an ICAP 200 OK response is received with an HTTP 403 Forbidden response inside, everything should work as expected in both REQMOD and RESPMOD. If it does not, please let me know or submit a bug report. Please note that all of the above scenarios only apply to an ICAP service that is either yet unprobed for options or is already considered up. A service is first probed when the first ICAP request needs to be made to that service. Thus, if your ICAP server returns ICAP 403 Forbidden for ICAP OPTIONS requests, and the service is optional (bypass=0), the service will be marked as down and will not be used for HTTP message adaptation. HTH, Alex.
Re: icap in either squid-3 and squid-2
On Tue, 2006-11-07 at 10:32 -0500, Jeremy Hall wrote: I'm trying to track down why the Content-Length header might not be sent to the icap server in squid-2's implementation of icap. ... I haven't tried this with squid-3 because I don't want to manually upgrade my config file if the problem isn't resolved. Can somebody give me some ideas on how best to proceed? If you tell me how to reproduce the Content-Length problem, I will try to check whether it still exists in Squid3. HTH, Alex.
Re: Squid 3 HEAD - ICAP Modifications
On Mon, 2006-12-04 at 12:57 +0100, Axel Westerhold wrote: Hi everyone, Second try this time hopefully complete. This is again patched against Squid 3 HEAD and includes 4 changes I would like to have when working with webwasher/squid systems. A.) ICAPServiceRep::TheSessionFailureLimit set through squid.conf B.) ICAPServiceRep delay for a down service set through squid.conf C.) Instead of hardcoding the Header used to transfer the username being able to set the used one through squid.conf D.) When using X-Authenticated-User in C I need the username to be base64 encoded so I added another option to turn on encoding if needed. I will work on adding these changes to the squid3-icap branch. Thank you, Alex.
Re: squid3 icap preview
On Tue, 2006-12-05 at 16:24 +0100, Gernot Tenchio wrote: Sometimes squid does not send a preview to the icap server even if requested. It seems to happen if squid doesn't know how large the response would be. Is this intended? As far as I can see, if the expected body size is unknown, Squid will send a zero-size preview. A comment in the code questions that logic. I do not know why Squid sets preview size to zero in that case; it could be an if-statement condition bug or just a leftover from earlier, less capable code. If you are willing to experiment, please try the attached patch. The patch removes the code that sets preview size to zero when the body size is not known. Please let me know whether the patch works for you. Thank you, Alex. Index: src/ICAP/ICAPModXact.cc === RCS file: /cvsroot/squid/squid3/src/ICAP/ICAPModXact.cc,v retrieving revision 1.1.2.17 diff -u -u -r1.1.2.17 ICAPModXact.cc --- src/ICAP/ICAPModXact.cc 25 Oct 2006 04:57:03 - 1.1.2.17 +++ src/ICAP/ICAPModXact.cc 6 Dec 2006 15:12:22 - @@ -1067,10 +1067,11 @@ // cannot preview more than we can backup size_t ad = XMIN(wantedSize, TheBackupLimit); -if (virginBody.expected() virginBody.knownSize()) -ad = XMIN(ad, virginBody.size()); // not more than we have +if (!virginBody.expected()) +ad = 0; // nothing to preview but headers else -ad = 0; // questionable optimization? +if (virginBody.knownSize()) +ad = XMIN(ad, virginBody.size()); // not more than we have debugs(93, 5, ICAPModXact should offer ad -byte preview (service wanted wantedSize ));
Re: Squid 3 HEAD - ICAP Modifications
On Mon, 2006-12-04 at 12:57 +0100, Axel Westerhold wrote: Hi everyone, Second try this time hopefully complete. This is again patched against Squid 3 HEAD and includes 4 changes I would like to have when working with webwasher/squid systems. A.) ICAPServiceRep::TheSessionFailureLimit set through squid.conf B.) ICAPServiceRep delay for a down service set through squid.conf C.) Instead of hardcoding the Header used to transfer the username being able to set the used one through squid.conf D.) When using X-Authenticated-User in C I need the username to be base64 encoded so I added another option to turn on encoding if needed. The above changes, with minor modifications are now committed to squid3-icap branch. The corresponding patch is attached. I took the liberty to rename some of your new squid.conf options as well as polish squid.conf comments and code. A negative value for the icap_service_failure_limit disables the limit feature. Please test and let me know whether any further changes are needed. Thank you, Alex. - Added icap_service_failure_limit squid.conf option. The limit specifies the number of failures that Squid tolerates when establishing a new TCP connection with an ICAP service. If the number of failures exceeds the limit, the ICAP service is not used for new ICAP requests until it is time to refresh its OPTIONS. The per-service failure counter is reset to zero each time Squid fetches new service OPTIONS. A negative value disables the limit. The limit used to be hardcoded to 10. - Added icap_service_revival_delay squid.conf option. The delay specifies the number of seconds to wait after an ICAP OPTIONS request failure before requesting the options again. The failed ICAP service is considered down until fresh OPTIONS are fetched. The actual delay cannot be smaller than the [still] hardcoded minimum delay of 60 seconds. - Added icap_client_username_header and icap_client_username_encode squid.conf options to control how the authenticated client username should be sent to the ICAP service. - All of the above changes are based on the patch by Axel Westerhold. Index: CONTRIBUTORS === RCS file: /cvsroot/squid/squid3/CONTRIBUTORS,v retrieving revision 1.7.8.3 diff -u -r1.7.8.3 CONTRIBUTORS --- CONTRIBUTORS 3 Oct 2006 05:24:00 - 1.7.8.3 +++ CONTRIBUTORS 14 Dec 2006 05:15:56 - @@ -105,5 +105,6 @@ Mark Bergsma [EMAIL PROTECTED] Tim Starling [EMAIL PROTECTED] Tsantilas Christos [EMAIL PROTECTED] + Axel Westerhold [EMAIL PROTECTED] Duane Wessels [EMAIL PROTECTED] Index: src/cf.data.pre === RCS file: /cvsroot/squid/squid3/src/cf.data.pre,v retrieving revision 1.79.2.12 diff -u -r1.79.2.12 cf.data.pre --- src/cf.data.pre 18 Oct 2006 21:26:10 - 1.79.2.12 +++ src/cf.data.pre 14 Dec 2006 05:16:03 - @@ -4966,6 +4966,39 @@ If you want to enable the ICAP module support, set this to on. DOC_END +NAME: icap_service_failure_limit +TYPE: int +IFDEF: ICAP_CLIENT +LOC: TheICAPConfig.service_failure_limit +DEFAULT: 10 +DOC_START + The limit specifies the number of failures that Squid tolerates + when establishing a new TCP connection with an ICAP service. If + the number of failures exceeds the limit, the ICAP service is + not used for new ICAP requests until it is time to refresh its + OPTIONS. The per-service failure counter is reset to zero each + time Squid fetches new service OPTIONS. + + A negative value disables the limit. Without the limit, an ICAP + service will not be considered down due to connectivity failures + between ICAP OPTIONS requests. +DOC_END + +NAME: icap_service_revival_delay +TYPE: int +IFDEF: ICAP_CLIENT +LOC: TheICAPConfig.service_revival_delay +DEFAULT: 180 +DOC_START + The delay specifies the number of seconds to wait after an ICAP + OPTIONS request failure before requesting the options again. The + failed ICAP service is considered down until fresh OPTIONS are + fetched. + + The actual delay cannot be smaller than the hardcoded minimum + delay of 60 seconds. +DOC_END + NAME: icap_preview_enable TYPE: onoff IFDEF: ICAP_CLIENT @@ -5026,8 +5059,29 @@ LOC: TheICAPConfig.send_client_username DEFAULT: off DOC_START -This adds the header X-Client-Username to ICAP requests -if proxy access is authentified. + This sends authenticated HTTP client username (if available) to + the ICAP service. The username value is encoded based on the + icap_client_username_encode option and is sent using the header + specified by the icap_client_username_header option. +DOC_END + +NAME: icap_client_username_header +TYPE: string +IFDEF: ICAP_CLIENT +LOC: TheICAPConfig.client_username_header +DEFAULT: X-Client-Username +DOC_START + ICAP request header name to use for send_client_username. +DOC_END + +NAME: icap_client_username_encode +TYPE: onoff +IFDEF: ICAP_CLIENT +COMMENT:
Re: Squid 3 HEAD - ICAP Modifications
On Thu, 2006-12-14 at 08:47 +0100, Axel Westerhold wrote: There is one more change I am currently testing. The problem with this patch: It does not follow any ICAP document but only enables squid to get rid of the DOMAIN part frpm an NTLM Auth so I can use the result string as a query on samaccountname. Would it be possible to add this too ? I assume we talking about modifying the authenticated client username, passed via the icap_client_username_header to the ICAP server. I believe the right thing to do here is similar to what Jeremy Hall did with the icap_auth_scheme in http://www.squid-cache.org/mail-archive/squid-dev/200611/0066.html I would suggest to add an icap_authenticated_user_header_value option that takes a string with the following supported substitutions: %U -- complete username, as is %N -- username without the domain part %% -- percent I think we do not need a default value here, but would not object to Local://%U if somebody insists on having a default. The current icap_client_username_header option should be renamed to icap_authenticated_user_header_name. The current icap_client_username_encode option should be renamed to icap_authenticated_user_header_encode. On should be the default, I guess. The three icap_authenticated_user_* options can be merged into one multivalued option: icap_authenticated_user_header [name:] [encoding(] value [)] For example, icap_authenticated_user_header X-Authenticated-User: base64(Local://%U) or icap_authenticated_user_header identity(%N) Thank you, Alex. Am 14.12.2006 6:25 Uhr schrieb Alex Rousskov unter [EMAIL PROTECTED]: On Mon, 2006-12-04 at 12:57 +0100, Axel Westerhold wrote: Hi everyone, Second try this time hopefully complete. This is again patched against Squid 3 HEAD and includes 4 changes I would like to have when working with webwasher/squid systems. A.) ICAPServiceRep::TheSessionFailureLimit set through squid.conf B.) ICAPServiceRep delay for a down service set through squid.conf C.) Instead of hardcoding the Header used to transfer the username being able to set the used one through squid.conf D.) When using X-Authenticated-User in C I need the username to be base64 encoded so I added another option to turn on encoding if needed. The above changes, with minor modifications are now committed to squid3-icap branch. The corresponding patch is attached. I took the liberty to rename some of your new squid.conf options as well as polish squid.conf comments and code. A negative value for the icap_service_failure_limit disables the limit feature. Please test and let me know whether any further changes are needed. Thank you, Alex.
Re: saving web page body to file... help needed
On Wed, 2007-01-17 at 19:07 +0530, Siddhesh PaiRaikar wrote: we are trying to develop a small enhancement to the existing application of squidguard using the squid proxy server... which can later be embedded in to squid itself as a html web page body scanner for unwanted content. Please consider using an ICAP (RFC 3507, i-cap.org) server for content scanning, blocking, and manipulation instead of building this feature directly into Squid. To implement your functionality, you can modify one of the free or for-a-fee ICAP servers available on the web. Besides having to work with a much simpler and smaller code base, you will have an advantage of being compatible with other popular proxies because they all speak ICAP. Good luck, Alex.
Squid3 BodyReader changes
Folks,
Executive summary: I am trying to fix several bugs and complaints
related to your favorite class, the BodyReader. The changes I would like
to make are significant, so I decided to post them here first. Please
see the attached BodyPipe.h sketch. I will proceed with these changes
unless there are violent objections or better ideas. The email below
documents my concerns and explains the rationale behind BodyPipe.
Current state: The BodyReader class does not seem to read a body. It
helps to move the body content from the body producer to the consumer.
The class is essentially a collection of loosely coupled function
pointers with protected and bare data attached to them. The class calls
consumer-side functions on behalf of the producer (or vice versa), while
updating the body size state. BodyReader refcounting was meant to
communicate abort conditions.
Here are a few related/intermingled reasons behind BodyPipe, the
replacement design I am implementing (see the attached sketch).
A) The reason I want to get away from the current lose ends approach
with function pointers is to provide a more clear, easier to follow
implementation (IMO). I want classes like ConnStateData or
ICAPClientReqmodPrecache to inherit from BodyProducer and just implement
the two or three required methods. No wrappers or bare data pointers at
the high level, where all the changes and enhancements are
taking place.
B) The reason I want explicit provider and consumer pointers inside
BodyPipe is to be able to abort either side if the other side is having
trouble. The current idea of using refcounting to abort something does
not and cannot work (if there are two refcounted pointers, the second
pointer will not know that the first has been set to NULL). I also want
joint ownership of the BodyReader so that the producer and the consumer
know that they must clear/detach themselves before forgetting about the
pipe. Refcounting will just help to destroy the abandoned pipe.
C1) The reason I want asynchronous calls is to separate producer from
consumer and break the call-me-back-while-I-am-calling-you loops. I
believe that at least the abortedSize assertion (bug #1862) is
caused, in part, by such loops.
C2) Another reason for separating producer and consumer is to allow the
producer to disconnect when done, leaving a functional object (with the
remaining/last body bytes) behind. This situation happens when, for
example, an ICAP transaction is finished but the HttpStateData is still
writing the body to the origin server.
Again, please take a look at the attached sketch.
I have seen this before: While sketching the design, I realized that
what I want is very similar to MsgPipe, a class used by ICAP-related
code to shovel HTTP messages between Squid core and an ICAP transaction.
If you look at MsgPipe.cc and ignore the low-level debugging macros, you
will see that there is virtually nothing there. The class simply
converts notifications from one pipe end to events and then calls the
other end when the event fires.
I do not plan on reusing MsgPipe itself because its simple design does
not allow one object to be the end of two pipes -- there would be no way
to know which pipe the messages are coming from. I do not want to
complicate MsgPipe and its users to support multi-pipe ends. I think it
would be better to have a very similar but distinct BodyPipe class, with
method names specific to the problem at hand and extra code to handle
body size calculations. The design duplication will be there, but code
duplication should be negligible.
I am worried that I see every problem as a nail though, which is
another motivation for this message. Again, if you have better ideas on
how to fix BodyReader-related problems, please stop me and contribute!
Thank you,
Alex.
// interface for those who want to provide body content to BodyPipe
class BodyProvider {
public:
virtual BodyProvider() {}
virtual provideMoreBody(BodyPipe *pipe) = 0;
virtual noteBodyConsumerAbort(BodyPipe *pipe) = 0;
};
// interface for those who want to consume body content from BodyPipe
class BodyConsumer {
public:
virtual BodyConsumer() {}
virtual consumeMoreBody(BodyPipe *pipe) = 0;
virtual noteBodyProviderAbort(BodyPipe *pipe) = 0;
};
// Connects those who produce message body content with those who
// consume it. For example, connects ConnStateData and FtpStateData OR
// ICAPClientReqmodPrecache and HttpStateData.
class BodyPipe: public RefCountable {
public:
typedef RefCountBodyPipe Pointer;
public:
BodyPipe(size_t aLength, Provider *aProvider);
~BodyPipe(); // clear provider and consumer first
// called by providers
void clearProvider(bool keepPiping);
size_t spaceSize() const;
size_t putMoreData(const char *buf, size_t size);
// called by consumers
void setConsumer(Consumer *aConsumer);
void clearConsumer(bool keepPiping);
size_t
clientReplyContext::replyStatus and proxy_keepalive
Hello,
If you understand clientReplyContext::replyStatus, please see if you
can help me with the following problem.
Background: I am making aborted POSTs to work with the new BodyPipe.
Under certain ICAP failure conditions, the client side needs to produce
an error response. We did not have any code for that case.
I found disabled #if ICAP_HARD_ERROR code inside
ClientHttpRequest::abortAdapting, removed the yet-unsupported errno
argument, and moved the code outside the if-statement. That wonderful,
albeit mysterious, piece if code seems to produce an error just fine.
The error is sometimes generated before the entire POST body is read and
discarded.
Problem: The following piece of client_side_reply code appears to be
sometimes confused by the client-side produced error and/or my other
bugs. It returns STREAM_UNPLANNED_COMPLETE, causing various problems on
the client_request side (e.g., entering closing state twice).
clientStream_status_t
clientReplyContext::replyStatus()
{
...
if (!done) {
debug(88, 5) (clientReplyStatus: closing, !done, but read 0
bytes\n
return STREAM_FAILED;
}
if (!http-gotEnough()) {
debug(88, 5) (clientReplyStatus: client didn't get all it
expected\
return STREAM_UNPLANNED_COMPLETE;
}
if (http-request-flags.proxy_keepalive) {
debug(88, 5) (clientReplyStatus: stream complete and can
keepalive\
return STREAM_COMPLETE;
}
debug(88, 5) (clientReplyStatus: stream was not expected to
complete!\n
return STREAM_UNPLANNED_COMPLETE;
Could somebody please explain the logic here? Specifically, I do not
understand why proxy_keepalive flag is required to get a STREAM_COMPLETE
result. I am getting STREAM_UNPLANNED_COMPLETE (from the last return
statement) because the request apparently does not have that flag set.
What does it mean to have a complete stream and why do I need a
proxy_keepalive flag with that?
A related question: Is replyStatus() expected to work correctly when the
connection is in a closing state?
Thank you,
Alex.
Re: squid cache information - help
On Thu, 2007-02-08 at 14:05 -0800, ccmail111 wrote: I have: squid-3.0.PRE5-20061215 on Linux. I am looking to understand how SQUID3.x stores (in no cache mode) the responses received from Http server - HTTP payload (including images etc. received and retrieved from different sites). I need this information to tie it to my backend program which does audit and such. I have some APIs like stdlib (linux) and would like to call them from appropriate locations from SQUID. Any help/pointers is appreciated ! Similar questions have been discussed a few times recently. You may want to search the list archive for the word ICAP. While not all suggested solutions involve ICAP, the protocol is usually mentioned at least once in the relevant thread. Here is an example of a thread offering up to four starting points for your question: http://thread.gmane.org/gmane.comp.web.squid.devel/4048/ This recent thread may also be relevant: http://thread.gmane.org/gmane.comp.web.squid.devel/4197/ HTH, Alex.
Re: icap in squid3
On Fri, 2007-02-09 at 15:55 -0500, Jeremy Hall wrote: Hello, I can't remember. What was the decided path for what was once the icap_auth_scheme? I recall there was some concern about my suggestion of having the ability to use ldap://hostname/cn=%u,dc=%d,dc=name,dc=int but I don't remember what the outcome was. I think you were on the right track with your patch[1,2]. If you can provide a separate %escape character for the username without the domain part, that would address Axel's needs, I think[2]. Perhaps your patch already does that -- I did not check. Thanks, Alex. [1] http://thread.gmane.org/gmane.comp.web.squid.devel/4091/ [2] http://thread.gmane.org/gmane.comp.web.squid.devel/4130 [3] http://article.gmane.org/gmane.comp.web.squid.devel/4068
Re: icap in squid3
On Mon, 2007-02-12 at 08:38 +0100, Axel Westerhold wrote:
Well, the syntax you are proposing is somewhat limited.
Here are my comments:
1.) cn=%u assumes that the used username equals the assigned CN which is
most of the time wrong. Normally the UID (or in AD the samaccountname) is
used for authentication. This will lead to a failure using cn=%u
2.) The given URI is not flexible enough as it assumes that all user object
will always be located within the same root object. The used syntax provides
fast access because it will avoid search operations but will fail as soon as
the object is located in a sub OU or a totally different tree.
3.) LDAP allows for all kinds of unicode chars we would need to encode
properly. While this is definately possible I wonder if there really should
be another encoding scheme impüplemented into squid.
It seems to me that Jeremy is not proposing any syntax, encoding, or URI
format (except perhaps for some default values). He wants to add ability
to use any URI, with any LDAP (or not LDAP) tags. The patch gives user a
set of supported substitutions. The user can use whatever substitution
codes they need in whatever opaque text filling they need. Please see
below for examples.
We should be able to agree on that set without much trouble because
adding more substitutions is not a problem. For example, if somebody
needs a username without a domain, there should be a substitution for
it.
If there are more than 5-7 substitutions, we may want to argue whether
single letter %S substitutions are better than easier-to-remember and
harder-to-mistype ${LongNames}. I would probably vote for the latter,
but it is not a big deal.
Alternative encodings should be supported, of course, perhaps as
$encodingName(string-to-be-encoded) substitution, where
string-to-be-encoded may have variable substitutions? Again, there is an
example below.
What I think might work better is as follows:
A.) A user authenticates using a proper DN authenticating against an LDAP
Server. In this case the username will be the DN and can be transmitted.
B.) The user authenticates using a uid (samaccountname). Either this uid is
already usable on it's own an we can transfer it without any changes just
encoding it base64 if requested (which will keep us out of trouble with
UTF-8 or Unicode chars). In case we get this stuff from a windows user
sending us a domain prefix, we should be able to split the username into
domain and username. The hard part will be to find some kind of abstract how
to transfer this.
Encoding aside, can the above two requirements be expressed as a set of
substitutions?
What we definately need are the following configuration entries:
A.) Do we need to split the username into parts and if so using which
seperator. ('' = off or '\' or '+' etc.)
Can the separator be up to the admin? Do we need to define it?
B.) The X- Header used to transfer the username (bare username without any
instruction on how to use it (X-Authenticated-User, X-User, X-MyUser, X-Blah
etc.)
Agreed. The icap_client_username_header option controls that now, but
please see (C) below.
C.) The X-Header used to transfer the prefix if any.
Should we just support an arbitrary set of user-configurable header
names, with user-configurable values? If we add substitutions support,
then Squid should not really care about the meaning of the header. For
example,
icap_client_add_header X-Username $base64($UserName)
icap_client_add_header X-Prefix bar=$base64($Foo+$Bar)foo=blah
...
D.) Something to force base64 Encoding on above headers
See for a suggestion above.
This ensures that the ICAP Client get's all the info we might have for the
user authenticating. This works fine if the ICAP Client will only deal with
one squid and it's auth scheme. As soon as we have x squids authenticating
to various sources but only one icap client we need to add some additional
information for the client to find the correct auth source. So we need to
tell the ICAP client the used auth (LDAP,WINNT etc) and where the target
(hostname:port) is. From there the client should use all infos received to
build it's internal request.
Can substitutions handle this? Or do we need dynamic support for
selecting an appropriate set of X-Headers, depending on how the user
authenticated?
Cheers,
Alex.
Am 09.02.2007 21:55 Uhr schrieb Jeremy Hall unter
[EMAIL PROTECTED]:
Hello,
I can't remember. What was the decided path for what was once the
icap_auth_scheme? I recall there was some concern about my suggestion of
having the ability to use ldap://hostname/cn=%u,dc=%d,dc=name,dc=int
but I don't remember what the outcome was.
_J
Re: icap in squid3
On Tue, 2007-02-13 at 08:09 +0100, Axel Westerhold wrote:
One comment on a nice feature I would like to have but still considering for
security reasons:
When an ICAP Server requieres auth for user mapping to rules/policies you
sometimes run into a problem with sources with can't auth or destinations
you do not want to require auth for. While you can use ACL's to get this
done easily on squid sometimes the icap clients won't play ball. As a result
some destinations are not using the icap virus scanner/ content system to
make it work. So, maybe but just as a thought, it would be nice to use ACL's
to automatically assign a username für those services so that they can be
properly matched.
Will a default user name work? As a stop-gap measure, it is probably
much easier to implement than ACL-defined user name.
A.) Do we need to split the username into parts and if so using which
seperator. ('' = off or '\' or '+' etc.)
Can the separator be up to the admin? Do we need to define it?
Must be configurable so empty string turns off and non-empty turns on and
defines sperator. Samba for instance allows for Seperator modifications.
Also, this gives squid some flexibility.
I was thinking that the user-configurable opaque text around individual
substitutions will define separators, if any. Is that too cumbersome?
Should we just support an arbitrary set of user-configurable header
names, with user-configurable values? If we add substitutions support,
then Squid should not really care about the meaning of the header. For
example,
icap_client_add_header X-Username $base64($UserName)
icap_client_add_header X-Prefix bar=$base64($Foo+$Bar)foo=blah
...
I like this from a technical point of view. But I can also see my customers
which are most of the time used to windows gui and stuff and already
complain about the squid conf, freaking out. Maybe, for common tasks we need
some kind of macro producing the above code. At all, just as a thought, we
might stay with a sfprint syntax first defining a format and then adding the
values like
icap_client_add_header X-Username %s=%b,bar,$username
Icap_cleint_add_header X-Auth_Scheme
LDAP://ldap.test.com:389/cn=%u,dc=%u,dc=test,dc=com,$username,$domain
Where %s is a string without encoding while %b is base64 encoded and maybe
$u is URI encoded etc. I know this asks for additional parsing but it might
lessen the learning curve for many people.
I doubt GUI folks would freak out _more_ if we add shell-style
substitutions than if we add printf-style ones. GUI folks need a GUI
configuration tool that hides all this.
FWIW, I find shell-style more intuitive and less error-prone than
printf, but since I am not going to implement or pay for this, I would
not argue one way or the other. The developer should decide :-).
In either case, it would be great if the configuration code checked that
all substitutions are known/supported. In case of printf, it would be
nice to check that the %escapes match the arguments as well.
Thank you,
Alex.
Re: httpProcessReplyHeader question
On Tue, 2007-02-13 at 19:45 +0800, Adrian Chadd wrote:
There's this in httpProcessReplyHeader:
hdr_size = headersEnd(httpState-reply_hdr.buf, hdr_len);
if (hdr_size)
hdr_len = hdr_size;
if (hdr_len Config.maxReplyHeaderSize) {
debug(11, 1) (httpProcessReplyHeader: Too large reply header\n);
/* [ahc] XXX not sure how to handle this and we should think about
it; so fail out for now */
storeAppend(entry, httpState-reply_hdr.buf,
httpState-reply_hdr.size);
memBufClean(httpState-reply_hdr);
reply-sline.status = HTTP_HEADER_TOO_LARGE;
httpState-reply_hdr_state += 2;
ctx_exit(ctx);
return size;
}
Why is there a storeAppend with no http parse?
Could it be because the header is so large that it was considered
dangerous or even impossible to parse when the above code was written?
Alex.
Re: Squid3 BodyReader changes
Hi there, I have committed the BodyPipe changes described below (with a few modifications) to the squid3-icap branch. Using the new BodyPipe class, I was able to eliminate many ICAP-related classes and code complexities, not to mention a few memory leaks and bugs. I will be working on a few remaning XXXs next week, but please test the current code if you can, especially if you are using ICAP. **WARNING** The committed changes are significant and probably introduce new bugs! The code passed some Web Polygraph and manual surfing tests, with and without ICAP. FTP-specific changes are untested, but the old code probably did not work well with ICAP either. Please test and submit bug reports. Thank you, Alex. On Tue, 2007-01-30 at 14:33 -0700, Alex Rousskov wrote: Folks, Executive summary: I am trying to fix several bugs and complaints related to your favorite class, the BodyReader. The changes I would like to make are significant, so I decided to post them here first. Please see the attached BodyPipe.h sketch. I will proceed with these changes unless there are violent objections or better ideas. The email below documents my concerns and explains the rationale behind BodyPipe. Current state: The BodyReader class does not seem to read a body. It helps to move the body content from the body producer to the consumer. The class is essentially a collection of loosely coupled function pointers with protected and bare data attached to them. The class calls consumer-side functions on behalf of the producer (or vice versa), while updating the body size state. BodyReader refcounting was meant to communicate abort conditions. Here are a few related/intermingled reasons behind BodyPipe, the replacement design I am implementing (see the attached sketch). A) The reason I want to get away from the current lose ends approach with function pointers is to provide a more clear, easier to follow implementation (IMO). I want classes like ConnStateData or ICAPClientReqmodPrecache to inherit from BodyProducer and just implement the two or three required methods. No wrappers or bare data pointers at the high level, where all the changes and enhancements are taking place. B) The reason I want explicit provider and consumer pointers inside BodyPipe is to be able to abort either side if the other side is having trouble. The current idea of using refcounting to abort something does not and cannot work (if there are two refcounted pointers, the second pointer will not know that the first has been set to NULL). I also want joint ownership of the BodyReader so that the producer and the consumer know that they must clear/detach themselves before forgetting about the pipe. Refcounting will just help to destroy the abandoned pipe. C1) The reason I want asynchronous calls is to separate producer from consumer and break the call-me-back-while-I-am-calling-you loops. I believe that at least the abortedSize assertion (bug #1862) is caused, in part, by such loops. C2) Another reason for separating producer and consumer is to allow the producer to disconnect when done, leaving a functional object (with the remaining/last body bytes) behind. This situation happens when, for example, an ICAP transaction is finished but the HttpStateData is still writing the body to the origin server. Again, please take a look at the attached sketch. I have seen this before: While sketching the design, I realized that what I want is very similar to MsgPipe, a class used by ICAP-related code to shovel HTTP messages between Squid core and an ICAP transaction. If you look at MsgPipe.cc and ignore the low-level debugging macros, you will see that there is virtually nothing there. The class simply converts notifications from one pipe end to events and then calls the other end when the event fires. I do not plan on reusing MsgPipe itself because its simple design does not allow one object to be the end of two pipes -- there would be no way to know which pipe the messages are coming from. I do not want to complicate MsgPipe and its users to support multi-pipe ends. I think it would be better to have a very similar but distinct BodyPipe class, with method names specific to the problem at hand and extra code to handle body size calculations. The design duplication will be there, but code duplication should be negligible. I am worried that I see every problem as a nail though, which is another motivation for this message. Again, if you have better ideas on how to fix BodyReader-related problems, please stop me and contribute! Thank you, Alex.
Squid3 ICAP references
Hello, Could somebody update the following pages to add references to built-in ICAP support in Squid3? http://devel.squid-cache.org/projects.html#icap http://devel.squid-cache.org/icap/ Thank you, Alex.
Re: Squid3 BodyReader changes
On Wed, 2007-02-14 at 21:17 +0200, Tsantilas Christos wrote: Hi Alex, I try some test's. It is not so bad, but I saw that there are problems. After surfing the web for a while the icap-client of squid failed and the squid returned an error page to the web browser (ICAP protocoll error ICAP server is not reachable ..) The icap client sends an respmod request with preview data and after the the server responds with 100 Continue the squid closes the connection. I have the same error on some sites (e.g freshmeat.net) using c-icap icap server. I compiled squid using gcc 4.1.2. I did not have enough time to look more on it. I will try to find time next days for tests and debuging. Nice logs, thank you. It looks like Squid is sending a 43 byte preview of a 43 byte body and does not expect 100 Continue because there is nothing to continue with. I suspect Squid should have sent ieof but I need to read the RFC and to think about it. We can also make Squid more robust and send a last-chunk even if the ICAP server should not have sent 100 Continue. Thank you, Alex. My squid logs are: 2007/02/14 20:21:19.587| ICAPModXact should offer 43-byte preview (service wanted 1024) 2007/02/14 20:21:19.587| ICAPModXact ICAP will write [FD 13r;Rrw(1)P(43)B/ icapx338]: RESPMOD icap://192.168.1.4:1344/srv_clamav ICAP/1.0 Host: 192.168.1.4:1344 Date: Wed, 14 Feb 2007 18:21:19 GMT Encapsulated: req-hdr=0, res-hdr=597, res-body=821 Preview: 43 Allow: 204 GET http://ads.in.gr/adlog.php?bannerid=815clientid=224zoneid=119source=251block=0capping=0cb=993fc456072f1bc18ea9ab8de58b1da9 HTTP/1.1 Host: ads.in.gr User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20061023 SUSE/2.0-30 Firefox/2.0 Accept: image/png,*/*;q=0.5 Accept-Language: el,en-us;q=0.7,en;q=0.3 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-7,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Proxy-Connection: keep-alive Referer: http://www.in.gr/news/category.asp?lngDtrID=251 Via: 1.0 fortune.home.chtsanti.net (C-ICAP/120207 Clamav/Antivirus service ) HTTP/1.1 200 OK Date: Wed, 14 Feb 2007 18:19:51 GMT Server: Apache X-Powered-By: PHP/4.4.4 Pragma: no-cache Cache-Control: private, max-age=0, no-cache Content-Length: 43 Connection: close Content-Type: image/gif 2007/02/14 20:21:19.587| ICAPModXact::noteCommConnected ended [FD 13wr;Rrw(2)P(43)B/ icapx338] 2007/02/14 20:21:19.588| ICAPModXact::noteCommWrote called [FD 13wr;Rrw(2)P(43)B/ icapx338] 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(149) Wrote 1014 bytes 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(177) checking whether to write more [FD 13r;Rrw(3)P(43)B/ icapx338] 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(216) will write Preview body from 0x8525f40*2 [FD 13r;Rrw(3)P(43)B/ icapx338] 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(255) will write up to 43 bytes of preview body 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(266) will write 43-byte chunk of preview body 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(278) will write last-chunk of preview body 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(283) will write 54 raw bytes of preview body 2007/02/14 20:21:19.588| ICAPModXact wrote entire Preview body [FD 13wr;Rrw(3)B/P icapx338] 2007/02/14 20:21:19.588| ICAPModXact::noteCommWrote ended [FD 13wr;Rrw(4)B/P icapx338] 2007/02/14 20:21:19.588| ICAPModXact::noteCommWrote called [FD 13wr;Rrw(4)B/P icapx338] 2007/02/14 20:21:19.589| ICAP/ICAPModXact.cc(149) Wrote 54 bytes 2007/02/14 20:21:19.589| ICAP/ICAPModXact.cc(177) checking whether to write more [FD 13r;Rrw(4)B/P icapx338] 2007/02/14 20:21:19.589| ICAPModXact::noteCommWrote ended [FD 13r;Rrw(4)B/P icapx338] 2007/02/14 20:21:19.599| ICAP/ICAPXaction.cc(58) 0x84d8ac8 read returned 25 2007/02/14 20:21:19.599| ICAPModXact::noteCommRead called [FD 13r;Rrw(4)B/P icapx338] 2007/02/14 20:21:19.599| ICAP/ICAPXaction.cc(328) read 25 bytes 2007/02/14 20:21:19.599| ICAP/ICAPModXact.cc(548) have 25 bytes to parse [FD 13;Rrw(4)B/P icapx338] 2007/02/14 20:21:19.599| ICAP/ICAPModXact.cc(549) ICAP/1.0 100 Continue 2007/02/14 20:21:19.599| ICAP/ICAPModXact.cc(577) parse ICAP headers 2007/02/14 20:21:19.599| ICAP/ICAPModXact.cc(780) have 25 head bytes to parse; state: 0 2007/02/14 20:21:19.600| HttpMsg::parse success (25 bytes) near 'ICAP/1.0 100 Continue ' 2007/02/14 20:21:19.600| ICAP/ICAPModXact.cc(792) parse success, consume 25 bytes, return true 2007/02/14 20:21:19.600| ICAP/ICAPModXact.cc(177) checking whether to write more [FD 13;Rrw(5)B/P icapx338] 2007/02/14 20:21:19.600| ICAPModXact::noteCommRead caught an exception: virginWriteClaim.active() [FD 13;Rrw(5)B/P icapx338] 2007/02/14 20:21:19.600| ICAPModXact will stop, reason: exception 2007/02/14 20:21:19.600| ICAPModXact::noteCommRead ends xaction [FD 13;Rrw(5)B/StoppedP icapx338] 2007/02/14 20:21:19.600| ICAP/ICAPModXact.cc(943) swan sings [FD 13;Rrw(5)B/StoppedP icapx338]
Re: Squid3 BodyReader changes
On Wed, 2007-02-14 at 13:19 -0700, Alex Rousskov wrote: On Wed, 2007-02-14 at 21:17 +0200, Tsantilas Christos wrote: Hi Alex, I try some test's. It is not so bad, but I saw that there are problems. After surfing the web for a while the icap-client of squid failed and the squid returned an error page to the web browser (ICAP protocoll error ICAP server is not reachable ..) The icap client sends an respmod request with preview data and after the the server responds with 100 Continue the squid closes the connection. I have the same error on some sites (e.g freshmeat.net) using c-icap icap server. I compiled squid using gcc 4.1.2. I did not have enough time to look more on it. I will try to find time next days for tests and debuging. Nice logs, thank you. It looks like Squid is sending a 43 byte preview of a 43 byte body and does not expect 100 Continue because there is nothing to continue with. I suspect Squid should have sent ieof but I need to read the RFC and to think about it. My suspicion was correct. The untested fix has been committed to squid3-icap. Squid should now append ieof extension to the last chunk whenever Preview contains the entire body. Please try again. Thank you, Alex. My squid logs are: 2007/02/14 20:21:19.587| ICAPModXact should offer 43-byte preview (service wanted 1024) 2007/02/14 20:21:19.587| ICAPModXact ICAP will write [FD 13r;Rrw(1)P(43)B/ icapx338]: RESPMOD icap://192.168.1.4:1344/srv_clamav ICAP/1.0 Host: 192.168.1.4:1344 Date: Wed, 14 Feb 2007 18:21:19 GMT Encapsulated: req-hdr=0, res-hdr=597, res-body=821 Preview: 43 Allow: 204 GET http://ads.in.gr/adlog.php?bannerid=815clientid=224zoneid=119source=251block=0capping=0cb=993fc456072f1bc18ea9ab8de58b1da9 HTTP/1.1 Host: ads.in.gr User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20061023 SUSE/2.0-30 Firefox/2.0 Accept: image/png,*/*;q=0.5 Accept-Language: el,en-us;q=0.7,en;q=0.3 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-7,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Proxy-Connection: keep-alive Referer: http://www.in.gr/news/category.asp?lngDtrID=251 Via: 1.0 fortune.home.chtsanti.net (C-ICAP/120207 Clamav/Antivirus service ) HTTP/1.1 200 OK Date: Wed, 14 Feb 2007 18:19:51 GMT Server: Apache X-Powered-By: PHP/4.4.4 Pragma: no-cache Cache-Control: private, max-age=0, no-cache Content-Length: 43 Connection: close Content-Type: image/gif 2007/02/14 20:21:19.587| ICAPModXact::noteCommConnected ended [FD 13wr;Rrw(2)P(43)B/ icapx338] 2007/02/14 20:21:19.588| ICAPModXact::noteCommWrote called [FD 13wr;Rrw(2)P(43)B/ icapx338] 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(149) Wrote 1014 bytes 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(177) checking whether to write more [FD 13r;Rrw(3)P(43)B/ icapx338] 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(216) will write Preview body from 0x8525f40*2 [FD 13r;Rrw(3)P(43)B/ icapx338] 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(255) will write up to 43 bytes of preview body 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(266) will write 43-byte chunk of preview body 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(278) will write last-chunk of preview body 2007/02/14 20:21:19.588| ICAP/ICAPModXact.cc(283) will write 54 raw bytes of preview body 2007/02/14 20:21:19.588| ICAPModXact wrote entire Preview body [FD 13wr;Rrw(3)B/P icapx338] 2007/02/14 20:21:19.588| ICAPModXact::noteCommWrote ended [FD 13wr;Rrw(4)B/P icapx338] 2007/02/14 20:21:19.588| ICAPModXact::noteCommWrote called [FD 13wr;Rrw(4)B/P icapx338] 2007/02/14 20:21:19.589| ICAP/ICAPModXact.cc(149) Wrote 54 bytes 2007/02/14 20:21:19.589| ICAP/ICAPModXact.cc(177) checking whether to write more [FD 13r;Rrw(4)B/P icapx338] 2007/02/14 20:21:19.589| ICAPModXact::noteCommWrote ended [FD 13r;Rrw(4)B/P icapx338] 2007/02/14 20:21:19.599| ICAP/ICAPXaction.cc(58) 0x84d8ac8 read returned 25 2007/02/14 20:21:19.599| ICAPModXact::noteCommRead called [FD 13r;Rrw(4)B/P icapx338] 2007/02/14 20:21:19.599| ICAP/ICAPXaction.cc(328) read 25 bytes 2007/02/14 20:21:19.599| ICAP/ICAPModXact.cc(548) have 25 bytes to parse [FD 13;Rrw(4)B/P icapx338] 2007/02/14 20:21:19.599| ICAP/ICAPModXact.cc(549) ICAP/1.0 100 Continue 2007/02/14 20:21:19.599| ICAP/ICAPModXact.cc(577) parse ICAP headers 2007/02/14 20:21:19.599| ICAP/ICAPModXact.cc(780) have 25 head bytes to parse; state: 0 2007/02/14 20:21:19.600| HttpMsg::parse success (25 bytes) near 'ICAP/1.0 100 Continue ' 2007/02/14 20:21:19.600| ICAP/ICAPModXact.cc(792) parse success, consume 25 bytes, return true 2007/02/14 20:21:19.600| ICAP/ICAPModXact.cc(177) checking whether to write more [FD 13;Rrw(5)B/P icapx338] 2007/02/14 20:21:19.600| ICAPModXact::noteCommRead caught an exception: virginWriteClaim.active() [FD 13;Rrw(5)B/P icapx338] 2007/02/14 20:21:19.600
Re: Squid3 BodyReader changes
On Sat, 2007-02-17 at 19:00 +0200, Tsantilas Christos wrote: Still exist problems in preview transaction between icap server and squid. There are cases in which the squid-icap does not send the 0; ieof sequence if all the http response body fits in the preview data but sends the 0\r\n\r\n. After that gets the 100 continue icap response but it stops the icap transaction becouse it fails in ICAPModXact::handle100Continue function in the Must exception: Must(state.writing == State::writingPaused); The state.writing here is equal to State::writingPreview According to your trace, Squid actually sent two last-chunks for the Preview, but it is probably the same bug. The following solves the problem for me (but maybe makes other problems). It is in function moveRequestChunk in file ICAPModXacct.cc: diff -u -r1.1.2.21 ICAPModXact.cc --- ICAPModXact.cc 14 Feb 2007 22:52:30 - 1.1.2.21 +++ ICAPModXact.cc 17 Feb 2007 16:32:40 - @@ -304,7 +304,8 @@ // if there is no active virginWriteClaim, // then there is no body left to write, // so we must have written everything -const bool wroteEof = !virginWriteClaim.active(); +const bool wroteEof = !virginWriteClaim.active() || +(virgin.body_pipe-productionEnded() claimSize(virginWriteClaim) = 0); preview.wrote(chunkSize, wroteEof); // even if wrote nothing } } The above change makes sense. The new condition is actually identical to calling the doneWithClaim() method. While reviewing your log and patch, I also noticed that the claimSize() method does not take into account that the buffer claim may be limited by the preview size. IMO, the right thing to do is to rename claimSize to match its code. Moreover, I believe I have over-engineered the entire claim concept. We do not need to maintain the size of the claim. Knowing the writing and sending offsets is sufficient! Finally, I suspect we should not abandon backup commitments when receiving a 100 Continue response to our Preview if we promised to allow 204s outside of Preview. I am streamlining and polishing related code and will commit all changes soon. BTW, do you use manual testing for this or do you have an automated collection of test cases? Thank you, Alex. I am also sending my squid log data (sorry for the long mail): 2007/02/17 18:45:27.929| ICAP/ICAPModXact.cc(377) consuming 195 out of 195 virgin body bytes 2007/02/17 18:45:27.929| ICAPModXact will not start sending [FD 12007/02/17 18:45:27.919| ICAPModXact should offer 1024-byte preview (service wanted 1024) 2007/02/17 18:45:27.919| ICAPModXact ICAP will write [FD 13r;Rrw(1)P(1024)B/ icapx6]: RESPMOD icap://192.168.1.4:1344/srv_clamav ICAP/1.0 Host: 192.168.1.4:1344 Date: Sat, 17 Feb 2007 16:45:27 GMT Encapsulated: req-hdr=0, res-hdr=518, res-body=883 Preview: 1024 GET http://www.google.com/ HTTP/1.1 Host: www.google.com User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20061023 SUSE/2.0-30 Firefox/2.0 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Language: el,en-us;q=0.7,en;q=0.3 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-7,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Proxy-Connection: keep-alive Via: 1.0 fortune.home.chtsanti.net (C-ICAP/120207 Clamav/Antivirus service ) HTTP/1.0 302 Moved Temporarily Location: http://www.google.gr/ Cache-Control: private Set-Cookie: PREF=ID=3bd23962963cb32c:TM=1171730638:LM=1171730638:S=cwcpju2ceoF4GUEV; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com Content-Type: text/html Server: GWS/2.1 Content-Encoding: gzip Date: Sat, 17 Feb 2007 16:43:58 GMT Connection: Close 2007/02/17 18:45:27.919| ICAPModXact::noteCommConnected ended [FD 13wr;Rrw(2)P(1024)B/ icapx6] 2007/02/17 18:45:27.920| ICAPModXact::noteCommWrote called [FD 13wr;Rrw(2)P(1024)B/ icapx6] 2007/02/17 18:45:27.920| ICAP/ICAPModXact.cc(149) Wrote 1066 bytes 2007/02/17 18:45:27.920| ICAP/ICAPModXact.cc(177) checking whether to write more [FD 13r;Rrw(3)P(1024)B/ icapx6] 2007/02/17 18:45:27.920| ICAP/ICAPModXact.cc(216) will write Preview body from 0x850dc00*2 [FD 13r;Rrw(3)P(1024)B/ icapx6] 2007/02/17 18:45:27.920| ICAP/ICAPModXact.cc(255) will write up to 195 bytes of preview body 2007/02/17 18:45:27.920| ICAP/ICAPModXact.cc(266) will write 195-byte chunk of preview body 2007/02/17 18:45:27.920| ICAP/ICAPModXact.cc(278) will write last-chunk of preview body 2007/02/17 18:45:27.920| ICAP/ICAPModXact.cc(283) will write 206 raw bytes of preview body 2007/02/17 18:45:27.920| ICAPModXact::noteCommWrote ended [FD 13wr;Rrw(3)P(829)B/ icapx6] 2007/02/17 18:45:27.920| ICAPModXact::noteCommWrote called [FD 13wr;Rrw(3)P(829)B/ icapx6] 2007/02/17 18:45:27.920| ICAP/ICAPModXact.cc(149) Wrote 206 bytes 2007/02/17 18:45:27.920| ICAP/ICAPModXact.cc(177) checking whether to write
Re: content adaptation
On Fri, 2007-02-16 at 21:45 +0330, MPZ World (Masoud Pour Zahmatkesh) wrote: i want when squid show website to my client add a customize html code to top of pages,only web pages. i do this task with redirect all urls to a program example phprpoxy or cgiproxy but this programs doesn't work with all websites. i think if i can change squid source code for add this HTML code ,it is the best way please help me ,which source file do this work and i must change it Many folks have content adaptation needs similar to yours. Please use the following recent message for pointers to implementation hints http://www.squid-cache.org/mail-archive/squid-dev/200702/0025.html Thank you, Alex.
locked out by partial cvsmerge
Hi there, I was running 'cvsmerge HEAD' from squid3-icap branch and got distracted. Meanwhile the internet connection went down. I am not sure where cvsmerge stopped, but it probably did not complete. If I try it again, I get: cvs rtag: [06:10:02] waiting for rousskov's lock in /cvsroot/squid/squid3/lib/cppunit-1.10.0/doc I tried to break my own lock, but, to my surprise, I get the same kind of result from cvs admin -u. It sounds like to break my lock, I need to obtain it first, which does not make sense to me: cvs admin: [23:22:56] waiting for rousskov's lock in /cvsroot/squid/squid3/lib/cppunit-1.10.0/doc FWIW, it is always the same directory that CVS gets stuck on. Do we have command-line access to SourceForge CVS so that the lock file can be manually removed? What should I do to unlock myself? Thank you, Alex.
Re: locked out by partial cvsmerge
On Sat, 2007-02-24 at 15:33 +1300, Amos Jeffries wrote: Adrian Chadd wrote: On Fri, Feb 23, 2007, Alex Rousskov wrote: Do we have command-line access to SourceForge CVS so that the lock file can be manually removed? What should I do to unlock myself? Shell.sourceforge.net is currently busted for some reason. I wonder whats going on. oh, grrr. It looks like the rest of us are also blocked by that lock. I just tried to merge the latest changes down into my branch and it gives me the same warning/error. cvs rtag: [02:21:29] waiting for rousskov's lock in /cvsroot/squid/squid3/lib/cppunit-1.10.0/doc I had to abort after a 30sec loop displaying that error for 10 minutes. Ouch. FWIW, I can get to shell.sourceforge.net, but I do not want to create more problems experimenting and probably do not have write access to the right CVS files anyway. Let's hope somebody will fix this soon. Sorry, Alex.
Re: squid3 comments
On Tue, 2007-02-27 at 13:27 +0200, Tsantilas Christos wrote: In the other hand I need a proxy with an icap client because I spent time (and continue spending) to an icap related project. Squid3 has a good icap client. The first problem someones can see in squid3 is that there are some parts derived from c-code, which are not (fully) converted to real c++ code. The second is a feeling that some parts of code are half-completed. I am thinking that maybe it is good practice for someone to start fixing this things first I agree that many Squid3 parts should be fixed, polished, or thrown away. However, I think that we should focus on making Squid3 stable first, and the performance/polishing work you are discussing should be done for v3.1. There are plenty of users who can use Squid3 that is stable but not very fast. An alternate for me is to try fix the bugs and rewrite some of the icap-related parts of the squid26-icap branch. I don't know This would be a bad idea from my biased point of view. While the code migration to Squid3 was poorly done, we are already at the point where we can make Squid3 work for your purposes instead of going back. Please do not forget that Squid2 has its own problems; it is not like you will be migrating to a great code that just needs a yet another ICAP patch! Alex.
Re: squid3 comments
On Tue, 2007-02-27 at 22:25 +0800, Adrian Chadd wrote: On Tue, Feb 27, 2007, Jeremy Hall wrote: Let me second this. When you start asking questions about squid3 and its stability, you get anything from it's stable to not for prime time and when you ask questions about using it in a production environment, most shy away from that. * noone's really stepped up to drag Squid-3 up to production quality. The bugs are relatively well-known and the issues with the codebase show up in bugzilla. I am working on dragging Squid3 to production quality and fixing bugs that are present in my environment. There are other folks doing that as well. Please do not try to persuade people to help you with your Squid2 projects by attacking Squid3. * People seem to think we can keep adding functionality without fixing the Squid core. Which is a mess, and in my opinion, needs to be fixed first. I agree. Personally, I am against adding new features to Squid 3.0. We need to spend time fixing the Squid internals and getting all of that fast, flexible and rock stable so stuff like ICAP can be implemented better. Agreed. I wish you could work on Squid3 internals instead of Squid2, but it is your call and I respect your choice. Let's just not assume that all work is done or should be done on Squid2. Thank you, Alex. P.S. FWIW, ICAP support is already pretty good in Squid3, regardless of the internals (that are getting better as well).
Re: squid3 comments
On Tue, 2007-02-27 at 23:00 +0800, Adrian Chadd wrote: On Tue, Feb 27, 2007, Jeremy Hall wrote: So are you saying those of us that need icap need to just wait? There's two parts. One: Alex is working on improvements to the ICAP code in Squid-3 which I hope will act as a kind of reference implementation to use in the future. Help him out any way you can. Two: I think the storework branch is one step along the right path to fixing up the codebase in the long term. The two things I need to sort it out is testing of the branch and some simple(!)ish test suite to implement tests of the client-side to make sure stuff isn't regressed as development continues. So some help dreaming up and coding up some test utilities to act as a test suite of the client side, along with some actual testing. No, its nowhere near ready to even sniff production traffic. :) The point of my Squid3/ICAP work is to make Squid3 usable in a content adaptation environment. I do not care much if ICAP code becomes a reference implementation or not. There are many parts. Some work on fixing Squid2 bugs, some work on optimizing Squid2, some work on fixing Squid3 bugs, some work on Windows port, etc. If people want to help optimizing Squid2, it is their choice, of course. Perhaps the storework branch will become a reference implementation to speedup Squid3 when the time comes :-). I would prefer that folks spent more energy on Squid 3.0 to make it stable faster (and only then optimize Squid 3.1), but I am not going to call every project that is not helping me dead. Alex.
Re: locked out by partial cvsmerge
On Sat, 2007-02-24 at 08:43 +0100, Henrik Nordstrom wrote: These locks gets cleared automatically after a while and the lock is now gone. Thanks for the explanation and snipped details. Unfortunately, it looks like the problem is back and has even become worse as I see more locks (from other developers) getting stuck now: cvs rtag: [21:28:08] waiting for rousskov's lock in /cvsroot/squid/squid3/lib/cppunit-1.10.0/doc cvs rtag: [21:28:38] waiting for serassio's lock in /cvsroot/squid/squid3/lib/cppunit-1.10.0/doc cvs rtag: [21:29:08] waiting for amosjeffries's lock in /cvsroot/squid/squid3/lib/cppunit-1.10.0/doc I am unable to merge with HEAD (using cvsmerge) because of this. Do you know whether there is something going on with SourceForge that increases the probability of these locks getting stuck? Anything we can do about it in the short term? Thank you, Alex.
Re: Squid3 BodyReader changes
On Tue, 2007-02-20 at 12:22 +0200, Tsantilas Christos wrote: So yes I believe it is the same bug, but I thing the problem begins because the state.writing is not updated correctly ... Right, but the state is not updated correctly because of the bug you have fixed. I have committed your change, simplified virgin body buffer maintenance (in hope to minimize the number of similar bugs), and probably fixed a bug with handling of post-preview 204 replies. The above was done many days ago, but I also wanted to sync with HEAD before asking you to test again. Since I am not able to do that because of stale CVS locks, I suggest that you try the squid3-icap branch before I sync with HEAD. Thank you, Alex.
Re: squid3 comments
On Wed, 2007-02-28 at 09:19 -0500, Jeremy Hall wrote: Is squid3 faster or slower than squid2? I will be doing performance tests with Squid3 shortly. Will post. Alex.
Re: squid3 comments
On Wed, 2007-02-28 at 20:48 +0200, Tsantilas Christos wrote: As an example of such changes I am attaching the rewritten parseHttpRequest, prepareTransparentURL and prepareAcceleratedURL A second example: again In parseHttpRequest we have the HttpParser struct which we are using it to parse the first line of request. The HttpRequest::parseFirstLine method (of HttpMsg derived HttpRequest class) does more or less the same. Maybe they can merged. A third example: in HttpStateData::processReplyHeader int http.cc file. I am seeing the line: const bool parsed = newrep-parse(readBuf, eof, error); and some lines after I am seeing: header_bytes_read = headersEnd(readBuf-content(), readBuf-contentSize()); What the hell parsing we did in previous line if we did not keep the end of headers? The easier we can do is to make HttpReply::parse to return the size of headers on success or zero else. Or just keep in an internall variable of HttpReply class the size of headers and then get them with a call like newrep-headersSize() I think such changes are small and can make squid3 to be a little faster and can simplify in some cases the code, which will help the debuging. I am not having a lot of time too but I can find some hours here or there to make such changes, if needed. If you are absolutely, positively, 100% sure that these changes are correct then it is only a question whether they will help us make Squid3 stable faster (by fixing bugs, improving the code, debugging flow, or whatever). If they will, we should implement them. My concern is that in many cases, an innocent-looking parsing function has a couple of well-hidden side effects. Removing a function call or changing calls order introduces subtle bugs. I have been bitten by this many times. I think we should start from stating the goal of these changes in Squid 3.0. If they are for performance improvement, I would suggest waiting until v3.1 or until performance tests indicate that we must improve Squid 3 performance before calling it stable. If the goal is to fix a common-case bug, we should make the necessary changes, of course. If the ultimate goal is different, let's discuss it. Thank you, Alex. P.S. I will try to look at your specific changes soon.
Re: locked out by partial cvsmerge
On Wed, 2007-02-28 at 10:35 +0100, Henrik Nordstrom wrote:
It might actually be the cvsmerge script.. the following line isn't
exactly kind to CVS..
o Check if there is any pending changes in the repository
diffl=`eecvs -q rdiff -kk -r ${mergetag} -r ${mergefrom} ${module} | head |
wc -l`
probably should remove the head from there, letting the rdiff run to
completion.
FWIW, I was able to merge after removing 'head'. Removing 'head' may
have nothing to do with merge success, but what a relief!
To speed things up, how about using the -s option with rdiff instead of
the 'head' trick:
If you use the -s option, no patch output is produced. Instead,
a summary of the changed or added files between the two releases
is sent to the standard output device. This is useful for find-
ing out, for example, which files have changed between two dates
or revisions.
Is it portable? Seems to be available on FreeBSD and Linux.
Thanks a lot,
Alex.
thank you for squid3 work
On Thu, 2007-03-01 at 21:51 +0100, Guido Serassio wrote: I'm very honored for this, but the results of my work on Squid 3 was very far from my expectations: my C++ knowledge is horribly low, and there was many bugs that I cannot resolve myself for this reason. I have always hoped than some other developer with a more strong C++ knowledge will try to fix them, but this was never happened: all Squid 3 peoples seems to like more the development of new features instead of fixing bugs. As example: I have arranged the TPROXY forward port done by Steven in November, but no one of the Squid 3 supporters seems to have tested it giving any kind of feedback, a little disappointing ... :-( So, sometimes I ask to me why I'm still wasting my time on Squid 3 ?. Guido and Henrik, Thank you very much for all your Squid3 efforts! Without your work Squid3 would not happen. I hope you feel the increased interest in and work on Squid3 these days. FWIW, I am very interested in fixing core bugs. For example, I rewrote request body handling to fix several core bugs (changes are currently in squid3-icap). With the exception of ICAP, I cannot spend time polishing optional features like TPROXY, but I am sure there are enough core bugs for me :-). Also, with the ICAP code becoming stable, I should be able to spend more time on core fixes. As for C++, if a willing person cannot understand some C++ concept in Squid, it should probably be removed or at least well documented. Personally, I am against hairy C++ code that only gurus can grok. For me, most of the C++ complexity in Squid3 comes from an unfortunate blend of C features like cbdata and C++ features like refcounting. Hopefully, we will fix that in Squid 3.1. Thank you, Alex.
Re: Question ICAP-client
On Wed, 2007-03-07 at 23:57 +0200, Tsantilas Christos wrote:
When an http request adapted using ICAP then the client and server
addresses and the authentication information does not copied to adapted
request.
This is will cause problems in any following access control lists
proccessing.
Should this copying be configurable? In some environments, an adapted
request is not a request authorized by the user. Can we always trust
the ICAP server to essentially impersonate the user?
I have no problems with committing a fix that always copies this
information as a temporary measure, but I think we need to at least put
some comments in the code that we are copying user-specific information
to a request that did not originate from the user.
In the ICAP 204 case, we should probably always copy (and eventually we
will be able simply reuse the same request object). For ICAP 200, we
should be more careful and at least put a TODO comment. If we want to
distinguish these two cases, then ICAPModXact should do the copying and
not client_side.
If you want to move the copying code to ICAPModXact, you can call it
from handle204NoContent() unconditionally and perhaps from
parseHttpHead() with a TODO: make this copying configurable comment.
I would be happy to commit this code one you think it is ready.
HTH,
Alex.
Looks that the following patch solves the problem. (But I am to tired
now for a good testing, tomorrow .)
Regards,
Christos
diff -u -r1.34.4.24 client_side_request.cc
--- client_side_request.cc 14 Feb 2007 07:19:43 - 1.34.4.24
+++ client_side_request.cc 7 Mar 2007 21:45:17 -
@@ -1112,6 +1112,15 @@
assert(msg);
if (HttpRequest *new_req = dynamic_castHttpRequest*(msg)) {
+ new_req-client_addr = request-client_addr;
+new_req-client_port = request-client_port;
+new_req-my_addr = request-my_addr;
+new_req-my_port = request-my_port;
+new_req-flags = request-flags;
+ if (request-auth_user_request) {
+new_req-auth_user_request = request-auth_user_request;
+new_req-auth_user_request-lock();
+}
/*
* Replace the old request with the new request.
*/
Re: redirection
On Fri, 2007-03-09 at 17:35 +0530, Siddhesh PaiRaikar wrote: while finding on ICAP servers i came accross this : there is a squid web proxy as an ICAP client available. in the above response where we are told to use the ICAP server, will it be helpful in any way to use this squid client in any way .. since we already have squid on our machines... or do we scrap squid altogether and use the ICAP. In that case i would need some suggestions on involving squid in this in some way. To use ICAP, one needs an ICAP client (usually an ICAP-capable HTTP proxy like Squid) and an ICAP server. Thus, if you decide to end the suffering of in-Squid content modification and use ICAP, you will still need Squid or a different ICAP-capable proxy. The point of adding an ICAP server into the mix is to avoid modifying or customizing the proxy (and to allow your solution to work with any ICAP-capable proxy). All your business logic will be in the ICAP server. For more details, you may want to read www.i-cap.org documents regarding the ICAP infrastructure. You do not have to write an ICAP server from scratch. There are two or three GPLed ICAP servers available. including the old reference implementation from NetApp, linked from the i-cap.org site. You can customize them. plugIf you want more support and less work, The Measurement Factory builds ICAP servers to suit and also provides Traffic Spicer, the bare-bones ICAP server where you can plug in your redirection logic./plug FWIW, I know of three primary ways to do ICAP-based redirection: 1) The ICAP server generates a 302 redirection response. The browser then goes to the new origin server. This is simple and the browser does most of the work. However, you need an origin server and you need to coordinate ICAP server and your origin server logic. The whole transaction takes longer. You need to be careful about loops. 2) The ICAP server rewrites the virgin request to go to a different server. This approach is similar to (1) but eliminates network delays. Its additional drawback is that it only works if you can make the redirection decision based on the request: When you receive a virgin response, it is impossible to rewrite that into a request to a different server. 3) The ICAP server can generate the final response. This approach eliminates the network delays like (2) and also allows you to have all processing logic in one place. On the other hand, this approach usually requires that the ICAP server becomes an HTTP server of sorts because the final response is likely to contain embedded images and similar objects. Thus, this approach works best when the final response is very simple and stateless. In general, I would recommend starting with (1). HTH, Alex.
stuff in the Squid3 pipeline
On Sat, 2006-05-06 at 12:08 +1200, Doug Dixon wrote: And as Henrik says, if you've got new stuff in the pipeline, please shout now. We just need to agree which PRE it goes into - whether 4 or later. The BodyPipe-related changes that currently live in squid3-icap branch are pretty much ready for being committed to HEAD. Those changes affect Squid core (as well as ICAP) and are meant to fix several POST-related bugs in Squid3. I hope these changes will be copied to HEAD in 14 days or so. Thank you, Alex.
Re: stuff in the Squid3 pipeline
On Sun, 2007-03-18 at 21:32 -0700, Alex Rousskov wrote: On Sat, 2006-05-06 at 12:08 +1200, Doug Dixon wrote: And as Henrik says, if you've got new stuff in the pipeline, please shout now. We just need to agree which PRE it goes into - whether 4 or later. The BodyPipe-related changes that currently live in squid3-icap branch are pretty much ready for being committed to HEAD. Those changes affect Squid core (as well as ICAP) and are meant to fix several POST-related bugs in Squid3. I hope these changes will be copied to HEAD in 14 days or so. Ha! Replying to messages sorted by subject can be humiliating: I did not realize the above thread is almost a year old until I sent the carefully crafted response :-). I should have listened to the inner voice being surprised by the low Squid PRE number being discussed on that thread! Sorry for the noise, Alex.
Re: stuff in the Squid3 pipeline
On Mon, 2007-03-19 at 17:01 +1200, [EMAIL PROTECTED] wrote: On Sun, 2007-03-18 at 21:32 -0700, Alex Rousskov wrote: On Sat, 2006-05-06 at 12:08 +1200, Doug Dixon wrote: And as Henrik says, if you've got new stuff in the pipeline, please shout now. We just need to agree which PRE it goes into - whether 4 or later. The BodyPipe-related changes that currently live in squid3-icap branch are pretty much ready for being committed to HEAD. Those changes affect Squid core (as well as ICAP) and are meant to fix several POST-related bugs in Squid3. I hope these changes will be copied to HEAD in 14 days or so. Ha! Replying to messages sorted by subject can be humiliating: I did not realize the above thread is almost a year old until I sent the carefully crafted response :-). I should have listened to the inner voice being surprised by the low Squid PRE number being discussed on that thread! Sorry for the noise, Alex. hehe, no worries. It's good news anyway ... I'm just wondering which POST-related errors are fixed by that upgrade I believe BodyPipe changes on squid3-icap branch fix all Squid3 PUT/POST-related errors in the bugzilla. Alex.
Re: help on a project
On Wed, 2007-03-14 at 15:43 +0200, Zeen Radwan wrote: My name is Zein Radwan, I'm developing a security project where i have to intercept the HTTP packets and apply further processing on them. in order to do that i downloaded SQUID source for windows, but the code is confusing, this is why i need to ask the following question: which class is responsible of forwarding the response to the client, because in this point i will intercept the response handle it with my class library and then forward it again to its destination. Many folks have content inspection or adaptation needs similar to yours. Please use the following recent message for pointers to implementation hints http://www.squid-cache.org/mail-archive/squid-dev/200702/0025.html Thank you, Alex.
Re: Requests, replies, HTTP messages.
On Sat, 2007-03-17 at 19:33 +0800, Adrian Chadd wrote: http://wiki.squid-cache.org/RequestQueues Just thinking about how to handle processing HTTP requests, replies and HTTP messages after my initial storework modifications. This stuff would lead itself to look like clientstreams but with all the messages abstracted out. It'd make it much easier to stuff processing in the middle. IIRC, we have discussed similar ideas at the dawn of Squid3. I still believe that the post the request and have the Request Resolution Engine orchestrate the next processing step design has its merits. However, do you really think that implementing such a huge change is now practical? Alex.
Re: Download time issue: Squid 2.6
On Tue, 2007-03-20 at 21:19 +0100, Henrik Nordstrom wrote: tis 2007-03-20 klockan 19:40 +0100 skrev Thomas-Martin Seck: I'd love to but I cannot easily distribute development snapshots via FreeBSD ports; Understood, so PRE releases is what you can use. I considered to chase Squid-3-PRE but since the interval between PRE releases is a bit long for a software in development, users would be permanently stuck with way outdated software and consistently be told to update when they report errors. The Squid-3 PRE release cycle matches the Squid-3 development quite well I would say.. If known good PRE-snapshots were released on a more regular time frame -- say once a month -- and be kept on the FTP/HTTP mirrors for some time I'd love to make them available for FreeBSD ports. The PRE releases is meant to serve this purpose. Just that we don't make new PRE releases unless there has been significant improvements since the last and that there is no known major blockers.. Squid-3 was known majorly broken for years, so no PRE releases was made.. and currently there is at least one blocker for PRE4 but with a patch pending. I don't think packaging PRE4 in ports when released is such a bad idea. But clearly labeled as a pre-release and not a stable supported release. I agree that it is a good idea to start tracking Squid3 PRE releases for FreeBSD ports (and such). Hopefully, the releases will correspond to major updates and will morph into a stable release in the foreseeable future. Also, for hot fixes of a PRE release, a FreeBSD port can contain patches. Many ports do that, of course. Thank you, Alex.
Re: Question ICAP-client
On Sat, 2007-03-10 at 16:00 +0200, Tsantilas Christos wrote: I think that client address/port and squid address/port must copied. They can not (and must not) changed by an ICAP server. The same about authentication information because referred to users authenticated on squid and this info must not changed by an ICAP server. rant We are looking at this from different angles. You perceive an ICAP server as an extension of a client: a client should have no problem with any adaptation of its request by such as friendly server. I am sure we will eventually see compromised or otherwise unfriendly ICAP servers that do nasty things. Such servers would love to do nasty things on behalf of a client, using client identity if possible. Thus, I have a problem with blindly copying sensitive client information into requests generated (originated from) the ICAP server. As an ICAP server developer and seller, I know that many proxy installations today use 3rd party ICAP servers, and many proxy admins have very little knowledge about what that ICAP server is actually doing. Add remote updates to ICAP server software, and you have a perfect location for a man-in-the-middle attack point. Not coping addresses and auth info, we are breaking the ACL checking in later steps (e.g in FwdState::fwdStart). From your point of view, the solution is to copy. From my point of view, this breakage is a sign that we should be extra careful with copying such information. That is, there may be a _good_ reason why ACLs do not just work for requests initiated by the ICAP server. /rant Also I think, there is not any problem with HttpRequest.flags. A small research about the flags show me that only the flags.redirected, flags.accelerated, flags.transparent, flags.internal, flags.internalclient are computed before the ICAP reqmod called. I do not think that any of this flags can modified by an ICAP server. The other flags set after the ICAP reqmod. (I kept a list with flags and section code they set) I was concerned about this too. Thank you for doing this research! To increase chances of preserving the above conclusion, I added a request_flags::cloneAdaptationImmune() method and a comment. Hopefully, when new flags are added, folks will read the comment and adjust cloneAdaptationImmune() if needed. I have no problems with committing a fix that always copies this information as a temporary measure, but I think we need to at least put some comments in the code that we are copying user-specific information to a request that did not originate from the user. I think it not so problem, the new request can appeared as originated by the same user and the same IP as the original request... Yes, the request will appear as if it was originated from the user. Whether that is a problem for requests that actually originated from the ICAP server depends on the environment. Christos, I have applied your patch, with minor modifications, to squid3-icap. Thanks a lot for writing the patch! Stefan, please test and file a bug report if something does not work right after this change. Alex.
Re: Question ICAP-client
On Wed, 2007-03-21 at 18:04 +0100, Stefan Bischof wrote: I don't see your point (probably I don't understood something). The ICAP-server already knows the clients username at this point, because of the REQMOD request. If the evil ICAP-server redirects the request to a evil HTTP-server, it wouldn't be a problem to send the username to the HTTP-server and then back to the ICAP-server via RESPMOD. This is not about transmitting a username to the ICAP server. It is about making Squid believe that the request that originated from the ICAP server is authenticated (with _client_ credentials). An ICAP server can do pretty much anything. However, whether the ICAP server can do something _while being authenticated as the client_ is up to the proxy. The current code grants such authenticated status to requests generated by the ICAP server. I agree that a client and an admin already have to trust the proxy. The question is whether the client is authenticating herself with the proxy under the assumption that her authenticated requests will not be altered substantially. For example, there may be a big difference between an anonymous employee surfing porn and a specific/authenticated employee surfing porn... The admin may have reasonable trust in Squid or another well-known proxy, but may not have the same level of trust in some obscure porn-blocking ICAP server being installed and maintained by a 3rd party. A complete solution to this problem is probably too big to implement in the foreseeable future. However, it would be nice to have a special mode in Squid where ICAP request alternations are not allowed (but request satisfaction and REQMOD are OK). This should take care of most problems in this context. Full support of such a safe no-request-alternation mode is not easy because many ICAP servers use ICAP 200 OK to return unmodified requests. However, we can limit the check to comparing request headers and ignoring POST/PUT issues. Another useful safe ICAP mode could prevent any adaptations (but allow blocking), with a how to detect a blocking message? problem. $0.02, Alex.
Re: Question ICAP-client
On Thu, 2007-03-22 at 16:26 +0100, Kinkie wrote: In this regard I see the ICAP server not to be any different from a proxy server, of which it is simply an extension. Whether the trust boundary includes both the proxy and the ICAP server depends on the setup. Being an extension is not always the same as being a trusted extension. And there may be several trust categories involved. I just fail to see any added security in not sending all the information that the proxy server has to the ICAP server. As I have tried to clarify, the problem we are discussing on this thread (and the problem that the now-committed patch works on) is _not_ about sending information to the ICAP server, but about treating requests generated by the ICAP server as if they were authenticated by the client. $0.02, Alex. P.S. Still, sending all the information that the proxy server has to the ICAP server is similar to sending all that information to another proxy server: Sometimes it is appropriate, sometimes it is not. The patch, however, does not affect what information is sent to the ICAP server.
Re: Download time issue: Squid 2.6
On Mon, 2007-03-26 at 22:35 +0200, Thomas-Martin Seck wrote: * Alex Rousskov ([EMAIL PROTECTED]): On Tue, 2007-03-20 at 21:19 +0100, Henrik Nordstrom wrote: The PRE releases is meant to serve this purpose. Just that we don't make new PRE releases unless there has been significant improvements since the last and that there is no known major blockers.. Squid-3 was known majorly broken for years, so no PRE releases was made.. and currently there is at least one blocker for PRE4 but with a patch pending. I don't think packaging PRE4 in ports when released is such a bad idea. But clearly labeled as a pre-release and not a stable supported release. I agree that it is a good idea to start tracking Squid3 PRE releases for FreeBSD ports (and such). Hopefully, the releases will correspond to major updates and will morph into a stable release in the foreseeable future. Also, for hot fixes of a PRE release, a FreeBSD port can contain patches. Many ports do that, of course. OK, my plan is to start tracking Squid 3 with PRE6. Are you OK if I keep it up to date by pulling in all changesets up to a certain date/changeset number (this would be roughly similar to what the FreeBSD vim port does) if needed? Needed would mean as indicated by developers on squid-users or -dev because critical issues had been fixed. Sounds like a good plan to me. I would not try to automate or even define any specific pull policy right now. Let's discuss what and how to pull when the time comes for such updates, after PRE6 is released. Thank you, Alex.
Re: Squid3 ICAP / PRE6
On Sun, 2007-04-01 at 16:02 +1200, Amos Jeffries wrote: You mentioned earlier that the ICAP integration into HEAD is going to be within the next few weeks. Exactly how close is it? I have a few bits of IPv6 I want to push up. But if you are very close I want to wait until your ICAP / PRE6 is released before I start changing HEAD on you. You are so courteous, thank you! You are supposed to commit as soon as possible before others will create conflicts for you. I will not be able to work on Squid until Monday, April 2. After that, getting ICAP changes committed to HEAD will be my top priority. Last time I checked, the changes were 95% ready: I need to review latest mailing list and bugzilla activity, commit a request satisfaction fix from Tsantilas Christos, and compile a change log entry. I hope to see ICAP changes in HEAD next week (by April 9th), but since I rely on others to do the actual HEAD commit, I cannot promise that. Thank you, Alex.
Re: direct responce from ICAP server in REQMOD
On Wed, 2007-03-28 at 21:03 +0300, Tsantilas Christos wrote: When an ICAP server responds with an http response in request modification, the web client waits more data from the proxy. Iam using the squid3-icap branch. An example of such ICAP response is: ICAP/1.0 200 OK Connection: keep-alive ISTag: CI0001-X Encapsulated: res-hdr=0, res-body=108 HTTP/1.0 403 Forbidden Content-Type: text/html Content-Language: en Connection: close 17 H1Permition deny!H1 0 The proxy server sends the correct data to the client, but does not close the connection after send all data. Adding the line storeEntry()-complete(); in ClientHttpRequest::endRequestSatisfaction method solves the problem but I think that the solution is not so simple, we need to know how to setup the StoreEntry... Nevertheless, I will commit your fix. Using the right sequences of entry-related calls has always eluded me, but calling complete() looks reasonable based on the existing code than handles real responses. If somebody knows whether any other entry-related things should be done to correctly finish an ICAP-made HTTP response, please chime in. Thank you, Alex.
Re: Squid3.0, ICAP server in python and respmod
On Tue, 2007-04-03 at 16:25 +0200, Jeremy Lardon wrote: Hey all, After some tests, I made my mind for squid3.0 and python based ICAP server. All works fine with reqmod precache but I cant' manage to get a response modification. There were many improvements in ICAP support on the squid3-icap CVS branch. Please consider running with that code or waiting for the next PRE release of Squid3 (which will include those ICAP improvements). If you find bugs in squid3-icap ICAP code, I will do my best to fix them. Thank you, Alex.
Re: squid3.0.PRE6 and bugzilla
On Tue, 2007-04-03 at 21:32 +0300, Tsantilas Christos wrote: Which is the goal for squid3.0.PRE6? Which bugs waiting a fix before released? Release recent improvements in a form convenient for testing and experimental deployment. I hate telling non-developers to use CVS. Can we assume that bug 1637 has a fix? http://www.squid-cache.org/bugs/show_bug.cgi?id=1637 I cannot tell. If I am not wrong most of the bugs causing crashes to squid fixed... I do not know of any reproducible crashes in Squid3 core or ICAP code, but there are a few known corner cases like squid -k shutdown. Alex.
Re: DEBUG Section for IPAddress
On Wed, 2007-04-04 at 13:02 +1200, Amos Jeffries wrote: Gah! I diffed the files backwards. Here is the actual patch for doc/debug-sections.txt Amos, The doc/mk-debugs.sh script you were using (or equivalent) needs to be fixed. It needs to grep files in subdirectories of src/. For example, it currently misses: DEBUG: section 93 ICAP (RFC 3507) Client Using 'find src -name *.cc -o -name *.h' instead of a src/*.cc mask may work. Alex.
Re: IPv6 developments for HEAD
On Sun, 2007-04-08 at 01:17 +1200, Amos Jeffries wrote: Guido Serassio wrote: You can try to run astyle on your sources, http://wiki.squid-cache.org/Squid3CodingGuidelines. Currently astyle formatting is not enforced on the Squid 3 HEAD CVS repository for some formatting side effects, but for a new file could be a good cosmetic thing to do before commit. Great, I had read the (old) requirement for indent 1.9. But have been unable to find a version earlier than 2.1 for my system. astyle at least is one I have. Last time I tried astyle, it created more problems than it solved. It may be OK for some simple files, but it made mane Squid sources look ugly and even marginally misleading, IMO. Please apply with care and double check the results. Formatting by hand also creates problems, of course. Some day I will write a perfect astyle replacement :-/. Alex.
Re: IPv6 developments for HEAD
On Sat, 2007-04-07 at 17:24 +1200, Amos Jeffries wrote: Attached are two patches which constitute part of the core developments for protocol-independent handling of IP addresses in squid3. In your opinion, should these be committed to Squid 3.0? Are they likely to cause short-term stability problems? Should they be applied to Squid 3.1 instead? Thank you, Alex.
Re: IPv6 developments for HEAD
On Wed, 2007-04-11 at 11:41 +1200, [EMAIL PROTECTED] wrote: I'm still a little fuzzy on the consensus of what the difference between 3.0 and 3.1 is. Squid 3.0 is the current Squid3 HEAD that we are making stable and want to release as such ASAP. Squid 3.1 is whatever comes after a stable 3.0 release. Open to experimentation. Not currently branched (but could be if needed). Question then becomes, where is the existing list of agreed features for 3.0-STABLE1 ?? Whatever features have been committed already minus unstable optional features. This is just my understanding, of course. Not claiming to express the elusive consensus here... Alex.
Re: IPv6 developments for HEAD
On Wed, 2007-04-11 at 11:41 +1200, [EMAIL PROTECTED] wrote: On Sat, 2007-04-07 at 17:24 +1200, Amos Jeffries wrote: Attached are two patches which constitute part of the core developments for protocol-independent handling of IP addresses in squid3. In your opinion, should these be committed to Squid 3.0? Are they likely to cause short-term stability problems? Should they be applied to Squid 3.1 instead? Thank you, Alex. Yes. No. both?. I would like to see them in 3.0. The new object I am submitting is isolated 'infrastructure' which does not affect the rest of squid in any way. It is itself the stable base needed for future work. Interesting. If IPAddress is not used by Squid before and after your patch, is there a reason to commit it now? Originally, I thought that you are modifying common code and want to commit ASAP to minimize future conflicts. Now I understand that IpAddress addition does not alter anything in Squid core (and you are not asking to commit the rest of your changes, which do). On one hand, I am tempted to vote immediate inclusion of IPAddress simply to satisfy a valuable developer. On the other hand, I do not understand why you want that file to be in HEAD if nothing is using it. Could you please clarify why you want to see IPAddress in HEAD? The update to rfc1035 is a reversal of previous changes which is again stabilising that area a little more. This is not my area. Any objections from others to committing the rfc1035 part of the patch? Thank you, Alex.
Re: IPv6 developments for HEAD
On Wed, 2007-04-11 at 23:19 +1200, Amos Jeffries wrote: Currently the branch is looking at nearly 5500 lines of code changed. With nearly 3000 lines removed from the core of squid so far. In complete agreement with Henrik and Adrians views that stability in 3.0 should not be risked. I am nevertheless trying to drop that huge difference in 'small' discrete isolated chunks. I have managed to find 3 areas that will do a large amount of reduction without changing or touching the core code in any way. RFCs were step 1. IPAddress is step 2. A new rfc3596 (DNS) library based non the RFCs is a third, but that still needs major testing and is weeks away I think. When the 'non-changes' are out of the branch and in HEAD waiting to be used. The actual core changes can push ahead cleanly for IPv6 in 3.1, both inside my branch and in any others who want to jump for the new ability while HEAD is closed to them. Thank you for the explanation. I still do not see much practical value in committing unused code. To me, committing 5500 lines with IPAddress is pretty much the same as committing 5200 lines without IPAddress because IPAddress is not used by the current code. However, I am not objecting to committing IPAddress. If others think IPAddress should be added to Squid3 now, then we should commit them. Thank you, Alex.
bug #1829 abort for shared memory (looks like)
On Sat, 2007-04-07 at 19:48 +0200, Guido Serassio wrote: There are some bugs probably ready to be commited/closed: #1889: If the latest Henrik's comment is true, this bug is still fixed and can be closed #1837: New patch available #1829: Probably the proposed fix could be enough #1900: New proposed patch available Out of the above, only bug #1829 abort for shared memory (looks like) remains open. I cannot review and commit the available fix. If somebody can, please do. Thank you, Alex.
Squid 3.0 bugs that need you.
Hi there, I went through the remaining Squid3 non-enhancement bug reports targeted for 3.0. For most bugs, I was able to close or comment in the bugzilla. The following bugs are special because the questions did not seem appropriate for bugzilla. Please review. http://www.squid-cache.org/bugs/show_bug.cgi?id=1345 Bug 1345 - Problem with tar-file Squid3 needs a volunteer to remove cppunit from the distribution and, probably, from CVS. Anybody is willing to do that? This is a perfect work item for folks who hate C++ development as it should not require much C++ programming... http://www.squid-cache.org/bugs/show_bug.cgi?id=1475 Bug 1475 - File descriptor limit does work. I am not sure what the final resolution for this bug is. Henrik? Adrian? ESI bugs targeted for 3.0 http://www.squid-cache.org/bugs/show_bug.cgi?id=951 http://www.squid-cache.org/bugs/show_bug.cgi?id=1088 If nobody is willing to work on ESI now, I would like to bump any 3.0 target milestone for ESI to 3.1 because ESI is an optional feature. Any objections? Also, there are quite a few bugs targeted for 3.1 and bugs without a specific target. I reviewed the ones without a target for possible 3.0 inclusion and updated some of the comments. If there is a Squid3 bug that is not targeted for 3.0 release, but you think it should, please flag it as such. Thank you, Alex.
Re: bug #1829 abort for shared memory (looks like)
On Thu, 2007-04-12 at 09:26 -0600, Alex Rousskov wrote: On Sat, 2007-04-07 at 19:48 +0200, Guido Serassio wrote: There are some bugs probably ready to be commited/closed: #1889: If the latest Henrik's comment is true, this bug is still fixed and can be closed #1837: New patch available #1829: Probably the proposed fix could be enough #1900: New proposed patch available Out of the above, only bug #1829 abort for shared memory (looks like) remains open. I cannot review and commit the available fix. If somebody can, please do. #1889 is closed after Duane's review. I currently see 21 bugs targeting 3.0 release, but that number is approximate because some targets will change. http://tinyurl.com/2c99um Thank you, Alex.
Re: Removing cppunit
On Fri, 2007-04-13 at 13:27 +1200, [EMAIL PROTECTED] wrote: On Fri, 2007-04-13 at 12:45 +1200, [EMAIL PROTECTED] wrote: It has not been linked to the cppunit testers, but I have written a specific test app to check each function and proivide for maula-eye check in increments from simplest up for both ipv4-pure then dual protocol paths. Then it has spent nearly a month in live run tests within squid and manual debug traces. That said, if anyone can think of any asserts that should be in there, point them out. If the test logic is not hooked into 'make check', then it will not be run regularly. Platform changes or subsequent untested changes to the code base can introduce regressions. I think it should most definately be hooked into the cppunit test suite. Um, which brings me to ... Alex: didn't you ask for cppunit to be removed from squid3? if not, I need a kick in the right direction on that. My understanding is that we want to remove cppunit (the program, framework, or whatever it is) itself but leave Squid-specific test cases. Somebody will need to install cppunit to actually use Squid test cases. This is similar to any other third party tool like GCC or valgrind. We do not distribute those with Squid, but have things that work with them. Running make check (or whatever it is) would run cppunit test cases if somebody has cppunit installed and should skip them (with a warning) if somebody does not have cppunit installed. If there is a special make target to force running cppunit test cases, that target will fail if there is no cppunit installed. HTH, Alex.
Re: Removing cppunit
On Fri, 2007-04-13 at 13:23 +1000, Robert Collins wrote: On Thu, 2007-04-12 at 21:01 -0600, Alex Rousskov wrote: Running make check (or whatever it is) would run cppunit test cases if somebody has cppunit installed and should skip them (with a warning) if somebody does not have cppunit installed. If there is a special make target to force running cppunit test cases, that target will fail if there is no cppunit installed. I think that make check should error if cppunit is not available: cppunit is core infrastructure for the test suite these days, and 'make check' cannot deliver robust results in its absence. I am not sure whether make check semantics (in Automake world) is that narrow, but I am sure of two things: 1) Many packages use make check or make distcheck without knowing anything about cppunit or any other form of unit testing. The standard installation sequence is: ./configure make make check make install Certainly, the above should not fail if cppunit is not available. 2) It is common to skip test cases that cannot be tested in a given environment. For example, Perl modules often skip POD test cases if some POD module is not installed. cppunit is not hard to install, even less so now than when we added it to the squid code base. That is good, but I do not think we should require it. Cppunit is a developer tool. Make check is a user-level reassurance that the package was built correctly. If cppunit is installed, make check should use it (although it would be nice to disable those checks with an environment variable or some such). If cppunit is not available, make check should succeed with a warning. These are just me thoughts and recommendations. I am fine with requiring cppunit for make check if folks think that is a good idea. Thank you, Alex.
Re: cvs commit: squid3/src main.cc
On Thu, 2007-04-12 at 18:16 -0600, Duane Wessels wrote: On Thu, 12 Apr 2007, Alex Rousskov wrote: rousskov2007/04/12 08:51:10 MDT Modified files: src main.cc Log: This change should fix bug #1837: Segfault on configuration error When quitting on a fatal error, such as a configuration error, Squid may need to write clean state/log files. Squid uses comm_ routines to do so. Thus, we must initialize comm_ before such fatal errors are discovered. Perhaps a better fix would be to avoid writing clean state/log files until the old ones become dirty? The last comment is correct. Squid should not write clean state files until existing state files have been entirely read. I fixed this bug a couple of days ago in the following revisions: 1.88 +2 -2 squid3/src/store_rebuild.cc 1.157 +9 -2 squid3/src/store_dir.cc store_dirs_rebuilding should be initialized to 1 store_dirs_rebuilding is initialized to _1_ as a hack so that storeDirWriteCleanLogs() doesn't try to do anything unless _all_ cache_dirs have been read. For example, without this hack, Squid will try to write clean log files if -kparse fails (becasue it calls fatal()). Sorry I did not realize there was a bugzilla entry for it. I'd suggest backing out your patch. With pleasure. I saw your store_dirs_rebuilding initialization hack but thought it was added before the bug report. Thank you, Alex.
Re: Removing cppunit
On Fri, 2007-04-13 at 14:29 +1000, Robert Collins wrote: On Thu, 2007-04-12 at 22:02 -0600, Alex Rousskov wrote: That is good, but I do not think we should require it. Cppunit is a developer tool. Make check is a user-level reassurance that the package was built correctly. I think this is the root of our disagreement. 'make check' to me is not, and never has been a tool for users: It does give reassurance that the package is built correctly, but its primarily for developers. I've never encountered an open source piece of software yet that considered the primary users of 'make check' to be the end user. Perl's CPAN is one example. A user cannot normally install a module using CPAN or CPANPLUS tools if the module fails make check. I think you should decide to either: - keep cppunit in the source tree or - have make check fail when its not installed. I think that cppunit should be removed from the source tree (especially if it is so easy to install and is not for end-users anyway). This step should not depend on the make check discussion outcome. I think others should decide whether make check should fail if there is no cppunit installed. Thank you, Alex.
Re: Squid 3.0 bugs that need you.
On Sat, 2007-04-14 at 03:27 +1200, Amos Jeffries wrote: Here is the patch for that additional ICAP problem that came to light testing cppunit removal. Committed. Thank you, Alex.
Re: debug statements to debugs statements
On Wed, 2007-04-18 at 11:30 +0300, Tsantilas Christos wrote: Here is a perl script which can be used to convert debug statements to new debugs statements :-) . It mostly works or just do standard errors. It does not formats the output. I think the astyle can do the rest work. Does not convert the debug statements, if the format string contains formating flags like %2.1f. If you get a chance, please modify the script debugging/logging to use one line for both the conversion status and the debug statement: Found: $line\n Converted: $mod_statement\n LeftAsIs: $line\n This way it is much easier to grep/see debug statements that were _not_ converted, for example. A few minor suggestions, if I may: You may want to replace \d* in a line format with \w+ because the arguments are required and sometimes contain variable names rather than constants. You may want to replace space and tab characters in the format with \s. If you are not using a matched string, no need to use parens. For example, this may work: if ( $line =~ /\sdebug\(\s*\w+\s*,\s*\w+\s*\)/ ) Thank you, Alex.
Re: Request for access to squid repository
On Thu, 2007-04-19 at 17:41 +0400, [EMAIL PROTECTED] wrote: I'm a software engeneer with interest in squid development. I've programmed various data processing servers for a few years. The access is needed to contribute to Squid3 testing and development, especially ICAP support. Henrik Nordstrom has granted you access. Welcome to the Squid community! Squid3 can definitely use your help. You will find a list of most important Squid3 bugs at http://tinyurl.com/2c99um Once that list is empty, we plan to make a fresh Squid3 PRE snapshot and will need to work on a flood of bug reports that may follow. Thank you, Alex.
Re: auth_ntlm broken?
On Fri, 2007-04-27 at 21:11 +0200, Gernot Tenchio wrote: Hi developers, I've noticed that auth_ntlm does not work in current squid3-icap branch. As far as I remember squid crashed right after printing AuthNTLMUserRequest::authenticate: authenticated user XYZ. After looking into sources i saw two changes to auth_ntlm.cc. One is dated 03-Jan-07 and the other is dated 20-Jan-07. They changing the way AUTHENTICATE_STATE_xxx is handled. After reverting these changes back to the version before 03-Jan-07, auth_ntlm works again as expected. What was the reason for these changes? Hello, Thank you for reporting the problem! It sounds like the changes you are talking about came from the Squid HEAD branch. Perhaps they created a conflict and the conflict was not resolved correctly. I will investigate and try to fix. Thank you, Alex. P.S. For the future, please post a diff of changes as it usually speeds things up considerably.
Re: ICAP orphan files ?
On Sat, 2007-04-28 at 21:15 +0200, Guido Serassio wrote: Hi, It seems to me that the following files in Squid 3 are orphaned: ICAP/ICAPClientStream.cc ICAP/ICAPClientStream.h It looks like the functions these files provide are called from client_side_reply.cc but the code is #ifdef-ed with ICAP_CLIENT_RESPMOD_POSTCACHE, which is not defined anywhere. It sounds like somebody was trying to get ICAP work with client streams long time ago but did not get far or perhaps just wanted to provide hooks for future use. If there are not objections, I am going to remove the files and the client_side_reply.cc code that references them. We should be able to restore them from CVS if needed. Thank you, Alex.
Re: Squid 3.0 PRE5 and ICAP
On Thu, 2007-05-03 at 14:20 +0200, Ghislain wrote: Hello, I test a Squid 3.0 PRE 5 and the nighty build 20070503. The problem is only present when I configure an ICAP service ( with a cache in ufs format or with null ) In ICAP configuration I have icap_preview_enable on icap_preview_size 0 If the service responds a 200 OK at the end of the preview ( then we don't have the body ) I get ( squid has been stated with squid -d 9 ): assertion failed: store.cc:817: store_status == STORE_PENDING Then Squid restart. Does someone have an idea I am not seeing this even though we are using zero-size previews. There is probably something specific to the HTTP message. I am sure we can fix this. Please file a bug report and, in addition the the stack trace suggested by Christos, provide the trace of the ICAP request and response (if you can). Please note that your cache.log should already contain relevant messages. Thank you, Alex.
Re: ICAP orphan files ?
On Sat, 2007-04-28 at 22:11 -0600, Alex Rousskov wrote: On Sat, 2007-04-28 at 21:15 +0200, Guido Serassio wrote: Hi, It seems to me that the following files in Squid 3 are orphaned: ICAP/ICAPClientStream.cc ICAP/ICAPClientStream.h It looks like the functions these files provide are called from client_side_reply.cc but the code is #ifdef-ed with ICAP_CLIENT_RESPMOD_POSTCACHE, which is not defined anywhere. It sounds like somebody was trying to get ICAP work with client streams long time ago but did not get far or perhaps just wanted to provide hooks for future use. If there are not objections, I am going to remove the files and the client_side_reply.cc code that references them. We should be able to restore them from CVS if needed. Done. Alex.
Re: Squid 2.6.STABLE12 and ICAP
On Tue, 2007-05-15 at 18:09 +0200, Ghislain wrote: I've used : cvs -d:pserver:[EMAIL PROTECTED]:/cvsroot/squid co - ricap-2_6 -d squid-2_6 squid Is it the best way to get a stable and recent version of squid icap? A stable and recent version of Squid ICAP client is not available for Squid2. You have to use Squid3 for that. Squid3 is not stable yet but it is getting there and many folks are using it in production already. If you experience ICAP-related bugs with Squid3, please file a bug report and they will probably be fixed soon. To try Squid3 (with ICAP) just use the nightly Squid3 snapshot. I believe the Squid-3 ICAP patch from devel.squid-cache.org that Henrik mentions below is currently stale and not needed. Henrik, anything I should do to make the auto-generated patch on devel.squid-cache.org reflect the current state of the ICAP branch? Thank you, Alex. Le 4 mai 07 à 13:13, Henrik Nordstrom a écrit : fre 2007-05-04 klockan 11:27 +0200 skrev Ghislain: Hi everyone, there is a compilation error on a squid 2.6.STABLE12 with the client ICAP. There is no maintained ICAP patch for Squid-2.6.STABLE. If you want Squid-2 with ICAP then please get Squid-2.HEAD and the ICAP patch from devel.squid-cache.org. But I would recommend you to try out Squid-3 ICAP instead. Get a nightly Squid-3 snapshot, and the Squid-3 ICAP patch from devel.squid- cache.org (the patch fixes a few problems with Squid-3). Regards Henrik
Re: A doubt about my ICAP server
On Wed, 2007-05-16 at 12:54 +0200, Jeremy Lardon wrote: Hey all, This mail is the following of some mails of this list. I tried to be clearer and add more attachments. As adviced by Tsantilas Christos and confirmed by Alex Rousskov, I installed the Squid-3 nightly snapshot on my squid box to test with my ICAP server. The fact is that I doubt the old ICAP server (last release: October 30, 2002) from http://icap-server.sourceforge.net/ that I chose for the possibility it offers to custom modification. The problem I'm facing is that, in respmod_precache (I haven't tried other vectoring points), the browser didn't display anything. So I turned on all debug for ICAP part (debug_options ALL,1 93,9) and get the cache.log attached. As you can see, Squid receives the ICAP response (lines 268-518) but stops the parsing and finally doesn't send anything (lines 524/525). This few lines make me doubt the ICAP server. My skill in C++ being limited, I can't understand this behavior. That's why I seek help for this problem. Does the problem come from the ICAP server as I guess ? Or is it something wrong in squid ? In the first case, I will try to fix the server, else I will gladly report the problem as a bug. Feel free to ask for more precision if I am not clear. Please file a bug report. It looks like Squid does not expect the body for the RESPMOD response, but I do not know why. Could be a Squid bug. Alex.
Re: cvs commit: squid3/src/ICAP ICAPConfig.cc ICAPConfig.h ....
On Sat, 2007-05-19 at 02:38 +1200, Amos Jeffries wrote:
Oh dear I was sure y test build had --enable-icap.
Seems not though, very sorry.
Blame evil #ifdefs!
Index: ICAP/ICAPXaction.cc
===
RCS file: /squid/squid3/src/ICAP/ICAPXaction.cc,v
retrieving revision 1.16
diff -u -r1.16 ICAPXaction.cc
--- ICAP/ICAPXaction.cc 18 May 2007 06:41:30 - 1.16
+++ ICAP/ICAPXaction.cc 18 May 2007 08:35:46 -
snip
@@ -118,14 +118,10 @@
disableRetries(); // we only retry pconn failures
- ICAPXaction.cc
if (connection 0) {
connection = comm_open(SOCK_STREAM, 0, getOutgoingAddr(NULL), 0,
COMM_NONBLOCKING, s.uri.c_str());
-===
-connection = comm_open(SOCK_STREAM, 0, getOutgoingAddr(NULL), 0,
-COMM_NONBLOCKING, s.uri.buf());
- 1.15
+}
This change is more of a problem. To me it looks like the latter bit
that you have removed is part of a previous conflict that got overlooked.
ALEX: am I right in thinking the part _after_ the is the new piece
that should be kept after converting to use c_str()?
Yes, I think you are right.
Thank you,
Alex.
Re: cvs commit: squid3/src/ICAP ICAPConfig.cc ICAPConfig.h ....
On Sat, 2007-05-19 at 03:33 +1200, Amos Jeffries wrote: Alex Rousskov wrote: On Sat, 2007-05-19 at 02:38 +1200, Amos Jeffries wrote: Oh dear I was sure y test build had --enable-icap. Seems not though, very sorry. Blame evil #ifdefs! I was going to, then I looked at the code more. I can't see _any_ in that file file. You are looking at the wrong code. ICAP/* code does not use evel #ifdefs, but code that includes ICAP (or not) does. It all starts with configure.in and Makefile.am I'm using the same g++ 4.1.2 on ubuntu. I just purged all the build created files, bootstrapped, configured with --enable-icap and rebuilt. (Am I missing an compile option for Xaction?) I do not know of any Xaction-specific compile options. How about running make -k distclean first? HTH, Alex.
Re: cvs commit: squid3/src HttpReply.cc
On Tue, 2007-05-22 at 19:14 +0200, Henrik Nordstrom wrote: tis 2007-05-22 klockan 10:37 -0600 skrev Alex Rousskov: Bug #1967 fix: avoid new strncmp() that silently converts char* buffers into Strings because String length is limited by 64KB and because it is an expensive conversion. Ugh... C++ magics strikes again. Thanks for finding this. Was a bit worried there for a while. And you should still be worried, unfortunately: I have not fixed the actual misapplication of magic problem, just stopped using strcmp in one particular piece of code. See bug #1970. Alex.
Re: cvs commit: squid3/lib MemPool.cc
On Thu, 2007-05-24 at 00:59 +1200, Amos Jeffries wrote: Alex Rousskov wrote: rousskov2007/05/22 10:40:06 MDT Modified files: lib MemPool.cc Log: Bug #1966 fix: Use rounded String MemPool sizes in the hard-coded pool config to avoid warnings that the configured pool size does not match the actual size. Revision ChangesPath 1.7 +7 -2 squid3/lib/MemPool.cc Did you run make check before committing any of the days changes? Nope. I never do. Will try to remember next time. Sorry, Alex. P.S. Henrik, thanks for fixing the test cases!
Re: SqString
On Thu, 2007-05-24 at 11:19 +0200, Henrik Nordstrom wrote: Should we back out SqString for now until the implicit cast issues have been analyzed in more detail, or try fixing it somehow for the 3.0 release? http://www.squid-cache.org/bugs/show_bug.cgi?id=1970 My vote is to defer the SqString change to 3.1, or at least after 3.0 has been branched from HEAD. Reasoning: More work is required to polish it, and it doesn't really add anything to the 3.0 release (beyond the bugfixes it triggered indirectly), mainly preparation for future work. I still think SqString API changes should not be in 3.0, but I do not have a strong opinion and do not want to be the one backing them out. We could adopt a middle-ground solution. The changes limited to renaming String methods to match std::string API can stay. The changes introducing new operators, conversions, or methods should be postponed until v3.1. Same for changes in the code that uses Strings: renaming is fine, rearranging and optimizing things is not. HTH, Alex.
Re: Squid 3.0.PRE6 - https connection breaking...
On Thu, 2007-05-24 at 16:58 +0200, Emmanuel Eyer wrote: It still crashes randomly, but at most once or twice a day. Thus now the HTTPS issue became visible (or popped up). Please consider filing bug reports for all crashes we do not know about. Thank you, Alex.
