[squid-users] 100% CPU
Hi folks, I'm using: Squid Cache version 3.0.STABLE13 for i686-suse-linux This cache has been running well for at least a couple years, or more. As of yesterday, my squid process started using 100% CPU and users on the proxy are experiencing extremely slow Internet speeds. I've disabled caching (thinking it was maybe a cache issue), but the problem remains. I've tried strace on the squid process, but I don't know what I'm even looking for. As for syscalls, I see lots of reads and writes. It seems to be a particular user, or number of users running something through the proxy that is causing the CPU to spike. This morning everything was find for a couple hours with users on it...CPU was behaving...then within a minute or so it spiked and remained pegged at 100%. Would a video or audio stream cause something like this? A very large download? Thanks, Matt
RE: [squid-users] POST method when using squid_kerb_auth and sending Yahoo mail attachment
Ok, to add yet another dimension to this... When I use IE with the configured proxy settings, and I run a packet capture at the moment I select Attach File in the Yahoo Mail compose message window, I capture the aforementioned POST method, which eventually fails or hangs etc. When I use Firefox with manually configured proxy settings (NOT the use system settings option) and I perform the same packet capture at the same point, I capture nothing to/from squid and the file is attached successfully. When I use Firefox with the use system settings proxy config, it behaves like IE above and fails. I know this is significant, but at this point I'm not sure what it means. Hank Date: Mon, 6 Feb 2012 20:00:24 +1300 From: squ...@treenet.co.nz To: squid-users@squid-cache.org Subject: Re: [squid-users] POST method when using squid_kerb_auth and sending Yahoo mail attachment On 6/02/2012 5:31 p.m., Hank Disuko wrote: Thanks Amos, What's happening is quite similar to the details described in the aforementioned Firefox bug filing. Of course. Authentication in HTTP has a flow of 4+ steps: 1) -- client request 2) -- server challenge 401 or 407 response 3) -- client request w/ credentials 4) -- server success/fail response ... This is the same for all authentication protocols. Possibly with a loop repeating 3 4 until a suitable set of credentials are agreed on or the client gives up. * The firefox bug was about firefox not sending Basic auth protocol credentials properly when challenged. * So far you have been talking around the edges of something that sounds like a client not sending Kerberos auth protocol credentials correctly when challenged, or possibly you misconfiguring a Kerberos helper to validate non-Kerberos credentials. The user watching gets to see only that the auth worked, a popup appeared, or the forbidden error page appeared. They are not forced to see what protocols are in use or how many retries were made. When the attach file function is started in the Yahoo Mail compose message window and a file is selected from the user's desktop filesystem, a new HTTP POST operation is initiated to squid. This is a new tcp session entirely. This would be step (1) above. The POST operation is a form served by host sp1.attach.mail.yahoo.com using a Shockwave Flash user-agent - so I'm assuming the browser itself sits this one out. Here's the first little bit of the request, it's followed by form-data such as filename and content-type etc. Aha. Now we are getting places. First item is to check whether Shockwave Flash supports Kerberos protocol which you are requiring of it. Chances are shockwave does but the applet it is running does not. It is very common to find web apps which cannot do auth even when SDK like Flash and Java have long provided API to do all the difficult parts. POST http://sp1.attach.mail.yahoo.com/ca.f431.mail.yahoo.com/ya/upload_with_cred?-- HTTP/1.1 Accept: text/* Content-Type: multipart/form-data; boundary=--cH2ae0gL6KM7ei4ei4ei4Ij5Ij5KM7 User-Agent: Shockwave Flash Host: sp1.attach.mail.yahoo.com Content-Length: 719794 Proxy-Connection: Keep-Alive Pragma: no-cache Cookie: B=dgrausd7a344rb=4d=vku6LippYFR6PRpZokl3s5qyCUJklnhtfiFfs=pti=A6MbHqjIfHzX9QIh5CDC; Yes definitely a step (1) client request with no credentials. However the thing to note here is that this POST request is using a simple HTTP format without the HTTP/1.1 chunked encoding or Expect features. That is fine but it means that Content-Length bytes of data MUST be transmitted for the body, regardess of what the server rsponds with. Squid responds to this initial POST operation with the predictable TCP_DENIED/407 Cache Access Denied message: from access.log: Sun Feb 5 22:29:16 2012 3 172.16.130.22 TCP_DENIED/407 5626 POST http://sp1.attach.mail.yahoo.com/ca.f431.mail.yahoo.com/ya/upload_with_cred? - NONE/- text/html HTTP/1.0 407 Proxy Authentication Required Server: squid/3.1.11 Mime-Version: 1.0 Date: Mon, 06 Feb 2012 03:29:16 GMT Content-Type: text/html Content-Length: 5206 X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0 Vary: Accept-Language Content-Language: en Proxy-Authenticate: Negotiate X-Cache: MISS from localhost X-Cache-Lookup: NONE from localhost:3128 Via: 1.0 localhost (squid/3.1.11) Connection: keep-alive So far everything is perfectly correct. Squid actually serves up the full 407 Denied webpage, but it's not presented to the user. That is correct. Displaying the error page is optional. Being kerberos authentication it SHOULD be able to locate the Kerberos credentials silently in the background without bothering the user at all. Instead, the Yahoo Flash user-agent seems to trample on instead and attempts to send the attachment anyway, without first re-sending the request with the credentials
RE: [squid-users] POST method when using squid_kerb_auth and sending Yahoo mail attachment
I think I see what may be going on. In Windows 7 with IE9, when you configure a proxy server it seems to use it for more than just IE. It would seem it's sending any HTTP traffic from the LAN interface, whether from the browser or not, through the proxy server. If I set IE to no proxy, and manually set Firefox to use a proxy, Firefox will only send it's own traffic to the proxy server. So when the Yahoo Mail attach file mechanism is activated, it uses Shockwave Flash as user-agent, not Firefox...so the connection is established directly with Yahoo server - not via the proxy. When a proxy is configured in IE, it will seem to send *all* HTTP traffic to it, regardless of user-agent (browser, flash, some other app etc). So we're back to the initial suspicion that the Yahoo flash agent does not support Proxy Authentication. Hank From: gourmetci...@hotmail.com To: squ...@treenet.co.nz; squid-users@squid-cache.org Date: Wed, 8 Feb 2012 09:33:31 -0500 Subject: RE: [squid-users] POST method when using squid_kerb_auth and sending Yahoo mail attachment Ok, to add yet another dimension to this... When I use IE with the configured proxy settings, and I run a packet capture at the moment I select Attach File in the Yahoo Mail compose message window, I capture the aforementioned POST method, which eventually fails or hangs etc. When I use Firefox with manually configured proxy settings (NOT the use system settings option) and I perform the same packet capture at the same point, I capture nothing to/from squid and the file is attached successfully. When I use Firefox with the use system settings proxy config, it behaves like IE above and fails. I know this is significant, but at this point I'm not sure what it means. Hank Date: Mon, 6 Feb 2012 20:00:24 +1300 From: squ...@treenet.co.nz To: squid-users@squid-cache.org Subject: Re: [squid-users] POST method when using squid_kerb_auth and sending Yahoo mail attachment On 6/02/2012 5:31 p.m., Hank Disuko wrote: Thanks Amos, What's happening is quite similar to the details described in the aforementioned Firefox bug filing. Of course. Authentication in HTTP has a flow of 4+ steps: 1) -- client request 2) -- server challenge 401 or 407 response 3) -- client request w/ credentials 4) -- server success/fail response ... This is the same for all authentication protocols. Possibly with a loop repeating 3 4 until a suitable set of credentials are agreed on or the client gives up. * The firefox bug was about firefox not sending Basic auth protocol credentials properly when challenged. * So far you have been talking around the edges of something that sounds like a client not sending Kerberos auth protocol credentials correctly when challenged, or possibly you misconfiguring a Kerberos helper to validate non-Kerberos credentials. The user watching gets to see only that the auth worked, a popup appeared, or the forbidden error page appeared. They are not forced to see what protocols are in use or how many retries were made. When the attach file function is started in the Yahoo Mail compose message window and a file is selected from the user's desktop filesystem, a new HTTP POST operation is initiated to squid. This is a new tcp session entirely. This would be step (1) above. The POST operation is a form served by host sp1.attach.mail.yahoo.com using a Shockwave Flash user-agent - so I'm assuming the browser itself sits this one out. Here's the first little bit of the request, it's followed by form-data such as filename and content-type etc. Aha. Now we are getting places. First item is to check whether Shockwave Flash supports Kerberos protocol which you are requiring of it. Chances are shockwave does but the applet it is running does not. It is very common to find web apps which cannot do auth even when SDK like Flash and Java have long provided API to do all the difficult parts. POST http://sp1.attach.mail.yahoo.com/ca.f431.mail.yahoo.com/ya/upload_with_cred?-- HTTP/1.1 Accept: text/* Content-Type: multipart/form-data; boundary=--cH2ae0gL6KM7ei4ei4ei4Ij5Ij5KM7 User-Agent: Shockwave Flash Host: sp1.attach.mail.yahoo.com Content-Length: 719794 Proxy-Connection: Keep-Alive Pragma: no-cache Cookie: B=dgrausd7a344rb=4d=vku6LippYFR6PRpZokl3s5qyCUJklnhtfiFfs=pti=A6MbHqjIfHzX9QIh5CDC; Yes definitely a step (1) client request with no credentials. However the thing to note here is that this POST request is using a simple HTTP format without the HTTP/1.1 chunked encoding or Expect features. That is fine but it means that Content-Length bytes of data MUST be transmitted for the body, regardess of what the server rsponds with. Squid responds to this initial POST operation with the predictable
RE: [squid-users] POST method when using squid_kerb_auth and sending Yahoo mail attachment
Thanks Amos, What's happening is quite similar to the details described in the aforementioned Firefox bug filing. When the attach file function is started in the Yahoo Mail compose message window and a file is selected from the user's desktop filesystem, a new HTTP POST operation is initiated to squid. This is a new tcp session entirely. The POST operation is a form served by host sp1.attach.mail.yahoo.com using a Shockwave Flash user-agent - so I'm assuming the browser itself sits this one out. Here's the first little bit of the request, it's followed by form-data such as filename and content-type etc. POST http://sp1.attach.mail.yahoo.com/ca.f431.mail.yahoo.com/ya/upload_with_cred?-- HTTP/1.1 Accept: text/* Content-Type: multipart/form-data; boundary=--cH2ae0gL6KM7ei4ei4ei4Ij5Ij5KM7 User-Agent: Shockwave Flash Host: sp1.attach.mail.yahoo.com Content-Length: 719794 Proxy-Connection: Keep-Alive Pragma: no-cache Cookie: B=dgrausd7a344rb=4d=vku6LippYFR6PRpZokl3s5qyCUJklnhtfiFfs=pti=A6MbHqjIfHzX9QIh5CDC; Squid responds to this initial POST operation with the predictable TCP_DENIED/407 Cache Access Denied message: from access.log: Sun Feb 5 22:29:16 2012 3 172.16.130.22 TCP_DENIED/407 5626 POST http://sp1.attach.mail.yahoo.com/ca.f431.mail.yahoo.com/ya/upload_with_cred? - NONE/- text/html HTTP/1.0 407 Proxy Authentication Required Server: squid/3.1.11 Mime-Version: 1.0 Date: Mon, 06 Feb 2012 03:29:16 GMT Content-Type: text/html Content-Length: 5206 X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0 Vary: Accept-Language Content-Language: en Proxy-Authenticate: Negotiate X-Cache: MISS from localhost X-Cache-Lookup: NONE from localhost:3128 Via: 1.0 localhost (squid/3.1.11) Connection: keep-alive Squid actually serves up the full 407 Denied webpage, but it's not presented to the user. Instead, the Yahoo Flash user-agent seems to trample on instead and attempts to send the attachment anyway, without first re-sending the request with the credentials required to access squid. I can see the pdf being uploaded to the squid server, but squid just ignores it. This manifests as a hanging upload window to the user. I guess I need to know where to look in order to find out why the request is not re-sent using the proper credentials. Is it the Yahoo user-agent? is it the browser? Thanks, Hank Date: Sat, 4 Feb 2012 18:39:23 +1300 From: squ...@treenet.co.nz To: squid-users@squid-cache.org Subject: Re: [squid-users] POST method when using squid_kerb_auth and sending Yahoo mail attachment On 4/02/2012 12:46 p.m., Hank Disuko wrote: Hello folks, I'm using squid 3.1.11-1 on Ubuntu Server 11. I am using /usr/lib/squid3/squid_kerb_auth to auth against a Windows 2008 directory. I am unable to upload attachments to emails when using the *new* Yahoo! Mail interface. The old interface seems to work fine. I've seen this problem reported around the internet. These older posts reveals some insight: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-NTML-and-auth-problems-with-POST-td2255704.html This is a well known problem with NTLM design. Kerberos was re-designed to avoid this. Since you are apparently Negotiate protocol with a Negotiate/kerberos helpers it is not relevant. http://www.squid-cache.org/mail-archive/squid-users/200506/0158.html ditto here. I made a POST_whitelist.txt for .yahoo.com and uploads work fine. But this is an ugly workaround. More recently, someone also experiencing this issue filed a Firefox bug. But I'm quite sure it's not a Firefox issue: https://bugzilla.mozilla.org/show_bug.cgi?id=679519 Any better fix for this out there? The bug reported to firefox seems to be about Basic authentication. Which is also irelevant. To provide any more help than that we will need to know exactly what is going on in your system. What is being requested from Squid, what Squid is responding with, anything Squid logs about the transaction, and how it is configured. Amos
[squid-users] POST method when using squid_kerb_auth and sending Yahoo mail attachment
Hello folks, I'm using squid 3.1.11-1 on Ubuntu Server 11. I am using /usr/lib/squid3/squid_kerb_auth to auth against a Windows 2008 directory. I am unable to upload attachments to emails when using the *new* Yahoo! Mail interface. The old interface seems to work fine. I've seen this problem reported around the internet. These older posts reveals some insight: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-NTML-and-auth-problems-with-POST-td2255704.html http://www.squid-cache.org/mail-archive/squid-users/200506/0158.html I made a POST_whitelist.txt for .yahoo.com and uploads work fine. But this is an ugly workaround. More recently, someone also experiencing this issue filed a Firefox bug. But I'm quite sure it's not a Firefox issue: https://bugzilla.mozilla.org/show_bug.cgi?id=679519 Any better fix for this out there? Thanks, Hank
