[squid-users] Re: Squid and Hangout (google) problem
Sorry, what are Google Hangouts video ? May be, you can provide an URL as an exaample to try/test ? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Re-Squid-and-Hangout-google-problem-tp4665683p4665877.html Sent from the Squid - Users mailing list archive at Nabble.com.
RE: [squid-users] Skype SSL is incompatible with OpenSSL
Hi Jay and others, If I am not mistaken based on your redirection description (transparent proxy) in order to get to the ACLs you described squid *first needs to SSL bump* the traffic to apply ACLs - so those ACLs will never exclude traffic from being SSL Bumped... One solution that comes to mind is to *not forward* traffic to those IPs you have in ACL (numeric_IPs) from your router to squid box. Of course you would need to keep up-to-date with the IP address changes and probably too much traffic will bypass proxy - but at least your Skype will start working... Raf From: Jay Jimenez j...@integralvox.com Sent: Thursday, May 8, 2014 5:49 AM To: squid-users@squid-cache.org Subject: Re: [squid-users] Skype SSL is incompatible with OpenSSL Hi Marcus and Pawel, Thank you very much for all the help. There is only 1 conclusion here. We cannot ssl bump Skype and therefore must be excluded. I can't find any solution to exclude Skype on my squid.conf file. I have tried to exclude ^skype browser and/or exclude .microsoft.com, .live.com and numeric IP addresses but no luck. Squid is always ssl bumping Skype traffic. This doesn't work for me... acl numeric_IPs dstdom_regex ^(([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)|(\[([0-9af]+)?:([0-9af:]+)?:([0-9af]+)?\])):443 acl Skype_UA browser ^skype acl broken dstdomain .skype.com .microsoft.com .live.com ssl_bump none numeric_IPs ssl_bump none Skype_UA ssl_bump none broken ssl_bump server-first all --- As a workaround, I just ssl bump numerous https websites only that are not allowed in the company like .facebook.com, .twitter.com, .linkedin.com, etc. acl sslsites dstdomain .facebook.com .twitter.com .linkedin.com ssl_bump server-first sslsites We don't have any video conferencing alternative than skype that's why it's allowed in the company. Regards, Jay On Thu, May 8, 2014 at 5:27 AM, Marcus Kool marcus.k...@urlfilterdb.com wrote: On 05/07/2014 10:55 AM, Pawel Mojski wrote: W dniu 2014-05-07 15:40, Marcus Kool pisze: [...] certificate chain: Certificate chain 0 s:/CN=*.gateway.messenger.live.com i:/DC=com/DC=microsoft/DC=corp/DC=redmond/CN=MSIT Machine Auth CA 2 1 s:/DC=com/DC=microsoft/DC=corp/DC=redmond/CN=MSIT Machine Auth CA 2 i:/CN=Microsoft Internet Authority 2 s:/CN=Microsoft Internet Authority i:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root There is a misunderstanding here. Skype does not use SSL, it only uses port 443 which is commonly used by SSL, but skype knows that there is a proxy and uses the CONNECT method to get a tunnel from Squid. Squid (without SSL-bump) than simply tunnels (i.e. passes everything from the client to the server and back). But _with_ ssl-bump Squid assumes that the CONNECT is for a SSL connection and this assumption is wrong. Sorry, but you are wrong. Skype *IS* using ssl like you can see on example above. That example was made on openssl -connect ip.from.sniffing.my.own.skype:443 and as you can see, it's a proper SSL connection. But, no one of us have any idea what is the native protocol, all what we can figure out it is SSL connection. This is some kind of protocol over SSL. Skype starts connecting to a node so a Skype node is a critical component. Lets look at a Skype node at 157.55.235.144 : openssl s_client -connect 157.55.235.144:443 CONNECTED(0003) 139691491997512:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:766: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 7 bytes and written 263 bytes There is no SSL. So at least part of Skype uses port 443 for non-SSL traffic. This observation matches the error messages in the original post: 2014/05/02 18:18:11 kid1| clientNegotiateSSL: Error negotiating SSL connection on FD 166: error:1408F10B:SSL so also Squid cannot negotiate an SSL connection because the server uses an other protocol. The design of Squid ssl-bump assumes that a CONNECT to a server always has an SSL-based communication channel and therefore any software that uses non-SSL traffic on port 443 fails to work with ssl-bump. Marcus
Re: [squid-users] Skype SSL is incompatible with OpenSSL
Hi Raf, As stated on my previous emal, I tried dstdom_regex to match all numeric IP addresses and it didn't help me acl numeric_IPs dstdom_regex ^(([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)|(\[([0-9af]+)?:([0-9af:]+)?:([0-9af]+)?\])):443 acl Skype_UA browser ^skype acl broken dstdomain .skype.com .microsoft.com .live.com ssl_bump none numeric_IPs ssl_bump none Skype_UA ssl_bump none broken ssl_bump server-first all Jay On Thu, May 8, 2014 at 3:48 PM, Rafael Akchurin rafael.akchu...@diladele.com wrote: Hi Jay and others, If I am not mistaken based on your redirection description (transparent proxy) in order to get to the ACLs you described squid *first needs to SSL bump* the traffic to apply ACLs - so those ACLs will never exclude traffic from being SSL Bumped... One solution that comes to mind is to *not forward* traffic to those IPs you have in ACL (numeric_IPs) from your router to squid box. Of course you would need to keep up-to-date with the IP address changes and probably too much traffic will bypass proxy - but at least your Skype will start working... Raf From: Jay Jimenez j...@integralvox.com Sent: Thursday, May 8, 2014 5:49 AM To: squid-users@squid-cache.org Subject: Re: [squid-users] Skype SSL is incompatible with OpenSSL Hi Marcus and Pawel, Thank you very much for all the help. There is only 1 conclusion here. We cannot ssl bump Skype and therefore must be excluded. I can't find any solution to exclude Skype on my squid.conf file. I have tried to exclude ^skype browser and/or exclude .microsoft.com, .live.com and numeric IP addresses but no luck. Squid is always ssl bumping Skype traffic. This doesn't work for me... acl numeric_IPs dstdom_regex ^(([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)|(\[([0-9af]+)?:([0-9af:]+)?:([0-9af]+)?\])):443 acl Skype_UA browser ^skype acl broken dstdomain .skype.com .microsoft.com .live.com ssl_bump none numeric_IPs ssl_bump none Skype_UA ssl_bump none broken ssl_bump server-first all --- As a workaround, I just ssl bump numerous https websites only that are not allowed in the company like .facebook.com, .twitter.com, .linkedin.com, etc. acl sslsites dstdomain .facebook.com .twitter.com .linkedin.com ssl_bump server-first sslsites We don't have any video conferencing alternative than skype that's why it's allowed in the company. Regards, Jay On Thu, May 8, 2014 at 5:27 AM, Marcus Kool marcus.k...@urlfilterdb.com wrote: On 05/07/2014 10:55 AM, Pawel Mojski wrote: W dniu 2014-05-07 15:40, Marcus Kool pisze: [...] certificate chain: Certificate chain 0 s:/CN=*.gateway.messenger.live.com i:/DC=com/DC=microsoft/DC=corp/DC=redmond/CN=MSIT Machine Auth CA 2 1 s:/DC=com/DC=microsoft/DC=corp/DC=redmond/CN=MSIT Machine Auth CA 2 i:/CN=Microsoft Internet Authority 2 s:/CN=Microsoft Internet Authority i:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root There is a misunderstanding here. Skype does not use SSL, it only uses port 443 which is commonly used by SSL, but skype knows that there is a proxy and uses the CONNECT method to get a tunnel from Squid. Squid (without SSL-bump) than simply tunnels (i.e. passes everything from the client to the server and back). But _with_ ssl-bump Squid assumes that the CONNECT is for a SSL connection and this assumption is wrong. Sorry, but you are wrong. Skype *IS* using ssl like you can see on example above. That example was made on openssl -connect ip.from.sniffing.my.own.skype:443 and as you can see, it's a proper SSL connection. But, no one of us have any idea what is the native protocol, all what we can figure out it is SSL connection. This is some kind of protocol over SSL. Skype starts connecting to a node so a Skype node is a critical component. Lets look at a Skype node at 157.55.235.144 : openssl s_client -connect 157.55.235.144:443 CONNECTED(0003) 139691491997512:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:766: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 7 bytes and written 263 bytes There is no SSL. So at least part of Skype uses port 443 for non-SSL traffic. This observation matches the error messages in the original post: 2014/05/02 18:18:11 kid1| clientNegotiateSSL: Error negotiating SSL connection on FD 166: error:1408F10B:SSL so also Squid cannot negotiate an SSL connection because the server uses an other protocol. The design of Squid ssl-bump assumes that a CONNECT to a server always has an SSL-based communication channel and therefore any software that uses non-SSL traffic on port 443 fails to work with ssl-bump. Marcus
RE: [squid-users] Skype SSL is incompatible with OpenSSL
Hi jay, If I am not mistaken dstdom_regex is matched against the *contents* of HTTP/HTTPS request - it means if first needs to be bumped. So it will never work in your case... You need to know not ssl bump traffic before looking into its contents - the only acl that domes to mind is dst - i.e. ip address of the remote server. So something like ssl_bump deny your_skype_ip_acl. But I may be mistaken, hopefully some one on the list will correct me if so. Raf From: Jay Jimenez j...@integralvox.com Sent: Thursday, May 8, 2014 10:21 AM To: squid-users@squid-cache.org Subject: Re: [squid-users] Skype SSL is incompatible with OpenSSL Hi Raf, As stated on my previous emal, I tried dstdom_regex to match all numeric IP addresses and it didn't help me acl numeric_IPs dstdom_regex ^(([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)|(\[([0-9af]+)?:([0-9af:]+)?:([0-9af]+)?\])):443 acl Skype_UA browser ^skype acl broken dstdomain .skype.com .microsoft.com .live.com ssl_bump none numeric_IPs ssl_bump none Skype_UA ssl_bump none broken ssl_bump server-first all Jay On Thu, May 8, 2014 at 3:48 PM, Rafael Akchurin rafael.akchu...@diladele.com wrote: Hi Jay and others, If I am not mistaken based on your redirection description (transparent proxy) in order to get to the ACLs you described squid *first needs to SSL bump* the traffic to apply ACLs - so those ACLs will never exclude traffic from being SSL Bumped... One solution that comes to mind is to *not forward* traffic to those IPs you have in ACL (numeric_IPs) from your router to squid box. Of course you would need to keep up-to-date with the IP address changes and probably too much traffic will bypass proxy - but at least your Skype will start working... Raf From: Jay Jimenez j...@integralvox.com Sent: Thursday, May 8, 2014 5:49 AM To: squid-users@squid-cache.org Subject: Re: [squid-users] Skype SSL is incompatible with OpenSSL Hi Marcus and Pawel, Thank you very much for all the help. There is only 1 conclusion here. We cannot ssl bump Skype and therefore must be excluded. I can't find any solution to exclude Skype on my squid.conf file. I have tried to exclude ^skype browser and/or exclude .microsoft.com, .live.com and numeric IP addresses but no luck. Squid is always ssl bumping Skype traffic. This doesn't work for me... acl numeric_IPs dstdom_regex ^(([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)|(\[([0-9af]+)?:([0-9af:]+)?:([0-9af]+)?\])):443 acl Skype_UA browser ^skype acl broken dstdomain .skype.com .microsoft.com .live.com ssl_bump none numeric_IPs ssl_bump none Skype_UA ssl_bump none broken ssl_bump server-first all --- As a workaround, I just ssl bump numerous https websites only that are not allowed in the company like .facebook.com, .twitter.com, .linkedin.com, etc. acl sslsites dstdomain .facebook.com .twitter.com .linkedin.com ssl_bump server-first sslsites We don't have any video conferencing alternative than skype that's why it's allowed in the company. Regards, Jay On Thu, May 8, 2014 at 5:27 AM, Marcus Kool marcus.k...@urlfilterdb.com wrote: On 05/07/2014 10:55 AM, Pawel Mojski wrote: W dniu 2014-05-07 15:40, Marcus Kool pisze: [...] certificate chain: Certificate chain 0 s:/CN=*.gateway.messenger.live.com i:/DC=com/DC=microsoft/DC=corp/DC=redmond/CN=MSIT Machine Auth CA 2 1 s:/DC=com/DC=microsoft/DC=corp/DC=redmond/CN=MSIT Machine Auth CA 2 i:/CN=Microsoft Internet Authority 2 s:/CN=Microsoft Internet Authority i:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root There is a misunderstanding here. Skype does not use SSL, it only uses port 443 which is commonly used by SSL, but skype knows that there is a proxy and uses the CONNECT method to get a tunnel from Squid. Squid (without SSL-bump) than simply tunnels (i.e. passes everything from the client to the server and back). But _with_ ssl-bump Squid assumes that the CONNECT is for a SSL connection and this assumption is wrong. Sorry, but you are wrong. Skype *IS* using ssl like you can see on example above. That example was made on openssl -connect ip.from.sniffing.my.own.skype:443 and as you can see, it's a proper SSL connection. But, no one of us have any idea what is the native protocol, all what we can figure out it is SSL connection. This is some kind of protocol over SSL. Skype starts connecting to a node so a Skype node is a critical component. Lets look at a Skype node at 157.55.235.144 : openssl s_client -connect 157.55.235.144:443 CONNECTED(0003) 139691491997512:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:766: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 7 bytes and written 263 bytes There is no SSL. So at least part of Skype uses port 443
Re: [squid-users] Skype SSL is incompatible with OpenSSL
On 8/05/2014 8:38 p.m., Rafael Akchurin wrote: Hi jay, If I am not mistaken dstdom_regex is matched against the *contents* of HTTP/HTTPS request - it means if first needs to be bumped. So it will never work in your case... You need to know not ssl bump traffic before looking into its contents - the only acl that domes to mind is dst - i.e. ip address of the remote server. So something like ssl_bump deny your_skype_ip_acl. But I may be mistaken, hopefully some one on the list will correct me if so. That is correct on both points regarding to intercepted port 443 traffic. The browser type ACL only works (sometimes) on the CONNECT requests when Skype is explicitly configured to use the proxy. In those same requests it can be used to prevent bumping. There is possibly a third option if Skype can be explicitly configured to use the proxy through a special port number. The myportname ACL can be used to prevent bumping any traffic received in that Squid port. This avoids having to pre-know all the IPs Skype will use but is likewise more risky than allowing non-bumped access to a set of whitelisted IPs as non-Skype applications might also sneak traffic through the port. Amos
Re: [squid-users] squid cpu problem
On 8/05/2014 12:33 a.m., a.afach wrote:
Hi amos
as i see the problem is still occurring with other errors in GDB the CPU
still goes to 100%
The problem is that very big objects do exist and occasionally need to
be moved from memory to disk.
this it the GDB :
Loaded symbols for /lib64/libnss_db.so.2
0x0050a1c8 in linklistPush (L=0x9fb429e8, p=0x53e0be0) at
list.cc:47
47 list.cc: No such file or directory.
in list.cc
(gdb) backtrace
#0 0x0050a1c8 in linklistPush (L=0x9fb429e8, p=0x53e0be0) at
list.cc:47
#1 0x00594841 in UFSStoreState::write (this=0xb7775918,
buf=0x723b7c70
I\223\324\004\245\201\315\306\354P\276\372e\373\r\235\250\311\033\275P\333\344\323\211\354\275\200\362A,
size=4096, aOffset=-1, free_func=
0x50f220 memNodeWriteComplete(void*)) at ufs/store_io_ufs.cc:247
#2 0x005436e0 in doPages (anEntry=optimized out) at
store_swapout.cc:160
snip
i tried to change config with no success
the problem occurs in peak times or when no load in random times.
how can i know if the problem is a hardware problem or squid
Neither and both.
It is a non-problem in that storing a large object to disk in small
incremental bits is going to take a lot of CPU cycles. The nature of the
task itself causes large CPU usage.
The Squid code doing this store is not great. It walks the linked-list
of memory blocks (N^2)/2 times during the store operation.
Also, the version you are using does not distinguish between objects
stored for future use and objects being discarded immediately. They all
go to disk on their way through Squid. So there is no way to avoid it by
configuring storage of smaller objects.
The hardware is not able to cope with that operation being done on the
size of objects you are proxying.
Amos
thanks
On 2014-04-05 03:37, Amos Jeffries wrote:
This looks like the CPU cycles are being consumed by walking one or more
very long lists of memory pieces and writing them to disk one by one.
Note the UFSStoreState::write parameter size=4096 in the backtrace for
how bit those memory pages are.
Which could happen if you cached a very big object in cache_mem and then
a random time later it needed swapping out to disk to free up memory.
It could also happen if Squid needed to suddenly swap out a large number
of smaller items to make memory space available for a large one which is
about to arrive.
So, have you configured Squid to allow very large objects (many MB or
GB) in memory storage?
Note these causes would not show up in the testing you mentioned unless
you had a very wide range of test object sizes being pumped randomly
through the proxy. A tool like web polygraph is best to test that
traffic behaviour accurately.
Amos
On 5/04/2014 1:59 a.m., a.afach wrote:
Dear all
i still have the CPU spikes even when i used
disable-strict-error-checking without using Cflags
this is the gdb backtrace while the CPU spikes
0x0051b348 in linklistPush (L=0x11853e188, p=0xce6d4300) at
list.cc:47
47 while (*L)
(gdb) backtrace
#0 0x0051b348 in linklistPush (L=0x11853e188, p=0xce6d4300) at
list.cc:47
#1 0x005a70a1 in UFSStoreState::write (this=0xb3970e28,
buf=0x11fe69ca0
!v\253r[/\307\232G\b\375`\237:\213\256^\335\373{\241%\232\363\021\071`\342\033\177a\202G\320{\323%\236K\342\243*\332\316\351\231=\360\370\313Ro=\317\262\243\315\027\351,\221\230\353Z\023\024q\QSC\036\214:M\242{@\351m\020\337Cw_\214\216\304\226\265\a\375\031\211\243V\222T\320\016\227\312-\211Sz\326^\346\230\251\327\222\n\373I\032\341\303==U\214\277\264\244\205\b1\346S=\230\215\204\245\254\312\223\066\336\230PpP\227\271\370\266;\362\226\242\036\225\235w\330\325\061\316{o_\364\021\062\351\376\062|\313\006`\357m\206FQ0\021\030C\224\004]\336\315\371\033h1\361\363\350d\366\066...,
size=4096, aOffset=-1, free_func=0x5203b0 memNodeWriteComplete(void*))
at ufs/store_io_ufs.cc:247
#2 0x00554ca0 in doPages (anEntry=optimized out) at
store_swapout.cc:160
#3 StoreEntry::swapOut (this=0x372ca10) at store_swapout.cc:279
#4 0x0054c986 in StoreEntry::invokeHandlers (this=0x372ca10) at
store_client.cc:714
#5 0x004dc1a7 in FwdState::complete (this=0xbb502b48) at
forward.cc:341
#6 0x005579a5 in ServerStateData::completeForwarding
(this=0xf8030588) at Server.cc:239
#7 0x005571bd in ServerStateData::serverComplete2
(this=0xf8030588) at Server.cc:207
#8 0x004ff3dc in HttpStateData::processReplyBody
(this=0xf8030588) at http.cc:1382
#9 0x004fd367 in HttpStateData::readReply (this=0xf8030588,
io=...) at http.cc:1161
#10 0x00503156 in JobDialerHttpStateData::dial
(this=0xde75ca50, call=...) at base/AsyncJobCalls.h:175
#11 0x00569ee4 in AsyncCall::make (this=0xde75ca20) at
AsyncCall.cc:34
#12 0x0056cb76 in AsyncCallQueue::fireNext (this=optimized
out) at AsyncCallQueue.cc:53
#13 0x0056ccf0
[squid-users] Re: Squid 3.3.8 does not work with mobile app
Amos Jeffries-2 wrote On 8/05/2014 10:33 p.m., 0bj3ct wrote: Eliezer Croitoru-2 wrote On 05/05/2014 10:38 AM, 0bj3ct wrote: I have a transparent Squid. But no one mobile app works with it. More details will be the basic answer: What is the IP topology of the network? What rules have you used in IPTALBES? is it a new machine? have you considered using 14.04? Eliezer Hello, Eliezer. I am using ubuntu 13.10, with squid 3.3.8. iptables/rules.v4 content: *filter #:PREROUTING ACCEPT [4258:491523] :INPUT ACCEPT [71:4706] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [388:25005] :INPUT_RULES - [0:0] -A INPUT -j INPUT_RULES -A FORWARD -j INPUT_RULES -A INPUT_RULES -i lo -j ACCEPT -A INPUT_RULES -p icmp -m icmp --icmp-type any -j ACCEPT -A INPUT_RULES -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT_RULES -p tcp -m tcp --dport 80 -j ACCEPT -A INPUT_RULES -p tcp -m tcp --dport 22 -j ACCEPT -A INPUT_RULES -p tcp -m tcp --dport 443 -j ACCEPT -A INPUT_RULES -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT_RULES -p tcp -m tcp --dport 67 -j ACCEPT -A INPUT_RULES -p tcp -m tcp --dport 8080 -j ACCEPT -A INPUT_RULES -p tcp -m tcp --dport 3128 -j ACCEPT #-A INPUT_RULES -j REJECT --reject-with icmp-host-prohibited COMMIT *nat :PREROUTING ACCEPT [0:0] :INPUT_RULES - [0:0] -A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j REDIRECT --to-port 3128 -A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j REDIRECT --to-port 3127 COMMIT I can reach http websites with squid, but https websites I must add exception before reaching the site. I've heard that mobile applications use HTTPS by default. How can I fix it? Thanks in advance! Sone do. Some don't. Some use port 80 and 443 for other things than HTTP and HTTPS. Those latter ones are abusing the ports and will always have problems with interception proxies. Amos Hello, Amos. Yes, You are right. I've noticed that some apps work with intercept proxy, but others do not. For example I can use stocks, weather, news feed, cinemas and this type applications, but more serious apps that need some authentification like facebook, google, linkedin, whatsapp and so on do not work with squid. What do you think, what is the problem? Thanks.. -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-3-3-8-does-not-work-with-mobile-app-tp4665820p4665885.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] Re: Squid 3.3.8 does not work with mobile app
On 8/05/2014 11:24 p.m., 0bj3ct wrote: Amos Jeffries-2 wrote On 8/05/2014 10:33 p.m., 0bj3ct wrote: I can reach http websites with squid, but https websites I must add exception before reaching the site. I've heard that mobile applications use HTTPS by default. How can I fix it? Thanks in advance! Sone do. Some don't. Some use port 80 and 443 for other things than HTTP and HTTPS. Those latter ones are abusing the ports and will always have problems with interception proxies. Amos Hello, Amos. Yes, You are right. I've noticed that some apps work with intercept proxy, but others do not. For example I can use stocks, weather, news feed, cinemas and this type applications, but more serious apps that need some authentification like facebook, google, linkedin, whatsapp and so on do not work with squid. What do you think, what is the problem? Thanks.. I have no idea what the problem could be. Those sounds like some compliated apps, and it could be anything at all that they are doing which breaks. You might be able to get more clues out of a tcpdump packet trace with full packet bodies (tcpdump options -s0 or -s65535). Amos
[squid-users] Re: Squid 3.3.8 does not work with mobile app
Amos, maybe I use self signed certificate, that is why I cannot use https in some mobile apps? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-3-3-8-does-not-work-with-mobile-app-tp4665820p4665886.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] feature requests
On 2/05/2014 10:01 a.m., Lawrence Pingree wrote: Hi Amos, Thanks for your help in understanding my request. I have attempted to create a rock store but was unsuccessful. There doesn't seem to be very good guidance on the proper step by step process of creating a rock store. Configure the directory: cache_dir rock /path/to/cache 1000 then run squid -z to format the database. All the other settings are supposed to be optional now AFAIK. NP: Very large rock caches (more than a few tens of GB) can take a while to load on startup. I came across crashes the last time i attempted it. Also, I am using an x86 platform (32 bit) with multiple cores, when I attempted to use SMP mode with multiple workers, instantly my intercept mode stopped functioning. I couldn't figure out what was wrong so I'd love to get better guidance on this as well. Strange. There is nothing in SMP which should interfere with interception from what I know of how they all operate. Amos -Original Message- From: Amos Jeffries On 29/04/2014 4:17 p.m., Lawrence Pingree wrote: I would like to request two features that could potentially help with performance. See item #1 Wait ... http://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feat ure.2C_enhance.2C_of_fix_something.3F Some comments to think about before you make the formal feature request bug. Don't let these hold you back, but they are the bigger details that will need to be overcome for these features to be accepted and useful. I will also suggest you test out the 3.HEAD Squid code to see what we have done recently with collapsed_forwarding, SMP support and large rock caches. Perhapse the startup issues that make you want these are now resolved. 1. I would like to specify a max age for memory-stored hot objects different than those specified in the generic cache refresh patterns. refresh_patterns are not generic. They are as targeted as the regex pattern you write. The only difference between memory, disk or network sources for a cache is access latency. Objects are promoted from disk to memory when used, and pushed from memory to disk when more memory space is needed. I suspect this feature will result in disk objects maximum age stabilizing at the same value as the memory cache is set to. - a memory age limit higher than disk objects needing to push to disk will get erased as they are too old. - a memory age limit lower than disk objects promoted from disk will get erased or revalidated to be within the memory limit (erasing the obsoleted disk copy). So either way anything not meeting the memory limit is erased. Disk will only be used for the objects younger than the memory limit which need to overspill into the slower storage area where they can age a bit beofre next use ... which is effectively how it works today. Additionally, there is the fact that objects *are* cached past their max-age values. All that happens in HTTP/1.1 when an old object is requested is a revalidation check over the network (used to be a re-fetch in HTTP/1.0). The revalidation MAY supply a whole new object, or just a few new headers. - a memory age limit higher than disk causes the disk (already slow) to have additional network lag for revalidation which is not applied to the in memory objects. - a memory age limit lower than disk places the extra network lag on memory objects. ... what benefit is gained from adding latency to one of the storage areas which is not applicable to the same object when it is stored to the other area? The overarching limit on all this is the *size* of the storage areas, not the object age. If you are in the habit of setting very large max-age value on refresh_pattern to increase caching take a look at your storage LRU/LFU age statistics sometime. You might be in for a bit of a surprise. 2. I would like to pre-load hot disk objects during startup so that squid is automatically re-launched with the memory cache populated. I'd limit this to the maximum memory cache size amount. This one is not so helpful as it seems when done by a cache. Loading on demand solves several performance problems which pre-loading encounter in full. 1) Loading the objects takes time. Resulting in a slower time until first request. Loading on-demand we can guarantee that the first client starts receiving its response as fast as possible. There is no waiting for GB of other objects to fully load first, or even the end of the current object to complete loading. 2) Loading based on previous experience is as best an educated guess. That can still load the wrong things, wasting the time spent. Loading on-demand guarantees that only the currently hot objects are loaded. Regardless of what was hot a few seconds, minutes or days ago when the proxy shutdown. Freeing up CPU cycles and disk waiting time for servicing more relevant requests. 3)
Re: [squid-users] Re: Squid 3.3.8 does not work with mobile app
On 8/05/2014 11:42 p.m., Samir Hasanov wrote: Amos, maybe I use self signed certificate, that is why I cannot use https in some mobile apps? Possibly. It is one of many things which they could be having trouble with. Amos
[squid-users] Re: Squid 3.3.8 does not work with mobile app
Thank your for your help, Amos! Regards, Samir Hasanov. -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-3-3-8-does-not-work-with-mobile-app-tp4665820p4665890.html Sent from the Squid - Users mailing list archive at Nabble.com.
[squid-users] Re: Squid 3.3.8 does not work with mobile app
Amos Jeffries-2 wrote On 8/05/2014 11:42 p.m., Samir Hasanov wrote: Amos, maybe I use self signed certificate, that is why I cannot use https in some mobile apps? Possibly. It is one of many things which they could be having trouble with. Amos Amos, I've got original certificate (not self signed that I used before) and added to the cert folder in linux. But it still does not work. My squid.conf file's cert part: http_port 3129 http_port 3128 intercept https_port 3127 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/usr/local/squid3/ssl_cert/myCa.pem ssl_bump server-first all sslproxy_cert_error allow all sslproxy_cipher HIGH:MEDIUM:!ADH:!DSS:!SSLv2:+3DES sslproxy_flags DONT_VERIFY_PEER,NO_DEFAULT_CA sslproxy_options NO_SSLv2 sslcrtd_program /usr/lib/squid3/ssl_crtd -s /usr/lib/squid3/ssl_db -M 4MB sslcrtd_children 5 Is there something wrong with configuration? Hope I'll be able to solve this problem. Thanks in advance! -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-3-3-8-does-not-work-with-mobile-app-tp4665820p4665891.html Sent from the Squid - Users mailing list archive at Nabble.com.
[squid-users] WIFI Implementation
Hi All I have configured the squid proxy server in CentOS 6.5 for LAN. The adsl-router provided by the ISP[Linksys WRT54G2] is connected to eth0 of squid proxy server, the second interface eth1 is connected to LAN switch. Both eth0 and eth1 are attached to squid proxy running on CentOS. The adsl-router has a wifi, so users can directly connect to adsl-router outside LAN. Now i want to bring all wifi users under squid proxy. Any ideas how should i proceed so that wifi users can be controlled under squid proxy ? Thanks in advance D Dey ISP | | | ADSL-Router[192.168.1.1]---Wifi is here -- | | | | |eth0 [192.168.1.x] | | | Squid proxy on CentOS| | | |eth1 [192.168.0.x] | | | | | LAN Switch | | | | | | | | PC1 PC2 PC3PC4PC5 | | | | | --| | || | PC 6 || | || -- | --- PC6, IPAD1,Nexus are outside the | | | --- | IPAD1|LAN and connect to wifi of ADSL router | |directly - || |Nexus | || -
[squid-users] Using Squid to replace apache rewrite rule and mod_proxy rewrite rule
Hello, I would like to use squid to fetch content on a webpage and also be able to rewrite url's just like Apache. Specifically I would like to selectively proxy a content and selectively rewrite URL. Where would be a good point to start on this any quick leads or example would be of great help. I already have a squid server setup and functional. Apache rewriteRule : RewriteRule ^suggest(.*)$ http://bla.bla.bla/suggest$1 [L,NC] Apache proxy rule RewriteRule ^(.*)$ http://bla.bla.bla/$1 [L,NC, P] Thank you! Amit
[squid-users] Intermittent slowness
I'm trying to debug an intermittent slowness problem with Squid 3.4.4. Unfortunately I haven't been able to figure out how to reproduce the problem, it just occurs every so often on a production server. I've not yet tried Squid 3.4.5, but there's nothing in the change log that would lead me to believe that this problem has been addressed. I've got an example web fetch from this morning: The network traffic shows: 09:01:54.489515 client - server TCP SYN 09:01:54.489541 client - server TCP SYN/ACK 09:01:54.489555 client - server TCP ACK 09:01:54.490059 client - server HTTP GET request 09:01:54.490074 client - server TCP ACK 09:02:09.492576 client - server TCP FIN (client times out, tears down) 09:02:09.53 client - server TCP ACK 09:02:35.371911 client - server TCP FIN (server tears down connection) The client is port 58469, the server is 3128. As you can see, Squid never replies to the GET request (and actually, in this case the GET request didn't require Squid to contact another server - the authentication credentials were invalid, so it should have produced a 407). Examining the Squid logs (http://persephone.nexusuk.org/~steve/cache.log.trimmed), I can see that Squid didn't accept the connection until 09:02:35.370 What seems to be happening is that helperStatefulHandleRead is being called, and taking several seconds to complete - if this happens frequently enough then the incoming connections get queued up and significantly delayed. See the log below: 2014/05/08 09:01:53.489 kid1| comm.cc(167) comm_read: comm_read, queueing read for local=[::] remote=[::] FD 66 flags=1; asynCall 0x7f9c174c4c00*1 2014/05/08 09:01:53.489 kid1| ModEpoll.cc(139) SetSelect: FD 66, type=1, handler=1, client_data=0x7f9bf4425328, timeout=0 2014/05/08 09:01:53.489 kid1| AsyncCallQueue.cc(53) fireNext: leaving helperHandleRead(local=[::] remote=[::] FD 66 flags=1, data=0x7f9c0a1dad78, size=120, buf=0x7f9c09f4d8a0) 2014/05/08 09:01:53.489 kid1| AsyncCallQueue.cc(51) fireNext: entering helperStatefulHandleRead(local=[::] remote=[::] FD 53 flags=1, data=0x7f9c08ae5928, size=281, buf=0x7f9bfa3e11d0) 2014/05/08 09:01:53.489 kid1| AsyncCall.cc(30) make: make call helperStatefulHandleRead [call44614375] 2014/05/08 09:01:58.329 kid1| AsyncCall.cc(18) AsyncCall: The AsyncCall helperDispatchWriteDone constructed, this=0x7f9c1a39bd40 [call44614625] 2014/05/08 09:01:58.329 kid1| Write.cc(29) Write: local=[::] remote=[::] FD 68 flags=1: sz 188: asynCall 0x7f9c1a39bd40*1 FDs 66 and 68 are connections to external ACL helpers, but the network traffic shows that the external ACLs are answering immediately, so as far as I can tell this delay isn't caused by the helpers themselves. (FD 66 received a query at 09:01:53.480992 and replied at 09:01:53.484808; FD 68 received a query at 09:01:58.334608 and replied at 09:01:58.334661). I *think* FD 53 might be a connection to the ICAP service. I have noticed that squid often seems to use a lot of CPU time when this problem is occurring. Unfortunately I don't know where to go with the debugging now - The current amount of debug logging produces a lot of data, but isn't really detailed enough for me to work out what's going on. But as mentioned, since I can't reproduce this problem in a test environment, I have no choice but to just leave debug logging turned on on a production server. Any suggestions / help from people more familiar with the Squid internals would certainly be helpful. -- - Steve Hill Technical Director Opendium Limited http://www.opendium.com Direct contacts: Instant messager: xmpp:st...@opendium.com Email:st...@opendium.com Phone:sip:st...@opendium.com Sales / enquiries contacts: Email:sa...@opendium.com Phone:+44-844-9791439 / sip:sa...@opendium.com Support contacts: Email:supp...@opendium.com Phone:+44-844-4844916 / sip:supp...@opendium.com
Re: [squid-users] Using Squid to replace apache rewrite rule and mod_proxy rewrite rule
On 9/05/2014 3:21 a.m., Kumar, Amit wrote: Hello, I would like to use squid to fetch content on a webpage and also be able to rewrite url's just like Apache. Specifically I would like to selectively proxy a content and selectively rewrite URL. Where would be a good point to start on this any quick leads or example would be of great help. I already have a squid server setup and functional. Apache rewriteRule : RewriteRule ^suggest(.*)$ http://bla.bla.bla/suggest$1 [L,NC] AFAICT this rule is invalid and never matches anything because no URL can start with 'suggest' like that. Apache proxy rule RewriteRule ^(.*)$ http://bla.bla.bla/$1 [L,NC, P] This is configured in Squid-3.2+ with: acl bla src all deny_info http://bla.bla.bla%R bla http_access deny bla However note that context for the lines is significant in both Apache and Squid. If you have the above embeded in a VirtualHost in Apache then replace the src all in my example with dstdomain followed by the FQDN of that VirtualHost. Particular to Squid ordering is important. Those lines must be in that order, and the http_access position relative to all other http_access determines whether it affects a request or not (they are tested top-down in a first match wins basis). Amos
[squid-users] configuring Eliezer RPMs for CentOS 6 for SMP
Hi there,
First of all, thanks Elizer for providing the packages!
The package installed and run fine with the default, single process
config. I'm trying to follow best-practices like keeping SELinux enabled
(enfocing) and I was happy to see the packages follow the standard
targete policies included on CentOS 6.5.
But, when I tried a SMP setup, I got some SELinux AVCs.
Those were related to the socket files used for IPC betweeb workers and
coordinator, and shm files for shared cache_mem and (i guess) rock
store.
Here are the revelant settings from my squid.conf (this is my test
machine, the production one has more RAM, cores and disk):
--- SNIP ---
workers 2
cache_mem 1024 MB
cache_dir rock /cache/shared 2000 min-size=1 max-size=31000
max-swap-rate=250 swap-timeout=350
cache_dir aufs /cache/worker${process_number} 2000 16 256
min-size=31001
max-size=346030080
--- SNIP ---
I followed standard advice and run audit2allow to build aditional
SELinux
policy modules. After everything was working fine, I took the time to
build and test a consolidated policy module. Here it is:
--- squid-snmp.te ---
module squid-smp 1.0;
# for ipc channels between kids/workers
require {
type var_run_t;
type squid_t;
class sock_file { create write };
}
allow squid_t var_run_t:sock_file { create write };
allow squid_t var_run_t:sock_file create;
# for shm used by cache_mem and rock store(?)
require {
type squid_t;
type tmpfs_t;
class dir { remove_name add_name write };
class file { unlink create };
}
allow squid_t tmpfs_t:dir { remove_name write add_name };
allow squid_t tmpfs_t:file { unlink create };
--- squid-snmp.te ---
To compile and install this policy module, please run:
# checkmodule -M -m -o squid-smp.mod squid-smp.te
# semodule_package -o squid-smp.pp -m squid-smp.mod
# semodule -i squid-smp.pp
I hope Eliezer can add this to the next update to his packages, so
others can try SMP without fighting with SELinux.
But, as I'm not so experienced with SELinux, I'd be glad to receive
advice on how to make this policy more strict, more secure.
For example, I guess I could use squid_var_run_t instead of using
var_run_t, to have a more specific fcontext for squid IPC files. I know
how to do this manually:
# semanage fcontext -a -t squid_var_run_t /var/run/squid(/.*)?
# restorecon -Fv /var/run/squid/
But I don't know hot to add this to the policy file.
I also guess I could be more strict about the shm files (tmpfs_t) but I
don't know how to do this.
Another hurdle I had with the RPM packages was about setting ulimit
nofile. I'm used to add them to /etc/security/limits.conf and this work
fo r CentOS init.d scripts, but not for the squid script.
My solution was to a line to /etc/sysconfig/squid:
ulimit -n 32768
But maybe I should edit the init.d/squid script to use function
daemon() from /etc/init.d/functions or runuser like other CentOS init.d
scripts do.
Well, I hope my findins and ideas can be useful to ther CentOS/RHEL
admins, and someone there can help me making the squid RPM more Red
Hat-ish.
[]s, Fernando Lozano
[squid-users] Advise about cache store in SMP mode, single disk
Hi there, I have a server configured to run in SMP mode with two cache stores: a shared rock store and a dedicated aufs store for each worker. But I have only one physical disk (actually a hardware raid). My preliminary tests show that having the shared rock store brings better response times than having only the aufs stores. But, as I still have some disk contention between workers, I wonder if I should replace the aufs stores by a diskd one. Squid docs are not clear about how diskd is supposed to work in a SMP setup: will I get only one diskd process, and so eliminate the disk contention from aufs stores? Or will squid start a diskd daemon for each worker? I also guess building a CARP setup wound not be so benefical because there would be still many workers competing for the same physical disk. []s, Fernando Lozano
[squid-users] How to throttle wireless clients MISS
When I have squid installed on a system with a wireless upstream link, how to throttle downloads to clients with TCP_MISS only, not to saturate the upstream link ? Actually I am using delay pools for the clients, but these also unnecessarily throttle TCP_HITs, I guess. -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/How-to-throttle-wireless-clients-MISS-tp4665898.html Sent from the Squid - Users mailing list archive at Nabble.com.
