Re: [squid-users] Re: Sibling cache peer for a HTTPS reverse proxy
On 26/07/2014 11:44 a.m., Makson wrote: Thanks for your reminder, i think the HTML RAW tag caused the problem, send the log again. Some records found in access.log in server b, 1406185920.441 1282 172.17.210.5 TCP_MISS/200 814 GET https://serverb.domain:9443/ccm/service/com.ibm.team.scm.common.IVersionedContentService/content/com.ibm.team.filesystem/FileItem/_houAAK2yEeOvOJ84krOqLg/_EPGIsq20EeOEJLtkkn17bg/h2LjUv8WJVDwJ3rcbA6_u3fNuJylQ0sQlSZdRL_IMkA - FIRSTUP_PARENT/172.17.96.148 application/octet-stream 1406185921.151 46349 172.17.210.5 TCP_MISS/200 219202 GET https://serverb.domain:9443/ccm/service/com.ibm.team.scm.common.IVersionedContentService/content/com.ibm.team.filesystem/FileItem/_hpCwIK2yEeOvOJ84krOqLg/_EN-HVK20EeOEJLtkkn17bg/rnslrsXloPXpudCIXRFjShexoc97mr7-2RxWPs7pVnI - FIRSTUP_PARENT/172.17.96.148 application/octet-stream All records found in access.log in server a, 1406185543.094 0 172.17.192.145 UDP_MISS/000 124 ICP_QUERY https://serverb.domain:9443/ccm/authenticated/identity?redirectPath=%2Fccm%2Fjauth-issue-token - HIER_NONE/- - 1406185544.871 0 172.17.192.145 UDP_MISS/000 79 ICP_QUERY https://serverb.domain:9443/ccm/auth/authrequired - HIER_NONE/- - 1406185565.202 0 172.17.192.145 UDP_MISS/000 124 ICP_QUERY https://serverb.domain:9443/ccm/authenticated/identity?redirectPath=%2Fccm%2Fjauth-issue-token - HIER_NONE/- - 1406185566.732 0 172.17.192.145 UDP_MISS/000 79 ICP_QUERY https://serverb.domain:9443/ccm/auth/authrequired - HIER_NONE/- - 1406185615.090 0 172.17.192.145 UDP_MISS/000 124 ICP_QUERY https://serverb.domain:9443/ccm/authenticated/identity?redirectPath=%2Fccm%2Fjauth-issue-token - HIER_NONE/- - Showing that server B is in fact qeuerying server A for the objects. But it would seem that server A did not have them cached. It may be that these responses use Vary: header. ICP does not handle that type of response properly. You may get better behaviour using HTCP instead of ICP between the siblings. I also note that you have 40GB of RAM allocated to each of these Squid instances. Do you actually have over 100GB of RAM on those machines (*excluding* swap space)? Amos
[squid-users] Re: YouTube Resolution Locker
Hi All, Feel free to modify the script (client side) to do not send all requests. As Cassiano said, only the YouTube urls need to be rewritten... Bye Fred -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/YouTube-Resolution-Locker-tp4667042p4667054.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] Re: YouTube Resolution Locker
On 07/26/2014 12:05 PM, Stakres wrote: Hi All, Feel free to modify the script (client side) to do not send all requests. As Cassiano said, only the YouTube urls need to be rewritten... My point here is that you have not mentioned anywhere that your script collects information. Script is made by Unveiltech and it sends all data to Unveiltech servers. Your server can very easily send redirection to their own server and fetch username OR password of any site. (If end user is not technically sound) For example your server can easily redirect http://login.google.com to http://storeid.unveiltech.com/login.google.com/ (which looks exactly same as Google login page). End user will not even know what is happening. Not sure if you did this on purpose OR you are new to programming that you did not realize this huge security and privacy angle. Additionally your script is one small function modification EXAMPLE redirector script. A real script would include full logic of youtube resolution locker (what your storeid server does currently). No offence meant, please. I am just warning other users if they try to use this php script, there is huge security risk. Regards, PS: Sorry for being off-topic on squid mailing list. AMM
[squid-users] Re: YouTube Resolution Locker
HI Amm, Everyone is free to modify the script (client side) by sending YouTube urls only, no need to send all the Squid traffic. Then, we collect nothing, the requests are reviewed by the script and it returns modified urls to lock the YouTube resolutions. We do not make any statistics, we do not share data with internal or external teams. We're not new to programming and we DO realize security and privacy issues, you're free to use the API or not, we force nobody. Everyone is free to spend time for developing a similar function or use ours for a quick solution. The one small functionis for free to all, we spent time to develop this API and we're a commercial company. So, do you work for free ? we do not. If you are interested by the complete API, no problem just contact us and I'm sure we will find an arrangement No problem for the No offence, all comments are welcome. PS: Sorry for being off-topic on squid mailing list, too. Bye Fred -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/YouTube-Resolution-Locker-tp4667042p4667056.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] Re: YouTube Resolution Locker
On 26/07/2014 8:36 p.m., Stakres wrote: HI Amm, Everyone is free to modify the script (client side) by sending YouTube urls only, no need to send all the Squid traffic. Then, we collect nothing, the requests are reviewed by the script and it returns modified urls to lock the YouTube resolutions. We do not make any statistics, we do not share data with internal or external teams. We're not new to programming and we DO realize security and privacy issues, you're free to use the API or not, we force nobody. Everyone is free to spend time for developing a similar function or use ours for a quick solution. The one small functionis for free to all, we spent time to develop this API and we're a commercial company. So, do you work for free ? we do not. If you are interested by the complete API, no problem just contact us and I'm sure we will find an arrangement No problem for the No offence, all comments are welcome. PS: Sorry for being off-topic on squid mailing list, too. Bye Fred It would be better practice to publish a script which is pre-restricted to the YT URLs which your server is useful for and your initial advertisement stated its purpose was. That would protect your servers from excessive bandwidth from naive administrators, help to offer better security by default, and protect your company from this type of complaint and any future legal accusations that may arise from naive use of the script. Amos
[squid-users] Re: YouTube Resolution Locker
Hi Amos, You're totaly right here. We'll prepare a version 1.01 including ACL restrictions... Bye Fred -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/YouTube-Resolution-Locker-tp4667042p4667058.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] Tproxy immediately closing connection
On 25/07/2014 10:02 a.m., Jan Krupa wrote: Hi all, I've been struggling to configure transparent proxy for IPv6 on my Raspberry Pi acting as a router following the guide: http://wiki.squid-cache.org/Features/Tproxy4 Despite all my efforts, all I got was squid squid immediately closing connection after it was established (not rejecting connection, three-way handshake is successful and then the client receives RST packet). Do you have libcap2 installed and libcap2-dev used to build Squid? there have been a few issues where its absence were not notified by Squid. Amos
Re: [squid-users] Re: YouTube Resolution Locker
On 07/26/2014 02:36 PM, Amos Jeffries wrote: On 26/07/2014 8:36 p.m., Stakres wrote: HI Amm, Everyone is free to modify the script (client side) by sending YouTube urls only, no need to send all the Squid traffic. ... Bye Fred It would be better practice to publish a script which is pre-restricted to the YT URLs which your server is useful for and your initial advertisement stated its purpose was. That would protect your servers from excessive bandwidth from naive administrators, help to offer better security by default, and protect your company from this type of complaint and any future legal accusations that may arise from naive use of the script. Amos Yes and also mention on top of script that Script sends URL data to your servers and giving link to privacy policy and if / how you use the URL data. Otherwise you may really have legal issue for capturing data without permission. (Even if you directly throw it in dustbin you can still be sued. - just my two cents) Amm.
RE: [squid-users] YouTube Resolution Locker
Hi All, Free API to lock resolution in YouTube players via your prefered Squid Cache. https://sourceforge.net/projects/youtuberesolutionlocker/ Very easy to use Does it actually lock resolution or limit resolution to = required resolution? Locking the resolution too high can cause problems if there isn't enough bandwidth. Youtube does its own automatic bandwidth detection and chooses an appropriate resolution based on that. It might be better to simply get squid to limit bandwidth from youtube URL's and let the problem take care of itself... Just a thought. James
[squid-users] RE: YouTube Resolution Locker
Hi James, All is in the title YouTube Resolution Locker, so... The API is there to lock the resolution to low value, we think the API to help in reducing bandwidth as HD videos consume a lot. Regarding bandwidth limiter with YouTube, do you think Squid admins need a special script ? There are plenty of samples explaining how to use Quota with Squid... ;) Bye Fred -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/YouTube-Resolution-Locker-tp4667042p4667062.html Sent from the Squid - Users mailing list archive at Nabble.com.
RE: [squid-users] RE: YouTube Resolution Locker
Hi James, All is in the title YouTube Resolution Locker, so... The API is there to lock the resolution to low value, we think the API to help in reducing bandwidth as HD videos consume a lot. Regarding bandwidth limiter with YouTube, do you think Squid admins need a special script ? There are plenty of samples explaining how to use Quota with Squid... ;) Actually after I hit send I realised that it probably wouldn't work anyway, unless youtube really did use a consistent url domain name for their content delivery network... the tricky bit wouldn't be the quota itself, it would be identifying the urls/ip's to limit. A script may be required to identify the current list. James
[squid-users] RE: YouTube Resolution Locker
it probably wouldn't work anyway, unless youtube really did use a consistent url domain name for their content delivery network.. Not correct. It is possible to cache youtubes content using StoreID. Additionally, locking the resolution is much more trivial, as the requested youtube-URL contains the requested resolution as one of the ... pars. Just needs modification. I run a free youtube proxy myself, for years already, and also do the resolution locking myself, to low res, to avoid overload of the proxy. -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/YouTube-Resolution-Locker-tp4667042p4667064.html Sent from the Squid - Users mailing list archive at Nabble.com.
[squid-users] Re: kerberos authentication with load balancers
Hi Giorgi, It would be msktutil -c -b CN=COMPUTERS -s HTTP/proxy1.domain.com -h proxy1.domain.com -k /root/keytab/PROXY.keytab --computer-name PROXY1-K --upn HTTP/proxy1.domain.com--server addc03.domain.com --verbose --enctypes 28 msktutil -c -b CN=COMPUTERS -s HTTP/proxy2.domain.com -h proxy2.domain.com -k /root/keytab/PROXY.keytab --computer-name PROXY2-K --upn HTTP/proxy2.domain.com --server addc03.domain.com --verbose --enctypes 28 and one for DNS RR record msktutil -c -b CN=COMPUTERS -s HTTP/proxy.mia.gov.ge -h proxy1.domain.com -k /root/keytab/PROXY.keytab --computer-name PROXY-K --upn HTTP/proxy.mia.gov.ge --server addc03.domain.com --verbose --enctypes 28 The -h value is not really used. So for the DNS RR you can use either name. Regards Markus Giorgi Tepnadze wrote in message news:53d219ea.1010...@mia.gov.ge... Hi Markus Excuse me for posting in old list, but I have a small question: So I have 2 squid servers (proxy1.domain.com and proxy2.domain.com) and one DNS RR record (proxy.mia.gov.ge). Regarding your recommendation how should I create keytab file. msktutil -c -b CN=COMPUTERS -s HTTP/proxy1.domain.com -h proxy1.domain.com -k /root/keytab/PROXY.keytab --computer-name PROXY1-K --upn HTTP/proxy1.mia.gov.ge --server addc03.domain.com --verbose --enctypes 28 msktutil -c -b CN=COMPUTERS -s HTTP/proxy2.domain.com -h proxy2.domain.com -k /root/keytab/PROXY.keytab --computer-name PROXY2-K --upn HTTP/proxy2.mia.gov.ge --server addc03.domain.com --verbose --enctypes 28 and one for DNS RR record msktutil -c -b CN=COMPUTERS -s HTTP/proxy.domain.com -h proxy1.domain.com -k /root/keytab/PROXY.keytab --computer-name PROXY2-K --upn HTTP/proxy.mia.gov.ge --server addc03.domain.com --verbose --enctypes 28 But there is problem with last one, which server name should I put in -s, -h, --upn and --computer-name? Many Thanks George On 07/02/14 01:26, Markus Moeller wrote: Hi Joseph, it is all possible :-) Firstly I suggest not to use samba tools to create the squid keytab, but use msktutil (see http://wiki.squid-cache.org/ConfigExamples/Authenticate/Kerberos). Then create a keytab for the loadbalancer name ( that is the one configured in IE or Firefox). use this keytab on both proxy servers and use negotiate_kerberos_auth with -s GSS_C_NO_NAME When you say multiple realms, do you have trust between the AD domains or are they separate ? If the domains do not have trust do you intend to use the same loadbalancer name for the users of both domains ? Markus Joseph Spadavecchia wrote in message news:2b43c569f8254a4e82c948ce4c247ed5158...@blx-ex01.alba.local... Hi there, What is the recommended way to configure Kerberos authentication behind two load balancers? AFAIK, based on the mailing lists, I should 1) Create a user account KrbUser on the AD server and add an SPN HTTP/loadbalancer.example.com for the load balancer 2) Join the domain with Kerberos and kinit 3) net ads keytab add HTTP/loadbalancer.example.com@REALM -U KrbUser 4) update squid.conf with an auth helper like negotiate_kerberos_auth -s HTTP/loadbalancer.example.com@REALM Unfortunately, when I try this it fails. The only way I could get it to work at all was by removing the SPN from the KrbUser and associating the SPN with the machine trust account (of the proxy behind the loadbalancer) However, this is not a viable solution since there are two machines behind the load balancer and AD only allows you to associate a SPN with one account. Furthermore, given that I needed step (4) above, is it possible to have load balanced Kerberos authentication working with multiple realms? If so, then how? Many thanks.
[squid-users] Re: Sibling cache peer for a HTTPS reverse proxy
Amos Jeffries wrote Showing that server B is in fact qeuerying server A for the objects. But it would seem that server A did not have them cached. It may be that these responses use Vary: header. ICP does not handle that type of response properly. You may get better behaviour using HTCP instead of ICP between the siblings. I also note that you have 40GB of RAM allocated to each of these Squid instances. Do you actually have over 100GB of RAM on those machines (*excluding* swap space)? Amos Hi Amos, Thanks for your reply, i am now using HTCP, still don't get it work :-( , here are the configurations, # Squid Server A cache_replacement_policy lru memory_replacement_policy lru maximum_object_size 1024 MB maximum_object_size_in_memory 16 MB cache_dir aufs /usr/local/squid/var/cache 307200 256 256 cache_mem 4096 MB cache_store_log none cache_peer app.domain parent 9443 0 no-query originserver ssl login=PASS sslflags=DONT_VERIFY_PEER cache_peer_access app.domain allow all coredump_dir /usr/local/squid/var/cache http_port 3128 http_access allow all htcp_port 4827 htcp_access allow all htcp_clr_access allow all https_port 9443 cert=/usr/local/squid/etc/server.pem accel key=/usr/local/squid/etc/privkey.pem vhost refresh_pattern . 0 20% 4320 cache_mgr admin cachemgr_passwd 123456 all buffered_logs on cache_store_log stdio:/usr/local/squid/var/logs/store.log # Squid Server B cache_replacement_policy lru memory_replacement_policy lru maximum_object_size 1024 MB maximum_object_size_in_memory 16 MB cache_dir aufs /usr/local/squid/var/cache 307200 256 256 cache_mem 4096 MB cache_store_log none cache_peer app.domain parent 9443 0 no-query originserver ssl login=PASS sslflags=DONT_VERIFY_PEER cache_peer servera.domain sibling 3128 4827 htcp cache_peer_access app.domain allow all cache_peer_access servera.domain allow all coredump_dir /usr/local/squid/var/cache http_port 3128 http_access allow all htcp_port 4827 htcp_access allow all htcp_clr_access allow all https_port 9443 cert=/usr/local/squid/etc/server.pem accel key=/usr/local/squid/etc/privkey.pem vhost refresh_pattern . 0 20% 4320 cache_mgr admin cachemgr_passwd 123456 all buffered_logs on cache_store_log stdio:/usr/local/squid/var/logs/store.log And here are access logs, # Squid Server A 1406380411.702 0 172.17.192.145 UDP_MISS/000 0 HTCP_CLR https://serverb.domain:9443/ccm/service/com.ibm.team.scm.common.IScmService - HIER_NONE/- - 1406380414.619 0 172.17.192.145 UDP_MISS/000 0 HTCP_CLR https://serverb.domain:9443/ccm/service/com.ibm.team.scm.common.IScmService - HIER_NONE/- - 1406380415.128 0 172.17.192.145 UDP_MISS/000 0 HTCP_CLR https://serverb.domain:9443/ccm/service/com.ibm.team.scm.common.IScmService - HIER_NONE/- - 1406380416.212 0 172.17.192.145 UDP_MISS/000 0 HTCP_CLR https://serverb.domain:9443/ccm/service/com.ibm.team.filesystem.common.IFilesystemService - HIER_NONE/- - # Squid Server B ... ... 1406380429.286 90 172.17.210.5 TCP_MISS/200 664 GET https://serverb.domain:9443/ccm/service/com.ibm.team.scm.common.IVersionedContentService/content/com.ibm.team.filesystem/FileItem/_TUaCUK2xEeOvOJ84krOqLg/_aN3NdK2xEeOEJLtkkn17bg/DjuXbV8AG7VOyHf-ds_UzKy02yApE33wddUBirD98Lo - FIRSTUP_PARENT/172.17.96.148 application/octet-stream 1406380429.290137 172.17.210.5 TCP_MISS/200 11040 GET https://serverb.domain:9443/ccm/service/com.ibm.team.scm.common.IVersionedContentService/content/com.ibm.team.filesystem/FileItem/_TVSMEa2xEeOvOJ84krOqLg/_aPuOrK2xEeOEJLtkkn17bg/mogLTY-2R4AJu2OHAShQtBaydLlMgHo34-Cqkzyaxws - FIRSTUP_PARENT/172.17.96.148 application/octet-stream -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Sibling-cache-peer-for-a-HTTPS-reverse-proxy-tp4667011p4667066.html Sent from the Squid - Users mailing list archive at Nabble.com.
[squid-users] Re: Sibling cache peer for a HTTPS reverse proxy
Would it be related to the content type? I can not find any HTCP/ICP query records in server a for objects with application/octet-stream content type. -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Sibling-cache-peer-for-a-HTTPS-reverse-proxy-tp4667011p4667067.html Sent from the Squid - Users mailing list archive at Nabble.com.
