Re: [squid-users] SSL Terminating Reverse Proxy with Referral Tracking
On 9/14/21 6:09 PM, Amos Jeffries wrote: b) If those upstream servers are embedding URLs for clients to directly contact the XaaS services. Then your desire is not possible without redesigning the upstream service(s) such that they stop exposing their use of the XaaS. Which often also means redesigning the XaaS service itself too. I don't know about Squid, but I do know that it's possible to manipulate traffic with Apache in a similar role. I've done so a number of times using the mod_proxy and associated mod_proxy_html modules. This allows Apache to re-write content as it's passing through the Apache proxy. I wonder if Squid's ICAP support might allow something to modify traffic as it passes through the Squid proxy. That is not possible for a reverse-proxy to do. It will never see the third-party traffic, as mentioned by (b) above. Sure it is. }:-) -- Grant. . . . unix || die smime.p7s Description: S/MIME Cryptographic Signature ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] SSL Terminating Reverse Proxy with Referral Tracking
On 9/14/21 7:12 PM, Grant Taylor wrote: I have concerns about "SSL terminating". It sounds to me like you are decidedly outside of the typical enterprise or home network scenario where you are wanting to terminate / intercept / bump-in-the-wire TLS connections. As such, I have *SERIOUS* /concerns/ about the security implications of this. -- But, I'm going to assume that you are well aware of the implications and are addressing them properly. But I'd be remiss to not say something. Moving on. I meant to add, I'm not convinced that you /need/ to do TLS termination. Or said another way, I'm not convinced that simply proxying CONNECT requests isn't sufficient. Do you actually /need/ to terminate the TLS? Or is simply proxying the CONNECT request sufficient? Can you stay out of the TLS stream, thereby avoiding any and all security concerns associated with TLS termination? Proxies have been passing TLS traffic for decades without TLS termination. -- Grant. . . . unix || die smime.p7s Description: S/MIME Cryptographic Signature ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] SSL Terminating Reverse Proxy with Referral Tracking
On 9/12/21 10:16 PM, Mehrdad Fatemi wrote: Hi Everyone, Hi, TL;DR: Proxy Auto Configuration I'm looking for an elegant technology option to have telcos zero-rate all of the traffic to a set of online destinations. I assume that "zero rating" means that specific destinations, e.g. the proxy server, are no-charge from the telco's customers point of view. If this is not what you are meaning, please correct me. Using an SSL terminating reverse proxy could be a potential answer to this as we can focus on zero-rating the proxy's downstream traffic with each ISP/Telco without worrying about upstream servers. I have concerns about "SSL terminating". It sounds to me like you are decidedly outside of the typical enterprise or home network scenario where you are wanting to terminate / intercept / bump-in-the-wire TLS connections. As such, I have *SERIOUS* /concerns/ about the security implications of this. -- But, I'm going to assume that you are well aware of the implications and are addressing them properly. But I'd be remiss to not say something. Moving on. Aside: I sat on this message for a few days while messing with my own TLS bump-in-the-wire /in/ /my/ /house/ on my /home/ network. As such, I'm perfectly fine with TLS termination within environments that have the authority to do so. ;-) -- I sat on the message while working on my own Proxy Auto Configuration script to have multiple clients do what I want. Further aside: I'm *EXTREMELY* happy with Squid's support for TLS bump-in-the-wire for my use cases; allowing ancient clients to use Squid as an encryption gateway between SSL3 / TLSv1 / TLSv1.1 and TLSv1.2 / TLSv1.3. The ability to filter various things like tracking pixels, and the caching is wonderful. -- I can't quite wrap my head around why Squid improves performance on a GPON connection, but it does. I would have thought that 1 Gbps connection would negate the need for local caching. There are two challenges to address here though: 1) Modern web applications on the upstream servers use many 3rd party and X-a-a-S resources (e.g. embedded media, libraries, etc) that we also want to pass through the proxy to ensure they are zero-rated. That's going to be a game of Whack-a-Mole. There's also the possibility that you will proxy ~> zero-rate some common library that many other sites that don't pass through your infrastructure will use. So I suspect it's an impure WaM game at best. 2) For a user to complete an end-to-end process they may get referred to 3rd party websites (like a payment gateway) that we only want to zero-rate if the referral is from one of our designated upstream servers. I suspect that trying to integrate conditional behavior based on account balance is going to be ... tricky, if not problematic. I'd suggest worrying about that after the fact or at a later point in the process. Any advice on whether and how Squid and other related technologies could help is much appreciated. I feel like a judicious use of a Proxy Auto Configuration (PAC) file / script may be a good start. It should be relatively easy for subscribers to configure their devices to utilize. Then you can update the PAC file as the WaM game requires. The PAC has the added advantage that you can direct proxy traffic to different proxy servers as necessary. As for normal (forward) vs reverse proxy is concerned, it seems to me like your proxy will be acting as both, a reverse proxy / accelerator /and/ a /conditional/ forward proxy. The conditionality is based on the result of the PAC file's FindProxyForURL() function. You are in some ways acting as a reverse proxy / accelerator for specific sites. You are also acting as a forward proxy for clients. The behaviors just overlap in your use case. The secret sauce is in the PAC file; what does and does not get sent to your proxy. Seeing as how you are dealing with subscribers, you probably do not want the closely related / largely overlapping Web Proxy Auto Discovery (WPAD) functionality. IMHO WPAD points to a PAC file. -- Grant. . . . unix || die smime.p7s Description: S/MIME Cryptographic Signature ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] SSL Terminating Reverse Proxy with Referral Tracking
On 13/09/21 4:16 pm, Mehrdad Fatemi wrote: Hi Everyone, I'm looking for an elegant technology option to have telcos zero-rate all of the traffic to a set of online destinations. Can you clarify what you mean exactly by "zero rate" ? What does it have to do with actions the proxy is performing? Using an SSL terminating reverse proxy could be a potential answer to this as we can focus on zero-rating the proxy's downstream traffic with each ISP/Telco without worrying about upstream servers. There are two challenges to address here though: 1) Modern web applications on the upstream servers use many 3rd party and X-a-a-S resources (e.g. embedded media, libraries, etc) that we also want to pass through the proxy to ensure they are zero-rated. To be clear; "reverse proxy" is just an old term for CDN frontend. It requires public DNS records for the domains it services point all their traffic to the CDN/proxy. I'm not sure you are talking about the same thing. Maybe you are needing an interception proxy or other QoS related systems. In general; a) If those upstream servers are doing XaaS fetches as their internal operations there is no relevance to the gateway. It simply passes traffic to upstream and they do their thing. b) If those upstream servers are embedding URLs for clients to directly contact the XaaS services. Then your desire is not possible without redesigning the upstream service(s) such that they stop exposing their use of the XaaS. Which often also means redesigning the XaaS service itself too. 2) For a user to complete an end-to-end process they may get referred to 3rd party websites (like a payment gateway) that we only want to zero-rate if the referral is from one of our designated upstream servers. That is not possible for a reverse-proxy to do. It will never see the third-party traffic, as mentioned by (b) above. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] logformat odd values
On 15/09/21 7:04 am, Moti Berger wrote:
Hi
I have the followings in squid.conf:
logformat metrics %icap::tt %adapt::all_trs %adapt::sum_trs
%{service_req_a}adapt::sum_trs %{service_resp_a}adapt::sum_trs
%{service_req_b}adapt::sum_trs %{service_resp_b}adapt::sum_trs
access_log daemon:/var/log/squid/metrics.log metrics
icap_service service_req_a reqmod_precache bypass=1 on-overload=wait
routing=1 icap://a.y:12345/request
icap_service service_req_b reqmod_precache bypass=1 on-overload=wait
icap://b.y:10101/request
adaptation_service_chain svcRequest service_req_a service_req_b
adaptation_access svcRequest deny manager
adaptation_access svcRequest allow all
icap_service service_resp_a respmod_precache bypass=1
on-overload=wait routing=1 icap://a.y:12345/response
icap_service service_resp_b respmod_precache bypass=1
on-overload=wait icap://b.y:10101/response
adaptation_service_chain svcResponse service_resp_a service_resp_b
adaptation_access svcResponse deny manager
adaptation_access svcResponse allow all
I see in metrics.log lines like this:
4 4,180 4,180 4 180 - -
Now I wonder how come the value of %icap:tt isn't at least as the sum of
all the numbers appear on %adapt::all_trs or %adapt::sum_trs (assuming
no failed transactions)?
%icap:: codes are relevant to ICAP.
%adapt:: codes are relevant to all adaptation types: ICAP, eCAP, and (in
theory, though often not really) redirector and URL re-writer helpers.
NP: failed transactions timers are included in all three metrics.
If %icap:tt isn't at least the sum of all ICAPs processing time, what is?
AFAIK you are right - it is supposed to be the sum of all processing
time for ICAP related things.
It does seem to be a bit odd that only the first one ("4") is counted.
Likely a bug IMO.
Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Compile/Rebuilding on debian bullseye (or buster)
On 14/09/21 9:22 pm, L.P.H. van Belle wrote: Hai Amos, Im attempting to make a squid 5.1 build based on the bullseye squid/debian folder. ( ps. Im building with sbuilder ) Now, this "normaly" worked since squid 3.2 for me, copy the debian folder, make minor adjustments if needed, Just with latest adjustments, well, i cant make it work. This was my last adjustmated.. * Used build : squid-5.1-20210804-r1f9e52827 of 04 Aug 2021 * Refreshed patches, removed patches already included. * d/control lower debhelper to 12 to allow building Changed d/rules, added. -srcdir=. --disable-dependency-tracking Any suggestions what i can do here? I am a bit stuck here myself with this build style, that is part of why the package is not already updated. The Debian auto-build system adds those options and AFAICT does not provide an easy way to avoid. And i tested also with my last settings from my backport to Debian buster and squid 4.16(ssl enabled) Resulting in : configure: Samba TrivialDB library support: no configure: error: external acl helper time_quota ... found but cannot be built make: *** [/usr/share/cdbs/1/class/autotools.mk:46: debian/stamp-autotools] Error 1 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2 Debian used to use the BerkeleyDB, so the normal Squid build dependencies pulls the library in for that. If libdb-dev is no longer available you will need to install libtdb-dev package. Also, (for now) if you are using the ext_time_quote_acl or ext_session_acl helpers you will need to manually purge their databases on install/upgrade. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] logformat odd values
Hi
I have the followings in squid.conf:
> logformat metrics %icap::tt %adapt::all_trs %adapt::sum_trs
> %{service_req_a}adapt::sum_trs %{service_resp_a}adapt::sum_trs
> %{service_req_b}adapt::sum_trs %{service_resp_b}adapt::sum_trs
> access_log daemon:/var/log/squid/metrics.log metrics
>
> icap_service service_req_a reqmod_precache bypass=1 on-overload=wait
> routing=1 icap://a.y:12345/request
> icap_service service_req_b reqmod_precache bypass=1 on-overload=wait
> icap://b.y:10101/request
> adaptation_service_chain svcRequest service_req_a service_req_b
> adaptation_access svcRequest deny manager
> adaptation_access svcRequest allow all
> icap_service service_resp_a respmod_precache bypass=1 on-overload=wait
> routing=1 icap://a.y:12345/response
> icap_service service_resp_b respmod_precache bypass=1 on-overload=wait
> icap://b.y:10101/response
> adaptation_service_chain svcResponse service_resp_a service_resp_b
> adaptation_access svcResponse deny manager
> adaptation_access svcResponse allow all
I see in metrics.log lines like this:
> 4 4,180 4,180 4 180 - -
>
Now I wonder how come the value of %icap:tt isn't at least as the sum of
all the numbers appear on %adapt::all_trs or %adapt::sum_trs (assuming no
failed transactions)?
If %icap:tt isn't at least the sum of all ICAPs processing time, what is?
Thanks,
Moti
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
[squid-users] Compile/Rebuilding on debian bullseye (or buster)
Hai Amos,
Im attempting to make a squid 5.1 build based on the bullseye squid/debian
folder.
( ps. Im building with sbuilder )
Now, this "normaly" worked since squid 3.2 for me, copy the debian folder, make
minor adjustments if needed,
Just with latest adjustments, well, i cant make it work.
This was my last adjustmated..
* Used build : squid-5.1-20210804-r1f9e52827 of 04 Aug 2021
* Refreshed patches, removed patches already included.
* d/control lower debhelper to 12 to allow building
Changed d/rules, added. -srcdir=. --disable-dependency-tracking
Any suggestions what i can do here?
If you want/need, i have the complete build log available.
The confirgure part and error.
configure: exit 1
dh_auto_configure: ./configure --build=x86_64-linux-gnu --prefix=/usr
--includedir=\${prefix}/include --mandir=\${prefix}/share/man
--infodir=\${prefix}/share/info --sysconfdir=/etc --localstatedir=/var
--disable-silent-rules --libdir=\${prefix}/lib/x86_64-linux-gnu
--runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking
"BUILDCXXFLAGS=-g -O2 -fdebug-prefix-map=/<>=.
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
-D_FORTIFY_SOURCE=2 -Wl,-z,relro -Wl,-z,now " BUILDCXX=g\+\+
--with-build-environment=default "--enable-build-info=Debian linux"
--datadir=/usr/share/squid --sysconfdir=/etc/squid --libexecdir=/usr/lib/squid
--mandir=/usr/share/man --enable-inline --disable-arch-native
--enable-async-io=8 --enable-storeio=ufs,aufs,diskd,rock
--enable-removal-policies=lru,heap --enable-delay-pools --enable-cache-digests
--enable-icap-client --enable-follow-x-forwarded-for --enable-au
th-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB
--enable-auth-digest=file,LDAP --enable-auth-negotiate=kerberos,wrapper
--enable-auth-ntlm=fake,SMB_LM
--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,time_quota,unix_group,wbinfo_group
--enable-security-cert-validators=fake --enable-storeid-rewrite-helpers=file
--enable-url-rewrite-helpers=fake --enable-eui --enable-esi --enable-icmp
--enable-zph-qos --enable-ecap --disable-translation
--with-swapdir=/var/spool/squid --with-logdir=/var/log/squid
--with-pidfile=/run/squid.pid --with-filedescriptors=65536 --with-large-files
--with-default-user=proxy --srcdir=. --disable-dependency-tracking
--enable-linux-netfilter --with-systemd --with-gnutls returned exit code 1
make[1]: *** [debian/rules:79: override_dh_auto_configure] Error 2
make[1]: Leaving directory '/<>'
make: *** [debian/rules:71: build] Error 2
And i tested also with my last settings from my backport to Debian buster and
squid 4.16(ssl enabled)
Resulting in :
configure: Samba TrivialDB library support: no
configure: error: external acl helper time_quota ... found but cannot be built
make: *** [/usr/share/cdbs/1/class/autotools.mk:46: debian/stamp-autotools]
Error 1
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
So im a bit stuck here..
And yes, a ./configure does configure it all without problems..
Any tips on this ?
Greetz,
Louis
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
