Re: [squid-users] logformat odd values
Hi
It's possible that in some cases they will run concurrently, however not in
my setup. The ICAPs are chained. If the chain doesn't guarantee it please
let me know since I rely on it.
Regarding what you said about things that can happen in parallel, I'm OK
with it since what I care the most is the extra time my ICAPs add to the
user latency and this is what I would like to measure.
Thanks
On Wed, Sep 15, 2021, 20:47 Alex Rousskov
wrote:
> On 9/14/21 3:04 PM, Moti Berger wrote:
>
> > I have the followings in squid.conf:
> >
> > logformat metrics %icap::tt %adapt::all_trs %adapt::sum_trs
> > %{service_req_a}adapt::sum_trs %{service_resp_a}adapt::sum_trs
> > %{service_req_b}adapt::sum_trs %{service_resp_b}adapt::sum_trs
> > access_log daemon:/var/log/squid/metrics.log metrics
> >
> >
> >
> > icap_service service_req_a reqmod_precache bypass=1 on-overload=wait
> > routing=1 icap://a.y:12345/request
> > icap_service service_req_b reqmod_precache bypass=1 on-overload=wait
> > icap://b.y:10101/request
> > adaptation_service_chain svcRequest service_req_a service_req_b
> > adaptation_access svcRequest deny manager
> > adaptation_access svcRequest allow all
> > icap_service service_resp_a respmod_precache bypass=1
> > on-overload=wait routing=1 icap://a.y:12345/response
> > icap_service service_resp_b respmod_precache bypass=1
> > on-overload=wait icap://b.y:10101/response
> > adaptation_service_chain svcResponse service_resp_a service_resp_b
> > adaptation_access svcResponse deny manager
> > adaptation_access svcResponse allow all
> >
> >
> > I see in metrics.log lines like this:
> >
> > 4 4,180 4,180 4 180 - -
> >
> >
> > Now I wonder how come the value of %icap:tt isn't at least as the sum of
> > all the numbers appear on %adapt::all_trs or %adapt::sum_trs (assuming
> > no failed transactions)?
>
> There is probably a bug somewhere, but please note that %icap:tt may not
> be the sum of individual transaction response times (in _some_ cases)
> even after that bug is fixed because those individual transactions may
> run _concurrently_ (i.e. partially overlap in time).
>
>
> > If %icap:tt isn't at least the sum of all ICAPs processing time, what is?
>
> Bugs notwithstanding, it is approximate time a master transaction spent
> doing adaptation (including checking whether adaptation is necessary).
> This stopwatch ticks when adaptation_access ACLs are checked and also
> when at least one adaptation transaction associated with that master
> transaction is in progress.
>
> Please note that a master transaction can do a lot of different things
> at once or in parallel. For example, it can communicate with an HTTP
> client while communicating with an FTP server while communicating with
> an eCAP REQMOD adaptation service while communicating with a DNS server
> to decide whether to start communicating with an ICAP RESPMOD service.
>
>
> HTH,
>
> Alex.
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
[squid-users] logformat odd values
Hi
I have the followings in squid.conf:
> logformat metrics %icap::tt %adapt::all_trs %adapt::sum_trs
> %{service_req_a}adapt::sum_trs %{service_resp_a}adapt::sum_trs
> %{service_req_b}adapt::sum_trs %{service_resp_b}adapt::sum_trs
> access_log daemon:/var/log/squid/metrics.log metrics
>
> icap_service service_req_a reqmod_precache bypass=1 on-overload=wait
> routing=1 icap://a.y:12345/request
> icap_service service_req_b reqmod_precache bypass=1 on-overload=wait
> icap://b.y:10101/request
> adaptation_service_chain svcRequest service_req_a service_req_b
> adaptation_access svcRequest deny manager
> adaptation_access svcRequest allow all
> icap_service service_resp_a respmod_precache bypass=1 on-overload=wait
> routing=1 icap://a.y:12345/response
> icap_service service_resp_b respmod_precache bypass=1 on-overload=wait
> icap://b.y:10101/response
> adaptation_service_chain svcResponse service_resp_a service_resp_b
> adaptation_access svcResponse deny manager
> adaptation_access svcResponse allow all
I see in metrics.log lines like this:
> 4 4,180 4,180 4 180 - -
>
Now I wonder how come the value of %icap:tt isn't at least as the sum of
all the numbers appear on %adapt::all_trs or %adapt::sum_trs (assuming no
failed transactions)?
If %icap:tt isn't at least the sum of all ICAPs processing time, what is?
Thanks,
Moti
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] New line in logformat
Is there a performance hit by using a logformat per metric? Let's say I
need 5-10 metrics.
On Thu, Sep 2, 2021 at 8:15 PM His Shadow wrote:
> I think it would be easier to just write a logfile daemon, than
> patching squid. The protocol is very easy.
> http://www.squid-cache.org/Doc/config/logfile_daemon/
> You'd just ignore everything but lines starting with L.
>
> чт, 2 сент. 2021 г. в 16:49, Amos Jeffries :
> >
> > On 3/09/21 12:07 am, Moti Berger wrote:
> > > Hi
> > >
> > > I want to send metrics to statsd, so I tried doing the following:
> > >
> > > logformat metrics_statsd
> > >
> my_service_latency:%{my_service}adapt::sum_trs|ms\nicap_total:%icap::tt|ms
> > >
> > > access_log tcp://1.2.3.4:8125 <http://1.2.3.4:8125>
> > > logformat=metrics_statsd
> > >
> > >
> > > In statsd daemon I see only the first (it supports multiple metrics
> > > separated by \n -
> >
> > Ouch. You may be able to work around this with a logformat for each
> > metric you want to send and a access_log line sending that metric.
> >
> > Otherwise you are looking at a patch for either Squid or the statsd
> daemon.
> >
> > I am happy to assist with a patch for Squid if you want to. Contact me
> > directly for rates etc.
> >
> > Amos
> > ___
> > squid-users mailing list
> > squid-users@lists.squid-cache.org
> > http://lists.squid-cache.org/listinfo/squid-users
>
>
>
> --
> HisShadow
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
[squid-users] New line in logformat
Hi
I want to send metrics to statsd, so I tried doing the following:
> logformat metrics_statsd
> my_service_latency:%{my_service}adapt::sum_trs|ms\nicap_total:%icap::tt|ms
>
access_log tcp://1.2.3.4:8125 logformat=metrics_statsd
In statsd daemon I see only the first (it supports multiple metrics
separated by \n -
https://github.com/statsd/statsd/blob/master/docs/server.md)
>From http://www.squid-cache.org/Doc/config/logformat/ I understand that
'\n' is what I need to use in order to get a new line, but it seems I get
two characters instead - '\' and 'n'.
What is the proper way to have a new line?
Thanks,
Moti
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
[squid-users] Log to statsd
Hi Is there a way to configure Squid to output the logs to statsd rather than a file? Today I have this: > +logformat my_metrics %icap::tt % +access_log daemon:/var/log/squid/my_metrics.log my_metrics > However I would like to avoid the overhead in parsing the log file by using statsd or something similar. Thanks, Moti ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] Sharing info from eCAP adapter with other adapters
Hi
I have an env with Squid 4 with several ICAPs.
In this env, I set in squid.conf the following:
adaptation_masterx_shared_names X-My-Header
>
and I add an ICAP header X-My-Header with some data in one ICAP and next
ICAPs can use it (I also see it in tcpdump output).
Now I'm writing an eCAP and I did the following (for testing):
static const libecap::Name myHeader("X-My-Header",
> libecap::Name::NextId());
const libecap::Area Adapter::Xaction::option(const libecap::Name &name)
> const
> {
>
> if (name == myHeader) {
std::string t = "popo";
return libecap::Area(t.data(), t.size());
>
}
return libecap::Area();
> }
>
>
> void Adapter::Xaction::visitEachOption(libecap::NamedValueVisitor
> &visitor) const
> {
>
> std::string t = "popo";
>
> visitor.visit(myHeader, libecap::Area(t.data(), t.size()));
>
}
It is much like the clamAV eCAP example.
What I see in tcpdump output is the following:
X.qZREQMOD icap://my.proxy:1234/request ICAP/1.0
> Host: my.proxy:1234
> Date: Wed, 14 Jul 2021 20:37:33 GMT
> X-My-Header: .
> Encapsulated: req-hdr=0, null-body=427
> Preview: 0
> Allow: 204
Meaning, it indeed added the X-My-Header as ICAP header for the benefit of
the ICAP server on the chain but it seems the value is just a dot.
What am I doing wrong?
BTW, I'm struggling to find a decent eCAP interface documentation. Can you
please help me understand what is the difference between 'option' and
'visitEachOption' methods?
Thanks,
Moti
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
[squid-users] Refrain from Cache Manager API requests to reach ICAPs
Hi I established an environment with Squid and Datadog. It periodically calls the endpoint: /squid-internal-mgr/counters > Those requests are also sent to the ICAPs. Is there a way to make Squid not to pass those requests to the ICAPs? Thanks, Moti ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] HTTP requests queue
Hi, I'm using Squid with ICAPs and ECAPs. Some of them are faster than others. Does squid hold one queue for each chain (REQMOD/RESPMOD) or per ICAP/ECAP? Meaning, if I have one ECAP and one ICAP in a chain where the faster ECAP is first. Will the ECAP keep getting new HTTP requests to adapt even if the ICAP that comes after it in the chain hasn't finished adapting the previous requests? Or is it the other way around where a new HTTP request will be sent to adaptation only after the previous one was handled by all ICAP/ECAPs in the chain? Thanks, Moti ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] cacheProtoClientHttpRequests OID
Hi, If I have the cache disabled: > cache deny all Can we be sure the OID cacheProtoClientHttpRequests really counts the HTTP requests received by Squid (v4.15)? https://wiki.squid-cache.org/Features/Snmp Thanks, Moti ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] Sporadic FATAL on squid 4.14
Hi I'm occasionally having the following error (entire compressed cache.log is 150K, I'll attach it if required). > 2021/04/25 09:42:36.226| 33,2| AsyncCallQueue.cc(55) fireNext: entering > clientListenerConnectionOpened(local=0.0.0.0:3128 remote=[::] flags=9, err=0, > HTTP Socket port=0x55ba5f6e3a50) > 2021/04/25 09:42:36.226| 33,2| AsyncCall.cc(38) make: make call > clientListenerConnectionOpened [call19] > 2021/04/25 09:42:36.226| 0,3| client_side.cc(3563) OpenedHttpSocket: check > failed: NHttpSockets > 0 > exception location: client_side.cc(3563) OpenedHttpSocket > > 2021/04/25 09:42:36.226| FATAL: check failed: NHttpSockets > 0 > exception location: client_side.cc(3563) OpenedHttpSocket > > 2021/04/25 09:42:36.226| 24,8| SBuf.cc(70) ~SBuf: SBuf1801 destructed > 2021/04/25 09:42:36.226| 24,9| MemBlob.cc(82) ~MemBlob: destructed, > this=0x55ba5f668a10 id=blob1050 capacity=40 size=2 > 2021/04/25 09:42:36.226| 24,8| SBuf.cc(30) SBuf: SBuf3438 created > 2021/04/25 09:42:36.226| 24,8| SBuf.cc(38) SBuf: SBuf3439 created from id > SBuf3438 > 2021/04/25 09:42:36.226| 24,8| SBuf.cc(70) ~SBuf: SBuf3438 destructed > > Could not figure out what may cause it. Can you please advise me on a set of actions I can perform to gather more information? Output of 'squid --version': > Squid Cache: Version 4.14 > Service Name: squid > > This binary uses OpenSSL 1.1.1d 10 Sep 2019. For legal restrictions on > distribution see https://www.openssl.org/source/license.html > > configure options: '--prefix=/usr' '--localstatedir=/var' > '--libexecdir=/usr/lib/squid' '--datadir=/usr/share/squid' > '--sysconfdir=/etc/squid' '--with-default-user=proxy' > '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' > '--mandir=/usr/share/man' '--enable-inline' '--disable-arch-native' > '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' > '--enable-removal-policies=lru,heap' '--enable-delay-pools' > '--enable-cache-digests' '--enable-icap-client' > '--enable-follow-x-forwarded-for' > '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' > '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' > '--enable-auth-ntlm=fake,SMB_LM' > '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' > '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' > '--enable-zph-qos' '--enable-ecap' '--disable-translation' > '--with-swapdir=/var/spool/squid' '--with-filedescriptors=65536' > '--with-large-files' '--enable-linux-netfilter' '--enable-ssl' > '--enable-ssl-crtd' '--with-openssl' > Many thanks, Moti ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid 5.0.4 crash
Hi I would like to share also that we haven't encountered this error before, also when using 5.0.4. We see that this error is in correlation with an error from networkd-dispatcher: > ERROR:Unknown interface index 37 seen even after reload We looked a month back in our logs for this error from networkd-dispatcher and we found none. The changes we made before seeing the error for the first time were: 1. Add a new ICAP server to RESPMOD (we already had two on REQMOD and one on RESPMOD). 2. Start using the ICAP header X-Next-Services 3. Start using the adaptation_masterx_shared_names conf. Maybe you can think of something related to above that might caused it? Thanks, Moti On Sun, Apr 4, 2021 at 4:14 PM Amos Jeffries wrote: > On 4/04/21 11:44 pm, Moti Berger wrote: > > Hi > > > > I noticed Squid sporadically crashes with the following error (taken > > from cache.log): > > > > 2021/04/01 21:58:03| FATAL: check failed: > > !request->pinnedConnection() > > > > exception location: FwdState.cc(1055) connectStart > > > > current master transaction: master4104 > > > > This is <https://bugs.squid-cache.org/show_bug.cgi?id=5090> > > > Amos > ___ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users > ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] Squid 5.0.4 crash
Hi I noticed Squid sporadically crashes with the following error (taken from cache.log): > 2021/04/01 21:58:03| FATAL: check failed: !request->pinnedConnection() > > exception location: FwdState.cc(1055) connectStart > > current master transaction: master4104 > > 2021/04/01 21:58:03| Removing PID file (/var/run/squid.pid) > > current master transaction: master4104 > > > We run it in a container in a t3.small EC2 instance. It crashes dozens of times a day, can't tell what triggers it (always with the same error, beside the master transaction number). I can attach the logs and squid.conf, around 7MB compressed. Please advise what's the preferred way of sharing them (or part of them). Thanks, Moti ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] X-Next-Services
Hi I want to be able to skip all subsequent ICAP servers defined in squid based on some logic I have in one of my ICAP servers. I used the X-Next-Services and it seems to control only the current ICAP chain. I also saw it while configuring two ICAP servers to handle requests and one ICAP server to handle responses. When the header was sent from the first ICAP on the request chain, the ICAP request didn't arrive at the second ICAP server handling requests but did get to the ICAP server that handles responses. I wish to also skip the ICAP which handles the responses. Is that possible? If not, is it possible for an ICAP server to add an ICAP header during request handling to be read by the other ICAP servers that come after it when they handle the same request or the same request's response? I'm aware I can add a header to the encapsulated HTTP request, but this is something I want to avoid because I don't want to have a performance hit because I need to copy all HTTP request bodies for this header (I work in zero-bytes preview mode). Thanks, Moti ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Adding headers in ICAP server with no preview
Thank you very much, that really helped!
On Mon, Jan 18, 2021 at 6:48 PM Alex Rousskov <
rouss...@measurement-factory.com> wrote:
> On 1/18/21 6:45 AM, Moti Berger wrote:
>
> > If the ICAP server sets 'Preview: 0' in the OPTIONS it means that when
> > the ICAP client sends a request, it should not contain the body.
>
> The above summary may mislead many readers. I would describe the
> protocol operation differently:
>
> * Preview in an OPTIONS response indicates that the server supports
> Preview in general and specifies the maximum Preview size the client
> should use (e.g., Preview:0 limits Preview to HTTP headers).
>
> * The Preview mode for a specific REQMOD or RESPMOD transaction is
> signaled in the corresponding REQMOD or RESPMOD request (not a previous
> OPTIONS response) by adding a Preview:N ICAP request header (Preview:0
> specifies a headers-only Preview for the current transaction).
>
> * The REQMOD or RESPMOD transaction with a Preview:0 request header is
> split into two phases. During the first phase, the client must not send
> the virgin body. During the second phase, if any, the client must send
> the virgin body. Both phases comprise a single ICAP transaction, with a
> single ICAP request and a single ICAP response. Thus, one cannot say
> that this transaction (as a whole) "should not contain a body".
>
>
> > This is the REQMOD request:
> >
> > F..n...DREQMOD icap://censor-req.proxy:14590/request ICAP/1.0
> > Host: censor-req.proxy:14590
> > Date: Mon, 18 Jan 2021 11:34:54 GMT
> > Encapsulated: req-hdr=0, req-body=222
> > Preview: 0
> > Allow: 204, trailers
> > X-custom-header: data
> >
> > POST http://www.dst-server.com:2/v1/test HTTP/1.1
> > User-Agent: python-requests/2.25.1
> > Accept-Encoding: gzip, deflate
> > Accept: */*
> > Content-Length: 10
> > Content-Type: application/json
> > Host: www.dst-server.com:2 <http://www.dst-server.com:2>
>
> > The ICAP 'Encapsulated' header has a req-body even though no 'body'
> > should be in this request.
>
> Not exactly. The request may not be over at this point. Please see my
> third bullet above for details.
>
>
> > The ICAP server has no way to encapsulate the HTTP request body if it
> > didn't get it.
>
> To get the request body in Preview:0 mode, the ICAP server must respond
> with ICAP 100 (Continue).
>
>
> > I want to avoid sending the body because the adaptation is body agnostic.
>
> Yes, I know, but you have to work within the ICAP protocol boundaries.
> ICAP simply does not optimize your use case! After you have the basics
> working well, you can invest in implementing a use-original-body ICAP
> extension[1] that, in _some_ cases, can prevent the body exchange while
> adapting HTTP headers.
>
> Alternatively, you can use an existing (extendible) ICAP server to do
> the legwork for you [2]. Many individuals and companies have learned the
> hard way that implementing an ICAP service correctly from scratch is
> very difficult and often prohibitively expensive.
>
> [1]
>
> http://www.icap-forum.org/documents/specification/draft-icap-ext-partial-content-07.txt
>
> [2] https://wiki.squid-cache.org/Features/ICAP#ICAP_Servers
>
>
> HTH,
>
> Alex.
>
>
>
> > On Sun, Jan 17, 2021 at 11:34 PM Alex Rousskov wrote:
> >
> > On 1/17/21 3:08 PM, Moti Berger wrote:
> > > What should the ICAP response look like?
> >
> > The vast majority off ICAP responses containing an HTTP POST message
> > will look like ICAP header + HTTP header + HTTP body. Please see RFC
> > 3507 and its errata for examples of and discussion about those three
> > components. It should help avoid guessing and developing by examples
> > (which usually leads to bugs, especially where ICAP is involved).
> >
> >
> > > What I do is to reply like this:
> > >
> > > (dI./M..ICAP/1.0 200 OK
> > > ISTag: "SjIzlRA4te41axxcDOoiSl6rBRg4ZK"
> > > Date: Sun, 17 Jan 2021 19:34:12 GMT
> > > Server: BaseICAP/1.0 Python/3.6.12
> > > Encapsulated: req-hdr=0, req-body=360
> > >
> > > POST http://www.dst-server.com:2/v1/test HTTP/1.1
> > > x-new-header: {"key": "value"}
> > > user-agent: python-requests/2.25.1
> > > accept-encoding: gzip, deflate
> > > accept: */*
> > > content-length: 16
> > &
Re: [squid-users] Adding headers in ICAP server with no preview
Hi
If the ICAP server sets 'Preview: 0' in the OPTIONS it means that when the
ICAP client sends a request, it should not contain the body.
This is the REQMOD request:
> F..n...DREQMOD icap://censor-req.proxy:14590/request ICAP/1.0
> Host: censor-req.proxy:14590
> Date: Mon, 18 Jan 2021 11:34:54 GMT
> Encapsulated: req-hdr=0, req-body=222
> Preview: 0
> Allow: 204, trailers
> X-custom-header: data
>
> POST http://www.dst-server.com:2/v1/test HTTP/1.1
> User-Agent: python-requests/2.25.1
> Accept-Encoding: gzip, deflate
> Accept: */*
> Content-Length: 10
> Content-Type: application/json
> Host: www.dst-server.com:2
>
The ICAP 'Encapsulated' header has a req-body even though no 'body' should
be in this request.
I wonder why in this case the 'Encapsulated' header doesn't contain
null-body.
I could not find any reference to this case in the RFC3507.
The ICAP server has no way to encapsulate the HTTP request body if it
didn't get it.
I want to avoid sending the body because the adaptation is body agnostic.
On Sun, Jan 17, 2021 at 11:34 PM Alex Rousskov <
rouss...@measurement-factory.com> wrote:
> On 1/17/21 3:08 PM, Moti Berger wrote:
> > What should the ICAP response look like?
>
> The vast majority off ICAP responses containing an HTTP POST message
> will look like ICAP header + HTTP header + HTTP body. Please see RFC
> 3507 and its errata for examples of and discussion about those three
> components. It should help avoid guessing and developing by examples
> (which usually leads to bugs, especially where ICAP is involved).
>
>
> > What I do is to reply like this:
> >
> > (dI./M..ICAP/1.0 200 OK
> > ISTag: "SjIzlRA4te41axxcDOoiSl6rBRg4ZK"
> > Date: Sun, 17 Jan 2021 19:34:12 GMT
> > Server: BaseICAP/1.0 Python/3.6.12
> > Encapsulated: req-hdr=0, req-body=360
> >
> > POST http://www.dst-server.com:2/v1/test HTTP/1.1
> > x-new-header: {"key": "value"}
> > user-agent: python-requests/2.25.1
> > accept-encoding: gzip, deflate
> > accept: */*
> > content-length: 16
> > content-type: application/json
> > host: www.dst-server.com:2 <http://www.dst-server.com:2>
>
>
> FYI: The above incomplete ICAP response promises an HTTP request body,
> both on the ICAP level (req-body) and on the HTTP level (content-length:
> 16).
>
>
> > As I said, I use 'Preview: 0' since I don't mind the body. The question
> > is whether declaring the body starts at X (req-body=X) is OK even though
> > I don't have a body to send?
>
> It is not OK not to send the body. Encapsulated:req-body does more than
> declaring where the encapsulated headers end. It also promises an
> embedded HTTP body after those headers. You must encapsulate the body if
> the HTTP message should have one. You cannot adapt the header of an HTTP
> message with a body without also sending the HTTP body (virgin or adapted).
>
> Preview is pretty much irrelevant in this context -- the ICAP protocol
> does not care how the ICAP service gets the HTTP body to include in the
> ICAP response.
>
> There are unofficial ICAP extensions that make it possible to tell the
> ICAP client to reuse the body it has buffered while adapting the header,
> but you should get the baseline case working before bothering with those
> extensions -- they are optimizations that are not applicable to some
> transactions.
>
>
> > I think having req-null=X is bad since it
> > probably tells squid that I decided the adapted request should have no
> > body, but that's only a guess.
>
> If you meant to say "null-body", then you guessed correctly -- null-body
> means the adapted HTTP message has no body. That is not what you want to
> say when adapting most HTTP POST messages.
>
>
> HTH,
>
> Alex.
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
[squid-users] Making destination IP available in ICAP REQMOD request
Hi My goal is to obtain the destination IP when sending an HTTP request for my ICAP server so it would be able to decide the kind of adaptation required based on it. Looking at squid (5.0.4) code I discovered the following: It seems that "everything" starts at ClientRequestContext. I've noticed that noteAdaptationAclCheckDone calls startAdaptation which calls more methods, eventually getting to Adaptation::Icap::ModXact::makeRequestHeaders where it iterates over headers defined by the adaptation_meta configurations in squid.conf. For each, it calls the 'match' method where it tries to format (and assemble) it. There it seems that the value is taken from an AccessLogEntry: >case LFT_SERVER_IP_ADDRESS: > if (al->hier.tcpServer) > out = al->hier.tcpServer->remote.toStr(tmp, sizeof(tmp)); > break; > So the AccessLogEntry object seems to be the key. At REQMOD time, I don't get the value of the destination IP. Looking further I found that the DNS resolving happens when it's decided that the request should be forwarded to the destination server. So I tracked the flow and it seems to start from FwdState::Start method which gets an AccessLogEntryPointer. Then it calls methods that eventually do the DNS resolving (Dns::nbgethostbyname) and ending in (FwdState::connectStart) which have the IP to connect to. So it seems that this flow will populate the AccessLogEntry. This seems right since during RESPMOD, the same code above (in Adaptation::Icap::ModXact::makeRequestHeaders) is running and this time the `match` method eventually gets the destination IP. I added logs that prints the AccessLogEntryPointer and in the FwdState.cc the log says address 0x5592ab521e30*12 and in the Notes.cc the log says address 0x5592ab521e30*25. Two things that I haven't found yet: 1. The place where the AccessLogEntry is populated 2. Where after the adaptation, the forwarding to the destination server occured (assuming it should be forwarded) I couldn't figure out a way to start the DNS resolving just before the startAdaptation starts as it requires all sorts of objects that seem to be unavailable there. I wonder if you can help me to find a way to do it. Thanks, Moti ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] Adding headers in ICAP server with no preview
Hi
I have an environment with squid version 5.0.4 with ICAP server adapting
requests by adding an header.
When I'm trying to send a POST request with a body I'm having an issue of a
stuck connection.
What should the ICAP response look like?
What I do is to reply like this:
> (dI./M..ICAP/1.0 200 OK
> ISTag: "SjIzlRA4te41axxcDOoiSl6rBRg4ZK"
> Date: Sun, 17 Jan 2021 19:34:12 GMT
> Server: BaseICAP/1.0 Python/3.6.12
> Encapsulated: req-hdr=0, req-body=360
>
> POST http://www.dst-server.com:2/v1/test HTTP/1.1
> x-new-header: {"key": "value"}
> user-agent: python-requests/2.25.1
> accept-encoding: gzip, deflate
> accept: */*
> content-length: 16
> content-type: application/json
> host: www.dst-server.com:2
>
Please assume the number in req-body=360 is correct (I trimmed here the
content of the new header).
As I said, I use 'Preview: 0' since I don't mind the body. The question is
whether declaring the body starts at X (req-body=X) is OK even though I
don't have a body to send? I think having req-null=X is bad since it
probably tells squid that I decided the adapted request should have no
body, but that's only a guess.
When the ICAP doesn't adapt the request, everything looks fine.
When it adapts the request I see that the POST request squid sends to
www.dst-server.com doesn't contain the body.
On the logs of the server behind www.dst-server.com I see an entry for the
API request only after I abort the request I sent.
I use python's requests module to make the request:
import requests
> s = requests.Session()
> s.proxies = {'http': 'localhost:8000', 'https': 'localhost:8000'}
resp = s.post('http://www.dst-server.com:2/v1/test',
> allow_redirects=False, json={'key': 'value'})
>
I'll highly appreciate any help.
Thanks,
Moti
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Connection occasionally not ending after adapting response with ICAP
I'm sorry for the bad format and for the uncompressed file. Indeed adding a Content-Length:0 solved the issue. This is BTW the preferred way for me since the URL I'm redirecting to explains why the redirection occurred. Thanks a lot for your help. On Wed, Dec 30, 2020 at 6:48 PM Alex Rousskov < rouss...@measurement-factory.com> wrote: > On 12/30/20 10:40 AM, Moti Berger wrote: > > >>>> I don't see the "0 CR LF CR LF" at all, not only for those URLs that > > cause the curl to hang. > > I suspect that none of those ICAP responses have HTTP bodies. An ICAP > response without an HTTP body, in itself, is not a bug, but it probably > explains the problem you are having. See below for more details. > > > > I'm attaching a tcpdump file. > > FWIW, my wireshark cannot read your packet capture file: "The capture > file appears to be damaged or corrupt. File has 1852785440-byte packet, > bigger than maximum of 262144". In the future, you should compress such > files before sharing them via email. > > However, looking at the ASCII parts of that capture file, I probably see > the likely culprit. The ICAP response below can be considered invalid > or, at least, rather difficult to handle correctly: > > > ICAP/1.0 200 OK > > ISTag: "tmrCy72n1Qgsn1mxBipIb4jdEYdce3" > > Date: Wed, 30 Dec 2020 15:07:52 GMT > > Server: BaseICAP/1.0 Python/3.6.12 > > Encapsulated: res-hdr=0, null-body=140 > > > > HTTP/1.1 307 Temporary Redirect > > location: http://www.redirect.url > > Cache-Control: no-cache, no-store, must-revalidate > > Pragma: no-cache > > > The HTTP 307 response header sent by the ICAP server promises an HTTP > response body (per HTTP rules), but the ICAP response has no HTTP body > (per the "null-body" value in the Encapsulated header). > > I do not remember what Squid does in this problematic case, but I > suspect the outcome is a stuck HTTP transaction. > > You should fix the ICAP service to either stop promising an HTTP > response body or provide one. The easiest (but ugly) fix may be to add > Content-Length:0 to the HTTP header returned by the ICAP service. A > better fix would be to include a real body that explains why the request > was redirected or provides some reference to the corresponding decision, > for triage. > > > HTH, > > Alex. > ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Connection occasionally not ending after adapting response with ICAP
On Wed, Dec 30, 2020 at 4:54 PM Alex Rousskov < rouss...@measurement-factory.com> wrote: > On 12/30/20 9:09 AM, Moti Berger wrote: > > I have a setup with squid 5.0.4 with ICAP server handling responses. > > FTR: This part seems to be about an ICAP RESPMOD service generating a > 307 redirect (AFAICT). > > > > When it hangs, I see this as the output of cURL > > > > GET / HTTP/1.1 > > > Host: www.one.co.il <http://www.one.co.il> > > > User-Agent: curl/7.58.0 > > > Accept: */* > > > < HTTP/1.1 307 Temporary Redirect > > < Location: > > < Cache-Control: no-cache, no-store, must-revalidate > > < Pragma: no-cache > > < Date: Wed, 30 Dec 2020 13:58:51 GMT > > < X-Cache: MISS from a0e59ea22cf8 > > < X-Cache-Lookup: MISS from a0e59ea22cf8:3128 > > < Transfer-Encoding: chunked > > < Via: 1.1 a0e59ea22cf8 (squid/5.0.4) > > < Connection: keep-alive > > Did curl receive the last-chunk from Squid? If not, did Squid received > the last-chunk from the ICAP service? See below for more details. > > > > For some of the URLs where it hangs, I saw that running the same cURL > > command with the --compressed switch, makes cURL exit as expected: > > Squid ignores Accept-Encoding so the --compressed switch is not relevant > to Squid. Perhaps the ICAP service or the origin server react to it, but > the headers you have provided do not show any important differences: > > > > GET / HTTP/1.1 > > > Host: www.one.co.il <http://www.one.co.il> > > > User-Agent: curl/7.58.0 > > > Accept: */* > > > Accept-Encoding: deflate, gzip > > > < HTTP/1.1 307 Temporary Redirect > > < Location: > > < Cache-Control: no-cache, no-store, must-revalidate > > < Pragma: no-cache > > < Date: Wed, 30 Dec 2020 14:00:31 GMT > > < X-Cache: MISS from a0e59ea22cf8 > > < X-Cache-Lookup: MISS from a0e59ea22cf8:3128 > > < Transfer-Encoding: chunked > > < Via: 1.1 a0e59ea22cf8 (squid/5.0.4) > > < Connection: keep-alive > > > When I skip the adaptation in REQMOD, I get the page and the connection > > is terminated. > > ... but REQMOD does not redirect when you do _not_ skip it, right? That > is, it is RESPMOD that always redirects (until a separate test below)? > If that is what happens, then it is possible that something goes wrong > inside the ICAP service, and turning off REQMOD that does not redirect > fixes the redirect in RESPMOD. However, it is a bit difficult for me to > follow the REQMOD/RESPMOD roles in your email so I may be > misunderstanding something. >>> My ICAP server handles both REQMOD and RESPMOD. I can switch on and off the adaptation in both modes. The adaptation, when take place, is to do redirects. So, for example, when I turn on the adaptation in REQMOD, meaning, redirecting, the connection terminates like expected. When the adaptation in REQMOD is turned off (meaning, the ICAP server still receives requests to modify, but it doesn't change them), and RESPMOD is turned on and adaptation occurs (meaning, redirecting) for some URLs, it get stuck and for some it doesn't. Regarding no adaptation in REQMOD that might affect the RESPMOD, I removed from squid.conf the related configs and the issue persists. > > > > When the ICAP works in REQMOD and redirects on the same URLs, everything > > seems to work properly. > > OK. Redirection in REQMOD is quite a different beast -- a lot of things > change, on both Squid and ICAP sides. > > > > What could make squid not to terminate the connection? Could it be that > > it still holds connection with the HTTP server? > > The actual question here is not about the connections but about the > messages inside those connections. It sounds like Squid does not send > the entire redirect response body to curl in some cases. _That_ could be > caused by the ICAP service not sending the entire redirect response body > to Squid. > > You can probably confirm or deny the last bit by looking at ICAP traffic > in wireshark. Does the ICAP response contain the last-chunk (which is > usually the "0 CR LF CR LF" sequence in the chunked-encoded message body)? > > >>> I don't see the "0 CR LF CR LF" at all, not only for those URLs that cause the curl to hang. > * If the last-chunk is missing, then Squid is waiting for the ICAP > service. I do not know what the ICAP RESPMOD service is waiting for, but > it could be the last-chunk of the virgin HTTP response that Squid is >
[squid-users] Connection occasionally not ending after adapting response with ICAP
I have a setup with squid 5.0.4 with ICAP server handling responses. The
ICAP server redirects based on some parameters of the response.
To test this setup, I use cURL like this:
> curl -k -s --proxy localhost:8000 -o /dev/null -v
Now, for some URLs, cURL hangs and for others it exits after receiving the
307 response.
When it hangs, I see this as the output of cURL (I removed what seemed to
me as non-related logs from the beginning):
> } [5 bytes data]
> * TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> } [1 bytes data]
> > GET / HTTP/1.1
> > Host: www.one.co.il
> > User-Agent: curl/7.58.0
> > Accept: */*
> >
> { [5 bytes data]
> * TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
> { [1 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [217 bytes data]
> * TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
> { [1 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [217 bytes data]
> * TLSv1.3 (IN), TLS Unknown, Unknown (23):
> { [1 bytes data]
> < HTTP/1.1 307 Temporary Redirect
> < Location:
> < Cache-Control: no-cache, no-store, must-revalidate
> < Pragma: no-cache
> < Date: Wed, 30 Dec 2020 13:58:51 GMT
> < X-Cache: MISS from a0e59ea22cf8
> < X-Cache-Lookup: MISS from a0e59ea22cf8:3128
> < Transfer-Encoding: chunked
> < Via: 1.1 a0e59ea22cf8 (squid/5.0.4)
> < Connection: keep-alive
> <
Using tcpdump I didn't see squid send any other ICAP requests (besides
OPTIONS which the ICAP server replied to properly).
For some of the URLs where it hangs, I saw that running the same cURL
command with the --compressed switch, makes cURL exit as expected:
> } [5 bytes data]
> * TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> } [1 bytes data]
> > GET / HTTP/1.1
> > Host: www.one.co.il
> > User-Agent: curl/7.58.0
> > Accept: */*
> > Accept-Encoding: deflate, gzip
> >
> { [5 bytes data]
> * TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
> { [1 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [217 bytes data]
> * TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
> { [1 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [217 bytes data]
> * TLSv1.3 (IN), TLS Unknown, Unknown (23):
> { [1 bytes data]
> < HTTP/1.1 307 Temporary Redirect
> < Location:
> < Cache-Control: no-cache, no-store, must-revalidate
> < Pragma: no-cache
> < Date: Wed, 30 Dec 2020 14:00:31 GMT
> < X-Cache: MISS from a0e59ea22cf8
> < X-Cache-Lookup: MISS from a0e59ea22cf8:3128
> < Transfer-Encoding: chunked
> < Via: 1.1 a0e59ea22cf8 (squid/5.0.4)
> < Connection: keep-alive
> <
> { [5 bytes data]
> * TLSv1.3 (IN), TLS Unknown, Unknown (23):
> { [1 bytes data]
> * Connection #0 to host localhost left intact
When I skip the adaptation in REQMOD, I get the page and the connection is
terminated.
When the ICAP works in REQMOD and redirects on the same URLs, everything
seems to work properly.
What could make squid not to terminate the connection? Could it be that it
still holds connection with the HTTP server?
Thanks
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
