Re: [squid-users] Caching application/octet-stream

[Michael Varun]

There is a bug in the SSL-Bump implementation we have not sorted out
yet, which makes the "ssl-bump" on this port enable reverse-proxy mode
handling. That seems to be leading to Surrogate feature being enabled
and the Authorization:Bearer being removed when it should be relayed to
the server.


Can you refer to the BUD ID if there is one already opened  if not
should i submit one for reference ?


On Fri, Sep 30, 2016 at 1:06 PM, Amos Jeffries  wrote:

> On 30/09/2016 8:10 p.m., Michael Varun wrote:
> > Here is the snippet of debug logs
> > I dont get to see anything missing out there . It does a GET call to the
> > docker registry on behalf of the requesting client The registry listens
> on
> > 443 so squid mimicks  client TLS connections post which does a GET call
> to
> > the docker registry on the requested blobs
>
> Well firstly, going by your earlier config file the client is *not*
> performing TLS connections. Your proxy is configured to receive
> plain-text HTTP at port 443.
>
> You seem to have stumbled onto two bugs in Squid which are combining to
> be problematic.
>
> There is a bug in the SSL-Bump implementation we have not sorted out
> yet, which makes the "ssl-bump" on this port enable reverse-proxy mode
> handling. That seems to be leading to Surrogate feature being enabled
> and the Authorization:Bearer being removed when it should be relayed to
> the server.
>
> The existence of Authorization header on the request combined with lack
> of Cache-Control:public on the response means these reponses are private
> responses associated with that auth credentials token. They cannot be
> cached and given to anyone else.
>
> That brings up what I think may be a second bug. Since the request to
> the server was sent without Auth header then Squid should be considering
> it a non-auth response and treating it as cacheable anyway. But probably
> is just using the client request for that decision.
>
>
> You could try adding the "login=PASSTHRU" option to your cache_peer
> line. If the server sends "Cache-Control:public" that should enable
> caching.
>
> Amos
>
>

-- 
_
The information contained in this communication is intended solely for the 
use of the individual or entity to whom it is addressed and others 
authorized to receive it. It may contain confidential or legally privileged 
information. If you are not the intended recipient you are hereby notified 
that any disclosure, copying, distribution or taking any action in reliance 
on the contents of this information is strictly prohibited and may be 
unlawful. If you have received this communication in error, please notify 
us immediately by responding to this email and then delete it from your 
system. The firm is neither liable for the proper and complete transmission 
of the information contained in this communication nor for any delay in its 
receipt.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Caching application/octet-stream

[Michael Varun]

Here is the snippet of debug logs
I dont get to see anything missing out there . It does a GET call to the
docker registry on behalf of the requesting client The registry listens on
443 so squid mimicks  client TLS connections post which does a GET call to
the docker registry on the requested blobs


Logs
=


-
GET
/v2/idp/mtd-log-viewer/blobs/sha256:7f24efe711014be87fc45d3ccdc15bb419f45f26ab1f7dd4652b83890ba16294
HTTP/1.1
Host: dockerregistry.com
User-Agent: docker/1.12.1 go/go1.6.3 git-commit/23cf638
kernel/3.16.0-71-generic os/linux arch/amd64
UpstreamClient(Docker-Client/1.12.1 \(linux\))
Authorization: Bearer
AKCp2V5yYCcLhiuTrm6gvr2h6At9FfzhpxEB61CncxQsQxoRNT4tmcSLbWjecmnSFT4gtvF1q
Accept-Encoding: gzip
Connection: close


--
2016/09/30 07:05:26.100| 11,2| http.cc(2201) sendRequest: HTTP Server local=
10.88.92.129:56249 remote=xx.xx.xx.xx:443 FD 16 flags=1
2016/09/30 07:05:26.100| 11,2| http.cc(2202) sendRequest: HTTP Server
REQUEST:
-
GET
/v2/idp/mtd-log-viewer/blobs/sha256:7f24efe711014be87fc45d3ccdc15bb419f45f26ab1f7dd4652b83890ba16294
HTTP/1.1
User-Agent: docker/1.12.1 go/go1.6.3 git-commit/23cf638
kernel/3.16.0-71-generic os/linux arch/amd64
UpstreamClient(Docker-Client/1.12.1 \(linux\))
Host: dockerregistry.com
Via: 1.1 proxyserver (squid/3.5.20-20160817-r14075)
Surrogate-Capability: proxyserver="Surrogate/1.0"
X-Forwarded-For: 10.88.84.102
Cache-Control: max-age=600
Connection: keep-alive


--
2016/09/30 07:05:27.046| ctx: enter level  0: '
https://dockerregistry.com/v2/blobs/sha256:3071a8ec2f6e96d06697a8069a385f889fa318ef479a0c2557baddb0e06a9b2c
'
2016/09/30 07:05:27.046| 11,2| http.cc(717) processReplyHeader: HTTP Server
local=10.88.92.129:56250 remote=xx.xx.xx.xx:443 FD 21 flags=1
2016/09/30 07:05:27.046| 11,2| http.cc(718) processReplyHeader: HTTP Server
REPLY:
-
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 30 Sep 2016 07:06:12 GMT
Content-Type: application/octet-stream
Content-Length: 1256
Connection: keep-alive
X-Artifactory-Id: fb4bb806eccd564b:-7d2d7c9a:156cf92100e:-8000
Last-Modified: Tue, 13 Sep 2016 06:03:06 GMT
ETag: ca5755fd72e65e6cc42e608eb7524d4956c9a358
X-Checksum-Sha1: ca5755fd72e65e6cc42e608eb7524d4956c9a358
Accept-Ranges: bytes
X-Checksum-Md5: a15d3dcf76f3334177fa44e6ad0af578
X-Artifactory-Filename:
sha256__3071a8ec2f6e96d06697a8069a385f889fa318ef479a0c2557baddb0e06a9b2c
Docker-Distribution-Api-Version: registry/2.0
Docker-Content-Digest:
sha256:3071a8ec2f6e96d06697a8069a385f889fa318ef479a0c2557baddb0e06a9b2c

�
--
2016/09/30 07:05:27.046| ctx: exit level  0
2016/09/30 07:05:27.046| 11,2| client_side.cc(1392) sendStartOfMessage:
HTTP Client local=10.88.92.129:443 remote=10.88.84.102:41429 FD 18 flags=1
2016/09/30 07:05:27.046| 11,2| client_side.cc(1393) sendStartOfMessage:
HTTP Client REPLY:
-
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 30 Sep 2016 07:06:12 GMT
Content-Type: application/octet-stream
Content-Length: 1256
X-Artifactory-Id: fb4bb806eccd564b:-7d2d7c9a:156cf92100e:-8000
Last-Modified: Tue, 13 Sep 2016 06:03:06 GMT
ETag: ca5755fd72e65e6cc42e608eb7524d4956c9a358
X-Checksum-Sha1: ca5755fd72e65e6cc42e608eb7524d4956c9a358
Accept-Ranges: bytes
X-Checksum-Md5: a15d3dcf76f3334177fa44e6ad0af578
X-Artifactory-Filename:
sha256__3071a8ec2f6e96d06697a8069a385f889fa318ef479a0c2557baddb0e06a9b2c
Docker-Distribution-Api-Version: registry/2.0
Docker-Content-Digest:
sha256:3071a8ec2f6e96d06697a8069a385f889fa318ef479a0c2557baddb0e06a9b2c
X-Cache: MISS from proxyserver
<===
MISS
X-Cache-Lookup: MISS from
proxyserver:443<=
LOOKUP MISS
Via: 1.1 proxyserver (squid/3.5.20-20160817-r14075)
Connection: close

On Wed, Sep 28, 2016 at 7:54 PM, Amos Jeffries  wrote:

> On 28/09/2016 11:17 p.m., Michael Varun wrote:
> > Team -
> >
> > Would like to know is there any specific config that we need to enable to
> > cache MIME attachement of application/octet-stream type
> >
> > We are trying to caching docker image blobs which is of
> > application/octet-stream  and everytime we hit the docker registry
> via)GET
> > call squid throws up TCP_MISS/200  I had never got a CACHE HIT or MEM HIT
> > for these contents. Can someone show pointers to how to succeed in cache
> > hit
>
> The object mime type does not matter for caching. What does matter are
> the URL and identifier headers, plus headers about the objects age,
> Vary, Expires and Cache-Control.
>
> You can configure "debug_options 11,2" to get a cache.log trace of the
> messages going through the proxy. Taking a closer look at those message
> headers to find out why caching is not happening.
>
> You could also use the tool at redbot.org. It takes a public URL and
> displays the cacheability situation for it.
>
> Amos
>
> ___
> squid-users mailing list
> 

[squid-users] Caching application/octet-stream

[Michael Varun]

Team -

Would like to know is there any specific config that we need to enable to
cache MIME attachement of application/octet-stream type

We are trying to caching docker image blobs which is of
application/octet-stream  and everytime we hit the docker registry via)GET
call squid throws up TCP_MISS/200  I had never got a CACHE HIT or MEM HIT
for these contents. Can someone show pointers to how to succeed in cache
hit


Logs

1475056789.667  0 10.0.0.160 TAG_NONE/200 0 CONNECT :443 - HIER_NONE/- -
1475056789.706   1633 10.0.0.160 TCP_MISS/200 1114 GET https://docker
registry/v2/blobs/sha256:36e0da2b6e166c5e7b1f243e9257590d4fbd1be729773df7f3d6a779941b22d9
- FIRSTUP_PARENT/IP.xx.xx.xxapplication/octet-stream
1475056789.708  0 10.0.0.160 TAG_NONE/200 0 CONNECT 443 - HIER_NONE/- -
1475056791.259   1545 10.0.0.160 TCP_MISS/200 1698 GET https://docker
registry/v2/blobs/sha256:2a00f8fcea92b16de0de328c557da476e930aa038ccee482daf91a62de5afa33
- FIRSTUP_PARENT/IP.xx.xx.xx application/octet-stream


Configuration
=
http_port docker registry:443 ssl-bump cert=/etc/ssl/private/docker
registry.certkey=/etc/ssl/private/docker registry.key
maximum_object_size 8 GB
maximum_object_size_in_memory 102400 KB
quick_abort_min -1
cache_dir ufs /data/squid/cache/squid 204800 16 256
cache_mem 51200 MB
cache allow all
cache_peer docker registry parent 443 0 no-query originserver no-digest
name=upstream ssl sslflags=DONT_VERIFY_PEER
never_direct allow all
#
acl site dstdomain /etc/mirror-dstdomain.acl
acl site1 dstdomain docker-registry.xx.xx
http_access allow site
http_access allow site1
cache_peer_access upstream allow site1
cache_peer_access upstream deny all
ssl_bump bump site1
ssl_bump splice all
sslproxy_flags DONT_VERIFY_PEER

-- 
_
The information contained in this communication is intended solely for the 
use of the individual or entity to whom it is addressed and others 
authorized to receive it. It may contain confidential or legally privileged 
information. If you are not the intended recipient you are hereby notified 
that any disclosure, copying, distribution or taking any action in reliance 
on the contents of this information is strictly prohibited and may be 
unlawful. If you have received this communication in error, please notify 
us immediately by responding to this email and then delete it from your 
system. The firm is neither liable for the proper and complete transmission 
of the information contained in this communication nor for any delay in its 
receipt.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users