Re: [squid-users] allow request to cloudfront after 302 redirection.

2021-05-27 Thread Eliezer Croitoru 
Thanks Alex,

I assume you do remember that 301 is not the same as 302.
Depends on the status code ie 301/302 307/307 etc the ttl of the cached 
location would be decides.
The basic idea is to use either redis or memcached  on another key-value DB to 
persist the Location response.
(This key-value DB would only be a "cache" with a ttl)

For a 301 the basic assumption is that the Location header contains the 
static/permanent redirection.
For a 302 it's not a static, I think it's important to have the status code 
sent to the helper.

About the http_reply_access, I actually forgot it exists and indeed it makes 
more sense...

I am working on an example.
Eliezer

-Original Message-
From: squid-users  On Behalf Of Alex 
Rousskov
Sent: Monday, May 24, 2021 9:26 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] allow request to cloudfront after 302 redirection.

On 5/24/21 5:52 AM, Eliezer Croitoru wrote:

> Following up this thread I was wondering about an example how to do
> that with an external_acl helper. With ICAP I can do that easily to
> some degree. With an external_acl helper I am not sure what values to
> send.

AFAICT, the external ACL helper(s) should be sent the Location URI
values the helper(s) should cache (at response time) and the request URI
values the helper(s) should examine (at request time). More information
may be sent/cached, of course, depending on the exact decision logic.


> I would guess that the response code and response Location header
> might be the ones which should be passed to the helper?

I do not see the value in sending the response status code (because
Squid can check that itself), but the correct answer depends on the
exact decision logic.


> What do you think about the next acls, should do the trick?

> acl redirect http_status 301-308
> acl gitlab_package dstdomain package.gitlab.com

> external_acl_type openlocation children=15 %DST %SRC %<{Location} 
> /usr/local/bin/location-openner.rb
> acl location_openner external openlocation
> http_access deny gitlab_package redirect location_openner
> http_access allow location_openner

The above sketch does not make sense to me because it uses response
information (e.g., % -Original Message-
> From: squid-users  On Behalf Of 
> Alex Rousskov
> Sent: Wednesday, April 21, 2021 8:49 PM
> To: squid-users@lists.squid-cache.org
> Subject: Re: [squid-users] allow request to cloudfront after 302 redirection.
> 
> On 4/21/21 12:48 PM, Miroslaw Malinowski wrote:
>> Is it possible to create a whitelist that allows cloudfront 302
>> redirections, e.g. gitlab is using cloudfront as CDN and when we
>> whitelist package.gitlab.com the URL is redirected (302) to
>> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c15d
> 
> 
> Yes, it is possible to allow future requests to Location-listed URLs,
> but since we are talking about two (or more) independent HTTP
> transactions, on two (or more) TCP connections, you will need to store
> the allowed Location values (at least) somewhere, maintain that storage
> (e.g., remove stale entries), and (optionally) determine whether the
> request for an allowed cloudfront URL came from the same user agent as
> the gitlab request that was redirected to that URL.
> 
> Storing, maintenance, and checking of allowed Locations/etc. can be done
> using external ACLs and/or eCAP/ICAP adaptation services. It cannot be
> reliably done using built-in ACLs alone AFAICT.
> 
> 
> HTH,
> 
> Alex.
> 
> 
>> I could whitelist a whole .cloudfront.net <http://cloudfront.net> domain
>> or url_regex, but what I would like to achieve, I don't know if
>> possible, is a chain of events like:
>> If packages.gitlab.com <http://packages.gitlab.com> return 302 Location
>> .cloudfront, then allow
>> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c
>> <https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c>
>> request.
>> I've been playing around with http_reply_access and rep_headers, but I
>> can only go as far as allow replay of the first request to
>> package.gitlab.com <http://package.gitlab.com>, but then a GET to
>> cloudfront is blocked anyway as it's not on our whitelist.
>> e.g.
>> 1619022938.916   423 172.16.230.237 NONE/200 0 CONNECT 54.153.54.194:443
>> <http://54.153.54.194:443> - ORIGINAL_DST/54.153.54.194
>> <http://54.153.54.194> -
>> 1619022939.074   153 172.16.230.237 TCP_MISS/302 758 GET
>> https://packages.gitlab.com/gitlab/gitlab-ee/packages/ubuntu/bionic/gitlab

Re: [squid-users] allow request to cloudfront after 302 redirection.

2021-05-24 Thread Alex Rousskov 
On 5/24/21 5:52 AM, Eliezer Croitoru wrote:

> Following up this thread I was wondering about an example how to do
> that with an external_acl helper. With ICAP I can do that easily to
> some degree. With an external_acl helper I am not sure what values to
> send.

AFAICT, the external ACL helper(s) should be sent the Location URI
values the helper(s) should cache (at response time) and the request URI
values the helper(s) should examine (at request time). More information
may be sent/cached, of course, depending on the exact decision logic.


> I would guess that the response code and response Location header
> might be the ones which should be passed to the helper?

I do not see the value in sending the response status code (because
Squid can check that itself), but the correct answer depends on the
exact decision logic.


> What do you think about the next acls, should do the trick?

> acl redirect http_status 301-308
> acl gitlab_package dstdomain package.gitlab.com

> external_acl_type openlocation children=15 %DST %SRC %<{Location} 
> /usr/local/bin/location-openner.rb
> acl location_openner external openlocation
> http_access deny gitlab_package redirect location_openner
> http_access allow location_openner

The above sketch does not make sense to me because it uses response
information (e.g., % -Original Message-
> From: squid-users  On Behalf Of 
> Alex Rousskov
> Sent: Wednesday, April 21, 2021 8:49 PM
> To: squid-users@lists.squid-cache.org
> Subject: Re: [squid-users] allow request to cloudfront after 302 redirection.
> 
> On 4/21/21 12:48 PM, Miroslaw Malinowski wrote:
>> Is it possible to create a whitelist that allows cloudfront 302
>> redirections, e.g. gitlab is using cloudfront as CDN and when we
>> whitelist package.gitlab.com the URL is redirected (302) to
>> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c15d
> 
> 
> Yes, it is possible to allow future requests to Location-listed URLs,
> but since we are talking about two (or more) independent HTTP
> transactions, on two (or more) TCP connections, you will need to store
> the allowed Location values (at least) somewhere, maintain that storage
> (e.g., remove stale entries), and (optionally) determine whether the
> request for an allowed cloudfront URL came from the same user agent as
> the gitlab request that was redirected to that URL.
> 
> Storing, maintenance, and checking of allowed Locations/etc. can be done
> using external ACLs and/or eCAP/ICAP adaptation services. It cannot be
> reliably done using built-in ACLs alone AFAICT.
> 
> 
> HTH,
> 
> Alex.
> 
> 
>> I could whitelist a whole .cloudfront.net <http://cloudfront.net> domain
>> or url_regex, but what I would like to achieve, I don't know if
>> possible, is a chain of events like:
>> If packages.gitlab.com <http://packages.gitlab.com> return 302 Location
>> .cloudfront, then allow
>> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c
>> <https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c>
>> request.
>> I've been playing around with http_reply_access and rep_headers, but I
>> can only go as far as allow replay of the first request to
>> package.gitlab.com <http://package.gitlab.com>, but then a GET to
>> cloudfront is blocked anyway as it's not on our whitelist.
>> e.g.
>> 1619022938.916   423 172.16.230.237 NONE/200 0 CONNECT 54.153.54.194:443
>> <http://54.153.54.194:443> - ORIGINAL_DST/54.153.54.194
>> <http://54.153.54.194> -
>> 1619022939.074   153 172.16.230.237 TCP_MISS/302 758 GET
>> https://packages.gitlab.com/gitlab/gitlab-ee/packages/ubuntu/bionic/gitlab-ee_11.0.1-ee.0_amd64.deb/download.deb
>> <https://packages.gitlab.com/gitlab/gitlab-ee/packages/ubuntu/bionic/gitlab-ee_11.0.1-ee.0_amd64.deb/download.deb>
>> - ORIGINAL_DST/54.153.54.194 <http://54.153.54.194> text/html
>> 1619022939.10820 172.16.230.237 NONE/200 0 CONNECT 52.84.90.34:443
>> <http://52.84.90.34:443> - ORIGINAL_DST/52.84.90.34 <http://52.84.90.34> -
>> 1619022939.114 2 172.16.230.237 TCP_DENIED/403 19053 GET
>> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb 
>> <https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb>?
>> - HIER_NONE/- text/html
> 
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
> 
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
> 

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] allow request to cloudfront after 302 redirection.

2021-05-24 Thread Eliezer Croitoru 
Hey Alex,

Following up this thread I was wondering about an example how to do that with 
an external_acl helper.
With ICAP I can do that easily to some degree.
With an external_acl helper I am not sure what values to send.
I would guess that the response code and response Location header might be the 
ones which should be passed to the helper?

What do you think about the next acls, should do the trick?(code to be followed)

acl redirect http_status 301-308
acl gitlab_package dstdomain package.gitlab.com

external_acl_type openlocation children=15 %DST %SRC %<{Location} 
/usr/local/bin/location-openner.rb
acl location_openner external openlocation
http_access deny gitlab_package redirect location_openner
http_access allow location_openner


Thanks,

Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: ngtech1...@gmail.com


-Original Message-
From: squid-users  On Behalf Of Alex 
Rousskov
Sent: Wednesday, April 21, 2021 8:49 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] allow request to cloudfront after 302 redirection.

On 4/21/21 12:48 PM, Miroslaw Malinowski wrote:
> Is it possible to create a whitelist that allows cloudfront 302
> redirections, e.g. gitlab is using cloudfront as CDN and when we
> whitelist package.gitlab.com the URL is redirected (302) to
> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c15d


Yes, it is possible to allow future requests to Location-listed URLs,
but since we are talking about two (or more) independent HTTP
transactions, on two (or more) TCP connections, you will need to store
the allowed Location values (at least) somewhere, maintain that storage
(e.g., remove stale entries), and (optionally) determine whether the
request for an allowed cloudfront URL came from the same user agent as
the gitlab request that was redirected to that URL.

Storing, maintenance, and checking of allowed Locations/etc. can be done
using external ACLs and/or eCAP/ICAP adaptation services. It cannot be
reliably done using built-in ACLs alone AFAICT.


HTH,

Alex.


> I could whitelist a whole .cloudfront.net <http://cloudfront.net> domain
> or url_regex, but what I would like to achieve, I don't know if
> possible, is a chain of events like:
> If packages.gitlab.com <http://packages.gitlab.com> return 302 Location
> .cloudfront, then allow
> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c
> <https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c>
> request.
> I've been playing around with http_reply_access and rep_headers, but I
> can only go as far as allow replay of the first request to
> package.gitlab.com <http://package.gitlab.com>, but then a GET to
> cloudfront is blocked anyway as it's not on our whitelist.
> e.g.
> 1619022938.916   423 172.16.230.237 NONE/200 0 CONNECT 54.153.54.194:443
> <http://54.153.54.194:443> - ORIGINAL_DST/54.153.54.194
> <http://54.153.54.194> -
> 1619022939.074   153 172.16.230.237 TCP_MISS/302 758 GET
> https://packages.gitlab.com/gitlab/gitlab-ee/packages/ubuntu/bionic/gitlab-ee_11.0.1-ee.0_amd64.deb/download.deb
> <https://packages.gitlab.com/gitlab/gitlab-ee/packages/ubuntu/bionic/gitlab-ee_11.0.1-ee.0_amd64.deb/download.deb>
> - ORIGINAL_DST/54.153.54.194 <http://54.153.54.194> text/html
> 1619022939.10820 172.16.230.237 NONE/200 0 CONNECT 52.84.90.34:443
> <http://52.84.90.34:443> - ORIGINAL_DST/52.84.90.34 <http://52.84.90.34> -
> 1619022939.114 2 172.16.230.237 TCP_DENIED/403 19053 GET
> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb 
> <https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb>?
> - HIER_NONE/- text/html

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] allow request to cloudfront after 302 redirection.

2021-04-21 Thread Alex Rousskov 
On 4/21/21 12:48 PM, Miroslaw Malinowski wrote:
> Is it possible to create a whitelist that allows cloudfront 302
> redirections, e.g. gitlab is using cloudfront as CDN and when we
> whitelist package.gitlab.com the URL is redirected (302) to
> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c15d


Yes, it is possible to allow future requests to Location-listed URLs,
but since we are talking about two (or more) independent HTTP
transactions, on two (or more) TCP connections, you will need to store
the allowed Location values (at least) somewhere, maintain that storage
(e.g., remove stale entries), and (optionally) determine whether the
request for an allowed cloudfront URL came from the same user agent as
the gitlab request that was redirected to that URL.

Storing, maintenance, and checking of allowed Locations/etc. can be done
using external ACLs and/or eCAP/ICAP adaptation services. It cannot be
reliably done using built-in ACLs alone AFAICT.


HTH,

Alex.


> I could whitelist a whole .cloudfront.net  domain
> or url_regex, but what I would like to achieve, I don't know if
> possible, is a chain of events like:
> If packages.gitlab.com  return 302 Location
> .cloudfront, then allow
> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c
> 
> request.
> I've been playing around with http_reply_access and rep_headers, but I
> can only go as far as allow replay of the first request to
> package.gitlab.com , but then a GET to
> cloudfront is blocked anyway as it's not on our whitelist.
> e.g.
> 1619022938.916   423 172.16.230.237 NONE/200 0 CONNECT 54.153.54.194:443
>  - ORIGINAL_DST/54.153.54.194
>  -
> 1619022939.074   153 172.16.230.237 TCP_MISS/302 758 GET
> https://packages.gitlab.com/gitlab/gitlab-ee/packages/ubuntu/bionic/gitlab-ee_11.0.1-ee.0_amd64.deb/download.deb
> 
> - ORIGINAL_DST/54.153.54.194  text/html
> 1619022939.108    20 172.16.230.237 NONE/200 0 CONNECT 52.84.90.34:443
>  - ORIGINAL_DST/52.84.90.34  -
> 1619022939.114     2 172.16.230.237 TCP_DENIED/403 19053 GET
> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb 
> ?
> - HIER_NONE/- text/html

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] allow request to cloudfront after 302 redirection.

2021-04-21 Thread Miroslaw Malinowski 
Is it possible to create a whitelist that allows cloudfront 302
redirections, e.g. gitlab is using cloudfront as CDN and when we whitelist
package.gitlab.com the URL is redirected (302) to
https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c15d.
I could whitelist a whole .cloudfront.net domain or url_regex, but what I
would like to achieve, I don't know if possible, is a chain of events like:
If packages.gitlab.com return 302 Location .cloudfront, then allow
https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c
request.
I've been playing around with http_reply_access and rep_headers, but I can
only go as far as allow replay of the first request to package.gitlab.com,
but then a GET to cloudfront is blocked anyway as it's not on our whitelist.
e.g.
1619022938.916   423 172.16.230.237 NONE/200 0 CONNECT 54.153.54.194:443 -
ORIGINAL_DST/54.153.54.194 -
1619022939.074   153 172.16.230.237 TCP_MISS/302 758 GET
https://packages.gitlab.com/gitlab/gitlab-ee/packages/ubuntu/bionic/gitlab-ee_11.0.1-ee.0_amd64.deb/download.deb
- ORIGINAL_DST/54.153.54.194 text/html
1619022939.10820 172.16.230.237 NONE/200 0 CONNECT 52.84.90.34:443 -
ORIGINAL_DST/52.84.90.34 -
1619022939.114 2 172.16.230.237 TCP_DENIED/403 19053 GET
https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?
- HIER_NONE/- text/html

Thanks,
Mirek
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users