Re: [squid-users] Get IP of denied request

[Alex Rousskov]

On 2023-10-27 07:14, ma...@web.de wrote:

Am 26.10.23 um 21:11 schrieb Alex Rousskov:

On 2023-10-26 08:37, ma...@web.de wrote:

TL;DR: is there a way to get/log the resolved ip of a denied request?


TLDR: Bugs notwithstanding, use %


%

Sorry, my first response was wrong: As you have correctly explained, %is the destination address of the used next hop connection while dst ACL 
uses an address derived from the request-target. The two addresses do 
not have to match and, more importantly, it is wrong to expect a dst ACL 
to update/store that "used next hop address" information when the ACL 
itself does not open or reuse a connection to that address.




 acl matchDst1 dst 127.0.0.1
 acl markDst1 note matched=127.0.0.1
 http_access deny matchDst1 markDst1

 acl matchDst2 dst 127.0.0.2
 acl markDst2 note matched=127.0.0.2
 http_access deny matchDst2 markDst2

 logformat myFormat ... matched_dst=%note{matched}
 access_log ...

For long dst lists, the above approach will require scripting the
generation of the corresponding squid.conf portions or include files, of
course.


I don't think this scales to blacklists with 6-digit count sizes 


I think that depends on the definition of "to scale".



and it also doesn't work for blacklisted networks :-(


Agreed. This approach can only log which dst ACL data matched, and that 
data may be different from the IP.




I hoped there would be a way to get the ip as some kind of variable like
the header fields in logformat.


Yes, I know. I was just documenting an existing workaround for cases 
where it is usable.




Any ideas?


I have three:

1. Enhance Squid to resolve transaction destination address once (on 
first use/need). Remember/reuse resolved IP addresses. Log them via some 
new %resolved_dst and %dst_resolution_detail codes.


This improvement will help address a few use cases unrelated to this 
discussion, but it will _not_ tell you which of the resolved addresses 
actually matched which ACL. You will be able to guess in many cases, but 
there will be exceptions.



2. Add a Squid feature where an evaluated ACL can be configured to 
annotate the transaction with the information about that evaluation.


To start with, we can support annotations for _matched_ dst ACLs. For 
example:


# When matches, sets the following master transaction annotations:
# * badDestination_input: used destination address (IP or name)
# * badDestination_match: the matched destination IP
# * badDestination_value: the matched ACL parameter value
# * badDestination_ips: resolved destination IP(s)
# * badDestination_errors: DNS resolution and other error(s)
# * badDestination_matches: number of matches (this transaction)
# * badDestination_evals: number of evaluations (this transaction)
acl badDestination dst --on-match=annotate 127.0.0.0/24 10.0.0.1

If the same ACL matches more than once, the last(?) match wins, but the 
aclfoo_matches annotation can be used to detect these cases. The 
aclfoo_evals annotation can be used to detect whether this ACL was 
"reached" at all.


If really needed, we can support turning individual annotations on and 
off, but I doubt that complexity is worth associated performance 
savings. After all, most ACLs will only match once per transaction 
lifetime (for correctly written configurations). Access.log will be 
configured to only log annotations of interest to the admin, of course.



The above approach can be extended to provide ACL debugging aid:

# Dumps every mismatch information to cache.log at level 1
acl goodDestination dst --on-mismatch=log 127.0.0.0/24

# Dumps every evaluation information to cache.log at level 1
acl redDestination dst --on-eval=log 127.0.0.0/24


3. Add a Squid feature where Squid (optionally) maintains an internal 
database of recent ACL evaluation history and makes that information 
accessible via cache manager queries like "which ACLs matched 
transaction X?" (where X is logged %master_xaction ID).



The three sketched options are not mutually exclusive, of course. All 
require non-trivial code changes.


Would any of the above options address your needs? Any preferences or 
spec adjustments?




Thank you,

Alex.



Is there any way to get the ip logged that was used in the dst-acl aside
from debug logging? Maybe through some annotation mechanism?

Squid version is 6.2, as 6.4 crashes with assertion errors here, too.

thanks,
Martin


___
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Get IP of denied request

[magri]

Hi Alex,


Am 26.10.23 um 21:11 schrieb Alex Rousskov:

On 2023-10-26 08:37, ma...@web.de wrote:


TL;DR: is there a way to get/log the resolved ip of a denied request?


TLDR: Bugs notwithstanding, use %

%http://127.0.0.1:3128 https_proxy=http://127.0.0.1:3128 wget
http://www.example.org
--2023-10-27 09:52:27--  http://www.example.org/
Connecting to 127.0.0.1:3128... connected.
Proxy request sent, awaiting response... 403 Forbidden
2023-10-27 09:52:31 ERROR 403: Forbidden.

Log contains:
1698393151.066   3903 127.0.0.1 TCP_DENIED/- 3889 GET
http://www.example.org/ - HIER_NONE/- text/html HTTP/1.1 37030 182 "-"
"Wget/1.21.3" ERR_ACCESS_DENIED "-" 53




We have a rather large ip based malware blacklist (dst acl) and
sometimes a destination is blocked inadvertantly because of a false
positive entry in this list.
This happens most often with CDNs where the ips of a destination change
often and even move between different sites.

Because of this rapid change it's difficult to determine the blocked ip
in hindsight when analyzing access problems and makes it impossible to
correct the blacklist.

For normal requests the resolved and accessed ip is be logged with %


If a request was denied by a dst ACL based on its successfully resolved
destination IP address but %

I don't think this scales to blacklists with 6-digit count sizes and it
also doesn't work for blacklisted networks :-(
I hoped there would be a way to get the ip as some kind of variable like
the header fields in logformat.



If a request was denied by a dst ACL because its destination IP address
could not be resolved, then %

I'm not familiar with the code but from staring at it I get that '%remote and this isn't necessarily the ip
of the resolved URI FQDN (e.g. when using a parent proxy).

There doesn't seem to be any code that stores or reuses a once resolved
ip in acl/DestinationIP.cc (at least if a request exists).
As far as I understand it for every dst ACL the ip is fetched
(asyncronously) from the ipcache and compared to the ACL but never
stored for later use.

Any ideas?
Martin




HTH,

Alex.




Is there any way to get the ip logged that was used in the dst-acl aside
from debug logging? Maybe through some annotation mechanism?

Squid version is 6.2, as 6.4 crashes with assertion errors here, too.

thanks,
Martin

___
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users


___
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users
http_port 3128

acl SSL_ports port 443
acl Safe_ports port 80
acl Safe_ports port 443

acl CONNECT method CONNECT

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

acl www_example_org dst 93.184.216.34

http_access deny www_example_org
http_access allow all

never_direct deny all
always_direct allow all

logformat debug_squid %ts.%03tu %6tr %>a %Ss/%p %>st "%{Referer}>h" "%{User-Agent}>h" %err_code "%err_detail" 
%master_xaction
access_log daemon:/var/log/squid/test.log logformat=debug_squid
cache_log /var/log/squid/cache.log
debug_options ALL,2
___
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Cache NTLM Authenticaion

[Amos Jeffries]

On 27/10/23 14:08, Andre Bolinhas wrote:

Hi

It's possible squid cache NTLM authentication from users?



NTLM tokens are unique per TCP connection. So no, caching is a pointless 
waste of CPU and memory. The best that can be done already is.



My goal is to store the credentials in cache in order to reduce the 
request to Active Directory.




The only way to do that is to reduce unique TCP connections between 
clients and Squid.


Check that 
 
directive is either absent or turned "on" explicitly.




I'm trying guide from this squid : auth_param configuration directive 
(squid-cache.org)  
but there is no information relative to cache the authentication / 
credentials.


Also, in NTLM did you recommend to use the keep_alive option?


If it works, yes. Though be aware it only affects the initial request of 
the NTLM handshake.



Cheers
Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users