Re: [squid-users] Best practice for adding or removing ACLs dynamically ?

[Eliezer Croitor]

Following up.

Well, github and google is full of horrible ways of implementing the 
tcp_outgoing_address selection so..
I have written a sketch of the concept I was writing about based on ncsa_auth 
which I must admit is far more .. simple then using a DB like mysql or LDAP.

The config I found was using either:
- multiple inbound squid ports -> mapped each to a single tcp_outgoing_addrees
- multiple static usernames mapped to a specific address(based on either 
username or usernamer+dst-domain.. else ACLs types)
- Others which are not relevant since these break the internet

What I am offering is an example which offloads the actual selection 
code/algorithm and config into an external_acl_type helper.
The result would be that squid will only listen(when possible and required) 
only on one port.
(which means that adding more ports can multiply/power up the service config 
"capacity", like vlans ontop of vlans)
Also the authentication mechanism would be ncsa_auth which is good for up to 
1000s++ of users since the helpers store the htaccess file In RAM.
Even if the file will contain 10k users and the auth helpers in use would be 
let say 40 and up, the proxy would still consume less Memory on the
storage of the htaccess file in RAM compared to a MySQL or LDAP DB.

To change an address for a specific existing user you can use a simple sed 
command:
sed -i -E "s@user80[\ \t]+[0-9]+@user80 1@g" /etc/squid/user-to-ip.txt

and to change password for a user you can use a simple htaccess command:
htpasswd -b /etc/squid/htpasswd  user80 1234


Within my external_acl_helper I have used a similar approach to the one the 
ncsa_auth helper uses to reload modified files from disk.
(If the mtime stamp is being changed from the old one the helper would reload 
the mapping config file from disk)
* 
https://github.com/elico/vagrant-squid-outgoing-addresses/blob/master/shared/note.rb

To demonstrate this piece of code I have used Vagrant on-top of Virtualbox.
(I have verified that on the current Windows 10 PRO installations both 
Virtualbox and Hyper-v can be used simultaneously with CPU Virtualization 
support ON)

The example lab is at:
https://github.com/elico/vagrant-squid-outgoing-addresses

The test is composed of two nodes:
* Squid
* Web

The Squid instance will be populated with more then 20 IP addresses on the same 
subnet.
For simplicity I have added a simple /32 address to the Squid+Web 
private/closed network interface.
There are other ways which can be used in production when Routing daemons are 
in the picture.

To start the demo/test you can use:
vagrant up
vagrant ssh squid --command /vagrant/test-random-ips-assignment.sh

The user to ip config mapping file is at:
/etc/squid/user-to-ip.txt

I have used a simple "  " mapping syntax.

Technically speaking this setup can work with scales of 1000s and up of 
addresses and clients.
I am pretty sure it's good enough for proxy services which uses multiple 
routing tunnels and/or BGP feeds.

If anyone wants to ask about this setup I will be more than happy to receive 
emails about it here in the list.

All The Bests,


Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: ngtech1...@gmail.com

-Original Message-
From: Eliezer Croitor  
Sent: Sunday, November 1, 2020 2:41 AM
To: squid-users@lists.squid-cache.org
Cc: 'roee klinger' 
Subject: RE: [squid-users] Best practice for adding or removing ACLs 
dynamically ?

Hey Roee,

I am trying to understand what part of squid.conf you want to be able to change 
without a reconfigure/reload?
If you have many users, ie above 50 you should probably not use a simple 
ncsa_auth although  it's possible and in more then one case is preferable.
You could probably write your own basic auth helper that will interact with a 
DB which will probably simplify your whole setup.
(You can use existing basic auth helpers with mysql or ldap)

As for the tcp_outgoing_address, it’s a whole different story.
Since it's a "fast" acl type the options to do something dynamic with it are an 
issue.
(Maybe eCAP/ICAP service or a "pre-cooked" note or other factor to the acl can 
be used)

I am pretty sure that if an authentication service can reply with a note ie 
connection annotation then it can be used for the address selection.
One issue with it is that It will be valid for the next X ttl 
seconds/minutes/hours.

I do believe that there should be a way to allow something like external_acl 
helper to affect this squid feature.
I was thinking that an eCAP or an ICAP service or an external_acl helper can 
add a note for a connection based on couple other factors like:
* src ip
* auth username
* request domain or request sni
* ...

So let say the proxy will have a set of 100 addresses, each will have a single 
specific matching acl for a request header or connection annotation/note.
This way the selection of a tcp_outgoing_address would be a little less complex 
the it is today.

Re: [squid-users] Best practice for adding or removing ACLs dynamically ?

[Amos Jeffries]

On 1/11/20 12:27 pm, roee klinger wrote:

Thanks Amos!

I updated "auth_param basic credentialsttl" according to your advice and 
it is working great.


I am still having issues with the "tcp_outgoing_address 192.168.8.12 
acl_for_user3002" part, you mentioned:

 > For ACLs with values that are expected to change often it is best to use
 > an external_acl_type helper that manages the updates or fetches from
 > somewhere the updates are handled without a reload.

My script updates the authenticator successfully, but when I update "acl 
acl_for_user3002 proxy_auth user2" to the new username I have to 
reconfigure to take effect.
I read online for hours but to my best understanding external_acl_type 
are for auth and access control, but they don't work for my needs I believe.


Is there any way to use external_acl_type in a way I don't understand to 
solve this problem? Do I have to reconfigure every time I make changes 
to an ACL in squid.conf?



Some directives have to produce allow/deny result immediately, without 
waiting for a helper to respond. The details are documented here:

 

In modern Squid you can use a helper to set annotations which are 
checked with the "note" ACL type in the fast checks.




It sounds a bit like you are trying to tie IPs to individual users. 
Please be aware that breaks the multiplexing and persistence features of 
HTTP, which is a major performance loss.


Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Best practice for adding or removing ACLs dynamically ?

[Eliezer Croitor]

Hey Roee,

I am trying to understand what part of squid.conf you want to be able to change 
without a reconfigure/reload?
If you have many users, ie above 50 you should probably not use a simple 
ncsa_auth although  it's possible and in more then one case is preferable.
You could probably write your own basic auth helper that will interact with a 
DB which will probably simplify your whole setup.
(You can use existing basic auth helpers with mysql or ldap)

As for the tcp_outgoing_address, it’s a whole different story.
Since it's a "fast" acl type the options to do something dynamic with it are an 
issue.
(Maybe eCAP/ICAP service or a "pre-cooked" note or other factor to the acl can 
be used)

I am pretty sure that if an authentication service can reply with a note ie 
connection annotation then it can be used for the address selection.
One issue with it is that It will be valid for the next X ttl 
seconds/minutes/hours.

I do believe that there should be a way to allow something like external_acl 
helper to affect this squid feature.
I was thinking that an eCAP or an ICAP service or an external_acl helper can 
add a note for a connection based on couple other factors like:
* src ip
* auth username
* request domain or request sni
* ...

So let say the proxy will have a set of 100 addresses, each will have a single 
specific matching acl for a request header or connection annotation/note.
This way the selection of a tcp_outgoing_address would be a little less complex 
the it is today.

I have couple other ideas for implementations which I have experimented with 
but the proxy admin need to learn how these work which might be
a bit complicated some times.

Eliezer

Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: ngtech1...@gmail.com

-Original Message-
From: squid-users  On Behalf Of roee 
klinger
Sent: Saturday, October 31, 2020 2:35 AM
To: squid-users@lists.squid-cache.org
Subject: [squid-users] Best practice for adding or removing ACLs dynamically ?


Hey,
I have Squid configured to send users to different outgoing interface like so:

..
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/htpassword
acl acl_for_user3002 proxy_auth user2
tcp_outgoing_address 192.168.8.12 acl_for_user3002
http_port 3002 name=3002
http_access allow authenticated
..

When I wanted to change the username:password for user2, I run a bash script to 
change it in squid.conf and also in htpassword and then I run "squid -k 
reconfigure", if I don't reconfigure the old user still has access to the proxy 
and the new one doesn't for about 30 minutes.

I am expecting to have 100s of users soon that will change credentials often, 
and also I would like to blacklist websites often and on the fly, so I was 
searching for a better way to manage this without reconfiguring every time, 
since sometimes a reconfigure can take up to 10-15 seconds.

I am new to Squid and wasn't able to find any info on this, am I doing this 
currently or there is a better way to change users/ACLs on the fly without 
reloading Squid?

Thanks,
Roee Klinger
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Best practice for adding or removing ACLs dynamically ?

[roee klinger]

Thanks Amos!

I updated "auth_param basic credentialsttl" according to your advice and it
is working great.

I am still having issues with the "tcp_outgoing_address 192.168.8.12
acl_for_user3002" part, you mentioned:
> For ACLs with values that are expected to change often it is best to use
> an external_acl_type helper that manages the updates or fetches from
> somewhere the updates are handled without a reload.

My script updates the authenticator successfully, but when I update "acl
acl_for_user3002 proxy_auth user2" to the new username I have to
reconfigure to take effect.
I read online for hours but to my best understanding external_acl_type are
for auth and access control, but they don't work for my needs I believe.

Is there any way to use external_acl_type in a way I don't understand to
solve this problem? Do I have to reconfigure every time I make changes to
an ACL in squid.conf?

Thanks again for your help.

On Sat, Oct 31, 2020 at 5:48 PM Amos Jeffries  wrote:

> On 31/10/20 1:34 pm, roee klinger wrote:
> > 
> > Hey,
> > I have Squid configured to send users to different outgoing interface
> like so:
> >
> > ..
> > auth_param basic program /usr/lib/squid/basic_ncsa_auth
> /etc/squid/htpassword
> > acl acl_for_user3002 proxy_auth user2
> > tcp_outgoing_address 192.168.8.12 acl_for_user3002
> > http_port 3002 name=3002
>
>
> No need to name a *_port like this. The default name is the first
> parameter string ("3002" on this line).
>
>
> > http_access allow authenticated
> > ..
> >
> > When I wanted to change the username:password for user2, I run a bash
> script to change it in squid.conf and also in htpassword and then I run
> "squid -k reconfigure", if I don't reconfigure the old user still has
> access to the proxy and the new one doesn't for about 30 minutes.
> >
>
> No need to restart for that change. The helper you have there will
> automatically detect changes to the htpassword file and reload it.
>
> It is a little odd that the new user was not able to authenticate. Check
> that your test did not lookup and cache a non-existence result for them
> prior to being added.
>
>
> The delay is due to the credentials being valid for a period of time. To
> reduce workload on the auth system Squid caches credential details for a
> while.
>
> Set "auth_param basic credentialsttl " to shorter values to reduce the
> delay (default is 2hrs).
>
>
> > I am expecting to have 100s of users soon that will change credentials
> often, and also I would like to blacklist websites often and on the fly, so
> I was searching for a better way to manage this without reconfiguring every
> time, since sometimes a reconfigure can take up to 10-15 seconds.
> >
>
> This helper does not need a reconfigure at all as far as I can tell from
> the code.
>
> All the reconfigure was doing for you previously was triggering an early
> prune of the records in the credentials cache. Probably why you saw
> about 30min delay instead of about 2hrs.
>
>
> > I am new to Squid and wasn't able to find any info on this, am I doing
> this currently or there is a better way to change users/ACLs on the fly
> without reloading Squid?
> >
>
> Config changes in squid.conf itself needs a reconfigure or sometimes a
> restart.
>
>
> For auth and ACLs whose values that come into Squid from a helper it
> depends on the helper itself. Most can auto-detect changes to their
> background databases and not need anything from Squid to update the
> outputs. All helpers do have some form of caching of their results by
> Squid, so there are settings in squid.conf to tune that to your needs -
> as you can see from the auth issue above.
>
>
> For ACLs with values that are expected to change often it is best to use
> an external_acl_type helper that manages the updates or fetches from
> somewhere the updates are handled without a reload.
>
>
>
> Amos
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Best practice for adding or removing ACLs dynamically ?

[Amos Jeffries]

On 31/10/20 1:34 pm, roee klinger wrote:


Hey,
I have Squid configured to send users to different outgoing interface like so:

..
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/htpassword
acl acl_for_user3002 proxy_auth user2
tcp_outgoing_address 192.168.8.12 acl_for_user3002
http_port 3002 name=3002



No need to name a *_port like this. The default name is the first 
parameter string ("3002" on this line).




http_access allow authenticated
..

When I wanted to change the username:password for user2, I run a bash script to change it 
in squid.conf and also in htpassword and then I run "squid -k reconfigure", if 
I don't reconfigure the old user still has access to the proxy and the new one doesn't 
for about 30 minutes.



No need to restart for that change. The helper you have there will 
automatically detect changes to the htpassword file and reload it.


It is a little odd that the new user was not able to authenticate. Check 
that your test did not lookup and cache a non-existence result for them 
prior to being added.



The delay is due to the credentials being valid for a period of time. To 
reduce workload on the auth system Squid caches credential details for a 
while.


Set "auth_param basic credentialsttl " to shorter values to reduce the 
delay (default is 2hrs).




I am expecting to have 100s of users soon that will change credentials often, 
and also I would like to blacklist websites often and on the fly, so I was 
searching for a better way to manage this without reconfiguring every time, 
since sometimes a reconfigure can take up to 10-15 seconds.



This helper does not need a reconfigure at all as far as I can tell from 
the code.


All the reconfigure was doing for you previously was triggering an early 
prune of the records in the credentials cache. Probably why you saw 
about 30min delay instead of about 2hrs.




I am new to Squid and wasn't able to find any info on this, am I doing this 
currently or there is a better way to change users/ACLs on the fly without 
reloading Squid?



Config changes in squid.conf itself needs a reconfigure or sometimes a 
restart.



For auth and ACLs whose values that come into Squid from a helper it 
depends on the helper itself. Most can auto-detect changes to their 
background databases and not need anything from Squid to update the 
outputs. All helpers do have some form of caching of their results by 
Squid, so there are settings in squid.conf to tune that to your needs - 
as you can see from the auth issue above.



For ACLs with values that are expected to change often it is best to use 
an external_acl_type helper that manages the updates or fetches from 
somewhere the updates are handled without a reload.




Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Best practice for adding or removing ACLs dynamically ?

[roee klinger]


Hey,
I have Squid configured to send users to different outgoing interface like so:

..
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/htpassword
acl acl_for_user3002 proxy_auth user2
tcp_outgoing_address 192.168.8.12 acl_for_user3002
http_port 3002 name=3002
http_access allow authenticated
..

When I wanted to change the username:password for user2, I run a bash script to 
change it in squid.conf and also in htpassword and then I run "squid -k 
reconfigure", if I don't reconfigure the old user still has access to the proxy 
and the new one doesn't for about 30 minutes.

I am expecting to have 100s of users soon that will change credentials often, 
and also I would like to blacklist websites often and on the fly, so I was 
searching for a better way to manage this without reconfiguring every time, 
since sometimes a reconfigure can take up to 10-15 seconds.

I am new to Squid and wasn't able to find any info on this, am I doing this 
currently or there is a better way to change users/ACLs on the fly without 
reloading Squid?

Thanks,
Roee Klinger
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] ..::Best Practice::..

[Alfonso Alejandro Reyes Jimenez]

Hi list.

We would like to know whats the best practice when we have more than 3000 rules 
on our squid. Right now squid takes more than 15 minutes to start or restart, 
we think that's because we have a lot of rules.

The question is:

If we change all that rules to a text file, squid will start faster?

Right now we are not able to add rules or change it, we need to wait until 
lunch time to apply changes.

Any ideas?

Thanks in advance.

Saludos. 
  
Alfonso

Re: [squid-users] ..::Best Practice::..

[Luis Daniel Lucio Quiroz]

HOla :) consultoria!!


We did have the same problem,  from 15 mins we reduce up to 30 seconds,

Squid 3.0 and 3.1 has a feature, that they stop all traffic until the hole set 
of ACL are processed, there is not much to dup than acl 
optimization/reducction in a single-box scenario.

LD

Le mardi 24 mai 2011 13:20:09, Alfonso Alejandro Reyes Jimenez a écrit :
 Hi list.
 
 We would like to know whats the best practice when we have more than 3000
 rules on our squid. Right now squid takes more than 15 minutes to start or
 restart, we think that's because we have a lot of rules.
 
 The question is:
 
 If we change all that rules to a text file, squid will start faster?
 
 Right now we are not able to add rules or change it, we need to wait until
 lunch time to apply changes.
 
 Any ideas?
 
 Thanks in advance.
 
 Saludos.
  
 Alfonso

Re: [squid-users] ..::Best Practice::..

[Eliezer Croitoru]

it depends on the machine...

to make sure that the 3000 rules loading is the problem you can run the 
squid server on a verbose mode to see how long it takes to load the rules.


from what i have seen the stop process depends less on the amount of 
rules but on the amount of connections used on this server.



Regards

Eliezer


On 24/05/2011 21:20, Alfonso Alejandro Reyes Jimenez wrote:


Hi list.

We would like to know whats the best practice when we have more than 3000 rules 
on our squid. Right now squid takes more than 15 minutes to start or restart, 
we think that's because we have a lot of rules.

The question is:

If we change all that rules to a text file, squid will start faster?

Right now we are not able to add rules or change it, we need to wait until 
lunch time to apply changes.

Any ideas?

Thanks in advance.

Saludos.
  
Alfonso

Re: [squid-users] best practice for intercept

[Amos Jeffries]

On 10/12/10 08:02, jeff donovan wrote:

okay made a few changes to the conf file to bring up to 3.1 terminology. things 
are still bogging down.

I just erased and reset the cache  squid -z  and my internet speeds are back 
to normal.


ill run for 24 hours and see how things go. This jam up has only been with 
3.1.9.
-j


Hmm, strange that that should fix it.

It's likely to come back unless it was some cache corruption. If/when 
you see it again, there are a few other things to check:

 - the size of the disk usage matches what you configured (bug 3068)
 - whether squid is running garbage collection, particularly on big 
caches the swap_high and swap_log thresholds should be close together.

 - if you have delay pools active bug 3096 will be biting you.
 - check the box is not swapping. index may be more than the box can 
cope with or bug 3113 and/or 3017 may also be biting.


Amos
--
Please be using
  Current Stable Squid 2.7.STABLE9 or 3.1.9
  Beta testers wanted for 3.2.0.3

Re: [squid-users] best practice for transparent

[Amos Jeffries]

On 10/12/10 02:37, BASDarchive wrote:


On Dec 7, 2010, at 10:35 PM, Amos Jeffries wrote:


On Tue, 7 Dec 2010 19:35:08 -0500, BASDarchive
basdarch...@beth.k12.pa.us
wrote:

On Dec 7, 2010, at 5:13 PM, Amos Jeffries wrote:


On 08/12/10 05:32, donovan jeffrey j wrote:

greetings

i recently updated my transparent proxy to sq 3.1.9, which also uses
squidguard for url filters.


First best practice is to use the right terminology.

sorry i forgot we changed that ;)


Your log traces says Accepting  intercepted HTTP connections at
10.0.2.3:3128  So they are NAT interception connections.


yes I am using NAT after Squid.

client ---  [ squid ]   [ NAT ] ---



??
interception proxy is done with NAT before squid. Doing NAT on the
outside looping back into Squid could be causing the long waits you saw.

clients--  NAT snips --  World
 \NAT--  Squid--  World


thanks for the reply.

So should I have my squid box after my firewall ? my clients access through the 
squid box and through the NAT firewall

say client 10.10.1.1 --- [ squid 10.10.1.2  --- 10.11.1.2 ] ---  
[ NAT Firewall ] -  [ bgp router to internet ]
ive had this setup for years. the 10.11.1.2 has a Static NAT translation so all 
clients pass through the squid.



It sounds like you are trying to describe a traffic flow of:
 client -- 10.11.1.2 --/NAT/-- Squid --/NAT/-- Firewall --- Internet

In order to do NAT interception (aka transparent proxy) the relevant 
DNAT or REDIRECT has to be done between the client and Squid.


The traffic going out from Squid has to void being looped back to Squid 
but that is all that matters.

snip


#no cache settings
no_cache deny noc
no_cache deny admin
no_cache deny hs
no_cache deny ms
no_cache deny ele
no_cache deny all


no_cache has been renamed to cache.


so,

i can use just cache deny all


Yes if you really want that.





NP: Following a list of denials with deny all is a waste of CPU cycles.
The rules all collapse down to a single deny all action.



http_access allow manager localhost
#http_access allow manager apache
http_access allow noc
http_access allow admin
http_access allow hs
http_access allow ms
http_access allow ele
http_access deny all

#Squid's user and group
cache_effective_user squid squid


Only one entry on this line. Second one is dropped.


which one is dropped ? should it only be cache_effective_user squid 


Yes, it should be cache_effective_user squid
snip


For NAT interception proxy in 3.1 it should now be this:

  http_port 3128
  http_port 3129 intercept

(3129 being some unusual port only known between NAT and Squid)


so even this  http_port 10.0.1.2:3128 transparent  is outdated ?



Yes, the confusing transparent keyword is deprecated.


Amos
--
Please be using
  Current Stable Squid 2.7.STABLE9 or 3.1.9
  Beta testers wanted for 3.2.0.3

[squid-users] best practice for transparent

[donovan jeffrey j]

greetings

i recently updated my transparent proxy to sq 3.1.9, which also uses squidguard 
for url filters.

this have been bogging down.  browser always says ,.. waiting for google,... 
or waiting for www.abc.com
I could have a dns issue or I could have a cache swap issue or a squidguard 
issue. I first wanted to make sure that running 3.1.9 transparent is the best 
version for the job. I had read a while back in a thread that v2.7 might be 
better than 3.1.x. Can anyone confirm ?

heres a snap shot of cache.log

hook2:bin root# ulimit -n 2048
hook2:bin root# squid -d1x
hook2:bin root# 2010/12/07 11:12:34| Starting Squid Cache version 3.1.9 for 
i686-apple-darwin9.8.0...
2010/12/07 11:12:34| Process ID 5210
2010/12/07 11:12:34| With 2048 file descriptors available
2010/12/07 11:12:34| Initializing IP Cache...
2010/12/07 11:12:34| DNS Socket created at [::], FD 6
2010/12/07 11:12:34| DNS Socket created at 0.0.0.0, FD 7
2010/12/07 11:12:34| Adding domain beth.k12.pa.us from /etc/resolv.conf
2010/12/07 11:12:34| Adding nameserver 8.8.8.8 from /etc/resolv.conf
2010/12/07 11:12:34| Adding nameserver 209.96.96.2 from /etc/resolv.conf
2010/12/07 11:12:34| helperOpenServers: Starting 100/100 'squidGuard' processes
2010/12/07 11:12:35| Unlinkd pipe opened on FD 212
2010/12/07 11:12:35| Store logging disabled
2010/12/07 11:12:35| Swap maxSize 134215680 + 262144 KB, estimated 1038 
objects
2010/12/07 11:12:35| Target number of buckets: 517222
2010/12/07 11:12:35| Using 524288 Store buckets
2010/12/07 11:12:35| Max Mem  size: 262144 KB
2010/12/07 11:12:35| Max Swap size: 134215680 KB
2010/12/07 11:12:35| Version 1 of swap file with LFS support detected... 
2010/12/07 11:12:35| Rebuilding storage in /Volumes/cache2/cache (CLEAN)
2010/12/07 11:12:35| Version 1 of swap file with LFS support detected... 
2010/12/07 11:12:35| Rebuilding storage in /Volumes/cache3/cache (CLEAN)
2010/12/07 11:12:35| Using Least Load store dir selection
2010/12/07 11:12:35| Current Directory is /usr/bin
2010/12/07 11:12:35| Loaded Icons.
2010/12/07 11:12:35| Accepting  intercepted HTTP connections at 10.0.2.3:3128, 
FD 217.
2010/12/07 11:12:35| HTCP Disabled.
2010/12/07 11:12:35| Squid modules loaded: 0
2010/12/07 11:12:35| Ready to serve requests.
2010/12/07 11:12:35| Store rebuilding is 0.12% complete
2010/12/07 11:13:25| Done reading /Volumes/cache3/cache swaplog (3496117 
entries)
2010/12/07 11:13:25| Store rebuilding is 99.79% complete
2010/12/07 11:13:25| Done reading /Volumes/cache2/cache swaplog (3510803 
entries)
2010/12/07 11:13:25| Finished rebuilding storage from disk.
2010/12/07 11:13:25|   7006920 Entries scanned
2010/12/07 11:13:25| 0 Invalid entries.
2010/12/07 11:13:25| 0 With invalid flags.
2010/12/07 11:13:25|   7006920 Objects loaded.
2010/12/07 11:13:25| 0 Objects expired.
2010/12/07 11:13:25| 0 Objects cancelled.
2010/12/07 11:13:25| 0 Duplicate URLs purged.
2010/12/07 11:13:25| 0 Swapfile clashes avoided.
2010/12/07 11:13:25|   Took 50.69 seconds (138218.19 objects/sec).
2010/12/07 11:13:25| Beginning Validation Procedure
2010/12/07 11:13:26|   262144 Entries Validated so far.
2010/12/07 11:13:26|   1310720 Entries Validated so far.
2010/12/07 11:13:26|   1572864 Entries Validated so far.
2010/12/07 11:13:26|   3407872 Entries Validated so far.
2010/12/07 11:13:26|   3670016 Entries Validated so far.
2010/12/07 11:13:26|   4718592 Entries Validated so far.
2010/12/07 11:13:26|   4980736 Entries Validated so far.
2010/12/07 11:13:26|   6291456 Entries Validated so far.
2010/12/07 11:13:26|   6553600 Entries Validated so far.
2010/12/07 11:13:27|   6815744 Entries Validated so far.
2010/12/07 11:13:27|   8388608 Entries Validated so far.
2010/12/07 11:13:27|   8650752 Entries Validated so far.
2010/12/07 11:13:27|   8912896 Entries Validated so far.
2010/12/07 11:13:27|   9699328 Entries Validated so far.
2010/12/07 11:13:27|   9961472 Entries Validated so far.
2010/12/07 11:13:27|   10223616 Entries Validated so far.
2010/12/07 11:13:27|   10485760 Entries Validated so far.
2010/12/07 11:13:27|   10747904 Entries Validated so far.
2010/12/07 11:13:27|   12845056 Entries Validated so far.
2010/12/07 11:13:28|   Completed Validation Procedure
2010/12/07 11:13:28|   Validated 14013839 Entries
2010/12/07 11:13:28|   store_swap_size = 120801068

Re: [squid-users] best practice for transparent

[Amos Jeffries]

On 08/12/10 05:32, donovan jeffrey j wrote:

greetings

i recently updated my transparent proxy to sq 3.1.9, which also uses squidguard 
for url filters.


First best practice is to use the right terminology.

Your log traces says Accepting  intercepted HTTP connections at 
10.0.2.3:3128  So they are NAT interception connections.





this have been bogging down.  browser always says ,.. waiting for google,... 
or waiting for www.abc.com
I could have a dns issue or I could have a cache swap issue or a squidguard 
issue. I first wanted to make sure that running 3.1.9 transparent is the best 
version for the job. I had read a while back in a thread that v2.7 might be 
better than 3.1.x. Can anyone confirm ?


2.7 should not be.

I've not had anyone explicitly mention whether the NAT logic upgrades to 
3.x worked or not in Mac. The BSD ones needed some extra fixes which 
were done back around 3.1.6



heres a snap shot of cache.log

hook2:bin root# ulimit -n 2048
hook2:bin root# squid -d1x
hook2:bin root# 2010/12/07 11:12:34| Starting Squid Cache version 3.1.9 for 
i686-apple-darwin9.8.0...

snip

2010/12/07 11:12:35| Loaded Icons.
2010/12/07 11:12:35| Accepting  intercepted HTTP connections at 10.0.2.3:3128, 
FD 217.
2010/12/07 11:12:35| HTCP Disabled.
2010/12/07 11:12:35| Squid modules loaded: 0
2010/12/07 11:12:35| Ready to serve requests.

snip

Amos
--
Please be using
  Current Stable Squid 2.7.STABLE9 or 3.1.9
  Beta testers wanted for 3.2.0.3

Re: [squid-users] best practice for transparent

[donovan jeffrey j]

On Dec 7, 2010, at 5:13 PM, Amos Jeffries wrote:

 On 08/12/10 05:32, donovan jeffrey j wrote:
 greetings
 
 i recently updated my transparent proxy to sq 3.1.9, which also uses 
 squidguard for url filters.
 
 First best practice is to use the right terminology.
sorry i forgot we changed that ;)
 
 Your log traces says Accepting  intercepted HTTP connections at 
 10.0.2.3:3128  So they are NAT interception connections.

yes I am using NAT after Squid.

client ---  [ squid ]  [ NAT ] ---

 
 
 
 this have been bogging down.  browser always says ,.. waiting for 
 google,... or waiting for www.abc.com
 I could have a dns issue or I could have a cache swap issue or a squidguard 
 issue. I first wanted to make sure that running 3.1.9 transparent is the 
 best version for the job. I had read a while back in a thread that v2.7 
 might be better than 3.1.x. Can anyone confirm ?
 
 2.7 should not be.
 
 I've not had anyone explicitly mention whether the NAT logic upgrades to 3.x 
 worked or not in Mac. The BSD ones needed some extra fixes which were done 
 back around 3.1.6

So basically 3.1.9 should be fine for my purposes then.

here is my config. does anything stand out as wrong or should be adjusted ?


# squid.conf
#

#
#
# set logging to the lowest level
debug_options ALL,1

#access to squid and the cache manager
#
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255

acl all src 0.0.0.0/0.0.0.0
acl noc src 10.3.1.0/24 10.135.0.0/16 10.235.0.0/16 10.35.1.0/24
acl admin src 10.139.0.0/16 10.136.0.0/16 10.103.0.0/16
acl hs src  10.150.0.0/16 10.149.0.0/16 10.151.0.0/16 10.152.0.0/16 
10.153.0.0/16
acl ms src  10.142.0.0/16 10.140.0.0/16
acl ele src 10.104.0.0/16

#no cache settings
no_cache deny noc
no_cache deny admin
no_cache deny hs
no_cache deny ms
no_cache deny ele
no_cache deny all

http_access allow manager localhost
#http_access allow manager apache
http_access allow noc
http_access allow admin
http_access allow hs
http_access allow ms
http_access allow ele
http_access deny all

#Squid's user and group
cache_effective_user squid squid

#visible hostname
visible_hostname T1-2

# set log directories
cache_access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log

# set cache directories of 16GB each
cache_dir ufs /usr/local/squid/var/cache 100 16 256


request_header_max_size 1000 KB


# set the cache memory target for the Squid process
cache_mem 100 MB

http_port 3128
httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on 
#
redirect_program /usr/local/bin/squidGuard
redirect_children 100
client_persistent_connections off
server_persistent_connections off

Re: [squid-users] best practice for transparent

[donovan jeffrey j]

On Dec 7, 2010, at 5:13 PM, Amos Jeffries wrote:

 Your log traces says Accepting  intercepted HTTP connections at 
 10.0.2.3:3128  So they are NAT interception connections.

question on terminology; which one do I use for 3.1

http_port 10.0.2.2:3128 transparent

or

http_port 10.0.2.2:3128 intercept 

tnx
-j

Re: [squid-users] best practice for transparent

[Amos Jeffries]

On Tue, 7 Dec 2010 19:35:08 -0500, BASDarchive
basdarch...@beth.k12.pa.us
wrote:
 On Dec 7, 2010, at 5:13 PM, Amos Jeffries wrote:
 
 On 08/12/10 05:32, donovan jeffrey j wrote:
 greetings
 
 i recently updated my transparent proxy to sq 3.1.9, which also uses
 squidguard for url filters.
 
 First best practice is to use the right terminology.
 sorry i forgot we changed that ;)
 
 Your log traces says Accepting  intercepted HTTP connections at
 10.0.2.3:3128  So they are NAT interception connections.
 
 yes I am using NAT after Squid.
 
 client ---  [ squid ]  [ NAT ] ---
 

??
 interception proxy is done with NAT before squid. Doing NAT on the
outside looping back into Squid could be causing the long waits you saw.

clients -- NAT snips -- World
 \NAT -- Squid -- World


 
 
 this have been bogging down.  browser always says ,.. waiting for
 google,... or waiting for www.abc.com
 I could have a dns issue or I could have a cache swap issue or a
 squidguard issue. I first wanted to make sure that running 3.1.9
 transparent is the best version for the job. I had read a while back
in
 a thread that v2.7 might be better than 3.1.x. Can anyone confirm ?
 
 2.7 should not be.
 
 I've not had anyone explicitly mention whether the NAT logic upgrades
to
 3.x worked or not in Mac. The BSD ones needed some extra fixes which
were
 done back around 3.1.6
 
 So basically 3.1.9 should be fine for my purposes then.
 
 here is my config. does anything stand out as wrong or should be
adjusted ?
 
 
 # squid.conf
 #
 
 #
 #
 # set logging to the lowest level
 debug_options ALL,1
 
 #access to squid and the cache manager
 #
 acl manager proto cache_object
 acl localhost src 127.0.0.1/255.255.255.255

In 3.1 that is:   acl src localhost 127.0.0.1 ::1

 
 acl all src 0.0.0.0/0.0.0.0

Remove the above line for 3.x. all is now built in.

 acl noc src 10.3.1.0/24 10.135.0.0/16 10.235.0.0/16 10.35.1.0/24
 acl admin src 10.139.0.0/16 10.136.0.0/16 10.103.0.0/16
 acl hs src10.150.0.0/16 10.149.0.0/16 10.151.0.0/16 10.152.0.0/16
 10.153.0.0/16
 acl ms src10.142.0.0/16 10.140.0.0/16
 acl ele src   10.104.0.0/16

NP: In 3.1 you may want to add the IPv6 ranges fe80::/16 and fc00::/7 for
local administration access.

 
 #no cache settings
 no_cache deny noc
 no_cache deny admin
 no_cache deny hs
 no_cache deny ms
 no_cache deny ele
 no_cache deny all

no_cache has been renamed to cache.

NP: Following a list of denials with deny all is a waste of CPU cycles.
The rules all collapse down to a single deny all action.

 
 http_access allow manager localhost
 #http_access allow manager apache
 http_access allow noc
 http_access allow admin
 http_access allow hs
 http_access allow ms
 http_access allow ele
 http_access deny all
 
 #Squid's user and group
 cache_effective_user squid squid

Only one entry on this line. Second one is dropped. Configure the OS user
squid to only be a member of the OS group squid.

 
 #visible hostname
 visible_hostname T1-2

This is ideally a full domain name with working DNS.
If gethostname() returns a properly DNS resolvable domain name this
setting is not needed either.

 
 # set log directories
 cache_access_log /var/log/squid/access.log

Directive name changed to access_log now.

 cache_log /var/log/squid/cache.log
 cache_store_log /var/log/squid/store.log

You can drop the cache_store_log line. Default is not to log it now.

 
 # set cache directories of 16GB each
 cache_dir ufs /usr/local/squid/var/cache 100 16 256
 
 
 request_header_max_size 1000 KB
 
 
 # set the cache memory target for the Squid process
 cache_mem 100 MB
 
 http_port 3128
 httpd_accel_host virtual
 httpd_accel_port 80
 httpd_accel_with_proxy on
 httpd_accel_uses_host_header on 

None of the httpd_accel_* settings are valid any more.

... the above config lines mean this:
   http_port 80 accel vhost
   http_port 3128

For NAT interception proxy in 3.1 it should now be this:

  http_port 3128
  http_port 3129 intercept

(3129 being some unusual port only known between NAT and Squid)


 #
 redirect_program /usr/local/bin/squidGuard
 redirect_children 100
 client_persistent_connections off
 server_persistent_connections off

You may want to reconsider that disabling now. Squid 3.1 is mostly
HTTP/1.1 compliant and pconn support is expected by servers.


Amos

[squid-users] Best practice for cache_dir

[Tony Mallin]

Hello

I'm using squid in a 200 user environment, processing approximately 50GB of 
traffic per day. 

Setup:

ESXi host, VM with 2 vCPUs assigned, 2gb ram, 40GB hdd space (stored on a SANS, 
speed shouldn’t be an issue)

I was wondering what the general opinion was for cache-related settings were in 
this sort of environment.

Config relevant to caching: 

Cache_mem 300 MB
Cache_dir ufs /usr/local/squid/cache 51200 64 255

A couple of thoughts I had: 

I already know that 51200 is over the total size of the hd assigned ,I need to 
either lower that value (would performance be hindered by lowering it?) or get 
a second drive attached and use that. 

So here come a few questions:

For the traffic and setup I have, is lowering this, to say, 25000 64 255 safe 
to do? 
Would there be any benefit to having it on a separate drive?
Does bigger cache equal better performance in the long run?

Thanks in advance
 



Summer 2010 Architext Now Available - 
http://www.stridetreglown.co.uk/architextsum2010.php
-
15th in this year's AJ Top 100...more info - 
http://www.stridetreglown.co.uk/aj1002010.php
--
Stride Treglown may monitor email traffic data and also the content of email 
for the purposes of security
Registered Office: Promenade House, The Promenade, Clifton Down, Bristol, BS8 
3NE
Registered in Cardiff. Registered Number: 1748850

###stl-sig-position-marker###

Re: [squid-users] Best practice for cache_dir

[Matus UHLAR - fantomas]

On 10.08.10 11:17, Tony Mallin wrote:
 I'm using squid in a 200 user environment, processing approximately 50GB
 of traffic per day.
 
 Setup:
 
 ESXi host, VM with 2 vCPUs assigned, 2gb ram, 40GB hdd space (stored on a
 SANS, speed shouldn’t be an issue)

The cache should reside on local disk, not on network disk.
otherwise forget about the disk cache and use memory-only.
However it takes much memory (which is still good to have of course)

 For the traffic and setup I have, is lowering this, to say, 25000 64 255
 safe to do?  Would there be any benefit to having it on a separate drive?
 Does bigger cache equal better performance in the long run?

bigger cache tends to give better hit ratio.
the cache should be on separate disk to have good performance.

-- 
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
There's a long-standing bug relating to the x86 architecture that
allows you to install Windows.   -- Matthew D. Fuller

Re: [squid-users] Best Practice for remote Active DirectoryAuthentication?

[Timothy Bushart]

Thats the problem, redhat is at squid-2.5.STABLE6-3:7.i386 for their latest 
supported version.

Thanks

 Henrik Nordstrom [EMAIL PROTECTED] 2/5/2007 3:59 PM 
mån 2007-02-05 klockan 09:58 -0500 skrev Timothy Bushart:
 For remote websites that require AD Authentication, what is the
 recommended configuration? We just exclude the domain in internet
 explorer proxy exclusion section. Is there a global way to detect this
 and exclude from going to through squid if a remote website requires a
 MS Login?

Works fine to proxy such sites with Squid-2.6.

Regards
Henrik



-
CONFIDENTIALITY NOTICE: This email and any attachments may contain
confidential information that is protected by law and is for the
sole use of the individuals or entities to which it is addressed.
If you are not the intended recipient, please notify the sender by
replying to this email and destroying all copies of the
communication and attachments. Further use, disclosure, copying,
distribution of, or reliance upon the contents of this email and
attachments is strictly prohibited. To contact Albany Medical
Center, or for a copy of our privacy practices, please visit us on
the Internet at www.amc.edu.

Re: [squid-users] Best Practice for remote Active DirectoryAuthentication?

[Henrik Nordstrom]

tis 2007-02-06 klockan 11:42 -0500 skrev Timothy Bushart:
 Thats the problem, redhat is at squid-2.5.STABLE6-3:7.i386 for their latest 
 supported version.

Then proxying of NTLM authentication won't work.. some things you have
to accept when running a 2.5 year old version no longer supported by the
Squid developers.

Regards
Henrik


signature.asc
Description: Detta är en digitalt signerad	meddelandedel

[squid-users] Best Practice for remote Active Directory Authentication?

[Timothy Bushart]

For remote websites that require AD Authentication, what is the
recommended configuration? We just exclude the domain in internet
explorer proxy exclusion section. Is there a global way to detect this
and exclude from going to through squid if a remote website requires a
MS Login?

Thanks


-
CONFIDENTIALITY NOTICE: This email and any attachments may contain
confidential information that is protected by law and is for the
sole use of the individuals or entities to which it is addressed.
If you are not the intended recipient, please notify the sender by
replying to this email and destroying all copies of the
communication and attachments. Further use, disclosure, copying,
distribution of, or reliance upon the contents of this email and
attachments is strictly prohibited. To contact Albany Medical
Center, or for a copy of our privacy practices, please visit us on
the Internet at www.amc.edu.

Re: [squid-users] Best Practice for remote Active Directory Authentication?

[Henrik Nordstrom]

mån 2007-02-05 klockan 09:58 -0500 skrev Timothy Bushart:
 For remote websites that require AD Authentication, what is the
 recommended configuration? We just exclude the domain in internet
 explorer proxy exclusion section. Is there a global way to detect this
 and exclude from going to through squid if a remote website requires a
 MS Login?

Works fine to proxy such sites with Squid-2.6.

Regards
Henrik



signature.asc
Description: Detta är en digitalt signerad	meddelandedel