Re: [squid-users] Squid + Squidguard Youtube URL video filtering

[Alex K]

On Thu, Aug 16, 2018, 22:51 Roberto Carna  wrote:

> Dear Amos, I've tried to sniff the HTTP requests when I ask for:
>
> https://www.youtube.com/embed/ff9sDLGtnK8?rel=0=0
>
> After that I've created a Squidguard exception list as below:
>
> ytimg.com
> googlevideo.com
> googleapis.com
> www.youtube.com/embed/ff9sDLGtnK8?rel=0=0

You may need to escape special characters (? is one). I would try:
www.youtube.com/embed/ff9sDLGtnK8
*

I assume you are putting above at the urls definition and rebuilding the
file?

What is access.log reporting?


>
> But I can't see the video yet.
>
> Please I need to know if using Squidguard it's just impossible to do
> this exception, so in this case I forget it.
>
> Thanks a lot again!!!
>
>
> 2018-08-16 10:17 GMT-03:00 Amos Jeffries :
> > On 17/08/18 00:43, Roberto Carna wrote:
> >> Dear, I have Squid + Squidguard working OK.
> >>
> >> Squidguard is filtering the entire www.youtube.com website.
> >>
> >> But now I have to permit just one video from Youtube:
> >>
> >> https://www.youtube.com/embed/ff9sDLGtnK8?rel=0=0
> >>
> >> I have added the below URL as an exception in Squidguard:
> >>
> >> www.youtube.com/embed/ff9sDLGtnK8?rel=0=0
> >>
> >> but after that I can't see it, still blocked.
> >>
> >> How can I enable just this URL from Squidguard preferently blocking
> >> the rest of Youtube ???
> >
> > Unfortunately only with a great deal of difficulty.
> >
> > The "?v=..." and "/embed/..." URLs are just public identifiers to access
> > the YouTube APIs. At the HTTP level they result in a quite long series
> > of sub-requests, redirections and the like bouncing all over the
> > youtube.* and googlevideos.* and googleapis.* domains.
> >  Yes all of them are involved multiple times. So whitelisting is an
> > all-or-nothing prospect, with other G services being implicitly
> > whitelisted as side effects.
> >
> >
> > Also, whenever the way to decipher the above maze of traffic gets
> > published so we can do things like what you ask. YT shortly afterwards
> > change how it operates - usually towards even more complexity. This has
> > happened too many times to be coincidence IMO.
> >
> >
> > Amos
> > ___
> > squid-users mailing list
> > squid-users@lists.squid-cache.org
> > http://lists.squid-cache.org/listinfo/squid-users
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid and ICMP

[Alex K]

Thanx Amos,

It is clear.

Alex

On Tue, Aug 7, 2018 at 9:20 PM, Amos Jeffries  wrote:

> On 08/08/18 04:56, Alex K wrote:
> > Hi all,
> >
> > I have a box with fairly restrictive firewall.
> > I see that the box blocks connections of squid to the requested sites
> > when squid tries to reach/send ICMP to them:
> >
> > 2018/08/07 16:51:57| Error sending to ICMP packet to 213.133.127.247.
> > ERR: (1) Operation not permitted
> > 2018/08/07 16:51:59| Error sending to ICMP packet to 194.55.30.166. ERR:
> > (1) Operation not permitted
> > 2018/08/07 16:52:00| Error sending to ICMP packet to 93.184.220.29. ERR:
> > (1) Operation not permitted
> > 2018/08/07 16:52:00| Error sending to ICMP packet to 72.21.202.25. ERR:
> > (1) Operation not permitted
> > 2018/08/07 16:52:02| Error sending to ICMP packet to 54.182.206.90. ERR:
> > (1) Operation not permitted
> > 2018/08/07 16:52:18| Error sending to ICMP packet to 54.239.220.40. ERR:
> > (1) Operation not permitted
> > 2018/08/07 16:52:18| Error sending to ICMP packet to 62.38.6.83. ERR:
> > (1) Operation not permitted
> > 2018/08/07 16:52:20| Error sending to ICMP packet to 13.32.16.243. ERR:
> > (1) Operation not permitted
> >
> > Anyone knows why squid is sending ICMP?
>
> To find the fastest route for its outbound HTTP messages when cache_peer
> are used, and to bootstrap the ARP and MTU discovery processes before
> server TCP connections have to use the information they provide.
>
> > Is this needed?
>
> Maybe. You can safely configure "pinger_enable off" if you don't care
> about a small (few milli- or micro-seconds) latency increase on TCP
> connection setup.
>
> Please note however that ICMP is not an optional protocol. It is
> mandatory for correct working of TCP. Only a few things like these echo
> packets are safely blocked.
>
> Amos
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Squid and ICMP

[Alex K]

Hi all,

I have a box with fairly restrictive firewall.
I see that the box blocks connections of squid to the requested sites when
squid tries to reach/send ICMP to them:

2018/08/07 16:51:57| Error sending to ICMP packet to 213.133.127.247. ERR:
(1) Operation not permitted
2018/08/07 16:51:59| Error sending to ICMP packet to 194.55.30.166. ERR:
(1) Operation not permitted
2018/08/07 16:52:00| Error sending to ICMP packet to 93.184.220.29. ERR:
(1) Operation not permitted
2018/08/07 16:52:00| Error sending to ICMP packet to 72.21.202.25. ERR: (1)
Operation not permitted
2018/08/07 16:52:02| Error sending to ICMP packet to 54.182.206.90. ERR:
(1) Operation not permitted
2018/08/07 16:52:18| Error sending to ICMP packet to 54.239.220.40. ERR:
(1) Operation not permitted
2018/08/07 16:52:18| Error sending to ICMP packet to 62.38.6.83. ERR: (1)
Operation not permitted
2018/08/07 16:52:20| Error sending to ICMP packet to 13.32.16.243. ERR: (1)
Operation not permitted

Anyone knows why squid is sending ICMP? Is this needed?
I am running 3.5.23 in tproxy mode with SSL splicing.


Thanx,
Alex
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid File descriptors warning

[Alex K]

On Tue, Aug 7, 2018 at 11:26 AM, Amos Jeffries  wrote:

> On 07/08/18 19:45, Alex K wrote:
> > Hi all,
> >
> > I observed the following warning at squid cache logs:
> >
> > WARNING! Your cache is running out of filedescriptors
> >
> > Googling around I tried to increase the default file descriptors of the
> > system (I am runnign Debian9 x64 bit), by setting at /etc/sysctl.conf:
> >
> ...
> >
> > I am running a compiled version 3.5.23.
> >
> > I am not sure I have done the correct steps or if I need to tweak the
> > ulimits also.
> > Any experience from your side?
>
> The init script installed with Squid 3.x on Debian sets the limit to 64K
> file descriptors.


> What is Squid startup logging for file descriptors available?
>  Probably in the system /var/log/messages log, if not cache.log.
>

After having set the parameter at squid.conf I get:

With 65535 file descriptors available

Without settings it I get:

With 4096 file descriptors available

Which seems to be the value I have set at /etc/systemd/system.conf and
/etc/systemd/user.conf:

DefaultLimitNOFILE=4096

Setting such values at /etc/security/limits.conf as below, doesn't seem to
affect anything:

proxysoftnofile  4096
proxyhardnofile  1048576

Without settings any limits I get:

With 1024 file descriptors available

which seems to be reflecting the default value reported from ulimit -a.



> Amos
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Squid File descriptors warning

[Alex K]

Hi all,

I observed the following warning at squid cache logs:

WARNING! Your cache is running out of filedescriptors

Googling around I tried to increase the default file descriptors of the
system (I am runnign Debian9 x64 bit), by setting at /etc/sysctl.conf:

fs.file-max=802762

Restarted system. Still was receiving the warnings.

When checking further I observed that I have the following default limits:


ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 15338
max locked memory   (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
*open files  (-n) 1024*
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 15338
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited


Where the "open files" seems to be the related one.

I set also the following at squid conf:

max_filedescriptors 65535

I am running a compiled version 3.5.23.

I am not sure I have done the correct steps or if I need to tweak the
ulimits also.
Any experience from your side?

Thanx,
alex
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Bounces

[Alex K]

I am using the same account to many other lists without such bounce
reports. It could be that others are not sending such reports. I will see
if i can find any logs at gtmail.

Thanx,
Alex



On Aug 4, 2018 19:18, "Goetz Schultz"  wrote:

It seems this happens from time to time. Despite my setting to "accept
all", I have drops from various mailing lists due to bounces. Why? No idea.


Thanks and regards

  Goetz R. Schultz


On 04/08/18 16:48, Matus UHLAR - fantomas wrote:
> On 04.08.18 18:29, Alex K wrote:
>> I have been receiving lately the following from the list:
>>
>> Your membership in the mailing list squid-users has been disabled due
>> to excessive bounces
>>
>> Any known issues with the list?
>
> I don't think so. It's possible that your mailserver (gmail) refused some
> mail in the last few days.
>
> Did not happen to me. I don't know if gmail provides logs related to your
> account...
>



>8--

  /"\
  \ /  ASCII Ribbon Campaign
   X   against HTML e-mail
  / \
 This message transmitted on 100% recycled electrons.
8<--

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Bounces

[Alex K]

Hi all,

I have been receiving lately the following from the list:

Your membership in the mailing list squid-users has been disabled due
to excessive bounces

Any known issues with the list?
Alex
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] PID file /var/run/squid.pid not readable AND Supervising process XXX which is not our child

[Alex K]

After proceeding with using the shipped service file, then systemctl
daemon-reload I do not experience any stuck reboots at the moment.

Alex

On Sat, Jun 9, 2018 at 4:30 PM, Alex K  wrote:

> Getting back to this, I face also issues that seems to be related with how
> systemd handles squid.
> Frequently when I try restart the VM the VM is stuch at stopping squid and
> it never restarts.
>
> Checking the differences between the autogenerated service file and the
> one shipped with squid I see:
>
> diff /run/systemd/generator.late/squid.service
> squid3-3.5.23/tools/systemd/squid.service
> 1c1,6
> < # Automatically generated by systemd-sysv-generator
> ---
> > ## Copyright (C) 1996-2016 The Squid Software Foundation and contributors
> > ##
> > ## Squid software is distributed under GPLv2+ license and includes
> > ## contributions from numerous individuals and organizations.
> > ## Please see the COPYING and CONTRIBUTORS files for details.
> > ##
> 4,14c9,10
> < Documentation=man:systemd-sysv-generator(8)
> < SourcePath=/etc/init.d/squid
> < Description=LSB: Squid HTTP Proxy version 3.x
> < Before=multi-user.target
> < Before=multi-user.target
> < Before=multi-user.target
> < Before=graphical.target
> < After=network-online.target
> < After=remote-fs.target
> < After=nss-lookup.target
> < Wants=network-online.target
> ---
> > Description=Squid Web Proxy Server
> > After=network.target
> 17,20c13,15
> < Type=forking
> < Restart=no
> < TimeoutSec=5min
> < IgnoreSIGPIPE=no
> ---
> > Type=simple
> > ExecStart=/usr/sbin/squid -sYC -N
> > ExecReload=/bin/kill -HUP $MAINPID
> 22,28c17,19
> < GuessMainPID=no
> < RemainAfterExit=no
> < PIDFile=/var/run/squid.pid
> < SuccessExitStatus=5 6
> < ExecStart=/etc/init.d/squid start
> < ExecStop=/etc/init.d/squid stop
> < ExecReload=/etc/init.d/squid reload
> ---
> >
> > [Install]
> > WantedBy=multi-user.target
>
> So do I just overwrite the  squid.service of the system with the one
> shipped with squid?
>
> Thanx,
> Alex
>
> On Thu, May 10, 2018 at 5:09 AM, Amos Jeffries 
> wrote:
>
>> On 10/05/18 11:53, Roberto Carna wrote:
>> > Dear, I have Squid/Dansguardian in a Debian 9 server.
>> >
>> > My Squid packages is from Debian repo, it is the stable version:
>> >
>> > squid  3.5.23-5+deb9u1
>> ...
>> >
>> > But when I read I notice two curious lines:
>> >
>> > systemd[1]: squid.service: PID file /var/run/squid.pid not readable
>> > (yet?) after start: No such file or directory
>> > systemd[1]: squid.service: Supervising process 895 which is not our
>> > child. We'll most likely not notice when it exits.
>> >
>> >
>> > Is it normal or do I have to solve these? I repeat Squid is running
>> OK...
>>
>> systemd cannot cope with daemons like Squid-3. All you can do for now is
>> ensure that you use the init.d scripts to manage Squid. Do not use the
>> "service ..." commands provided by systemd.
>>
>> Squid-4 packages that resolve these issues are in Debian experimental
>> awaiting an official upstream stable release.
>>  NP: the major bugs preventing upstream stable are not affecting the
>> Debian package features. You can use the Squid-4 package now if you wish
>> by adding that "experimental" repository to your apt sources.list,
>> update apt, then install/upgrade Squid with "apt-get -t experimental
>> install squid".
>>
>> Amos
>> ___
>> squid-users mailing list
>> squid-users@lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
>>
>
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] PID file /var/run/squid.pid not readable AND Supervising process XXX which is not our child

[Alex K]

Getting back to this, I face also issues that seems to be related with how
systemd handles squid.
Frequently when I try restart the VM the VM is stuch at stopping squid and
it never restarts.

Checking the differences between the autogenerated service file and the one
shipped with squid I see:

diff /run/systemd/generator.late/squid.service
squid3-3.5.23/tools/systemd/squid.service
1c1,6
< # Automatically generated by systemd-sysv-generator
---
> ## Copyright (C) 1996-2016 The Squid Software Foundation and contributors
> ##
> ## Squid software is distributed under GPLv2+ license and includes
> ## contributions from numerous individuals and organizations.
> ## Please see the COPYING and CONTRIBUTORS files for details.
> ##
4,14c9,10
< Documentation=man:systemd-sysv-generator(8)
< SourcePath=/etc/init.d/squid
< Description=LSB: Squid HTTP Proxy version 3.x
< Before=multi-user.target
< Before=multi-user.target
< Before=multi-user.target
< Before=graphical.target
< After=network-online.target
< After=remote-fs.target
< After=nss-lookup.target
< Wants=network-online.target
---
> Description=Squid Web Proxy Server
> After=network.target
17,20c13,15
< Type=forking
< Restart=no
< TimeoutSec=5min
< IgnoreSIGPIPE=no
---
> Type=simple
> ExecStart=/usr/sbin/squid -sYC -N
> ExecReload=/bin/kill -HUP $MAINPID
22,28c17,19
< GuessMainPID=no
< RemainAfterExit=no
< PIDFile=/var/run/squid.pid
< SuccessExitStatus=5 6
< ExecStart=/etc/init.d/squid start
< ExecStop=/etc/init.d/squid stop
< ExecReload=/etc/init.d/squid reload
---
>
> [Install]
> WantedBy=multi-user.target

So do I just overwrite the  squid.service of the system with the one
shipped with squid?

Thanx,
Alex

On Thu, May 10, 2018 at 5:09 AM, Amos Jeffries  wrote:

> On 10/05/18 11:53, Roberto Carna wrote:
> > Dear, I have Squid/Dansguardian in a Debian 9 server.
> >
> > My Squid packages is from Debian repo, it is the stable version:
> >
> > squid  3.5.23-5+deb9u1
> ...
> >
> > But when I read I notice two curious lines:
> >
> > systemd[1]: squid.service: PID file /var/run/squid.pid not readable
> > (yet?) after start: No such file or directory
> > systemd[1]: squid.service: Supervising process 895 which is not our
> > child. We'll most likely not notice when it exits.
> >
> >
> > Is it normal or do I have to solve these? I repeat Squid is running OK...
>
> systemd cannot cope with daemons like Squid-3. All you can do for now is
> ensure that you use the init.d scripts to manage Squid. Do not use the
> "service ..." commands provided by systemd.
>
> Squid-4 packages that resolve these issues are in Debian experimental
> awaiting an official upstream stable release.
>  NP: the major bugs preventing upstream stable are not affecting the
> Debian package features. You can use the Squid-4 package now if you wish
> by adding that "experimental" repository to your apt sources.list,
> update apt, then install/upgrade Squid with "apt-get -t experimental
> install squid".
>
> Amos
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid configuration sanity check

[Alex K]

Ok, clear.
Thank you Amos.

Alex

On Wed, May 16, 2018 at 3:33 PM, Amos Jeffries <squ...@treenet.co.nz> wrote:

> On 16/05/18 18:17, Alex K wrote:
> > Hi again,
> >
> > With this config I get:
> >
> > ERROR: No forward-proxy ports configured.
> >
> > I am wondering if I could just add a dummy entry:
> >
> > http_port 3130
> >
> > to suppress this error.
> >
> > But not sure how this is useful when reading:
> >
> > https://wiki.squid-cache.org/KnowledgeBase/NoForwardProxyPorts
> >
>
> As the wiki page says Squid generates URLs sometimes which require the
> client to contact the proxy directly for something(s). That cannot be
> done through a port used for TPROXY or NAT interception traffic.
>
> The port 3130 (if you choose that over the well-known 3128 port) should
> not be a "dummy" that does nothing. Squid *will* open and listen for
> traffic there. Clients will at times be told to fetch URLs from the
> Squid machines public hostname at that port.
>
> You can firewall the port off from all access if you really want to.
> Just be aware that will add error messages about the proxy port not
> being accessible to whatever problem the client is having that required
> direct contact with Squid in the first place (usually trying to display
> an error page).
>
> Amos
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid configuration sanity check

[Alex K]

Hi again,

With this config I get:

ERROR: No forward-proxy ports configured.

I am wondering if I could just add a dummy entry:

http_port 3130

to suppress this error.

But not sure how this is useful when reading:

https://wiki.squid-cache.org/KnowledgeBase/NoForwardProxyPorts

Alex

On Tue, May 8, 2018 at 7:49 PM, Amos Jeffries <squ...@treenet.co.nz> wrote:

> On 08/05/18 22:36, Alex K wrote:
> > Correction:
> >
> > On Tue, May 8, 2018 at 1:35 PM, Alex K wrote:
> >
> > Hi Amos,
> >
> > On Tue, May 8, 2018 at 8:55 AM, Amos Jeffries wrote:
> >
> > On 08/05/18 04:56, Alex K wrote:
> > > Hi Amos,
> > >
> > > On Mon, May 7, 2018 at 7:30 PM, Amos Jeffries wrote:
> > >
> > > On 08/05/18 00:24, Alex K wrote:
> > > > Hi all,
> > > >
> > ...
> > > > acl localhost src 192.168.200.1/32
> > >
> > > 192.168.200.1 is assigned to your lo interface?
> > >
> > > Yes, this is the IP of one of the interfaces of the device at
> the
> > > network where the users use squid to reach Internet.
> > >
> >
> > No, I mean specifically the interface named "lo" which has ::1
> and
> > 127.0.0.0/8 assigned by the system. It has
> > some special security
> > properties like hardware restriction preventing globally
> > routable IPs
> > being used as dst-IP of packets even routed through it result in
> > rejections.
> >
> > I have not assigned 192.168.200.1 at lo. It is assigned to an
> > interface (eth3 for example). localhost is here misleading. it could
> > say "proxy"
>
> Yes, it should be different. "localhost" ACL is used for some defaults.
> What you are doing here is adding 192.168.200.1 to the ::! etc
> definition of the predefined localhost ACL.
>
>
> >
> > >
> > > >
> > > > acl SSL_ports port 443
> > > > acl Safe_ports port 80
> > > > acl Safe_ports port 21
> > > > acl Safe_ports port 443
> > > > acl Safe_ports port 10080
> > > > acl Safe_ports port 10443
> > > > acl SSL method CONNECT
> > >
> > > The above can be quite deceptive,
> > >
> > > I removed port 21 as I don't think I am using FTP.
> > >
> >
> > Sorry, I missed out the last half of that text. I was meaning
> > the "SSL"
> > ACL definition specifically. CONNECT method is not restricted to
> SSL
> > protocol even when all you are doing is intercepting port 443
> (think
> > HTTP/2, WebSockets, QUIC, etc). It would be better to use the
> > provided
> > CONNECT ACL in place of "SSL" - they are identical in definition
> and
> > CONNECT is clearer to see if/when some access control is not as
> > tightly
> > restricted as "SSL" would make it seem.
> >
> > You mean remove  "acl SSL method CONNECT" and leave only "acl
> > CONNECT method CONNECT" ?
> >
>
> Yes. Exactly so.
>
> Amos
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Collecting squid logs to DB

[Alex K]

Thanx Eliezer and Amos for the feedback. I just saw the logformat directive
and will experiment with that.
Yes, I have a small group of users (up to 30 - 40 devices) but the hardware
is a relatively small appliance (4G RAM, 4 cores 2GHz, SSD).

Alex


On Sun, May 13, 2018, 11:37 Eliezer Croitoru <elie...@ngtech.co.il> wrote:

> To lose the stress on the DB you can use a custom format as Amos suggested
> but..
>
> I think that when you will define and write what you want to log exactly
> you will get what you need and want.
>
>
>
> The general squid access log is pretty lose and I believe that with these
> days hardware the difference will only be seen on systems with thousands or
> millions of clients requests.
>
> If this is a small place it’s not required.
>
>
>
> All The Bests,
>
> Eliezer
>
>
>
> 
>
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: elie...@ngtech.co.il
>
>
>
> *From:* Alex K <rightkickt...@gmail.com>
> *Sent:* Sunday, May 13, 2018 01:56
> *To:* Eliezer Croitoru <elie...@ngtech.co.il>
> *Cc:* squid-users@lists.squid-cache.org
> *Subject:* Re: [squid-users] Collecting squid logs to DB
>
>
>
> +++ Including list +++
>
> Hi Eliezer,
>
> I have used the following lines to instruct squid to log at mariadb:
>
> logfile_daemon /usr/lib/squid/log_db_daemon
> access_log daemon:/127.0.0.1/squid_log/access_log/squid/squid squid
>
> Through testing it seems that sometimes squid is not logging anything. I
> don't know why. After a restart it seems to unblock and write to DB.
>
> The access_log table is currently InnoDB and I am wondering if MyISAM will
> behave better.
>
>
>
> I would prefer if I could have real time access log. My scenario is that
> when a user disconnects from squid, an aggregated report of the sites that
> the user browsed will be available under some web portal where the user has
> access. Usually there will be up to 20 users connected concurrently so I
> have to check if this approach is scalable. If this approach is not stable
> then I might go with log parsing (perhaps logstash or some custom parser)
> which will parse and generate an aggregated report once per hour or day.
>
> Is there a way I format the log and pipe to DB only some interesting
> fields in order to lessen the stress to DB?
>
>
>
>
>
> On Sun, May 13, 2018 at 1:25 AM, Eliezer Croitoru <elie...@ngtech.co.il>
> wrote:
>
> Hey Alex,
>
>
>
> How did you used to log into the DB? What configuration lines have you
> used?
>
> Also what log format have you used?
>
> Is it important to have realtime data in the DB or a periodic parsing is
> also an option?
>
>
>
> Eliezer
>
>
>
> 
>
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: elie...@ngtech.co.il
>
>
>
> *From:* squid-users <squid-users-boun...@lists.squid-cache.org> *On
> Behalf Of *Alex K
> *Sent:* Saturday, May 5, 2018 01:20
> *To:* squid-users@lists.squid-cache.org
> *Subject:* [squid-users] Collecting squid logs to DB
>
>
>
> Hi all,
>
> I had a previous setup on Debian 7 with squid and I was using mysar to
> collect squid logs and store them to DB and provide some browsing report at
> the end of the day.
>
> Now at Debian 9, trying to upgrade the whole setup, I see that mysar does
> not compile.
>
> Checking around I found mysar-ng but this has compilation issues on Debian
> 9 also.
>
> Do you suggest any tool that does this job? Does squid support logging to
> DB natively? (I am using mysql/mariadb)
>
> Some other tool I stumbled on is https://github.com/paranormal/blooper.
>
>
>
> Thanx a bunch,
>
> Alex
>
>
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Collecting squid logs to DB

[Alex K]

+++ Including list +++

Hi Eliezer,

I have used the following lines to instruct squid to log at mariadb:

logfile_daemon /usr/lib/squid/log_db_daemon
access_log daemon:/127.0.0.1/squid_log/access_log/squid/squid squid

Through testing it seems that sometimes squid is not logging anything. I
don't know why. After a restart it seems to unblock and write to DB.
The access_log table is currently InnoDB and I am wondering if MyISAM will
behave better.

I would prefer if I could have real time access log. My scenario is that
when a user disconnects from squid, an aggregated report of the sites that
the user browsed will be available under some web portal where the user has
access. Usually there will be up to 20 users connected concurrently so I
have to check if this approach is scalable. If this approach is not stable
then I might go with log parsing (perhaps logstash or some custom parser)
which will parse and generate an aggregated report once per hour or day.

Is there a way I format the log and pipe to DB only some interesting fields
in order to lessen the stress to DB?


On Sun, May 13, 2018 at 1:25 AM, Eliezer Croitoru <elie...@ngtech.co.il>
wrote:

> Hey Alex,
>
>
>
> How did you used to log into the DB? What configuration lines have you
> used?
>
> Also what log format have you used?
>
> Is it important to have realtime data in the DB or a periodic parsing is
> also an option?
>
>
>
> Eliezer
>
>
>
> 
>
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: elie...@ngtech.co.il
>
>
>
> *From:* squid-users <squid-users-boun...@lists.squid-cache.org> *On
> Behalf Of *Alex K
> *Sent:* Saturday, May 5, 2018 01:20
> *To:* squid-users@lists.squid-cache.org
> *Subject:* [squid-users] Collecting squid logs to DB
>
>
>
> Hi all,
>
> I had a previous setup on Debian 7 with squid and I was using mysar to
> collect squid logs and store them to DB and provide some browsing report at
> the end of the day.
>
> Now at Debian 9, trying to upgrade the whole setup, I see that mysar does
> not compile.
>
> Checking around I found mysar-ng but this has compilation issues on Debian
> 9 also.
>
> Do you suggest any tool that does this job? Does squid support logging to
> DB natively? (I am using mysql/mariadb)
>
> Some other tool I stumbled on is https://github.com/paranormal/blooper.
>
>
>
> Thanx a bunch,
>
> Alex
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Help with WCCP: Cisco 1841 to Squid 3.5.25 on Ubuntu 16

[Alex K]

Is the ubuntu able to reach Internet?
Do you see any events at squid access log?

Alex


On Wed, May 9, 2018, 07:59 Ilias Clifton <adili...@gmx.com> wrote:

>
>  Hi Alex,
>
> On the wccp0 interface I only see traffic arriving in 1 direction -
> original client ip to destination ip.
>
> The ubuntu box only has a single ethernet interface -  Sorry, that should
> have been in my original question. I see the gre traffic arriving from the
> router, but again - no response.
>
> I tried adding a MASQUERADE line to the iptables rules, just to see if it
> made a difference.. but same result.
>
>
>
>
> Sent: Wednesday, May 09, 2018 at 2:37 PM
> From: "Alex K" <rightkickt...@gmail.com>
> To: "Ilias Clifton" <adili...@gmx.com>
> Cc: squid-users@lists.squid-cache.org
> Subject: Re: [squid-users] Help with WCCP: Cisco 1841 to Squid 3.5.25 on
> Ubuntu 16
>
> Hi,
>
> At the wccp0  interface do you see bidirectional http traffic? If the
> squid box has multiple interfaces, do you see traffic on its wan interface?
> That traffic might need NATing. Also I would check if squidbox drops any
> packages in case you have firewall configured on it.
>
> Alex
>
>
> On Wed, May 9, 2018, 07:22 Ilias Clifton <adili...@gmx.com[mailto:
> adili...@gmx.com]> wrote:
> Hello,
>
> I've been trying to get WCCP working but have been banging my head against
> a wall, so thought I would ask for help.
>
> There are 2 internal subnets that I would like to use the squid proxy:
> 172.28.30.128/25[http://172.28.30.128/25]
> <http://172.28.30.128/25%5Bhttp://172.28.30.128/25%5D> and
> 172.28.29.0/25[http://172.28.29.0/25]
> <http://172.28.29.0/25%5Bhttp://172.28.29.0/25%5D>
>
> I have squid v3.5.25 running on Ubuntu 16 : 172.28.28.252
>
> I have a Cisco 1841 - Adv IP - 12.4, see relevent config:
>
> #Inside Interface
> interface FastEthernet0/1
>  ip address 172.28.28.1 255.255.255.240
>  ip wccp web-cache redirect in
>  ip nat inside
>  ip virtual-reassembly max-reassemblies 64
>  no ip mroute-cache
>  duplex auto
>  speed auto
>
> #Loopback for wccp router ID
> interface Loopback0
>  ip address 172.28.28.33 255.255.255.255
>
> ip wccp web-cache redirect-list PROXY_USERS group-list SQUID
>
> ip access-list extended PROXY_USERS
>  deny   tcp host 172.28.28.252 any
>  permit tcp 172.28.30.128 0.0.0.127 any eq www
>  permit tcp 172.28.29.0 0.0.0.127 any eq www
>  deny   ip any any
>
> ip access-list standard SQUID
>  permit 172.28.28.252
>
>
>
> On the Ubuntu box, I have the squid with the following config:
>
> http_port 3128
> http_port 3129 intercept
> acl localnet src 172.28.28.0/22[http://172.28.28.0/22]
> <http://172.28.28.0/22%5Bhttp://172.28.28.0/22%5D>
> http_access allow localnet
> http_access allow localhost
> http_access deny all
> visible_hostname Squid
> wccp2_router 172.28.28.1
> wccp2_forwarding_method gre
> wccp2_return_method gre
> wccp2_service standard 0
>
> If clients are manually set to use the proxy on port 3128, they work
> correctly.
>
> Again on the Ubuntu box, I have setup the following gre tunnel.
>
> ip tunnel add wccp0 mode gre remote 172.28.28.33 local 172.28.28.252 dev
> ens33 ttl 255
>
> and the following redirect using iptables..
>
> iptables -t nat -A PREROUTING -i wccp0 -p tcp -m tcp --dport 80 -j
> REDIRECT --to-ports 3129
>
> In sysctl.conf, I have disabled reverse path filtering and enabled ip
> forarding.
>
> net.ipv4.conf.default.rp_filter=0
> net.ipv4.conf.all.rp_filter=0
> net.ipv4.ip_forward=1
>
> When starting squid, using tcpdump, i see traffic between the Ubuntu box
> and the router on udp port 2048
>
> 00:39:34.587799 IP 172.28.28.252.2048 > 172.28.28.1.2048: UDP, length 144
> 00:39:34.590399 IP 172.28.28.1.2048 > 172.28.28.252.2048: UDP, length 140
>
> I see the following message on the router..
> %WCCP-5-SERVICEFOUND: Service web-cache acquired on WCCP client
> 172.28.28.252
>
> So looks like it's working ok so far...
>
> When I try and browse to a site from a client..
> $ wget http://www.google.com[http://www.google.com]
>
> On the Ubuntu box, I see gre traffic on the ethernet interface..
> 00:44:22.340734 IP 172.28.28.33 > 172.28.28.252[http://172.28.28.252]:
> GREv0, length 72: gre-proto-0x883e
>
>
> I see the un-encapsulated traffic on the wccp0 interface:
> 00:56:26.888519 IP 172.28.29.4.52128 > 216.58.203.100.80
>
> Which is correctly showing original client IP and destination IP.
>
> I can see hits on the iptable redirect rule:
> pkts bytes target prot opt in out sourc

Re: [squid-users] Help with WCCP: Cisco 1841 to Squid 3.5.25 on Ubuntu 16

[Alex K]

Hi,

At the wccp0  interface do you see bidirectional http traffic? If the squid
box has multiple interfaces, do you see traffic on its wan interface? That
traffic might need NATing. Also I would check if squidbox drops any
packages in case you have firewall configured on it.

Alex



On Wed, May 9, 2018, 07:22 Ilias Clifton  wrote:

>
> Hello,
>
> I've been trying to get WCCP working but have been banging my head against
> a wall, so thought I would ask for help.
>
> There are 2 internal subnets that I would like to use the squid proxy:
> 172.28.30.128/25 and 172.28.29.0/25
>
> I have squid v3.5.25 running on Ubuntu 16 : 172.28.28.252
>
> I have a Cisco 1841 - Adv IP - 12.4, see relevent config:
>
> #Inside Interface
> interface FastEthernet0/1
>  ip address 172.28.28.1 255.255.255.240
>  ip wccp web-cache redirect in
>  ip nat inside
>  ip virtual-reassembly max-reassemblies 64
>  no ip mroute-cache
>  duplex auto
>  speed auto
>
> #Loopback for wccp router ID
> interface Loopback0
>  ip address 172.28.28.33 255.255.255.255
>
> ip wccp web-cache redirect-list PROXY_USERS group-list SQUID
>
> ip access-list extended PROXY_USERS
>  deny   tcp host 172.28.28.252 any
>  permit tcp 172.28.30.128 0.0.0.127 any eq www
>  permit tcp 172.28.29.0 0.0.0.127 any eq www
>  deny   ip any any
>
> ip access-list standard SQUID
>  permit 172.28.28.252
>
>
>
> On the Ubuntu box, I have the squid with the following config:
>
> http_port 3128
> http_port 3129 intercept
> acl localnet src 172.28.28.0/22
> http_access allow localnet
> http_access allow localhost
> http_access deny all
> visible_hostname Squid
> wccp2_router 172.28.28.1
> wccp2_forwarding_method gre
> wccp2_return_method gre
> wccp2_service standard 0
>
> If clients are manually set to use the proxy on port 3128, they work
> correctly.
>
> Again on the Ubuntu box, I have setup the following gre tunnel.
>
> ip tunnel add wccp0 mode gre remote 172.28.28.33 local 172.28.28.252 dev
> ens33 ttl 255
>
> and the following redirect using iptables..
>
> iptables -t nat -A PREROUTING -i wccp0 -p tcp -m tcp --dport 80 -j
> REDIRECT --to-ports 3129
>
> In sysctl.conf, I have disabled reverse path filtering and enabled ip
> forarding.
>
> net.ipv4.conf.default.rp_filter=0
> net.ipv4.conf.all.rp_filter=0
> net.ipv4.ip_forward=1
>
> When starting squid, using tcpdump, i see traffic between the Ubuntu box
> and the router on udp port 2048
>
> 00:39:34.587799 IP 172.28.28.252.2048 > 172.28.28.1.2048: UDP, length 144
> 00:39:34.590399 IP 172.28.28.1.2048 > 172.28.28.252.2048: UDP, length 140
>
> I see the following message on the router..
> %WCCP-5-SERVICEFOUND: Service web-cache acquired on WCCP client
> 172.28.28.252
>
> So looks like it's working ok so far...
>
> When I try and browse to a site from a client..
> $ wget http://www.google.com
>
> On the Ubuntu box, I see gre traffic on the ethernet interface..
> 00:44:22.340734 IP 172.28.28.33 > 172.28.28.252: GREv0, length 72:
> gre-proto-0x883e
>
>
> I see the un-encapsulated traffic on the wccp0 interface:
> 00:56:26.888519 IP 172.28.29.4.52128 > 216.58.203.100.80
>
> Which is correctly showing original client IP and destination IP.
>
> I can see hits on the iptable redirect rule:
> pkts bytes target prot opt in out source
>  destination
>   429 26280 REDIRECT   tcp  --  wccp0  any anywhere
>  anywhere tcp dpt:http redir ports 3129
>
>
> But there is no response from squid on the Ubuntu box :-(
>
> I don't see anything helpful in either access.log or cache.log.
>
> I'm not sure if there is anything else that could be dropping the packet
> apart from return path filtering..
>
> If someone could give me some pointers or any further debugging I could
> try, that would be great.
>
>
> Thanks.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid configuration sanity check

[Alex K]

Correction:

On Tue, May 8, 2018 at 1:35 PM, Alex K <rightkickt...@gmail.com> wrote:

> Hi Amos,
>
> On Tue, May 8, 2018 at 8:55 AM, Amos Jeffries <squ...@treenet.co.nz>
> wrote:
>
>> On 08/05/18 04:56, Alex K wrote:
>> > Hi Amos,
>> >
>> > On Mon, May 7, 2018 at 7:30 PM, Amos Jeffries wrote:
>> >
>> > On 08/05/18 00:24, Alex K wrote:
>> > > Hi all,
>> > >
>> ...
>> > > acl localhost src 192.168.200.1/32 <http://192.168.200.1/32>
>> >
>> > 192.168.200.1 is assigned to your lo interface?
>> >
>> > Yes, this is the IP of one of the interfaces of the device at the
>> > network where the users use squid to reach Internet.
>> >
>>
>> No, I mean specifically the interface named "lo" which has ::1 and
>> 127.0.0.0/8 assigned by the system. It has some special security
>> properties like hardware restriction preventing globally routable IPs
>> being used as dst-IP of packets even routed through it result in
>> rejections.
>>
> I have not assigned 192.168.200.1 at lo. It is assigned to an interface
> (eth3 for example). localhost is here misleading. it could say "proxy"
>
>
>>
>>
>> >
>> > >
>> > > acl SSL_ports port 443
>> > > acl Safe_ports port 80
>> > > acl Safe_ports port 21
>> > > acl Safe_ports port 443
>> > > acl Safe_ports port 10080
>> > > acl Safe_ports port 10443
>> > > acl SSL method CONNECT
>> >
>> > The above can be quite deceptive,
>> >
>> > I removed port 21 as I don't think I am using FTP.
>> >
>>
>> Sorry, I missed out the last half of that text. I was meaning the "SSL"
>> ACL definition specifically. CONNECT method is not restricted to SSL
>> protocol even when all you are doing is intercepting port 443 (think
>> HTTP/2, WebSockets, QUIC, etc). It would be better to use the provided
>> CONNECT ACL in place of "SSL" - they are identical in definition and
>> CONNECT is clearer to see if/when some access control is not as tightly
>> restricted as "SSL" would make it seem.
>
> You mean remove  "acl SSL method CONNECT" and leave only "acl CONNECT
> method CONNECT" ?
>
>>
>>
>> Cheers
>> Amos
>>
>
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid configuration sanity check

[Alex K]

Hi Amos,

On Tue, May 8, 2018 at 8:55 AM, Amos Jeffries <squ...@treenet.co.nz> wrote:

> On 08/05/18 04:56, Alex K wrote:
> > Hi Amos,
> >
> > On Mon, May 7, 2018 at 7:30 PM, Amos Jeffries wrote:
> >
> > On 08/05/18 00:24, Alex K wrote:
> > > Hi all,
> > >
> ...
> > > acl localhost src 192.168.200.1/32 <http://192.168.200.1/32>
> >
> > 192.168.200.1 is assigned to your lo interface?
> >
> > Yes, this is the IP of one of the interfaces of the device at the
> > network where the users use squid to reach Internet.
> >
>
> No, I mean specifically the interface named "lo" which has ::1 and
> 127.0.0.0/8 assigned by the system. It has some special security
> properties like hardware restriction preventing globally routable IPs
> being used as dst-IP of packets even routed through it result in
> rejections.
>
I have not assigned 192.168.200.1 at lo. It is assigned to an interface
(eth3 for example). localhost is here misleading. it could say "proxy"


>
>
> >
> > >
> > > acl SSL_ports port 443
> > > acl Safe_ports port 80
> > > acl Safe_ports port 21
> > > acl Safe_ports port 443
> > > acl Safe_ports port 10080
> > > acl Safe_ports port 10443
> > > acl SSL method CONNECT
> >
> > The above can be quite deceptive,
> >
> > I removed port 21 as I don't think I am using FTP.
> >
>
> Sorry, I missed out the last half of that text. I was meaning the "SSL"
> ACL definition specifically. CONNECT method is not restricted to SSL
> protocol even when all you are doing is intercepting port 443 (think
> HTTP/2, WebSockets, QUIC, etc). It would be better to use the provided
> CONNECT ACL in place of "SSL" - they are identical in definition and
> CONNECT is clearer to see if/when some access control is not as tightly
> restricted as "SSL" would make it seem.

You mean remove  "acl CONNECT method CONNECT" and leave only "acl CONNECT
method CONNECT" ?

>
>
> Cheers
> Amos
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid configuration sanity check

[Alex K]

Hi Amos,

On Mon, May 7, 2018 at 7:30 PM, Amos Jeffries <squ...@treenet.co.nz> wrote:

> On 08/05/18 00:24, Alex K wrote:
> > Hi all,
> >
> > I wanted to check with your accumulated wisdom the following squid
> > configuration.
> >
> > The config is working both for splice or bump (by
> > commenting/uncommenting the respective line) using TPROXY. It is a
> > config ported form an old installation of squid 3.1 for the new 3.5 and
> > although I did some cleanup I am wondering if I am misusing any
> > directive or missing any crucial one for better performance or just for
> > sake of cleanliness.
> >
> > At the moment for filtering I am using squidGuard and considering to go
> > with ufdbGuard instead as pointed from Amos (thanx for that).
> >
> > To avoid issues with some sites I am considering to use only splicing,
> > although this has some caveats as bumping also does. I could go with a
> > hybrid approach (splice some and bump all) but this sounds that this
> > will cause periodically more administrative overhead to sort out the
> > sites that need splicing.
> >
> > The config has also some ACLs as an attempt to block media streaming by
> > those seem to not work.
>
> The ACL checking for mms:// URL will not work because MMS protocol is
> not HTTP. Any client using that protocol will not be going through
> Squid. So quite likely none of the other checks will work for its
> non-proxied traffic either.
>
> "working" can also depend on what you are looking at. Your rules are
> only blocking *reply* access. Which means only that the client does not
> get the response delivered. It still gets fetched from the server -
> maybe in full. So checking your logs etc can still show things arriving
> and lots of bandwidth usage.
>
> The urlpath and req_mime_type can be checked in http_access instead to
> block those requests from ever happening. That MAY work better, but no
> guarantees.
>
>
>
> >
> > The hardware running the squid is somehow small with 4 GB of RAM, 4 CPU
> > cores and 100 GB SSD in case one wonders.
> >
> >
> > http_port 192.168.200.1:3128 tproxy
> > https_port 192.168.200.1:3129 tproxy \
> >   ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
> \
> >   cert=/etc/squid/ssl_cert/myCA.pem
> >
> > sslcrtd_program /usr/lib/squid/ssl_crtd -s
> > /usr/local/squid/var/lib/ssl_db -M 4MB
> > sslcrtd_children 5
> >
> > shutdown_lifetime 5 seconds
> >
> > # ACL
> > #acl ncsa_users proxy_auth REQUIRED
> > #acl all src 0.0.0.0/0.0.0.0
> > acl manager proto cache_object
>
> 'manager' ACL is now built-in, and has a different type signature. The
> above needs to be removed. Same with 'all'. It is not a good idea to
> leave them even commented out because the old definitions are no longer
> true.
>
> ok, removed these entries (ncsa_users, all, manager)
>


> > acl localhost src 192.168.200.1/32
>
> 192.168.200.1 is assigned to your lo interface?
>
Yes, this is the IP of one of the interfaces of the device at the network
where the users use squid to reach Internet.

>
> >
> > acl SSL_ports port 443
> > acl Safe_ports port 80
> > acl Safe_ports port 21
> > acl Safe_ports port 443
> > acl Safe_ports port 10080
> > acl Safe_ports port 10443
> > acl SSL method CONNECT
>
> The above can be quite deceptive,
>
I removed port 21 as I don't think I am using FTP.


>
> > acl CONNECT method CONNECT # multiling http
> > #acl block_url dstdomain "/etc/squid/block_url.squid"
> > #acl allow_url dstdomain "/etc/squid/allow_url.squid"
> > acl ELAN src 192.168.200.0/24
> >
> > acl QUERY urlpath_regex cgi-bin \?
>
> The QUERY is not being used. It is also no longer necessary so can be
> removed.
>
Removed.

>
> >
> > # SSL
> > always_direct allow all
>
> That should not be. You do not have any cache_peer configured.
>
> Removed


> >
> > # Video Streaming ACLs
> > acl media rep_mime_type ^.*mms.*
> > acl media rep_mime_type ^.*ms-hdr.*
> > acl media rep_mime_type ^.*x-fcs.*
> > acl media rep_mime_type ^.*x-ms-asf.*
> > acl media2 urlpath_regex dvrplayer mediastream mms://
> > acl media2 urlpath_regex \.asf$ \.afx$ \.flv$ \.swf$
> > acl flashvideo rep_mime_type -i video/flv
> > acl flashvideo rep_mime_type -i video/x-flv
> > acl shockwave rep_mime_type -i ^application/x-shockwave-flash$
>
> > acl x-type req_mime_type -i ^application/octet-stream$
> > acl x-type req_mime_type -i application/octet-stream
>

[squid-users] Squid configuration sanity check

[Alex K]

Hi all,

I wanted to check with your accumulated wisdom the following squid
configuration.

The config is working both for splice or bump (by commenting/uncommenting
the respective line) using TPROXY. It is a config ported form an old
installation of squid 3.1 for the new 3.5 and although I did some cleanup I
am wondering if I am misusing any directive or missing any crucial one for
better performance or just for sake of cleanliness.

At the moment for filtering I am using squidGuard and considering to go
with ufdbGuard instead as pointed from Amos (thanx for that).

To avoid issues with some sites I am considering to use only splicing,
although this has some caveats as bumping also does. I could go with a
hybrid approach (splice some and bump all) but this sounds that this will
cause periodically more administrative overhead to sort out the sites that
need splicing.

The config has also some ACLs as an attempt to block media streaming by
those seem to not work.

The hardware running the squid is somehow small with 4 GB of RAM, 4 CPU
cores and 100 GB SSD in case one wonders.


http_port 192.168.200.1:3128 tproxy
https_port 192.168.200.1:3129 tproxy ssl-bump generate-host-certificates=on
dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/myCA.pem

sslcrtd_program /usr/lib/squid/ssl_crtd -s /usr/local/squid/var/lib/ssl_db
-M 4MB
sslcrtd_children 5

shutdown_lifetime 5 seconds

# ACL
#acl ncsa_users proxy_auth REQUIRED
#acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 192.168.200.1/32

acl SSL_ports port 443
acl Safe_ports port 80
acl Safe_ports port 21
acl Safe_ports port 443
acl Safe_ports port 10080
acl Safe_ports port 10443
acl SSL method CONNECT
acl CONNECT method CONNECT # multiling http
#acl block_url dstdomain "/etc/squid/block_url.squid"
#acl allow_url dstdomain "/etc/squid/allow_url.squid"
acl ELAN src 192.168.200.0/24

acl QUERY urlpath_regex cgi-bin \?

# SSL
always_direct allow all

# Video Streaming ACLs
acl media rep_mime_type ^.*mms.*
acl media rep_mime_type ^.*ms-hdr.*
acl media rep_mime_type ^.*x-fcs.*
acl media rep_mime_type ^.*x-ms-asf.*
acl media2 urlpath_regex dvrplayer mediastream mms://
acl media2 urlpath_regex \.asf$ \.afx$ \.flv$ \.swf$
acl flashvideo rep_mime_type -i video/flv
acl flashvideo rep_mime_type -i video/x-flv
acl shockwave rep_mime_type -i ^application/x-shockwave-flash$
acl x-type req_mime_type -i ^application/octet-stream$
acl x-type req_mime_type -i application/octet-stream
acl x-type req_mime_type -i ^application/x-mplayer2$
acl x-type req_mime_type -i application/x-mplayer2
acl x-type req_mime_type -i ^application/x-oleobject$
acl x-type req_mime_type -i application/x-oleobject
acl x-type req_mime_type -i application/x-pncmd
acl x-type req_mime_type -i ^video/x-ms-asf$
acl x-type2 rep_mime_type -i ^application/octet-stream$
acl x-type2 rep_mime_type -i application/octet-stream
acl x-type2 rep_mime_type -i ^application/x-mplayer2$
acl x-type2 rep_mime_type -i application/x-mplayer2
acl x-type2 rep_mime_type -i ^application/x-oleobject$
acl x-type2 rep_mime_type -i application/x-oleobject
acl x-type2 rep_mime_type -i application/x-pncmd
acl x-type2 rep_mime_type -i ^video/x-ms-asf$

# Block Media Streaming
http_reply_access deny flashvideo
http_reply_access deny shockwave
http_reply_access deny media
http_reply_access deny media2
http_reply_access deny x-type
http_reply_access deny x-type2

#
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
#http_access deny block_url
#http_access allow allow_url
http_access allow LAN
http_access allow ELAN

http_access allow localhost
#http_access allow ncsa_users
http_reply_access allow all

deny_info ERR_CUSTOM LAN ELAN media media2 flashvideo shockwave x-type
x-type2
error_directory /usr/share/squid-langpack/en

#icp_access allow all

# Logging
logfile_daemon /usr/lib/squid/log_db_daemon
access_log daemon:/127.0.0.1/squid_log/access_log/squid/squid squid
icap_log stdio:/var/log/squid/icap.log squid
cache_store_log stdio:/var/log/squid/store.log

# DNS
dns_nameservers 127.0.0.1
positive_dns_ttl 8 hours
negative_dns_ttl 30 seconds
ipcache_size 2048
ipcache_low 95
ipcache_high 97
fqdncache_size 2048

# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid
cache_dir ufs /var/spool/squid 10240 16 256
minimum_object_size 0 KB
maximum_object_size 30 MB
maximum_object_size_in_memory 1024 KB

# HTTPS filtering
acl step1 at_step SslBump1

ssl_bump peek step1
ssl_bump splice all
#ssl_bump bump all

# SquidGuard
url_rewrite_program /usr/bin/squidGuard -c /etc/squidguard/squidGuard.conf
url_rewrite_children 5


Your input is highly appreciated.

Alex
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Collecting squid logs to DB

[Alex K]

Hi all,

I had a previous setup on Debian 7 with squid and I was using mysar to
collect squid logs and store them to DB and provide some browsing report at
the end of the day.
Now at Debian 9, trying to upgrade the whole setup, I see that mysar does
not compile.

Checking around I found mysar-ng but this has compilation issues on Debian
9 also.
Do you suggest any tool that does this job? Does squid support logging to
DB natively? (I am using mysql/mariadb)

Some other tool I stumbled on is https://github.com/paranormal/blooper.

Thanx a bunch,
Alex
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid with squidguard

[Alex K]

Thank you Amos for the feedback.

I did see an example online using ACL and that tricked me.
Removing the allow line, now squid is logging that squidguard is started
(though no squidguard processes are listed, it could be due to that I have
not tested yet with actual traffic)

I will check also ufdbguard as it seems promising.

Thanx,
Alex

On Thu, Apr 26, 2018 at 4:02 AM, Amos Jeffries <squ...@treenet.co.nz> wrote:

> On 25/04/18 23:44, Alex K wrote:
> > Hi all,
> >
> > I was using a squid (3.1.20) + squidguard setup (to filter out several
> > site categories) on Debian 7 and the setup worked. The squidguard was
> > invoked from squid.conf as below:
> >
> > redirect_program /usr/bin/squidGuard -c /etc/squidguard/squidGuard.conf
> > redirect_children 7
> >
> > I am now testing the setup on Debian 9 (with squid 3.5.23) with the
> > following lines in squid.conf:
> >
> > url_rewrite_access allow
>
> There are no ACLs on the above line. So it cannot match anything. The
> implicit default applies instead. Implicit default after any "allow"
> line is "deny all".
>
> Also, you did not configure any allow/deny previously. So why add it now?
>
> > url_rewrite_program /usr/bin/squidGuard -c /etc/squidguard/squidGuard.
> conf
> > url_rewrite_children 5
> >
> > But I get at squid logs:
> >
> > 2018/04/24 12:06:57 kid1| helperOpenServers: Starting 0/5 'squidGuard'
> > processes
> > 2018/04/24 12:06:57 kid1| helperOpenServers: No 'squidGuard' processes
> > needed.
>
> No traffic is allowed to go to the helper. So no SG processes necessary.
> Squid is correct.
>
>
> >
> > Seems that squid is ignoring and not starting squidguard.
> > I have read also some have mentioned that squidguard is not maintained
> > anymore.
> >
> > Any idea on the above?> Any better alternative to squidguard that you
> recommend?
>
> ufdbguard is much better than the outdated and no longer maintained
> SquidGuard (but is not packaged on Debian).
>
> Amos
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Squid with squidguard

[Alex K]

Hi all,

I was using a squid (3.1.20) + squidguard setup (to filter out several site
categories) on Debian 7 and the setup worked. The squidguard was invoked
from squid.conf as below:

redirect_program /usr/bin/squidGuard -c /etc/squidguard/squidGuard.conf
redirect_children 7

I am now testing the setup on Debian 9 (with squid 3.5.23) with the
following lines in squid.conf:

url_rewrite_access allow
url_rewrite_program /usr/bin/squidGuard -c /etc/squidguard/squidGuard.conf
url_rewrite_children 5

But I get at squid logs:

2018/04/24 12:06:57 kid1| helperOpenServers: Starting 0/5 'squidGuard'
processes
2018/04/24 12:06:57 kid1| helperOpenServers: No 'squidGuard' processes
needed.


Seems that squid is ignoring and not starting squidguard.
I have read also some have mentioned that squidguard is not maintained
anymore.

Any idea on the above?
Any better alternative to squidguard that you recommend?

thanx,
Alex
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] tproxy first time implementation on squid.

[Alex K]

You might be missing a NAT at last node before the packet is left to
Internet otherwise you need a public IP at the windows client.

On Oct 22, 2017 19:08, "Hanoch Hanoch K"  wrote:

> Hi
> I am trying to configure tproxy to expose the ip address i am using to
> internet sites and not the ip address of the squid server.
> I did read the wiki from the squid web site and acted upon.
> the environment i am using is test and i will need to deploy it into
> producton when test will work and all the subject will be clear to me.
> so the server is ubuntu 14.04.
> squid was compiled with netfilter prefix.
> the kernel is new and seems to have built in support in the tproxy.
> iptables rules where created as the wiki request.
> route option had been applied and the sysctl was configured as requested
> by the wiki.
> the client is windows 7 vmware vm and also the server is vmware vm with 2
> ethernet adapters.
> one of  the interfaces connect the windows 7 and one having ip from the
> built in dhcp server at the router and it is the internet interface.
> the server and the client both behind router and all have private ip.
> till now the setup.
> the problem is when I try to surf with this configuration i get time out.
> at the wiki it says it is routing problem.
> but digging the logs i do not understand where is my mistake.
> can i use this configuration? lets say can i send ip like 10.0.0.2 to be
> discovered at internet?
> is this configuration is legal?
> do not i need public ip on all the interfaces?
> if not what is wrong.
> i will be happy to supply any log or conf file.
> please try to help me.
>
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Enable tproxy in Squid 3.5 running on Debian 9

[Alex K]

You will need to transpareny redirect the traffic and not explicitly
pointing your browser to squid. Seems that the mentioned firewall rules are
correct. You will need a policy route also for the marked traffic.

On Oct 5, 2017 7:54 PM, "xpro6000"  wrote:

I'm back to square one then, and it looks like there is no way to tell
Squid to use the same connecting ip for the outgoing ip, which is what I
need.

On Thu, Oct 5, 2017 at 3:49 AM, Amos Jeffries  wrote:

> On 05/10/17 15:01, xpro6000 wrote:
>
>> I'm trying to setup tproxy with Squid 3.5 for the purpose of having the
>> same outgoing ip as the connecting ip. (I have thousands of IPs and I can
>> not add them one by one)
>>
>> I started with a fresh install of Debian 9, installed Squid by
>>
>> apt install squid
>>
>> then I added
>>
>> http_port 3129 tproxy
>>
>> to squid.conf
>>
>> I then ran the following commands for iptables
>>
>> iptables -t mangle -N DIVERT
>> iptables -t mangle -A DIVERT -j MARK --set-mark 1
>> iptables -t mangle -A DIVERT -j ACCEPT
>>
>> iptables  -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
>>
>> iptables  -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY
>> --tproxy-mark 0x1/0x1 --on-port 3129
>>
>>
>> I can use the proxy with no problems on port 3128, but on Firefox I get a
>> message "The proxy server is refusing connections" when I set the proxy to
>> port 3129. Did I miss any steps or am I doing something wrong?
>>
>
> You missed the fact that TPROXY is an MITM operation. You *cannot* setup
> the browser to use the proxy directly to its tproxy port. You have to route
> the packets to the proxy machine without any explicit browser or client
> configuration.
>
> Only the Squid machine bits (and thus behaviour) are different with TPROXY
> vs NAT interception.
>
> ...
>
>> http_access deny !Safe_ports
>> http_access deny CONNECT !SSL_ports
>> http_access allow localhost manager
>> http_access deny manager
>> http_access allow localhost
>> http_access allow all
>>
>
> Do not do "allow all" like this. Setup the localnet ACL to your LAN
> range(s) properly and only allow those clients through the proxy.
>
> Then you can use the recommended default:
>  http_access deny !Safe_ports
>  http_access deny CONNECT !SSL_ports
>  http_access allow localhost
>  http_access deny manager
>  http_access allow localnet
>  http_access deny all
>
> Amos
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>


___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users