RE: [squid-users] FW: Problem with server IO resource, need to reduce logging level by excluding specific sites from being logged
Sorry Marcus, new to this forum support. You mention cache_mem is small, excuse me noobness can you explain the impact. The Memory allocation to the VM is 4GB, and it has at present 4 VCPU (doesn't look like it being stressed at all). Version = Squid Cache: Version 3.2.5 Disk structure is as follows 20GB VDMK = System 40GB VMDK = SQUID only # # /etc/fstab # Created by anaconda on Mon Apr 23 16:24:28 2012 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/vg_008-lv_root / ext4defaults1 1 UUID=c13ba480-17e3-4df3-b6d3-9a2eb9cea766 /boot ext4 defaults1 2 # UUID=08301dc8-4e84-4cd9-a402-f4e71a461098 /squid ext4 defaults1 2 /dev/mapper/vg_008-lv_swap swapswapdefaults0 0 /dev/sdb/squid ext4defaults 1 2 -Original Message- From: Marcus Kool [mailto:marcus.k...@urlfilterdb.com] Sent: 25 July 2014 00:37 To: RYAN Justin Subject: Re: [squid-users] FW: Problem with server IO resource, need to reduce logging level by excluding specific sites from being logged Juz, The systems seems to have a very small config. 32 MB for cache_mem is very small indeed Do you have room/RAM to extend the in-memory cache of Squid? From the data that you posted it is not clear if /squid shares its disk with /. What version of Squid do you have (output of squid -v) ? What file system type and mount options are used for /squid ? You did not reply to the squid list. I suggest to include the squid list in the CC: and replace the cachemgr_passwd to XXX in the post. Marcus On 07/24/2014 10:39 AM, RYAN Justin wrote: Sorry Marcus, was a little light on background. Storage on 2 partitions [root@ ]# df -k Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 2057264 0 2057264 0% /dev tmpfs 2066040 0 2066040 0% /dev/shm tmpfs 2066040504 2065536 1% /run /dev/mapper/vg_008-lv_root 160623843864120 11382344 26% / tmpfs 2066040 0 2066040 0% /sys/fs/cgroup tmpfs 2066040 0 2066040 0% /media /dev/sdb 41284928 14322924 24864852 37% /squid /dev/sda2 495844 65891404353 15% /boot Below is the config http_port 3128 dns_nameservers 8.8.8.8 icp_port 0 acl QUERY urlpath_regex cgi-bin \? no_cache deny QUERY append_domain .phoenix.loc cache_mgr i...@pms.co.uk cachemgr_passwd * all buffered_logs on coredump_dir /squid/cache cache_access_log /squid/logs/access.log cache_log /squid/logs/cache.log logfile_rotate 60 cache_dir aufs /squid/cache 4096 16 256 cache_mem 32 MB maximum_object_size 64 MB maximum_object_size_in_memory 20 KB cache_effective_user squid max_filedesc 4096 # acl all src all # acl manager proto cache_object acl localhost src 127.0.0.1 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 # https acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http #acl SSL method CONNECT acl CONNECT method CONNECT acl webserver src 192.168.100.0/24 http_access allow manager webserver http_access allow manager localhost http_access deny manager http_access deny CONNECT !SSL_ports http_access deny !Safe_ports http_access allow localhost # --- auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp auth_param ntlm children 30 startup=30 # auth_param ntlm use_ntlm_negotiate on auth_param ntlm keep_alive off auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic auth_param basic children 10 startup=10 auth_param basic realm Squid proxy-caching web server auth_param basic credentialsttl 2 hours external_acl_type ADS children-max=30 children-startup=30 %LOGIN /usr/lib/squid/ext_wbinfo_group_acl acl block_all dstdomain /squid/rules/block-all acl malware dstdomain /squid/rules/malware.jr acl download_all urlpath_regex \.mp1$ \.mp2$ \.mp3$ \.mpa$ \.mpeg$ \.asf$
Re: [squid-users] FW: Problem with server IO resource, need to reduce logging level by excluding specific sites from being logged
Juz, The mount options rw,noatime reduce I/O a little for ext4 so they are recommended for /squid. Since the system has 4 GB memory it is recommendable to increase cache_mem from 32 MB to 512 MB and to change maximum_object_size_in_memory from 20 KB to 128 KB. Both options help to cache more in-memory instead of on-disk and hence reduce disk reads. But only increase the parameters if the system has enough free memory to give to Squid. Note that 512 MB memory cache 'translates' into roughly 1.4 GB total memory requirement for Squid. And last but not least, since the disk is a virtual disk, it is recommendable to see if the virtual disk can be allocated on a device with more I/O capacity. Marcus On 07/25/2014 05:52 AM, RYAN Justin wrote: Sorry Marcus, new to this forum support. You mention cache_mem is small, excuse me noobness can you explain the impact. The Memory allocation to the VM is 4GB, and it has at present 4 VCPU (doesn't look like it being stressed at all). Version = Squid Cache: Version 3.2.5 Disk structure is as follows 20GB VDMK = System 40GB VMDK = SQUID only # # /etc/fstab # Created by anaconda on Mon Apr 23 16:24:28 2012 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/vg_008-lv_root / ext4defaults1 1 UUID=c13ba480-17e3-4df3-b6d3-9a2eb9cea766 /boot ext4 defaults1 2 # UUID=08301dc8-4e84-4cd9-a402-f4e71a461098 /squid ext4 defaults1 2 /dev/mapper/vg_008-lv_swap swapswapdefaults0 0 /dev/sdb/squid ext4defaults 1 2 -Original Message- From: Marcus Kool [mailto:marcus.k...@urlfilterdb.com] Sent: 25 July 2014 00:37 To: RYAN Justin Subject: Re: [squid-users] FW: Problem with server IO resource, need to reduce logging level by excluding specific sites from being logged Juz, The systems seems to have a very small config. 32 MB for cache_mem is very small indeed Do you have room/RAM to extend the in-memory cache of Squid? From the data that you posted it is not clear if /squid shares its disk with /. What version of Squid do you have (output of squid -v) ? What file system type and mount options are used for /squid ? You did not reply to the squid list. I suggest to include the squid list in the CC: and replace the cachemgr_passwd to XXX in the post. Marcus On 07/24/2014 10:39 AM, RYAN Justin wrote: Sorry Marcus, was a little light on background. Storage on 2 partitions [root@ ]# df -k Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 2057264 0 2057264 0% /dev tmpfs 2066040 0 2066040 0% /dev/shm tmpfs 2066040504 2065536 1% /run /dev/mapper/vg_008-lv_root 160623843864120 11382344 26% / tmpfs 2066040 0 2066040 0% /sys/fs/cgroup tmpfs 2066040 0 2066040 0% /media /dev/sdb 41284928 14322924 24864852 37% /squid /dev/sda2 495844 65891404353 15% /boot Below is the config http_port 3128 dns_nameservers 8.8.8.8 icp_port 0 acl QUERY urlpath_regex cgi-bin \? no_cache deny QUERY append_domain .phoenix.loc cache_mgr i...@pms.co.uk cachemgr_passwd * all buffered_logs on coredump_dir /squid/cache cache_access_log /squid/logs/access.log cache_log /squid/logs/cache.log logfile_rotate 60 cache_dir aufs /squid/cache 4096 16 256 cache_mem 32 MB maximum_object_size 64 MB maximum_object_size_in_memory 20 KB cache_effective_user squid max_filedesc 4096 # acl all src all # acl manager proto cache_object acl localhost src 127.0.0.1 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 # https acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http #acl SSL method CONNECT acl CONNECT method CONNECT acl webserver src 192.168.100.0/24 http_access allow manager webserver http_access allow manager localhost http_access deny manager http_access deny CONNECT !SSL_ports http_access deny !Safe_ports http_access allow localhost #
RE: [squid-users] FW: Problem with server IO resource, need to reduce logging level by excluding specific sites from being logged
Cheers Marcus, I did see via googling a rule of thumb quote cache_mem = total physical memory / 3 - ref http://forums.justlinux.com/showthread.php?126396-Squid-cache-tuning there is a more complex formula quoted too. Money and access constraints negate the move to faster storage :) I will look into your recommendations. The question of removing noise from being logged still exists - would be a nice to have option -Original Message- From: Marcus Kool [mailto:marcus.k...@urlfilterdb.com] Sent: 25 July 2014 12:11 To: RYAN Justin Cc: 'squid-users@squid-cache.org' Subject: Re: [squid-users] FW: Problem with server IO resource, need to reduce logging level by excluding specific sites from being logged Juz, The mount options rw,noatime reduce I/O a little for ext4 so they are recommended for /squid. Since the system has 4 GB memory it is recommendable to increase cache_mem from 32 MB to 512 MB and to change maximum_object_size_in_memory from 20 KB to 128 KB. Both options help to cache more in-memory instead of on-disk and hence reduce disk reads. But only increase the parameters if the system has enough free memory to give to Squid. Note that 512 MB memory cache 'translates' into roughly 1.4 GB total memory requirement for Squid. And last but not least, since the disk is a virtual disk, it is recommendable to see if the virtual disk can be allocated on a device with more I/O capacity. Marcus On 07/25/2014 05:52 AM, RYAN Justin wrote: Sorry Marcus, new to this forum support. You mention cache_mem is small, excuse me noobness can you explain the impact. The Memory allocation to the VM is 4GB, and it has at present 4 VCPU (doesn't look like it being stressed at all). Version = Squid Cache: Version 3.2.5 Disk structure is as follows 20GB VDMK = System 40GB VMDK = SQUID only # # /etc/fstab # Created by anaconda on Mon Apr 23 16:24:28 2012 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/vg_008-lv_root / ext4defaults1 1 UUID=c13ba480-17e3-4df3-b6d3-9a2eb9cea766 /boot ext4 defaults1 2 # UUID=08301dc8-4e84-4cd9-a402-f4e71a461098 /squid ext4 defaults1 2 /dev/mapper/vg_008-lv_swap swapswapdefaults0 0 /dev/sdb/squid ext4defaults 1 2 -Original Message- From: Marcus Kool [mailto:marcus.k...@urlfilterdb.com] Sent: 25 July 2014 00:37 To: RYAN Justin Subject: Re: [squid-users] FW: Problem with server IO resource, need to reduce logging level by excluding specific sites from being logged Juz, The systems seems to have a very small config. 32 MB for cache_mem is very small indeed Do you have room/RAM to extend the in-memory cache of Squid? From the data that you posted it is not clear if /squid shares its disk with /. What version of Squid do you have (output of squid -v) ? What file system type and mount options are used for /squid ? You did not reply to the squid list. I suggest to include the squid list in the CC: and replace the cachemgr_passwd to XXX in the post. Marcus On 07/24/2014 10:39 AM, RYAN Justin wrote: Sorry Marcus, was a little light on background. Storage on 2 partitions [root@ ]# df -k Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 2057264 0 2057264 0% /dev tmpfs 2066040 0 2066040 0% /dev/shm tmpfs 2066040504 2065536 1% /run /dev/mapper/vg_008-lv_root 160623843864120 11382344 26% / tmpfs 2066040 0 2066040 0% /sys/fs/cgroup tmpfs 2066040 0 2066040 0% /media /dev/sdb 41284928 14322924 24864852 37% /squid /dev/sda2 495844 65891404353 15% /boot Below is the config http_port 3128 dns_nameservers 8.8.8.8 icp_port 0 acl QUERY urlpath_regex cgi-bin \? no_cache deny QUERY append_domain .phoenix.loc cache_mgr i...@pms.co.uk cachemgr_passwd * all buffered_logs on coredump_dir /squid/cache cache_access_log /squid/logs/access.log cache_log /squid/logs/cache.log logfile_rotate 60 cache_dir aufs /squid/cache 4096 16 256 cache_mem 32 MB maximum_object_size 64 MB maximum_object_size_in_memory 20 KB cache_effective_user squid max_filedesc 4096 # acl all src all # acl manager proto cache_object acl localhost src 127.0.0.1 acl to_localhost
Re: [squid-users] FW: Problem with server IO resource, need to reduce logging level by excluding specific sites from being logged
On 25/07/2014 11:28 p.m., RYAN Justin wrote: Cheers Marcus, I did see via googling a rule of thumb quote cache_mem = total physical memory / 3 - ref http://forums.justlinux.com/showthread.php?126396-Squid-cache-tuning there is a more complex formula quoted too. Money and access constraints negate the move to faster storage :) I will look into your recommendations. The question of removing noise from being logged still exists - would be a nice to have option Depends on what you mean by noise. I assume you mean entries in access.log ... The relevant directive is in your config file as cache_access_log. Nowdays that should be configured as: access_log /squid/logs/access.log squid the line can be followed by a list of ACL names, all of which must match for a transaction to be recorded in the log file. For example; in order to log only requests for example.com acl example1 dstdomain example.com access_log /squid/logs/access.log squid example1 ... or in order to omit all CONNECT requests: # ACL for CONNECT is already defined. access_log /squid/logs/access.log squid CONNECT Amos
Re: [squid-users] FW: Problem with server IO resource, need to reduce logging level by excluding specific sites from being logged
Juz, It helps if you describe the system in more detail. What is the configuration of Squid (squid.conf without the comments) and how are the data store file systems spread over the disks ? For immediate results, you can reduce the disk cache or even temporarily disable the disk cache. Marcus On 07/24/2014 09:36 AM, RYAN Justin wrote: Hi, Probably been answered before but my problem is this. We have a esx server out in our providers pop. The server has been in situ for quite sometime. We have noticed that the datastore writes latency has increased to on average 45ms. We know this is bad but have a problem both budgetry and timescale and require a quick solution. We propose to reduce IO load by excluding specific traffic from getting logged, but are now looking on how to achieve this. Understand a change to squid.conf is required plus the use of a access list. Problem being we have inherited this build and there is no real documentation. Holding our hands up, we are not experts and could really do with some community help. Thanks in advance. Juz PRIVACY CONFIDENTIALITY This e-mail is private and confidential. If you have, or suspect you have received this message in error please notify the sender as soon as possible and remove from your system. You may not copy, distribute or take any action in reliance on it. Thank you for your co-operation. Please note that whilst best efforts are made, neither the company nor the sender accepts any responsibility for viruses and it is your responsibility to scan the email and attachments (if any). This e-mail has been automatically scanned for viruses by Proofpoint (Inc.) Enterprise Protection for Email Security services. PHOENIX Healthcare Distribution Ltd. Reg. in England No. 129370 PHOENIX Medical Supplies Ltd. Reg. in England No. 3603234 Rivington Road, Whitehouse Ind. Est., Runcorn, Cheshire, WA7 3DJ VAT No. 741706443
