[squid-users] memory utilization
Hi All, I configured squid box to get good cache performance and for that i set cache_mem and object size in cache. cat /etc/squid/squid.conf | grep cache_mem cache_mem 6144 MB cat /etc/squid/squid.conf | grep -i maximum_object_size_in_memory maximum_object_size_in_memory 1 MB And whenever i check memory usage at OS level while squid is serving to traffic. it shows me free -mto total used free sharedbuffers cached Mem: 7995345 7650 0 30 78 Swap: 8999 0 8999 Total: 16995345 16650 So as per my squid setup i set 6GB , So why free -mto showing me that only 345 mb is used and 7650 mb is free. As per my understanding , if i assign 6GB RAM to squid then 6GB will be deducted from my actual memory and then OS has that remaining amount of memory. My perception is right ? What is the purpose of --enable-async-io option in squid? i read on internet that it enable more performance while using more thread with that option is it correct information which i have ? My squid version is Squid Cache: Version 3.1.15 i used squid rpm which i have from my fedora 15 64 bit os in that i can't have that option is enabled so this option is really useful for better performance in high network traffic ? Regards, Benjamin
Re: [squid-users] memory utilization
On 21/11/2011 1:33 a.m., benjamin fernandis wrote: Hi All, I configured squid box to get good cache performance and for that i set cache_mem and object size in cache. cat /etc/squid/squid.conf | grep cache_mem cache_mem 6144 MB cat /etc/squid/squid.conf | grep -i maximum_object_size_in_memory maximum_object_size_in_memory 1 MB And whenever i check memory usage at OS level while squid is serving to traffic. it shows me free -mto total used free sharedbuffers cached Mem: 7995345 7650 0 30 78 Swap: 8999 0 8999 Total: 16995345 16650 So as per my squid setup i set 6GB , So why free -mto showing me that only 345 mb is used and 7650 mb is free. As per my understanding , if i assign 6GB RAM to squid then 6GB will be deducted from my actual memory and then OS has that remaining amount of memory. My perception is right ? Yes your understanding is generally correct. There is one circumstance I can think of when the OS might show lower than cache_mem usage. That is when memory pooling control has been disabled when building Squid. In that case Squid will not pre-allocate any memory for use. What is the purpose of --enable-async-io option in squid? To enable Asynchrnous I/O (AIO) for threaded read/write to disks. This is required for AUFS. i read on internet that it enable more performance while using more thread with that option Yes. Up to the amount of AIO threads your disk controller can work with efficiently. There is a point at which the threads become too many for the controller and things get slow again. I'm not personally aware what that upper limit is. is it correct information which i have ? My squid version is Squid Cache: Version 3.1.15 i used squid rpm which i have from my fedora 15 64 bit os in that i can't have that option is enabled so this option is really useful for better performance in high network traffic ? See above. Only if you are disk caching using AUFS. Amos
Re: [squid-users] memory utilization
Hi Amos, Really your support is grateful to us.You always help us to understanding of squid and its work pattern.Thanks for your kind support. On 21/11/2011 1:33 a.m., benjamin fernandis wrote: Hi All, I configured squid box to get good cache performance and for that i set cache_mem and object size in cache. cat /etc/squid/squid.conf | grep cache_mem cache_mem 6144 MB cat /etc/squid/squid.conf | grep -i maximum_object_size_in_memory maximum_object_size_in_memory 1 MB And whenever i check memory usage at OS level while squid is serving to traffic. it shows me free -mto total used free sharedbuffers cached Mem: 7995345 7650 0 30 78 Swap: 8999 0 8999 Total: 16995345 16650 So as per my squid setup i set 6GB , So why free -mto showing me that only 345 mb is used and 7650 mb is free. As per my understanding , if i assign 6GB RAM to squid then 6GB will be deducted from my actual memory and then OS has that remaining amount of memory. My perception is right ? Yes your understanding is generally correct. There is one circumstance I can think of when the OS might show lower than cache_mem usage. That is when memory pooling control has been disabled when building Squid. In that case Squid will not pre-allocate any memory for use. If suppose memory pooling control has been disabled in that case, does squid will not have full control on memory utilization or suppose i assign 6 GB ram to squid, so does squid use 6gb ram for caching ? Memory pooling control is mandatory to enable in squid while we are talking for memory utilization or work flow of squid ? What is the purpose of --enable-async-io option in squid? To enable Asynchrnous I/O (AIO) for threaded read/write to disks. This is required for AUFS. Yes, i am using AUFS. i read on internet that it enable more performance while using more thread with that option Yes. Up to the amount of AIO threads your disk controller can work with efficiently. There is a point at which the threads become too many for the controller and things get slow again. I'm not personally aware what that upper limit is. i installed squid rpm which i got from my os distro.i used fedora 15 64 bit.But i can't find that option in squid -v command.so i guess that is not included while installing squid from rpm. So --enable-async-io is mandatory while we r using AUFS or while we are talking for performance how many thread is idle for squid. On internet i see 128 thread configuration at many blogs. Suppose --enable-async-io is not enabled with squid so in that case what is default thread for AUFS? is it correct information which i have ? My squid version is Squid Cache: Version 3.1.15 i used squid rpm which i have from my fedora 15 64 bit os in that i can't have that option is enabled so this option is really useful for better performance in high network traffic ? See above. Only if you are disk caching using AUFS. Amos Warm Regards, Benjamin
Re: [squid-users] memory utilization
On 21/11/2011 6:06 p.m., Benjamin wrote: Hi Amos, Really your support is grateful to us.You always help us to understanding of squid and its work pattern.Thanks for your kind support. On 21/11/2011 1:33 a.m., benjamin fernandis wrote: Hi All, I configured squid box to get good cache performance and for that i set cache_mem and object size in cache. cat /etc/squid/squid.conf | grep cache_mem cache_mem 6144 MB cat /etc/squid/squid.conf | grep -i maximum_object_size_in_memory maximum_object_size_in_memory 1 MB And whenever i check memory usage at OS level while squid is serving to traffic. it shows me free -mto total used free sharedbuffers cached Mem: 7995345 7650 0 30 78 Swap: 8999 0 8999 Total: 16995345 16650 So as per my squid setup i set 6GB , So why free -mto showing me that only 345 mb is used and 7650 mb is free. As per my understanding , if i assign 6GB RAM to squid then 6GB will be deducted from my actual memory and then OS has that remaining amount of memory. My perception is right ? Yes your understanding is generally correct. There is one circumstance I can think of when the OS might show lower than cache_mem usage. That is when memory pooling control has been disabled when building Squid. In that case Squid will not pre-allocate any memory for use. If suppose memory pooling control has been disabled in that case, does squid will not have full control on memory utilization or suppose i assign 6 GB ram to squid, so does squid use 6gb ram for caching ? Memory pooling control is mandatory to enable in squid while we are talking for memory utilization or work flow of squid ? No. Squid will continue to operate as normal and will allocate the 6GB cache_mem eventually, it just means the OS is aware of how much cache_mem area has (or not) been used at any given time. And squid will continuously be allocating/deallocating memory blocks. You can see the churn in the cachemgr memory allocation report. The benefits of pooling is speed of avoiding allocating new blocks of RAM. We have not had any actual speed measurements in many years to get actual numbers. But when I tested it casually about Mar this year the default malloc library most OS supply was still slower than pools. For higher speed you will be needing some modern library like talloc which seemed as good or better than pools (though no numbers means that may be an illusion). The other allocator libraries competing with talloc are also very close. What is the purpose of --enable-async-io option in squid? To enable Asynchrnous I/O (AIO) for threaded read/write to disks. This is required for AUFS. Yes, i am using AUFS. i read on internet that it enable more performance while using more thread with that option Yes. Up to the amount of AIO threads your disk controller can work with efficiently. There is a point at which the threads become too many for the controller and things get slow again. I'm not personally aware what that upper limit is. i installed squid rpm which i got from my os distro.i used fedora 15 64 bit.But i can't find that option in squid -v command.so i guess that is not included while installing squid from rpm. So --enable-async-io is mandatory while we r using AUFS or while we are talking for performance how many thread is idle for squid. On internet i see 128 thread configuration at many blogs. Suppose --enable-async-io is not enabled with squid so in that case what is default thread for AUFS? Well, its equivalent to --with-aufs-threads=N_THREADS --with-pthreads --enable-storeio=ufs,aufs, so they may have set those individually instead. Or just using the default of 16 per cache_dir. Amos
