Sorry forgot to paste the iostat readings in the below email

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.70    0.01    1.19    4.64    0.00   93.47

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               3.24       120.01       111.88    6330517    5901960
sda1              0.00         0.04         0.00       1876          8
sda2              3.24       119.96       111.88    6328145    5901952
sdb              10.45       133.64         4.21    7049934     221920
sdb1             10.45       133.62         4.21    7048838     221920
sdc              10.45       143.34         4.23    7561702     223264
sdc1             10.45       143.32         4.23    7560606     223264
sdd              29.48      4451.11       642.59  234804222   33897624
sdd1             29.48      4451.09       642.59  234803126   33897624
sde              91.10       757.60      1533.97   39964886   80919592
sde1             91.10       757.58      1533.97   39963790   80919592
dm-0             15.51       118.73       110.83    6263154    5846352
dm-1              0.29         1.23         1.05      64736      55600

Ragheb Rustom
Smart Telecom S.A.R.L


-----Original Message-----
From: Ragheb Rustom [mailto:rag...@smartelecom.org] 
Sent: Friday, July 22, 2011 3:10 PM
To: 'Amos Jeffries'; squid-users@squid-cache.org
Cc: 'Marcus Kool'
Subject: RE: [squid-users] I see this error in cache.log file no free membufs
Importance: High

Dear Markus and Amos,

I have done the changes you have proposed. I have dropped the max-size on COSS 
partition to 100KB so the COSS cache_dir line now reads as follows:

cache_dir coss /cache3/coss1 110000 max-size=102400 max-stripe-waste=32768 
block-size=8192 membufs=100
cache_dir aufs /cache1 115000 16 256 min-size=102401
cache_dir aufs /cache2 115000 16 256 min-size=102401
cache_dir aufs /cache4/cache1 240000 16 256 min-size=102401

After doing this I have noticed the following warnings every now and then 
(usually every 1 - 2 hours) in the cache.log file

squidaio_queue_request: WARNING - Queue congestion

What I also noticed using iostat is that the big HDD with AUFS dir is handling 
a lot of write requests while the other 2 HDDS with AUFS dirs rarely have disk 
writes. Is this normal behavior since I have 3 AUFS cache_dir shouldn't squid 
disk read and write access be somewhat equal between the 3 AUFS partitions? Do 
you think I should go for a higher max-size on the COSS partition to relieve 
the extra IO from the big AUFS cache_dir?

Thanks again for your excellent support.

Sincerely,

Ragheb Rustom
Smart Telecom S.A.R.L

-----Original Message-----
From: Amos Jeffries [mailto:squ...@treenet.co.nz] 
Sent: Thursday, July 21, 2011 2:00 AM
To: squid-users@squid-cache.org
Subject: Re: [squid-users] I see this error in cache.log file no free membufs

 On Wed, 20 Jul 2011 18:23:10 -0300, Marcus Kool wrote:
> The message indicates that the numbers of membufs should be
> because there are insufficent membufs to use for caching
> objects.  The reason for having 'insufficient membufs'
> is explained below.
>
> Given the fact that the average object size is 13 KB, the given
> configuration effectively puts a very large percentage of objects,
> most likely more than 90% in the COSS-based cache dir.  This puts
> a high pressure on (the disk with) COSS and I bet that the disk
> with COSS (/cache3) is 100% busy while the other three are mostly 
> idle.
>
> COSS is very good for small objects and AUFS is fine with larger 
> objects.
>
> There is one larger disk.  But this larger disk is not faster.
> It will perform worse with more objects on it than the other disks.
>
> To find out more about disk I/O and pressure on the disk with COSS, 
> one
> can evaluate the output of iostat or 'vmstat -d 5 5'
>
> I recommend to change the configuration, to utilise all disks in a
> more balanced way.  Be sure to also look at the output of iostat.
> My suggestion is to use COSS only for objects smaller than 64 KB.
> Depending on the average object size of your cache, this limit
> may be set lower.
>
> So I suggest:
>
> cache_dir coss /cache3 110000 max-size=65535 max-stripe-waste=32768
> block-size=8192 membufs=15
> cache_dir aufs /cache1 115000 16 256 min-size=65536
> cache_dir aufs /cache2 115000 16 256 min-size=65536
> cache_dir aufs /cache4 115000 16 256 min-size=65536
>
> And to observe the log and output of iostat.
> If the disk I/O is balanced and the message about membufs reappears 
> and
> you have sufficient free memory, you may increase membufs.  If the 
> I/O is
> not balanced, the limit of 64KB may be decreased to 16KB.
>
> Depending on the results and iostat, it may be better to
> have 2 disks with COSS and 2 disks with AUFS:
>
> cache_dir coss /cache3 110000 max-size=16383 max-stripe-waste=32768
> block-size=8192 membufs=15
> cache_dir aufs /cache1 110000 max-size=16383 max-stripe-waste=32768
> block-size=8192 membufs=15
> cache_dir aufs /cache2 115000 16 256 min-size=16384
> cache_dir aufs /cache4 115000 16 256 min-size=16384
>
> Marcus
>

 NP: use the cache manager "info" report to find the average object 
 size.
   squidclient mgr:info

 COSS handles things in 1MB slices. This is the main reason 
 max-size=1048575 is a bad idea, one object per file/slice is less 
 efficient than AUFS one object per file. So with 110GB of COSS dir will 
 be juggling a massive 110000 slices on and off of disk as things are 
 needed. I recommend using smaller COSS overall size and using the 
 remainder of each disk for AUFS storage of the larger objects. (COSS is 
 the exception to the one-dir-per-spindle guideline)

 Something like this with ~30GB COSS on each disk, double size on the 
 big disk = ~150GB of small objects:

 cache_dir coss /cache1coss 30000 max-size=65535 max-stripe-waste=32768 
 block-size=8192 membufs=15
 cache_dir aufs /cache1aufs 100000 16 256 min-size=65536

 cache_dir coss /cache2coss 30000 max-size=65535 max-stripe-waste=32768 
 block-size=8192 membufs=15
 cache_dir aufs /cache2aufs 100000 16 256 min-size=65536

 cache_dir coss /cache3coss 30000 max-size=65535 max-stripe-waste=32768 
 block-size=8192 membufs=15
 cache_dir aufs /cache3aufs 100000 16 256 min-size=65536

 cache_dir coss /cache4coss1 60000 max-size=65535 max-stripe-waste=32768 
 block-size=8192 membufs=15
 cache_dir aufs /cache4aufs 240000 16 256 min-size=65536

 This last one is a little tricky. You will need to test and see if its 
 is okay this big or needs reducing.

 On the size multiple dirs like this will means ~60MB of RAM consumed by 
 active COSS membufs instead of 15MB. You could bump membufs up to 20, 
 but changes like thse suggested by Marcus and myself above are needed to 
 make that worthwhile.

 Amos

>
> Ragheb Rustom wrote:
>> Dear All,
>> I have a squid cache proxy which is delivering content to around 
>> 3000+
>> users. After some problems with performance on AUFS under peak hour 
>> loads I
>> have used one of my cache_dirs as COSS disk following Amos Settings 
>> on
>> squid-cache website while left all others as AUFS that hold files 
>> bigger
>> than 1MB. After running perfectly for some time with coss and very 
>> beautiful
>> results I started seeing the below messages in my cache.log file:
>> storeCossCreateMemOnlyBuf: no free membufs.  You may need to 
>> increase the
>> value of membufs on the /cache3/coss1 cache_dir
>> here are my squid.conf settings:
>> cache_dir coss /cache3/coss1 110000 max-size=1048575 
>> max-stripe-waste=32768
>> block-size=8192 membufs=15
>> cache_dir aufs /cache1 115000 16 256 min-size=1048576
>> cache_dir aufs /cache2 115000 16 256 min-size=1048576
>> cache_dir aufs /cache4/cache1 240000 16 256 min-size=1048576
>> Please note all my Hdds are SAS 15k drives sizes as follows:
>> /cache1                                147GB
>> /cache2                                147GB
>> /cache3                                147GB
>> /cache4                                450GB
>> The system is dual Xeon quad core intel server with 16GB of physical 
>> Ram
>> Do you think I should increase the membufs value and what do you 
>> think the
>> best or optimal value for a such system should be?
>> Sincerely,
>>
>> Ragheb Rustom
>> Smart Telecom S.A.R.L
>>
>>
>>
>>
>>





Reply via email to