I hit similar problem ... Can I know what is the ingest Gbps into the SSDs when you hit the problem? and How many cached file nodes in cache-manager? i have millions ...
On Wed, May 21, 2014 at 7:57 PM, ixos <[email protected]> wrote: > I'm having problem with I/O performance. I'm running nginx as caching > reverse proxy server. > When cache size on disk exceeds max_size cache manager starts working, but > it causes two problems occur: > > 1) I/O %util reach 100% and nginx starts dropping connections > 2) cache manager process dosen't unlink files speed enough to delete old > file. So cache becomes bigger util the space on disk ends. > > Can you give me an idea how can I solve those problems. Below are some > details. > > #build on 20x 300GB SAS disks with 2 SSDs for Cachecade. > > # storcli64 /c0 show > VD LIST : > ======= > > ---------------------------------------------------------------- > DG/VD TYPE State Access Consist Cache Cac sCC Size Name > ---------------------------------------------------------------- > 1/2 RAID60 Optl RW Yes RaWBC R ON 4.357 TB > 2/1 Cac0 Optl RW Yes RaWTD - ON 557.875 GB > ---------------------------------------------------------------- > > # mount > /dev/sdb1 on /cache type ext4 (rw,noatime,data=ordered) > > # df -h /dev/sdb1 > /dev/sdb1 4.3T 3.2T 828G 80% /cache > > > # for pid in `pgrep nginx `;do ionice -p $pid ;done > unknown: prio 4 <- master > best-effort: prio 0 > best-effort: prio 0 > best-effort: prio 0 > best-effort: prio 0 > best-effort: prio 0 > best-effort: prio 0 > best-effort: prio 0 > best-effort: prio 0 > best-effort: prio 0 > best-effort: prio 0 > best-effort: prio 0 > best-effort: prio 0 <- workers > idle <- cache manager > > # grep proxy_cache_path nginx.conf > proxy_cache_path /cache zone=my-cache:20000msize=3355443m > > # netstat -sp|grep -i drop > 6335115 SYNs to LISTEN sockets dropped > > # iostat -dx 1 /dev/sdb |grep ^sdb | awk '{print $14}' > 24.40 > 31.20 > 26.80 > 23.60 > 26.80 > 16.00 > 34.80 > 35.20 > 29.60 > ... > 14.40 > 15.60 > 11.60 > 16.00 > 17.20 > 18.00 > 17.20 > 42.00 > 90.80 <- cache manager process starts > 100.00 > 100.00 > 29.20 > 100.00 > 100.00 > 100.00 > 52.00 > 100.00 > 100.00 > 100.00 > > Posted at Nginx Forum: > http://forum.nginx.org/read.php?2,250247,250247#msg-250247 > > _______________________________________________ > nginx mailing list > [email protected] > http://mailman.nginx.org/mailman/listinfo/nginx >
_______________________________________________ nginx mailing list [email protected] http://mailman.nginx.org/mailman/listinfo/nginx
