massive deleted open files in proxy cache

Mon, 20 Feb 2017 01:46:34 -0800 

Hi!

We are useing Ubuntu 16.04 with nginx version 1.10.0-0ubuntu0.16.04.4.

nginx.conf:

user  nginx;
worker_processes  auto;
worker_rlimit_nofile 20480; # ulimit open files per worker process

events {
        # Performance
        worker_connections  2048; # openfilelimits beachten
        multi_accept on;
        use epoll;
}

http {
        open_file_cache max=10000 inactive=1d;
        open_file_cache_valid 1d;
        open_file_cache_min_uses 1;
        open_file_cache_errors off;
proxy_cache_path /var/cache/nginx/proxy_cache levels=1:2 keys_zone=html_cache:30m max_size=8192m inactive=4h use_temp_path=off; proxy_cache_path /var/cache/nginx/wordpress_cache levels=1:2 keys_zone=wordpress_cache:1m max_size=256m inactive=24h use_temp_path=off; 

        proxy_buffer_size          128k;
        proxy_buffers              4 256k;
        proxy_busy_buffers_size    256k;
}


# df -h |grep cache
tmpfs      300M   74M  227M  25% /var/cache/nginx/wordpress_cache
tmpfs      9.0G  4.1G  5.0G  45% /var/cache/nginx/proxy_cache

# df -i |grep cache
tmpfs    20632978    5457 20627521    1% /var/cache/nginx/wordpress_cache
tmpfs    20632978   74613 20558365    1% /var/cache/nginx/proxy_cache

# grep cache /etc/fstab
tmpfs /var/cache/nginx/proxy_cache/ tmpfs rw,uid=109,gid=117,size=9G,mode=0755 0 0 tmpfs /var/cache/nginx/wordpress_cache/ tmpfs rw,uid=109,gid=117,size=300M,mode=0755 0 0 


# free -m
total used free shared buff/cache available Mem: 161195 112884 1321 4626 46988 42519 
Swap:          3903         211        3692


Problem:
========
We got massive open file handles from nginx user located inside proxy_cache_path with status deleted: 

# lsof -E -M -T > lsof.`date +"%Y%d%m-%H%M%S"`.out
nginx 3613 nginx 48r REG 0,42 148664 29697 /var/cache/nginx/proxy_cache/temp/5/23/04ca8002edd2daa3c538ada5b202d6eb (deleted) nginx 3613 nginx 50r REG 0,42 161618 19416 /var/cache/nginx/proxy_cache/temp/1/40/d8f0a3563d18af4fbf43242e19283b15 (deleted) 

# grep nginx lsof.20172002-085328.out |wc -l
69003

# grep nginx lsof.20172002-085328.out |grep deleted |wc -l
36312
# grep nginx lsof.20172002-085328.out |grep deleted |grep "/var/cache/nginx/proxy_cache/temp" |wc -l 
32004

The most of the 36k deleted files are located inside the temp cache folder.
My question is, why we got so much deleted files inside the cache. Why is nginx not freeing these files? 
Is there a problem with the proxy_cache_path option use_temp_path=off ?
I am in worry that the cache file system will be filled up with deleted files, and will reach open files limits. 

Or do we have a nginx misconfiguration somewhere?
Additionally we were often visited from the oom-killer, (in face of 20G free memory). If I restart nginx before we reach 80k open ngnix files, the oom-killer will not visit us! 

Has anybody simmilar findings regarding deleted open files?

br,
Marco
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Reply via email to