Re: Issue 1188 in ganeti: ganeti-mond in 2.15.2 appears to leak file descriptors
Comment #6 on issue 1188 by bpfo...@google.com: ganeti-mond in 2.15.2 appears to leak file descriptors https://code.google.com/p/ganeti/issues/detail?id=1188 Excellent. I'm glad you have a workaround for now, but I'd still like to fix it properly. Narrowing it down to the xen-cpu-avg-load and inst-status-xen collectors on non-xen systems is definitely helpful. I'll see if I can figure out what's going on here. -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings
Re: Issue 1188 in ganeti: ganeti-mond in 2.15.2 appears to leak file descriptors
Comment #5 on issue 1188 by zen2dr...@gmail.com: ganeti-mond in 2.15.2 appears to leak file descriptors https://code.google.com/p/ganeti/issues/detail?id=1188 I have stop the file descriptor leak by stopping monitor of xen-cpu-avg-load and inst-status-xen since we don't use xen (xen not installed and ganeti not compiled with xen support). So now I got: Data collectors: cpu-avg-load: active: True interval: 5.000s diskstats: active: True interval: 5.000s drbd: active: True interval: 5.000s inst-status-xen: active: False interval: 5.000s lv: active: True interval: 5.000s xen-cpu-avg-load: active: False interval: 5.000s Thanks to lead me on the right direction. -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings
Re: Issue 1188 in ganeti: ganeti-mond in 2.15.2 appears to leak file descriptors
Comment #4 on issue 1188 by zen2dr...@gmail.com: ganeti-mond in 2.15.2 appears to leak file descriptors https://code.google.com/p/ganeti/issues/detail?id=1188 # wget -qO - http://127.0.0.1:1815/1/list/collectors | /etc/ganeti/share/fmtjson [ [ "cpu-avg-load", "default", 0 ], [ "xen-cpu-avg-load", "default", 0 ], [ "diskstats", "storage", 0 ], [ "drbd", "storage", 1 ], [ "lv", "storage", 0 ] Don't hesitate to ask me if you need that I dig further. -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings
Re: Issue 1188 in ganeti: ganeti-mond in 2.15.2 appears to leak file descriptors
Comment #3 on issue 1188 by bpfo...@google.com: ganeti-mond in 2.15.2 appears to leak file descriptors https://code.google.com/p/ganeti/issues/detail?id=1188 I tried to reproduce this on my 2.17 setup using Xen and DRBD, but couldn't. I'll continue looking at it, but out of interest could you tell me which collectors are enabled on your system? You can do this by running the following on the master wget -qO - http://127.0.0.1:1815/1/list/collectors | /etc/ganeti/share/fmtjson For me, the only collectors that were enabled were drbd, inst-status-xen and diagnose. -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings
Re: Issue 1188 in ganeti: ganeti-mond in 2.15.2 appears to leak file descriptors
Comment #2 on issue 1188 by zen2dr...@gmail.com: ganeti-mond in 2.15.2 appears to leak file descriptors https://code.google.com/p/ganeti/issues/detail?id=1188 # gnt-cluster version Software version: 2.15.2 Internode protocol: 215 Configuration format: 215 OS api version: 20 Export interface: 0 VCS version: (ganeti) version v2.14.2-177-g003cd9a -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings
Re: Issue 1188 in ganeti: ganeti-mond in 2.15.2 appears to leak file descriptors
Comment #1 on issue 1188 by zen2dr...@gmail.com: ganeti-mond in 2.15.2 appears to leak file descriptors https://code.google.com/p/ganeti/issues/detail?id=1188 I take a look on my nodes in /proc/$(pidof ganeti-mond)/fd and there is 2 new pipe link that appears every 5s: l-wx-- 1 root root 64 Sep 14 22:53 820 -> pipe:[13902781] lr-x-- 1 root root 64 Sep 14 22:53 821 -> pipe:[13902782] /proc/15720/fd/820: broken symbolic link to pipe:[13902781] /proc/15720/fd/821: broken symbolic link to pipe:[13902782] Right after ganeti-mond is launched: lr-x-- 1 root root 64 Sep 14 22:57 0 -> /dev/null l-wx-- 1 root root 64 Sep 14 22:57 1 -> /var/log/ganeti/monitoring-daemon.log l-wx-- 1 root root 64 Sep 14 22:57 10 -> pipe:[13926595] lr-x-- 1 root root 64 Sep 14 22:57 11 -> pipe:[13926596] l-wx-- 1 root root 64 Sep 14 22:57 2 -> /var/log/ganeti/monitoring-daemon.log lr-x-- 1 root root 64 Sep 14 22:57 3 -> /var/lib/ganeti/ssconf_master_node lrwx-- 1 root root 64 Sep 14 22:57 4 -> socket:[13906864] lrwx-- 1 root root 64 Sep 14 22:57 6 -> /run/ganeti/ganeti-mond.pid l-wx-- 1 root root 64 Sep 14 22:57 7 -> /var/log/ganeti/monitoring-daemon-access.log l-wx-- 1 root root 64 Sep 14 22:57 8 -> /var/log/ganeti/monitoring-daemon-error.log lrwx-- 1 root root 64 Sep 14 22:57 9 -> socket:[13906866] /proc/5602/fd/0: symbolic link to /dev/null /proc/5602/fd/1: symbolic link to /var/log/ganeti/monitoring-daemon.log /proc/5602/fd/10: broken symbolic link to pipe:[13926595] /proc/5602/fd/11: broken symbolic link to pipe:[13926596] /proc/5602/fd/2: symbolic link to /var/log/ganeti/monitoring-daemon.log /proc/5602/fd/3: symbolic link to /var/lib/ganeti/ssconf_master_node /proc/5602/fd/4: broken symbolic link to socket:[13906864] /proc/5602/fd/6: symbolic link to /run/ganeti/ganeti-mond.pid /proc/5602/fd/7: symbolic link to /var/log/ganeti/monitoring-daemon-access.log /proc/5602/fd/8: symbolic link to /var/log/ganeti/monitoring-daemon-error.log /proc/5602/fd/9: broken symbolic link to socket:[13906866] So the process reach 1024 broken symbolic link to pipe in 1024/2/60*5 = 42.66 mn And since the watcher check and restart the process every 5mn by cron, the process is restarted every 45mn as observed. -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings
