On 26/09/16 17:04, to...@tuxteam.de wrote: > On Mon, Sep 26, 2016 at 04:38:01PM +0200, Tony van der Hoff wrote: >> On 26/09/16 16:03, to...@tuxteam.de wrote: >>> On Mon, Sep 26, 2016 at 12:54:49PM +0200, Erwan David wrote: >>> >>>> A possibility is that you have processes writing into deleted files. You >>>> can see them with lsof +L1 (as root) >>>
Thanks again to all who have helped with this. After the overnight run, I'm now seeing this: root@shell:~# lsof +L1 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME mysqld 3164 mysql 4u REG 254,16 0 0 589839 /tmp/ibvh7MKT (deleted) mysqld 3164 mysql 5u REG 254,16 473 0 589840 /tmp/ibT7D133 (deleted) mysqld 3164 mysql 6u REG 254,16 0 0 589845 /tmp/ibO9pgne (deleted) mysqld 3164 mysql 7u REG 254,16 0 0 589851 /tmp/ibOgMl6y (deleted) mysqld 3164 mysql 11u REG 254,16 0 0 589852 /tmp/ibON9JEJ (deleted) /usr/sbin 4643 www-data 2w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 4643 www-data 29w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 4643 www-data 30w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 4643 www-data 31w REG 254,16 0 0 2180306 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927 (deleted) /usr/sbin 4643 www-data 32w REG 254,16 0 0 2180396 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927 (deleted) /usr/sbin 4643 www-data 33w REG 254,16 0 0 2180397 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927 (deleted) /usr/sbin 4643 www-data 34w REG 254,16 10374 0 2180686 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927 (deleted) /usr/sbin 4643 www-data 35w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 4643 www-data 36w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 4643 www-data 37w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 4643 www-data 38w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13918 www-data 2w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13918 www-data 29w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13918 www-data 30w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13918 www-data 35w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13918 www-data 36w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13918 www-data 37w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13918 www-data 38w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13918 www-data 39w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13920 www-data 2w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13920 www-data 29w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13920 www-data 30w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13920 www-data 31w REG 254,16 0 0 2180306 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927 (deleted) /usr/sbin 13920 www-data 32w REG 254,16 0 0 2180396 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927 (deleted) /usr/sbin 13920 www-data 33w REG 254,16 0 0 2180397 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927 (deleted) /usr/sbin 13920 www-data 34w REG 254,16 10374 0 2180686 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927 (deleted) /usr/sbin 13920 www-data 35w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13920 www-data 36w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13920 www-data 37w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13920 www-data 38w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13920 www-data 39w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 13920 www-data 40w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 15766 www-data 2w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 15766 www-data 33w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 15766 www-data 34w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 15766 www-data 35w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 15766 www-data 36w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 15766 www-data 37w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 15766 www-data 38w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 15766 www-data 39w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16711 www-data 2w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16711 www-data 29w REG 254,16 0 0 2180306 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927 (deleted) /usr/sbin 16711 www-data 30w REG 254,16 0 0 2180396 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927 (deleted) /usr/sbin 16711 www-data 31w REG 254,16 0 0 2180397 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927 (deleted) /usr/sbin 16711 www-data 32w REG 254,16 10374 0 2180686 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927 (deleted) /usr/sbin 16711 www-data 33w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16711 www-data 34w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16711 www-data 35w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16711 www-data 36w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16711 www-data 37w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16768 www-data 2w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16768 www-data 29w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16768 www-data 30w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16768 www-data 31w REG 254,16 0 0 2180306 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927 (deleted) /usr/sbin 16768 www-data 32w REG 254,16 0 0 2180396 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927 (deleted) /usr/sbin 16768 www-data 33w REG 254,16 0 0 2180397 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927 (deleted) /usr/sbin 16768 www-data 34w REG 254,16 10374 0 2180686 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927 (deleted) /usr/sbin 16768 www-data 35w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16768 www-data 36w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16768 www-data 37w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16768 www-data 38w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16770 www-data 2w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16770 www-data 29w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16770 www-data 30w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16770 www-data 31w REG 254,16 0 0 2180306 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927 (deleted) /usr/sbin 16770 www-data 32w REG 254,16 0 0 2180396 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927 (deleted) /usr/sbin 16770 www-data 33w REG 254,16 0 0 2180397 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927 (deleted) /usr/sbin 16770 www-data 34w REG 254,16 10374 0 2180686 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927 (deleted) /usr/sbin 16770 www-data 35w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16798 www-data 2w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16798 www-data 29w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16798 www-data 30w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16798 www-data 31w REG 254,16 0 0 2180306 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927 (deleted) /usr/sbin 16798 www-data 32w REG 254,16 0 0 2180396 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927 (deleted) /usr/sbin 16798 www-data 33w REG 254,16 0 0 2180397 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927 (deleted) /usr/sbin 16798 www-data 34w REG 254,16 10374 0 2180686 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927 (deleted) /usr/sbin 16798 www-data 35w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 16798 www-data 36w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 18785 www-data 2w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 18785 www-data 29w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 18785 www-data 30w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 18785 www-data 31w REG 254,16 0 0 2180306 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927 (deleted) /usr/sbin 18785 www-data 32w REG 254,16 0 0 2180396 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927 (deleted) /usr/sbin 18785 www-data 33w REG 254,16 0 0 2180397 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927 (deleted) /usr/sbin 18785 www-data 34w REG 254,16 10374 0 2180686 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927 (deleted) /usr/sbin 18785 www-data 35w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 18929 www-data 2w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 18929 www-data 29w REG 254,16 22055 0 2180962 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927 (deleted) /usr/sbin 18929 www-data 31w REG 254,16 0 0 2180306 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927 (deleted) /usr/sbin 18929 www-data 32w REG 254,16 0 0 2180396 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927 (deleted) /usr/sbin 18929 www-data 33w REG 254,16 0 0 2180397 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927 (deleted) /usr/sbin 18929 www-data 34w REG 254,16 10374 0 2180686 /home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927 (deleted) So, I can see this growing to an impossible size, which is probably where my problem lies. These files are created by a apache-driven PERL application, and the log files are renamed/compressed overnight by a cron job. This setup has existed for a number of years unchanged, so what's caused it to start misbehaving? How to stop it happening? Should I be stopping Apache while compressing the logs? -- Tony van der Hoff | mailto:t...@vanderhoff.org Ariège, France |