Hi there, On Mon, 13 Aug 2007 Tomasz Chmielewski wrote:
Re: memory leak in clamd (0.91.1)? > Does clamd (0.91.1) have a memory leak of some sort? I use sendmail and clamav-milter, and I see something rather strange with 0.91.1. I haven't usesd versions between those, I moved from 0.88.7 directly to 0.91.1 since I don't like my servers to be on the bleeding edge. > After it's started, it takes just about 40-50 MB: > ... > After some time, the memory usage goes to about 100 MB, machine begins > to swap more and more: > ... > Is it normal that clamd grows so much after a couple of days running? I think it might be normal - but I don't think it should happen. :) The sudden changes in memory consumption seem to coincide with freshclam updates. Here are a couple of graphs of clamav-milter usage on one of my machines. The first covers March 18 to 16:40 on July 17, clamav is version 0.88.7: http://www.jubileegroup.co.uk/JOS/misc/clamav-milter-0.88.7.jpg The graph shows a fairly steady memory usage for a couple of months and then for three months it's actually reducing. I don't know why but I suspect it's something to do with bringing out 0.91.1. The second covers 16:45 on July 17 to Aug 13, clamav is version 0.91.1 which replaced 0.88.7 on this machine: http://www.jubileegroup.co.uk/JOS/misc/clamav-milter-0.91.1.jpg The graph shows two sudden increases in memory consumption, both consist of more than one step change. The first group of events is a change from 30 to 56 Megabytes at 04:50 on July 18 and from 56 to 61 Megabytes at 08:55 on the same day. The second is on July 27 when memory consumption changed from 62 to 91 Megabytes and 91 to 118 Megabytes on at 14:49 and 16:49 respectively. There is a striking similarity between the graphs around the two large changes in memory consumption; on July 18 and July 27. In both cases there is a large step increase followed by several smaller increases followed by a period of relative stability. Times are all British Summer Time, UTC+0100. Freshclam is updating every hour. The clamav configuration includes phishing detection. The clamav-milter daemon is started and left to run indefinitely, so each graph shows the memory consumption for a single process. If you'd like to see these graphs and you can't because I'm firewalling your IP, please post your email address to the List and I'll send them to you. They're only about 4kBytes each. The raw data is available if you want it. The sudden changes in memory consumption from 30 to 56 Megabytes at 04:50 and to 61 Megabytes at 08:55 on Jul 18 seem to coincide with database updates: Jul 18 04:48:09 mail3 freshclam[23917]: Received signal: wake up Jul 18 04:48:09 mail3 freshclam[23917]: Max retries == 5 Jul 18 04:48:09 mail3 freshclam[23917]: ClamAV update process started at Wed Jul 18 04:48:09 2007 Jul 18 04:48:09 mail3 freshclam[23917]: Querying current.cvd.clamav.net Jul 18 04:48:09 mail3 freshclam[23917]: TTL: 900 Jul 18 04:48:09 mail3 freshclam[23917]: Software version from DNS: 0.91.1 Jul 18 04:48:09 mail3 freshclam[23917]: main.cvd version from DNS: 43 Jul 18 04:48:09 mail3 freshclam[23917]: main.cvd is up to date (version: 43, sigs: 104500, f-level: 14, builder: sven) Jul 18 04:48:09 mail3 freshclam[23917]: daily.cvd version from DNS: 3690 Jul 18 04:48:09 mail3 freshclam[23917]: Retrieving http://db.uk.clamav.net/daily-3690.cdiff Jul 18 04:48:09 mail3 freshclam[23917]: Trying to download http://db.uk.clamav.net/daily-3690.cdiff (IP: 80.82.245.8) Jul 18 04:48:09 mail3 freshclam[23917]: Downloading daily-3690.cdiff [100%] Jul 18 04:48:09 mail3 freshclam[23917]: cdiff_apply: Parsed 7 lines and executed 7 commands Jul 18 04:48:09 mail3 freshclam[23917]: daily.cvd updated (version: 3690, sigs: 33953, f-level: 16, builder: ccordes) Jul 18 04:48:09 mail3 freshclam[23917]: Database updated (138453 signatures) from db.uk.clamav.net (IP: 80.82.245.8) Jul 18 04:48:09 mail3 freshclam[23917]: Clamd successfully notified about the update. Jul 18 04:48:09 mail3 freshclam[23917]: -------------------------------------- Jul 18 05:48:09 mail3 freshclam[23917]: Received signal: wake up Jul 18 05:48:09 mail3 freshclam[23917]: Max retries == 5 Jul 18 05:48:09 mail3 freshclam[23917]: ClamAV update process started at Wed Jul 18 05:48:09 2007 Jul 18 05:48:09 mail3 freshclam[23917]: Querying current.cvd.clamav.net Jul 18 05:48:09 mail3 freshclam[23917]: TTL: 900 Jul 18 05:48:09 mail3 freshclam[23917]: Software version from DNS: 0.91.1 Jul 18 05:48:09 mail3 freshclam[23917]: main.cvd version from DNS: 43 Jul 18 05:48:09 mail3 freshclam[23917]: main.cvd is up to date (version: 43, sigs: 104500, f-level: 14, builder: sven) Jul 18 05:48:09 mail3 freshclam[23917]: daily.cvd version from DNS: 3690 Jul 18 05:48:09 mail3 freshclam[23917]: daily.inc is up to date (version: 3690, sigs: 33953, f-level: 16, builder: ccordes) Jul 18 05:48:09 mail3 freshclam[23917]: -------------------------------------- Jul 18 06:48:09 mail3 freshclam[23917]: Received signal: wake up Jul 18 06:48:09 mail3 freshclam[23917]: Max retries == 5 Jul 18 06:48:09 mail3 freshclam[23917]: ClamAV update process started at Wed Jul 18 06:48:09 2007 Jul 18 06:48:09 mail3 freshclam[23917]: Querying current.cvd.clamav.net Jul 18 06:48:09 mail3 freshclam[23917]: TTL: 900 Jul 18 06:48:09 mail3 freshclam[23917]: Software version from DNS: 0.91.1 Jul 18 06:48:09 mail3 freshclam[23917]: main.cvd version from DNS: 43 Jul 18 06:48:09 mail3 freshclam[23917]: main.cvd is up to date (version: 43, sigs: 104500, f-level: 14, builder: sven) Jul 18 06:48:09 mail3 freshclam[23917]: daily.cvd version from DNS: 3690 Jul 18 06:48:09 mail3 freshclam[23917]: daily.inc is up to date (version: 3690, sigs: 33953, f-level: 16, builder: ccordes) Jul 18 06:48:09 mail3 freshclam[23917]: -------------------------------------- Jul 18 07:48:09 mail3 freshclam[23917]: Received signal: wake up Jul 18 07:48:09 mail3 freshclam[23917]: Max retries == 5 Jul 18 07:48:09 mail3 freshclam[23917]: ClamAV update process started at Wed Jul 18 07:48:09 2007 Jul 18 07:48:09 mail3 freshclam[23917]: Querying current.cvd.clamav.net Jul 18 07:48:09 mail3 freshclam[23917]: TTL: 900 Jul 18 07:48:09 mail3 freshclam[23917]: Software version from DNS: 0.91.1 Jul 18 07:48:09 mail3 freshclam[23917]: main.cvd version from DNS: 43 Jul 18 07:48:09 mail3 freshclam[23917]: main.cvd is up to date (version: 43, sigs: 104500, f-level: 14, builder: sven) Jul 18 07:48:09 mail3 freshclam[23917]: daily.cvd version from DNS: 3691 Jul 18 07:48:09 mail3 freshclam[23917]: Retrieving http://db.uk.clamav.net/daily-3691.cdiff Jul 18 07:48:09 mail3 freshclam[23917]: Trying to download http://db.uk.clamav.net/daily-3691.cdiff (IP: 80.82.245.8) Jul 18 07:48:10 mail3 freshclam[23917]: Downloading daily-3691.cdiff [100%] Jul 18 07:48:10 mail3 freshclam[23917]: cdiff_apply: Parsed 28 lines and executed 28 commands Jul 18 07:48:10 mail3 freshclam[23917]: Removing backup directory ./clamav-4869019beca2eb985cef7f59389d089f Jul 18 07:48:10 mail3 freshclam[23917]: daily.inc updated (version: 3691, sigs: 33971, f-level: 16, builder: ccordes) Jul 18 07:48:10 mail3 freshclam[23917]: Database updated (138471 signatures) from db.uk.clamav.net (IP: 80.82.245.8) Jul 18 07:48:10 mail3 freshclam[23917]: Clamd successfully notified about the update. Jul 18 07:48:10 mail3 freshclam[23917]: -------------------------------------- Jul 18 08:48:09 mail3 freshclam[23917]: Received signal: wake up Jul 18 08:48:09 mail3 freshclam[23917]: Max retries == 5 Jul 18 08:48:09 mail3 freshclam[23917]: ClamAV update process started at Wed Jul 18 08:48:09 2007 Jul 18 08:48:09 mail3 freshclam[23917]: Querying current.cvd.clamav.net Jul 18 08:48:10 mail3 freshclam[23917]: TTL: 900 Jul 18 08:48:10 mail3 freshclam[23917]: Software version from DNS: 0.91.1 Jul 18 08:48:10 mail3 freshclam[23917]: main.cvd version from DNS: 43 Jul 18 08:48:10 mail3 freshclam[23917]: main.cvd is up to date (version: 43, sigs: 104500, f-level: 14, builder: sven) Jul 18 08:48:10 mail3 freshclam[23917]: daily.cvd version from DNS: 3692 Jul 18 08:48:10 mail3 freshclam[23917]: Retrieving http://db.uk.clamav.net/daily-3692.cdiff Jul 18 08:48:10 mail3 freshclam[23917]: Trying to download http://db.uk.clamav.net/daily-3692.cdiff (IP: 80.82.245.8) Jul 18 08:48:10 mail3 freshclam[23917]: Downloading daily-3692.cdiff [100%] Jul 18 08:48:10 mail3 freshclam[23917]: cdiff_apply: Parsed 380 lines and executed 380 commands Jul 18 08:48:10 mail3 freshclam[23917]: Removing backup directory ./clamav-4f0756644092568d9cc34cf000c2392d Jul 18 08:48:10 mail3 freshclam[23917]: daily.inc updated (version: 3692, sigs: 34345, f-level: 16, builder: ccordes) Jul 18 08:48:10 mail3 freshclam[23917]: Database updated (138845 signatures) from db.uk.clamav.net (IP: 80.82.245.8) Jul 18 08:48:10 mail3 freshclam[23917]: Clamd successfully notified about the update. Jul 18 08:48:10 mail3 freshclam[23917]: -------------------------------------- The sudden change in memory consumption at 14:49 on Jul 27 seems to conicide with an unusual event: Jul 27 13:49:40 mail3 freshclam[23917]: Received signal: wake up Jul 27 13:49:40 mail3 freshclam[23917]: Max retries == 5 Jul 27 13:49:40 mail3 freshclam[23917]: ClamAV update process started at Fri Jul 27 13:49:40 2007 Jul 27 13:49:40 mail3 freshclam[23917]: Querying current.cvd.clamav.net Jul 27 13:49:40 mail3 freshclam[23917]: TTL: 300 Jul 27 13:49:40 mail3 freshclam[23917]: Software version from DNS: 0.91.1 Jul 27 13:49:40 mail3 freshclam[23917]: main.cvd version from DNS: 44 Jul 27 13:49:40 mail3 freshclam[23917]: main.inc is up to date (version: 44, sigs: 133163, f-level: 20, builder: sven) Jul 27 13:49:40 mail3 freshclam[23917]: daily.cvd version from DNS: 3781 Jul 27 13:49:40 mail3 freshclam[23917]: daily.inc is up to date (version: 3781, sigs: 7600, f-level: 20, builder: tkojm) Jul 27 13:49:40 mail3 freshclam[23917]: -------------------------------------- Jul 27 14:49:40 mail3 freshclam[23917]: Received signal: wake up Jul 27 14:49:40 mail3 freshclam[23917]: Max retries == 5 Jul 27 14:49:40 mail3 freshclam[23917]: ClamAV update process started at Fri Jul 27 14:49:40 2007 Jul 27 14:49:40 mail3 freshclam[23917]: Querying current.cvd.clamav.net Jul 27 14:49:40 mail3 freshclam[23917]: TTL: 300 Jul 27 14:49:40 mail3 freshclam[23917]: DNS record is older than 3 hours. Jul 27 14:49:40 mail3 freshclam[23917]: Invalid DNS reply. Falling back to HTTP mode. Jul 27 14:49:40 mail3 freshclam[23917]: Assuming modification time in the past Jul 27 14:49:40 mail3 freshclam[23917]: If-Modified-Since: Mon, 27 Dec 2004 03:52:10 GMT Jul 27 14:49:40 mail3 freshclam[23917]: Reading CVD header (main.cvd): Jul 27 14:49:40 mail3 freshclam[23917]: Connected to db.uk.clamav.net (IP: 80.82.245.8). Jul 27 14:49:40 mail3 freshclam[23917]: Trying to retrieve CVD header of http://db.uk.clamav.net/main.cvd Jul 27 14:49:40 mail3 freshclam[23917]: OK Jul 27 14:49:40 mail3 freshclam[23917]: main.inc is up to date (version: 44, sigs: 133163, f-level: 20, builder: sven) Jul 27 14:49:40 mail3 freshclam[23917]: Assuming modification time in the past Jul 27 14:49:40 mail3 freshclam[23917]: If-Modified-Since: Mon, 27 Dec 2004 03:52:10 GMT Jul 27 14:49:40 mail3 freshclam[23917]: Reading CVD header (daily.cvd): Jul 27 14:49:40 mail3 freshclam[23917]: Connected to db.uk.clamav.net (IP: 80.82.245.8). Jul 27 14:49:40 mail3 freshclam[23917]: Trying to retrieve CVD header of http://db.uk.clamav.net/daily.cvd Jul 27 14:49:40 mail3 freshclam[23917]: OK Jul 27 14:49:40 mail3 freshclam[23917]: Retrieving http://db.uk.clamav.net/daily-3782.cdiff Jul 27 14:49:40 mail3 freshclam[23917]: Trying to download http://db.uk.clamav.net/daily-3782.cdiff (IP: 80.82.245.8) Jul 27 14:49:40 mail3 freshclam[23917]: Downloading daily-3782.cdiff [100%] Jul 27 14:49:40 mail3 freshclam[23917]: cdiff_apply: Parsed 14 lines and executed 14 commands Jul 27 14:49:40 mail3 freshclam[23917]: Removing backup directory ./clamav-27a45031a7107ffbb43260b5c43b77e4 Jul 27 14:49:40 mail3 freshclam[23917]: daily.inc updated (version: 3782, sigs: 7595, f-level: 16, builder: sven) Jul 27 14:49:40 mail3 freshclam[23917]: Database updated (140758 signatures) from db.uk.clamav.net (IP: 80.82.245.8) Jul 27 14:49:40 mail3 freshclam[23917]: Clamd successfully notified about the update. Jul 27 14:49:40 mail3 freshclam[23917]: -------------------------------------- Jul 27 15:49:40 mail3 freshclam[23917]: Received signal: wake up Jul 27 15:49:40 mail3 freshclam[23917]: Max retries == 5 Jul 27 15:49:40 mail3 freshclam[23917]: ClamAV update process started at Fri Jul 27 15:49:40 2007 Jul 27 15:49:40 mail3 freshclam[23917]: Querying current.cvd.clamav.net Jul 27 15:49:41 mail3 freshclam[23917]: TTL: 300 Jul 27 15:49:41 mail3 freshclam[23917]: Software version from DNS: 0.91.1 Jul 27 15:49:41 mail3 freshclam[23917]: main.cvd version from DNS: 44 Jul 27 15:49:41 mail3 freshclam[23917]: main.inc is up to date (version: 44, sigs: 133163, f-level: 20, builder: sven) Jul 27 15:49:41 mail3 freshclam[23917]: daily.cvd version from DNS: 3782 Jul 27 15:49:41 mail3 freshclam[23917]: daily.inc is up to date (version: 3782, sigs: 7595, f-level: 16, builder: sven) Jul 27 15:49:41 mail3 freshclam[23917]: -------------------------------------- Jul 27 16:49:40 mail3 freshclam[23917]: Received signal: wake up Jul 27 16:49:40 mail3 freshclam[23917]: Max retries == 5 Jul 27 16:49:40 mail3 freshclam[23917]: ClamAV update process started at Fri Jul 27 16:49:40 2007 Jul 27 16:49:40 mail3 freshclam[23917]: Querying current.cvd.clamav.net Jul 27 16:49:41 mail3 freshclam[23917]: TTL: 300 Jul 27 16:49:41 mail3 freshclam[23917]: Software version from DNS: 0.91.1 Jul 27 16:49:41 mail3 freshclam[23917]: main.cvd version from DNS: 44 Jul 27 16:49:41 mail3 freshclam[23917]: main.inc is up to date (version: 44, sigs: 133163, f-level: 20, builder: sven) Jul 27 16:49:41 mail3 freshclam[23917]: daily.cvd version from DNS: 3783 Jul 27 16:49:41 mail3 freshclam[23917]: Retrieving http://db.uk.clamav.net/daily-3783.cdiff Jul 27 16:49:41 mail3 freshclam[23917]: Trying to download http://db.uk.clamav.net/daily-3783.cdiff (IP: 80.82.245.8) Jul 27 16:49:41 mail3 freshclam[23917]: Downloading daily-3783.cdiff [100%] Jul 27 16:49:41 mail3 freshclam[23917]: cdiff_apply: Parsed 50 lines and executed 50 commands Jul 27 16:49:41 mail3 freshclam[23917]: Removing backup directory ./clamav-21cf422e8d08aaf9d1c81406bb8a6722 Jul 27 16:49:41 mail3 freshclam[23917]: daily.inc updated (version: 3783, sigs: 7636, f-level: 16, builder: ccordes) Jul 27 16:49:41 mail3 freshclam[23917]: Database updated (140799 signatures) from db.uk.clamav.net (IP: 80.82.245.8) Jul 27 16:49:41 mail3 freshclam[23917]: Clamd successfully notified about the update. Jul 27 16:49:41 mail3 freshclam[23917]: -------------------------------------- Jul 27 17:49:41 mail3 freshclam[23917]: Received signal: wake up Jul 27 17:49:41 mail3 freshclam[23917]: Max retries == 5 Jul 27 17:49:41 mail3 freshclam[23917]: ClamAV update process started at Fri Jul 27 17:49:41 2007 Jul 27 17:49:41 mail3 freshclam[23917]: Querying current.cvd.clamav.net Jul 27 17:49:41 mail3 freshclam[23917]: TTL: 300 Jul 27 17:49:41 mail3 freshclam[23917]: Software version from DNS: 0.91.1 Jul 27 17:49:41 mail3 freshclam[23917]: main.cvd version from DNS: 44 Jul 27 17:49:41 mail3 freshclam[23917]: main.inc is up to date (version: 44, sigs: 133163, f-level: 20, builder: sven) Jul 27 17:49:41 mail3 freshclam[23917]: daily.cvd version from DNS: 3783 Jul 27 17:49:41 mail3 freshclam[23917]: daily.inc is up to date (version: 3783, sigs: 7636, f-level: 16, builder: ccordes) Jul 27 17:49:41 mail3 freshclam[23917]: -------------------------------------- Jul 27 18:49:41 mail3 freshclam[23917]: Received signal: wake up Jul 27 18:49:41 mail3 freshclam[23917]: Max retries == 5 Jul 27 18:49:41 mail3 freshclam[23917]: ClamAV update process started at Fri Jul 27 18:49:41 2007 Jul 27 18:49:41 mail3 freshclam[23917]: Querying current.cvd.clamav.net Jul 27 18:49:41 mail3 freshclam[23917]: TTL: 300 Jul 27 18:49:41 mail3 freshclam[23917]: Software version from DNS: 0.91.1 Jul 27 18:49:41 mail3 freshclam[23917]: main.cvd version from DNS: 44 Jul 27 18:49:41 mail3 freshclam[23917]: main.inc is up to date (version: 44, sigs: 133163, f-level: 20, builder: sven) Jul 27 18:49:41 mail3 freshclam[23917]: daily.cvd version from DNS: 3784 Jul 27 18:49:41 mail3 freshclam[23917]: Retrieving http://db.uk.clamav.net/daily-3784.cdiff Jul 27 18:49:41 mail3 freshclam[23917]: Trying to download http://db.uk.clamav.net/daily-3784.cdiff (IP: 80.82.245.8) Jul 27 18:49:41 mail3 freshclam[23917]: Downloading daily-3784.cdiff [100%] Jul 27 18:49:41 mail3 freshclam[23917]: cdiff_apply: Parsed 8 lines and executed 8 commands Jul 27 18:49:41 mail3 freshclam[23917]: Removing backup directory ./clamav-67bf3d2293eb9c8fc5c55e737e17f2c8 Jul 27 18:49:41 mail3 freshclam[23917]: daily.inc updated (version: 3784, sigs: 7638, f-level: 16, builder: diego) Jul 27 18:49:41 mail3 freshclam[23917]: Database updated (140801 signatures) from db.uk.clamav.net (IP: 80.82.245.8) Jul 27 18:49:41 mail3 freshclam[23917]: Clamd successfully notified about the update. Jul 27 18:49:41 mail3 freshclam[23917]: -------------------------------------- Jul 27 19:49:41 mail3 freshclam[23917]: Received signal: wake up Jul 27 19:49:41 mail3 freshclam[23917]: Max retries == 5 Jul 27 19:49:41 mail3 freshclam[23917]: ClamAV update process started at Fri Jul 27 19:49:41 2007 Jul 27 19:49:41 mail3 freshclam[23917]: Querying current.cvd.clamav.net Jul 27 19:49:41 mail3 freshclam[23917]: TTL: 300 Jul 27 19:49:41 mail3 freshclam[23917]: Software version from DNS: 0.91.1 Jul 27 19:49:41 mail3 freshclam[23917]: main.cvd version from DNS: 44 Jul 27 19:49:41 mail3 freshclam[23917]: main.inc is up to date (version: 44, sigs: 133163, f-level: 20, builder: sven) Jul 27 19:49:41 mail3 freshclam[23917]: daily.cvd version from DNS: 3784 Jul 27 19:49:41 mail3 freshclam[23917]: daily.inc is up to date (version: 3784, sigs: 7638, f-level: 16, builder: diego) Jul 27 19:49:41 mail3 freshclam[23917]: -------------------------------------- The clamav-milter log extract below includes updates at 12:49 and 18:49 on Jul 27 (when there were _not_ large changes in memory consumption) as well as 14:49 and 16:49. It isn't very interesting: Jul 27 12:49:45 mail3 clamav-milter[23585]: Database has changed, loading updated database Jul 27 12:49:49 mail3 clamav-milter[23585]: Loaded ClamAV 0.91.1/3781/Fri Jul 27 12:24:10 2007 Jul 27 12:49:49 mail3 clamav-milter[23585]: ClamAV: Protecting against 149504 viruses Jul 27 12:49:50 mail3 clamav-milter[23585]: Database correctly reloaded (149504 viruses) Jul 27 14:49:42 mail3 clamav-milter[23585]: Database has changed, loading updated database Jul 27 14:49:46 mail3 clamav-milter[23585]: Loaded ClamAV 0.91.1/3782/Fri Jul 27 14:21:57 2007 Jul 27 14:49:46 mail3 clamav-milter[23585]: ClamAV: Protecting against 149507 viruses Jul 27 14:49:47 mail3 clamav-milter[23585]: Database correctly reloaded (149507 viruses) Jul 27 16:49:50 mail3 clamav-milter[23585]: Database has changed, loading updated database Jul 27 16:49:55 mail3 clamav-milter[23585]: Loaded ClamAV 0.91.1/3783/Fri Jul 27 16:08:30 2007 Jul 27 16:49:55 mail3 clamav-milter[23585]: ClamAV: Protecting against 149548 viruses Jul 27 16:49:55 mail3 clamav-milter[23585]: Database correctly reloaded (149548 viruses) Jul 27 18:49:41 mail3 clamav-milter[23585]: Database has changed, loading updated database Jul 27 18:49:46 mail3 clamav-milter[23585]: Loaded ClamAV 0.91.1/3784/Fri Jul 27 18:35:18 2007 Jul 27 18:49:46 mail3 clamav-milter[23585]: ClamAV: Protecting against 149550 viruses Jul 27 18:49:46 mail3 clamav-milter[23585]: Database correctly reloaded (149550 viruses) The database didn't change on the odd hours between these log entries. I hope this helps someone... -- 73, Ged. _______________________________________________ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
