Re: [Ganglia-developers] gmond steadily memory usage increase
Well at least since version 3.3.1. Before that, we have been changing setup's so I don't know. Cheers, - Ramon. On 19-6-2012 18:05, Bernard Li wrote: Hi Ramon: Which version of Ganglia do you start seeing the issue? Thanks, Bernard On Tuesday, June 19, 2012, Ramon Bastiaans wrote: Hi Bernard, This collector daemon gathers metrics from 512 machines. Every node has ~100 metrics, this results in ~50.000 metrics. I do realize this is a somewhat large amount of metrics for 1 gmond and that this can require a lot of memory. Nevertheless I would expect the memory usage to level out at a certain point. We only use modpython on this machine. When I disable the modpython module and move all *pyconf*, the memory usage still climbs over time. I made a little script to print the amount of HOST/METRICs in memory every minute and the memory consumption of gmond. It seems whever there is a (temporary, small) decrease in METRICs in the gmond's XML, the next cycle the memory usage increases. This sounds like something is not free'd / cleaned. See the attached log. Cheers, - Ramon. On 18-6-2012 17:22, Bernard Li wrote: Hi Ramon: That's not good... are you by chance running any metric modules? How many nodes report to this collector? What if you turn off all except for one gmond reporting to it, does it still leak memory? Thanks, Bernard On Mon, Jun 18, 2012 at 7:43 AM, Ramon Bastiaans ramon.bastia...@sara.nl wrote: This also happens with version 3.4.0. - Ramon. On 18-6-2012 16:40, Ramon Bastiaans wrote: Hi, Is anyone still experiencing memory increase over time in gmond? At one of our systems the collector gmond reaches up to 2GB memory if we keep it running long enough, see attached graph. I'm no expert but when I run valgrind on gmond for a few minutes, amongst other things, it says: ==17668== LEAK SUMMARY: ==17668==definitely lost: 22,339 bytes in 50 blocks ==17668==indirectly lost: 2,206,722 bytes in 659 blocks ==17668== possibly lost: 0 bytes in 0 blocks ==17668==still reachable: 154,605 bytes in 2,745 blocks ==17668== suppressed: 0 bytes in 0 blocks ==17668== Reachable blocks (those to which a pointer was found) are not shown. ==17668== To see them, rerun with: --leak-check=full --show-reachable=yes I'm not sure how accurate or correct this is, but is anyone aware of memory usage issues? Kind regards, - Ramon. -- ing. R. Bastiaans, B.ICT * Senior Systems Programmer * Operations, Support and Development SARA Science Park 140 PO Box 94613 1098 XG Amsterdam NL 1090 GP Amsterdam NL P.+31 (0)20 592 3000 F.+31 (0)20 668 3167 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers -- ing. R. Bastiaans, B.ICT * Senior Systems Programmer * Operations, Support and Development SARA Science Park 140 PO Box 94613 1098 XG Amsterdam NL 1090 GP Amsterdam NL P.+31 (0)20 592 3000 F.+31 (0)20 668 3167 -- ing. R. Bastiaans, B.ICT * Senior Systems Programmer * Operations, Support and Development SARA Science Park 140 PO Box 94613 1098 XG Amsterdam NL 1090 GP Amsterdam NL P.+31 (0)20 592 3000 F.+31 (0)20 668 3167 smime.p7s Description: S/MIME Cryptographic Signature -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats.
Re: [Ganglia-developers] gmond steadily memory usage increase
I think this may be this issue: * http://bugzilla.ganglia.info/cgi-bin/bugzilla/show_bug.cgi?id=329 On 21-6-2012 11:38, Ramon Bastiaans wrote: Well at least since version 3.3.1. Before that, we have been changing setup's so I don't know. Cheers, - Ramon. On 19-6-2012 18:05, Bernard Li wrote: Hi Ramon: Which version of Ganglia do you start seeing the issue? Thanks, Bernard On Tuesday, June 19, 2012, Ramon Bastiaans wrote: Hi Bernard, This collector daemon gathers metrics from 512 machines. Every node has ~100 metrics, this results in ~50.000 metrics. I do realize this is a somewhat large amount of metrics for 1 gmond and that this can require a lot of memory. Nevertheless I would expect the memory usage to level out at a certain point. We only use modpython on this machine. When I disable the modpython module and move all *pyconf*, the memory usage still climbs over time. I made a little script to print the amount of HOST/METRICs in memory every minute and the memory consumption of gmond. It seems whever there is a (temporary, small) decrease in METRICs in the gmond's XML, the next cycle the memory usage increases. This sounds like something is not free'd / cleaned. See the attached log. Cheers, - Ramon. On 18-6-2012 17:22, Bernard Li wrote: Hi Ramon: That's not good... are you by chance running any metric modules? How many nodes report to this collector? What if you turn off all except for one gmond reporting to it, does it still leak memory? Thanks, Bernard On Mon, Jun 18, 2012 at 7:43 AM, Ramon Bastiaans ramon.bastia...@sara.nl wrote: This also happens with version 3.4.0. - Ramon. On 18-6-2012 16:40, Ramon Bastiaans wrote: Hi, Is anyone still experiencing memory increase over time in gmond? At one of our systems the collector gmond reaches up to 2GB memory if we keep it running long enough, see attached graph. I'm no expert but when I run valgrind on gmond for a few minutes, amongst other things, it says: ==17668== LEAK SUMMARY: ==17668==definitely lost: 22,339 bytes in 50 blocks ==17668==indirectly lost: 2,206,722 bytes in 659 blocks ==17668== possibly lost: 0 bytes in 0 blocks ==17668==still reachable: 154,605 bytes in 2,745 blocks ==17668== suppressed: 0 bytes in 0 blocks ==17668== Reachable blocks (those to which a pointer was found) are not shown. ==17668== To see them, rerun with: --leak-check=full --show-reachable=yes I'm not sure how accurate or correct this is, but is anyone aware of memory usage issues? Kind regards, - Ramon. -- ing. R. Bastiaans, B.ICT * Senior Systems Programmer * Operations, Support and Development SARA Science Park 140 PO Box 94613 1098 XG Amsterdam NL 1090 GP Amsterdam NL P.+31 (0)20 592 3000 F.+31 (0)20 668 3167 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers -- ing. R. Bastiaans, B.ICT * Senior Systems Programmer * Operations, Support and Development SARA Science Park 140 PO Box 94613 1098 XG Amsterdam NL 1090 GP Amsterdam NL P.+31 (0)20 592 3000 F.+31 (0)20 668 3167 -- ing. R. Bastiaans, B.ICT * Senior Systems Programmer * Operations, Support and Development SARA Science Park 140 PO Box 94613 1098 XG Amsterdam NL 1090 GP Amsterdam NL P.+31 (0)20 592 3000 F.+31 (0)20 668 3167 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint
Re: [Ganglia-developers] gmond steadily memory usage increase
Hi Ramon: Which version of Ganglia do you start seeing the issue? Thanks, Bernard On Tuesday, June 19, 2012, Ramon Bastiaans wrote: Hi Bernard, This collector daemon gathers metrics from 512 machines. Every node has ~100 metrics, this results in ~50.000 metrics. I do realize this is a somewhat large amount of metrics for 1 gmond and that this can require a lot of memory. Nevertheless I would expect the memory usage to level out at a certain point. We only use modpython on this machine. When I disable the modpython module and move all *pyconf*, the memory usage still climbs over time. I made a little script to print the amount of HOST/METRICs in memory every minute and the memory consumption of gmond. It seems whever there is a (temporary, small) decrease in METRICs in the gmond's XML, the next cycle the memory usage increases. This sounds like something is not free'd / cleaned. See the attached log. Cheers, - Ramon. On 18-6-2012 17:22, Bernard Li wrote: Hi Ramon: That's not good... are you by chance running any metric modules? How many nodes report to this collector? What if you turn off all except for one gmond reporting to it, does it still leak memory? Thanks, Bernard On Mon, Jun 18, 2012 at 7:43 AM, Ramon Bastiaans ramon.bastia...@sara.nl wrote: This also happens with version 3.4.0. - Ramon. On 18-6-2012 16:40, Ramon Bastiaans wrote: Hi, Is anyone still experiencing memory increase over time in gmond? At one of our systems the collector gmond reaches up to 2GB memory if we keep it running long enough, see attached graph. I'm no expert but when I run valgrind on gmond for a few minutes, amongst other things, it says: ==17668== LEAK SUMMARY: ==17668==definitely lost: 22,339 bytes in 50 blocks ==17668==indirectly lost: 2,206,722 bytes in 659 blocks ==17668== possibly lost: 0 bytes in 0 blocks ==17668==still reachable: 154,605 bytes in 2,745 blocks ==17668== suppressed: 0 bytes in 0 blocks ==17668== Reachable blocks (those to which a pointer was found) are not shown. ==17668== To see them, rerun with: --leak-check=full --show-reachable=yes I'm not sure how accurate or correct this is, but is anyone aware of memory usage issues? Kind regards, - Ramon. -- ing. R. Bastiaans, B.ICT * Senior Systems Programmer * Operations, Support and Development SARA Science Park 140 PO Box 94613 1098 XG Amsterdam NL 1090 GP Amsterdam NL P.+31 (0)20 592 3000 F.+31 (0)20 668 3167 --**--** -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/**sfrnl04242012/114/50122263/http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ __**_ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/**lists/listinfo/ganglia-**developershttps://lists.sourceforge.net/lists/listinfo/ganglia-developers -- ing. R. Bastiaans, B.ICT * Senior Systems Programmer * Operations, Support and Development SARA Science Park 140 PO Box 94613 1098 XG Amsterdam NL 1090 GP Amsterdam NL P.+31 (0)20 592 3000 F.+31 (0)20 668 3167 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] gmond steadily memory usage increase
This also happens with version 3.4.0. - Ramon. On 18-6-2012 16:40, Ramon Bastiaans wrote: Hi, Is anyone still experiencing memory increase over time in gmond? At one of our systems the collector gmond reaches up to 2GB memory if we keep it running long enough, see attached graph. I'm no expert but when I run valgrind on gmond for a few minutes, amongst other things, it says: ==17668== LEAK SUMMARY: ==17668==definitely lost: 22,339 bytes in 50 blocks ==17668==indirectly lost: 2,206,722 bytes in 659 blocks ==17668== possibly lost: 0 bytes in 0 blocks ==17668==still reachable: 154,605 bytes in 2,745 blocks ==17668== suppressed: 0 bytes in 0 blocks ==17668== Reachable blocks (those to which a pointer was found) are not shown. ==17668== To see them, rerun with: --leak-check=full --show-reachable=yes I'm not sure how accurate or correct this is, but is anyone aware of memory usage issues? Kind regards, - Ramon. -- ing. R. Bastiaans, B.ICT * Senior Systems Programmer * Operations, Support and Development SARA Science Park 140 PO Box 94613 1098 XG Amsterdam NL 1090 GP Amsterdam NL P.+31 (0)20 592 3000 F.+31 (0)20 668 3167 smime.p7s Description: S/MIME Cryptographic Signature -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
