Hi Peter, Thanks, these are some great pointers relating to Perl and memory use/profiling.
I'm very much of the school of thought that says when using Perl you should just to things "the Perl way" and not try to optimize memory use too much (avoiding obvious mistakes is about the best you can do), just because the benefits of Perl are to a large extent paid for by a ton of memory use. It's horrible really. :) But, the advantages of Perl are (for me) often enough to make the cost in memory bearable. But, there are times when I wonder - should I even bother to start this Perl program on this set of data? Is there any chance it will finish, or am I going to run out of memory after 4 days of running? That's really why I want to profile the memory use, not to try and optimize anything, but rather to project how much data I can expect to use on whatever hardware I have available. Right now the particular issue I'm looking at is the cost of "windowing" in the Ngram Statistics Package (http://ngram.sourceforge.org). As the window size grows, memory use also grows. But how much? I'm not sure of that. So, last night I set up some runs of NSP in one window and I watched 'top' in the other. :) It wasn't ideal, because from time to time I'd get bored and miss when a program finished. :) Hence my quest for a memory profiling too - to save me from getting bleary eyed staring at top output. Anyway, I'll be sending some results of that little experiment to the NSP list later today I think. The news isn't all bad actually - expensive, but not exponentially so. I think your proposed class sounds great, and yes, in the end if you really need to optimize memory use you really do need to jump over into C - I'd be happy to see a class like that offered, and might even work out a way to get down there for that since that's something I often think about doing but never actually get to figuring out. BTW, I came to the same conclusion about valgrind last night - it looks great for tracking down memory leaks and the like, but not so handy for just tracking memory usage. So I'll give some of these other suggestions a try, and I suppose I could write a Perl script to read top output automatically (that seems kind of twisted, but kind of fun too...) I'll keep you posted, Ted On Wed, Oct 21, 2009 at 10:52 PM, Peter Karman <[email protected]> wrote: > > > > ted_pedersen wrote on 10/21/09 7:46 PM: > > > But, I really don't want to start modifying code to do this...I'd rather > > be able to do something like you can do with execution time, which just > > involves using a command line option... > > > > perl -d:DProf myprogram.pl > > dprofpp > > > > And then you get a nice report like this, which shows where your program > > spent its time. > > > > I've recently converted to NYTProf: > > http://search.cpan.org/~timb/Devel-NYTProf-2.10/ > > which leaps light years over the tried-and-true DProf tool. > > > > > So I want to do the same thing with memory, that is find out where my > > programs are spending their memory... > > > > http://perl.active-venture.com/pod/perldebguts-perlmemory.html > > might prove useful. > > > The closest I've come so far is actually not a Perl specific tool, but > > is rather a generic performance monitoring tool (available on Linux at > > least...) called valgrind, found at > > > > http://valgrind.org <http://valgrind.org> > > > > It's also available via apt-get ... > > > > apt-get install valgrind > > > > I'm not entirely certain what it's giving me though, so I'm still looking. > > valgrind is very useful for finding mem leaks, but I haven't found it as > helpful > for profiling. iirc, it also requires that your Perl is compiled with some > special flags. > > Steven Lembark gave an interesting talk at Frozen Perl this year about memory > management with Perl. What I took away was "use arrays not hashes" if you are > trying to squeeze more out of Perl. I don't see slides but the talk is > described > here: > > http://www.frozen-perl.org/mpw2009/talk/1607 > > In general, though, having spent some significant time recently profiling some > of my own CPAN modules, I wonder why you are concerned about memory use. It's > Perl. Memory use sucks. :) Or are you simply running out of memory in some > cases > and trying to figure out where to cut back? > > NYTProf is excellent in any case, and might provide some clues. > > I've been in touch with MSI about teaching a special Perl class next semester, > and I've been thinking of talking about something along this same topic, i.e., > how to profile and optimize your Perl code, with particular examples using > NYTProf and then rewriting hot spots in C/XS (where of course memory usage is > moreorless under your own control). So if you do find useful ways of measuring > memory usage specifically, I would be very interested to hear. > > -- > Peter Karman . http://peknet.com/ . [email protected] > -- Ted Pedersen http://www.d.umn.edu/~tpederse
