Re: [ Memory ] Re: Thought
On Wed 02 Oct 2002 13:50, H.Merijn Brand [EMAIL PROTECTED] wrote:
As a start for a `normal' interface to the interperters internals, this time
no open hack, but a neat interface.
Very open to additions.
Useful?
Feedback please
Thank you Dan!
Devel::Size from Dan Sugalski is what we all want, don't we?
=head1 NAME
Devel::Size - Perl extension for finding the memory usage of Perl variables
=head1 SYNOPSIS
use Devel::Size qw(size total_size);
my $size = size(A string);
my @foo = (1, 2, 3, 4, 5);
my $other_size = size(\@foo);
my $foo = {a = [1, 2, 3],
b = {a = [1, 3, 4]}
};
my $total_size = total_size($foo);
=head1 DESCRIPTION
This module figures out the real sizes of Perl variables in bytes.
Call functions with a reference to the variable you want the size
of. If the variable is a plain scalar it returns the size of
the scalar. If the variable is a hash or an array, use a reference
when calling.
--
H.Merijn BrandAmsterdam Perl Mongers (http://amsterdam.pm.org/)
using perl-5.6.1, 5.8.0 633 on HP-UX 10.20 11.00, AIX 4.2, AIX 4.3,
WinNT 4, Win2K pro WinCE 2.11. Smoking perl CORE: [EMAIL PROTECTED]
http:[EMAIL PROTECTED]/ [EMAIL PROTECTED]
send smoke reports to: [EMAIL PROTECTED], QA: http://qa.perl.org
Re: [ Memory ] Re: Thought
At 4:45 PM +0200 10/11/02, Tels wrote: -BEGIN PGP SIGNED MESSAGE- Moin, On 11-Oct-02 H.Merijn Brand carved into stone: On Wed 02 Oct 2002 13:50, H.Merijn Brand [EMAIL PROTECTED] wrote: As a start for a `normal' interface to the interperters internals, this time no open hack, but a neat interface. Very open to additions. Useful? Feedback please Thank you Dan! Devel::Size from Dan Sugalski is what we all want, don't we? YES! Juhu! Well, then, you've got it. :) It's mostly done, but if someone wants to take it and finish the rest (It doesn't do formats right, nor does it completely handle IO handles or code refs) I certainly wouldn't mind. -- Dan --it's like this--- Dan Sugalski even samurai [EMAIL PROTECTED] have teddy bears and even teddy bears get drunk
Re: [ Memory ] Re: Thought
Michael G Schwern [EMAIL PROTECTED] writes: But you're not. You're just exposing sbrk(), which is a gory detail. My sbrk man page describes sbrk as being used to find the current location of the program break which means nothing to me. Nor does returns the current memory top. It'll be even worse on an OS which doesn't have sbrk, which means no sbrk(2) man page to look at. A very important point. Does sbrk() just return the current number of bytes allocated to the program? No. It returns the (virtual) address of the end of the data segment. Most (but not all) UNIX malloc()s get memory from the OS by moving this up to make data segment bigger. So differences in sbrk(0) returns indicate how much memory has been grabbed from OS that way (but not necessarily all used). And malloc may use mmap() instead/as well and may (but probably doesn't) return memory to OS ... -- Nick Ing-Simmons http://www.ni-s.u-net.com/
Re: [ Memory ] Re: Thought
H.Merijn Brand [EMAIL PROTECTED] writes: Ahh, someone on /my/ side. I guess most of the discussion boils down to - sbrk () being unknown to people not familiar with unix - Devel::Internals being to wide. [ Gimme another namespace proposal ] - MemUsed interface to polymorphistic So far, all I got was criticism. I asked for it. But no-one said it was useful. (Or I didn't read between the lines enough). Getting sbrk(0) return is occasionaly useful. I am worried though that this isn't really perl's internals but OS's internals. -- Nick Ing-Simmons http://www.ni-s.u-net.com/
Re: [ Memory ] Re: Thought
On Sat 05 Oct 2002 01:45, [EMAIL PROTECTED] wrote: On Fri, Oct 04, 2002 at 08:24:12PM +0200, Slaven Rezic wrote: Not a module, but a function which should work on FreeBSD and Linux: Why not package this up and stick it on CPAN? Proc::Memory or Wait a while. Ann Barcomb is working on this too. Let's collect all this into one module and not release twenty or so that try to do the same. something. It's a good start nitpickthough you'll probably want to do it File::Spec style with Proc::Memory::FreeBSD, etc.../nitpick -- H.Merijn BrandAmsterdam Perl Mongers (http://amsterdam.pm.org/) using perl-5.6.1, 5.8.0 633 on HP-UX 10.20 11.00, AIX 4.2, AIX 4.3, WinNT 4, Win2K pro WinCE 2.11. Smoking perl CORE: [EMAIL PROTECTED] http:[EMAIL PROTECTED]/ [EMAIL PROTECTED] send smoke reports to: [EMAIL PROTECTED], QA: http://qa.perl.org
Re: [ Memory ] Re: Thought
Merijn wrote: Wait a while. Ann Barcomb is working on this too. Let's collect all this into one module and not release twenty or so that try to do the same. I'm not actually working on it...I just had need of such a module Thursday, mentioned to Jos what I had been trying to work on, and he said Hmm, I think Merijn was working on something like that, you should talk to him. So I did, and he forwarded the relevant discussion. I hadn't looked in to how I could solve my question, and because it was given to me as a low priority task, I wasn't sure I was going to have a chance to either. But you can count me as someone who will be very happy about the module :) - Ann
Re: [ Memory ] Re: Thought
En op 5 oktober 2002 sprak Ann Barcomb: I hadn't looked in to how I could solve my question, and because it was given to me as a low priority task, I wasn't sure I was going to have a chance to either. But you can count me as someone who will be very happy about the module :) I noticed CPAN Proc::ProcessTable, which is worth a look, if only to peer at the code and note the portability, er, challenges. If the module that reports memory usage itself consumes significant memory, that is a pest when trying to figure out how much memory a Perl data structure is using. That is the nice thing about Merijn's original sbrk() -- it is lightweight in the extreme. /-\
Re: [ Memory ] Re: Thought
Nicholas Clark [EMAIL PROTECTED] writes: On Thu, Oct 03, 2002 at 02:35:14PM -0400, Benjamin Goldberg wrote: If what it does is merely return the value of the C sbrk() function, then IMHO, sbrk() is a perfectly good name. However, as to other possible names -- how about ProcessSize() ? I'm not sure if this is really a valid description of what sbrk() returns, though. I'm not certain it's as accurate as we might like to think. I am under the impression (quite possibly wrong, as I don't have the source code handy) that 1: Doug Lea's malloc will do anonymous mmap() to satisfy really big requests 2: glib malloc is a variant derived from Doug Lea's code glibc (glib has its own g_malloc function) You're right. With glibc-2.1.2, it seems that malloc falls back to mmap for requests larger than 128K. Which is quite cool and maybe should be implemented in perl's mymalloc. Hence is it plausible that systems exist where the malloc that perl calls may normally get memory from the system with sbrk(), but occasionally use somewhere other than the system heap. So perl code might cause more memory to be used, but if we report back sbrk() as process size we're actually claiming a bit more for our fact than we can actually be sure of. If we call it sbrk() we can divorce ourselves from actually claiming that it is the one true source of memory :-) Not that this is helpful - I guess we want a direct sbrk() interface if sbrk() is on the system, and a process size interface that makes the best estimate it can. But I'm not writing this thing, and so far I guess I've not wanted it *that* much. [mm. maybe I have. number 3 on my hitlist of what slows perl is RAM. So seeing if a data structure change reduces RAM is interesting to me, the encode compiler, and everyone building perl from source] Nicholas Clark Regards, Slaven -- Slaven Rezic - [EMAIL PROTECTED] babybike - routeplanner for cyclists in Berlin handheld (e.g. Compaq iPAQ with Linux) version of bbbike http://bbbike.sourceforge.net
Re: [ Memory ] Re: Thought
Tels [EMAIL PROTECTED] writes: -BEGIN PGP SIGNED MESSAGE- Moin, On 05-Oct-02 [EMAIL PROTECTED] carved into stone: En op 5 oktober 2002 sprak Ann Barcomb: I hadn't looked in to how I could solve my question, and because it was given to me as a low priority task, I wasn't sure I was going to have a chance to either. But you can count me as someone who will be very happy about the module :) I noticed CPAN Proc::ProcessTable, which is worth a look, if only to peer at the code and note the portability, er, challenges. If the module that reports memory usage itself consumes significant memory, that is a pest when trying to figure out how much memory a Perl data structure is using. That is the nice thing about Merijn's original sbrk() -- it is lightweight in the extreme. Yes, but more or less useless, since: * your system might not have it, so you either get nothing, or something incompatible (depending on how it is implemented, or not) in Devel::MemUses * it doesn't reflect the true usage of a perl structure anyway, it just reports a change in the heap, which might or might not be related. With usage of a perl structure, do you mean the size of an SV, plus the size of an XPV, plus the malloced area for the string, etc.? I think the heap usage is more important, because these is the memory taken from the system and not useable for other processes. sbrk *might* be usefull to someone, but I wouldn't use it to find out how much memory a perl structure consumes - the numbers may well be totally wrong. So, I wouldn't use it (part of that is that I didn't understand the man page of sbrk() - which means I would likely interepret the results wrong) (I mean, come on: Calling sbrk with an increment of 0 can be used to find the current location of the program break. Thats gibberish to the average perl programmer like me. It's probably not meaning that the program had a KitKat break and we are trying to locate it via it's breadcrumb trail...) As for ProcessTable, that sounds like a good idea, since looking at top's output is cumbersome. Also, you could (probably) do something like: #!/usr/bin/perl -w use Proc::Memory 'report'; report(); # baseline require Math::BigInt; Math::BigInt-import(); report(); # and now? So you get it in one go (assuming the (nonyetexistant?) Proc::Memory is lightwight :) I assume we need both --- a raw, lightweight interface with sbrk() and some more user-friendly interface (probably using my currmem() function and/or Proc::ProcessTable), but which is probably more inaccurate (because of Heisenberg). Regards, Slaven -- Slaven Rezic - [EMAIL PROTECTED] tkrevdiff - graphical display of diffs between revisions (RCS or CVS) http://ptktools.sourceforge.net/#tkrevdiff
Re: [ Memory ] Re: Thought
[EMAIL PROTECTED] writes:
On Fri, Oct 04, 2002 at 08:24:12PM +0200, Slaven Rezic wrote:
Not a module, but a function which should work on FreeBSD and Linux:
Why not package this up and stick it on CPAN? Proc::Memory or
something.
Because it'a another module to maintain :-) The function is in my
private perl function repository, and I thought about creating a
public function repository on CPAN...
Regards,
Slaven
It's a good start nitpickthough you'll probably want
to do it File::Spec style with Proc::Memory::FreeBSD, etc.../nitpick
=item currmem([$pid])
=for category System
Return ($mem, $realmem) of the current process or process $pid, if $pid
is given.
=cut
sub currmem {
my $pid = shift || $$;
if (open(MAP, /proc/$pid/map)) { # FreeBSD
my $mem = 0;
my $realmem = 0;
while(MAP) {
my(@l) = split /\s+/;
my $delta = (hex($l[1])-hex($l[0]));
$mem += $delta;
if ($l[11] ne 'vnode') {
$realmem += $delta;
}
}
close MAP;
($mem, $realmem);
} elsif (open(MAP, /proc/$pid/maps)) { # Linux
my $mem = 0;
my $realmem = 0;
while(MAP) {
my(@l) = split /\s+/;
my($start,$end) = split /-/, $l[0];
my $delta = (hex($end)-hex($start));
$mem += $delta;
if (!defined $l[5] || $l[5] eq '') {
$realmem += $delta;
}
}
close MAP;
($mem, $realmem);
} else {
undef;
}
}
__END__
--
Michael G Schwern [EMAIL PROTECTED] http://www.pobox.com/~schwern/
Perl Quality Assurance [EMAIL PROTECTED] Kwalitee Is Job One
--
Slaven Rezic - [EMAIL PROTECTED]
BBBike - route planner for cyclists in Berlin
WWW version: http://www.bbbike.de
Perl/Tk version: http://bbbike.sourceforge.net
Re: [ Memory ] Re: Thought
-BEGIN PGP SIGNED MESSAGE- Moin, On 05-Oct-02 Slaven Rezic carved into stone: Tels [EMAIL PROTECTED] writes: -BEGIN PGP SIGNED MESSAGE- Moin, On 05-Oct-02 [EMAIL PROTECTED] carved into stone: En op 5 oktober 2002 sprak Ann Barcomb: I hadn't looked in to how I could solve my question, and because it was given to me as a low priority task, I wasn't sure I was going to have a chance to either. But you can count me as someone who will be very happy about the module :) I noticed CPAN Proc::ProcessTable, which is worth a look, if only to peer at the code and note the portability, er, challenges. If the module that reports memory usage itself consumes significant memory, that is a pest when trying to figure out how much memory a Perl data structure is using. That is the nice thing about Merijn's original sbrk() -- it is lightweight in the extreme. Yes, but more or less useless, since: * your system might not have it, so you either get nothing, or something incompatible (depending on how it is implemented, or not) in Devel::MemUses * it doesn't reflect the true usage of a perl structure anyway, it just reports a change in the heap, which might or might not be related. With usage of a perl structure, do you mean the size of an SV, plus the size of an XPV, plus the malloced area for the string, etc.? I Yes, that is what I thought. think the heap usage is more important, because these is the memory taken from the system and not useable for other processes. But shouldn't that be just the same, or slightly more (if the memory is used in chunks of, let's say 16 bytes, it might alloc up to 15 more).? Or did I understand something wrong? Nevertheless, making your data structures smaller will help, even if in some particulare case it doesn't reduce the heap usage directly. Also, comparing how much mem an array if SVs takes to a string etc would be very usefull (for me). Cheers, Tels - -- perl -MDev::Bollocks -le'print Dev::Bollocks-rand()' revolutionarily customize unique technologies http://bloodgate.com/perl My current Perl projects PGP key available on http://bloodgate.com/tels.asc or via email -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.6 (GNU/Linux) Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl. iQEVAwUBPZ7uCHcLPEOTuEwVAQEjlQf8D40dG0AeWRl/MEmojQuhW/JUAU9C+HKs ihzVjxo/pMKx7sTvHfyag1f6NnOHCwOQBz68zqtJq0OPgQKl29zSmPms38Eb1MAu p+NKVxFFGYuOQAbUZsA9+duwhtkg55kArHeE2TEHkUfNQPR3L14gWybFc1jnLKEU JjWkUZYhTu/MWUxFlrvY5xt+vchGKiLVIz4UIQ5L7BIqY9aMnW0+tpEQwj/nkqaQ G3dkyOEJFAbIeYHPZB9LwEbOSER0x/a2uNzo9EwmjvjBuYa1hpwgPryt/F1Ffvpy nKt037n8XECBPV6dMmaRYg0zb0cysti9PlwU2A4OKnIPg2+6AR9fMQ== =onR3 -END PGP SIGNATURE-
Re: [ Memory ] Re: Thought
Tels [EMAIL PROTECTED] writes: Moin, On 05-Oct-02 Slaven Rezic carved into stone: Tels [EMAIL PROTECTED] writes: -BEGIN PGP SIGNED MESSAGE- Moin, On 05-Oct-02 [EMAIL PROTECTED] carved into stone: En op 5 oktober 2002 sprak Ann Barcomb: I hadn't looked in to how I could solve my question, and because it was given to me as a low priority task, I wasn't sure I was going to have a chance to either. But you can count me as someone who will be very happy about the module :) I noticed CPAN Proc::ProcessTable, which is worth a look, if only to peer at the code and note the portability, er, challenges. If the module that reports memory usage itself consumes significant memory, that is a pest when trying to figure out how much memory a Perl data structure is using. That is the nice thing about Merijn's original sbrk() -- it is lightweight in the extreme. Yes, but more or less useless, since: * your system might not have it, so you either get nothing, or something incompatible (depending on how it is implemented, or not) in Devel::MemUses * it doesn't reflect the true usage of a perl structure anyway, it just reports a change in the heap, which might or might not be related. With usage of a perl structure, do you mean the size of an SV, plus the size of an XPV, plus the malloced area for the string, etc.? I Yes, that is what I thought. think the heap usage is more important, because these is the memory taken from the system and not useable for other processes. But shouldn't that be just the same, or slightly more (if the memory is used in chunks of, let's say 16 bytes, it might alloc up to 15 more).? The malloc system will probably get larger chunks from the underlying virtual memory system (4K or 8K or so). Malloc itself will use smaller chunks (e.g. FreeBSD and glibc malloc use 16 bytes, I think perl's mymalloc uses even smaller chunks). That's fine, but memory gets fragmented. Suppose you make a large 100 byte malloc, then a 1 byte malloc, and finally free the large malloc'ed area. The process' memory usage will still be the same, because the malloc system can't give the large area back to the virtual memory system. This is only possible if the freed area is at the end of the heap. Somewhere in this thread I said that glibc's malloc is cool because it uses mmap() for large memory chunks. In this case, the large chunk won't be allocated on the heap, and after freeing the large chunk the memory may be returned to the system. Or did I understand something wrong? Nevertheless, making your data structures smaller will help, even if in some particulare case it doesn't reduce the heap usage directly. And tuning the malloc system may also help. Also, comparing how much mem an array if SVs takes to a string etc would be very usefull (for me). Cheers, Tels Regards, Slaven -- Slaven Rezic - [EMAIL PROTECTED] babybike - routeplanner for cyclists in Berlin handheld (e.g. Compaq iPAQ with Linux) version of bbbike http://bbbike.sourceforge.net
Re: [ Memory ] Re: Thought
On Sat, Oct 05, 2002 at 03:50:34PM +0200, Tels wrote: Nevertheless, making your data structures smaller will help, even if in some particulare case it doesn't reduce the heap usage directly. Yes, I found this to be true, but on my hitlist of things that make your perl script slow, it was number 3, with raw CPU usage as number 2, and Ops as number 1. [ie play Ops golf, even if you think you'll be using more CPU or more RAM with your way of doing it in less Ops. And unlike conventional perl golf, that's sum total Ops encountered at runtime, not total Ops used] I did a talk about optimising perl code at YAPC in Munich. I'm going to use this as an opportunity to blatantly plug the fact that I've put it online: http://www.ccl4.org/~nick/P/Fast_Enough/ because it might contain one or two ideas that people haven't thought of. People who saw it at YAPC may not want to read it *yet*, because I've not incorporated more bits based the several other good ideas *I* hadn't thought of that other people told me. Nicholas Clark -- Even better than the real thing:http://nms-cgi.sourceforge.net/
Re: [ Memory ] Re: Thought
-BEGIN PGP SIGNED MESSAGE- Moin, On 05-Oct-02 Slaven Rezic carved into stone: Tels [EMAIL PROTECTED] writes: But shouldn't that be just the same, or slightly more (if the memory is used in chunks of, let's say 16 bytes, it might alloc up to 15 more).? The malloc system will probably get larger chunks from the underlying virtual memory system (4K or 8K or so). Malloc itself will use smaller chunks (e.g. FreeBSD and glibc malloc use 16 bytes, I think perl's mymalloc uses even smaller chunks). That's fine, but memory gets fragmented. Suppose you make a large 100 byte malloc, then a 1 byte malloc, and finally free the large malloc'ed area. The process' memory usage will still be the same, because the malloc system can't give the large area back to the virtual memory system. This is only possible if the freed area is at the end of the heap. Yes, but this are implementation specific details, and from the Perl code you can't do anything about this. What you *can* do, though, is to look how large single elements (an SV, an AV etc) are, and then: * use something else in Perl * or try to fix Perl to use less memory. You are still at the mercy of the underlying malloc(), but that *should* take care of all the fragmentation/sizee/chunked issues for you. :) Cheers, Tels - -- perl -MDev::Bollocks -le'print Dev::Bollocks-rand()' adaptively transition total infrastructures http://bloodgate.com/perl My current Perl projects PGP key available on http://bloodgate.com/tels.asc or via email -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.6 (GNU/Linux) Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl. iQEVAwUBPZ9qRXcLPEOTuEwVAQHJ9Af9FUOt9DHcrefS/V55MdoRS50KDjWoe8wu JD5UfmmVHAAoRWp5Y881SoDRY3eoHnIzDfX29xDHz3bIjyeUMJT5rSFYh8gXO8La ljYl64pzYHC0Ebf53FOgHl6DHg3JsB3ljv7QD964t0a61gC5u2w1mhY4mYiuQljH DCM7zyqWZp/PqVWhVvqeURw/8PkdBQC7LCjDSaGVU+T5DjYeZdfdW6tOlD9B8fV/ m4jhWU23ICbneNw+yaqpYMMEbUrNvroBDB0ko/TxF+bAs0XouCta0LhT7SG1/GCA 9sqLbsOtz4ZqR2STZfynLJ0LtdLIJEtCpYkxUXyolsJodWNzx+pE/Q== =QNta -END PGP SIGNATURE-
Re: [ Memory ] Re: Thought
On Thu, Oct 03, 2002 at 02:35:14PM -0400, Benjamin Goldberg wrote: If what it does is merely return the value of the C sbrk() function, then IMHO, sbrk() is a perfectly good name. However, as to other possible names -- how about ProcessSize() ? I'm not sure if this is really a valid description of what sbrk() returns, though. I'm not certain it's as accurate as we might like to think. I am under the impression (quite possibly wrong, as I don't have the source code handy) that 1: Doug Lea's malloc will do anonymous mmap() to satisfy really big requests 2: glib malloc is a variant derived from Doug Lea's code Hence is it plausible that systems exist where the malloc that perl calls may normally get memory from the system with sbrk(), but occasionally use somewhere other than the system heap. So perl code might cause more memory to be used, but if we report back sbrk() as process size we're actually claiming a bit more for our fact than we can actually be sure of. If we call it sbrk() we can divorce ourselves from actually claiming that it is the one true source of memory :-) Not that this is helpful - I guess we want a direct sbrk() interface if sbrk() is on the system, and a process size interface that makes the best estimate it can. But I'm not writing this thing, and so far I guess I've not wanted it *that* much. [mm. maybe I have. number 3 on my hitlist of what slows perl is RAM. So seeing if a data structure change reduces RAM is interesting to me, the encode compiler, and everyone building perl from source] Nicholas Clark -- Even better than the real thing:http://nms-cgi.sourceforge.net/
RE: [ Memory ] Re: Thought
sbrk is very Unixish. It isn't in POSIX at all. Our system is highly POSIX compliant but doesn't have a function that directly maps to sbrk. We do have a way of determining heap max size and heap current usage, but it is a functional (subroutine) interface. If you had a function that just calculated heap usage it would be easier for me to replace it. Thanks PG
Re: [ Memory ] Re: Thought
On Thu 03 Oct 2002 15:19, Green, Paul [EMAIL PROTECTED] wrote:
sbrk is very Unixish.
I know. I'm not targetting Win32 (memory shortage is not an option, but a
feature) or OS/2. I was targetting my own problem/curiousity and probing the
usefulness of making it public.
My initial sbrk probe used
--8---
use Inline C = '
long MemUsed ()
{
return ((long)sbrk (0));
}'
--8---
Then I thought it was very likely that I would use this more than once, so I
turned it into a module, but before I installed it, well, you all saw it
It isn't in POSIX at all. Our system is highly POSIX
compliant but doesn't have a function that directly maps to sbrk. We do
have a way of determining heap max size and heap current usage, but it is a
functional (subroutine) interface. If you had a function that just
calculated heap usage it would be easier for me to replace it.
Thanks
PG
--
H.Merijn BrandAmsterdam Perl Mongers (http://amsterdam.pm.org/)
using perl-5.6.1, 5.8.0 633 on HP-UX 10.20 11.00, AIX 4.2, AIX 4.3,
WinNT 4, Win2K pro WinCE 2.11. Smoking perl CORE: [EMAIL PROTECTED]
http:[EMAIL PROTECTED]/ [EMAIL PROTECTED]
send smoke reports to: [EMAIL PROTECTED], QA: http://qa.perl.org
Re: [ Memory ] Re: Thought
At 01:01 PM 10/3/02 +0200, H.Merijn Brand wrote: So far, all I got was criticism. I asked for it. But no-one said it was useful. (Or I didn't read between the lines enough). Well, since I probably have been the person that set this off, I think I should say something here. Ideally, I think there should be a tool that you give a reference (scalar, list, hash, code or glob) and which will then tell you how many bytes the underlying structure is using within the system. Even better, maybe you should be a able to give it a module stash and it would calculate how much memory is used by that module. Having the ability to find out how much memory the Perl interpreter is using inside the system, is a nice first step towards this. Personally, I would vote against naming it something BSD centric as sbrk(), especially if there are other systems in which you would be able to access similar information, but in a different way. Liz
Re: [ Memory ] Re: Thought
En op 4 oktober 2002 sprak Michael G Schwern: The problem is when you put those two next to each other, one promising a friendly interface, one a bare-metal interface, it confuses the intent of the module. Is it for Joe End User or is it for Joe Core Hacker? A module to report memory usage of any process (not just yourself), is do-able: on some Unices you might trample thru /proc file system; on Windows, the Performance Registry, and so on. But has it been done? I searched CPAN and could not find such a module. And, if you were to write such a module, what should you call it? I would find such a module generally useful in that it gives a sysadmin view of a Perl process (is it a memory pig compared to a Java process, say). Admittedly, such a module may not be very useful in figuring out how much memory a perl data structure is using, but I don't see how sbrk is any better in that regard. For a more fine-grained view, you need hooks into Perl internals (such as the Perl malloc). /-\
RE: [ Memory ] Re: Thought
H.Merijn Brand [mailto:[EMAIL PROTECTED]] writes: So far, all I got was criticism. I asked for it. But no-one said it was useful. (Or I didn't read between the lines enough). In my experience, the phrase I really liked your idea and I hope you won't mind if I share a few ideas for improving it needs to be inserted before almost anything critical that an engineer says. I've just taken to assuming that the fact that people are giving narrow criticisms is a sign they like the overall picture. And my life is now better for realizing these two truths. :-) Kind of like you have to insert the elided question How is the quality, performance, and functional completeness of your source code? before the usual question When will you be done? when your boss shows up in your office. ;-) For the record, I think it is useful to be able to obtain the bytesize of a perl object. Are there systems where the sbrk () value /decreases/ after mallocs? Top-down stacks and heaps a.o.t. bottom-up. Our OS runs on 3 hardware architectures. On two of them, heaps grow up and stacks grow down. On the PA-RISC, the heap grows down and the stack grows up. The same heap allocator runs on all of them. The heap allocator has exactly one trick to make 'inverted' heaps work; it allocates and then frees the largest possible block. From then on, this humongous free block is carved up by the normal allocation algorithm. The size of the inverted heap is a constant value. Our malloc puts a system header before every allocated block that records its size. So, whether you allocate virgin storage or carve up a free block, we can always tell you how big it is. And when you go to free it up, we can verify that it is a pointer to a block of storage that we actually allocated rather than a pointer to random trash. (We also verify that it is still allocated). As I mentioned before, there is no concept of a break address, but you can presume the existance of a subroutine that, given a pointer to a block of malloc'ed storage, can return the number of bytes it occupies. PG
Re: [ Memory ] Re: Thought
On Thu, Oct 03, 2002 at 10:43:43AM +0100, Nicholas Clark wrote: The reason I'm saying it might not be much use to people unfamiliar with the internals of unix is precisely because it does return a precisely defined but not directly useful value - the highest address on the heap. If you read it, call perl code that creates a known perl structure, and read it again, doesn't go up directly and exactly related to the amount of memory that structure needed. Depending on how much other memory there is free, it may not go up at all, or it may go up more if the allocation system grabs a big block of free space. Ok, now I'm really confused. - sbrk() returns, effectively, the size of the heap. - the heap may not go up when you create a perl structure but MemUsed() is shown to be used like this: use Devel::Internals; my @array = (1..1); MemUsed creation of array; my @dup = @array; MemUsed duplication of array; and MemUsed, in scalar context, returns the sbrk value, the size of the heap. So I presume in the above code it simply returns the difference between the sbrk value between the two calls. But Nick just said this isn't actually the memory used, it's just the different in heap size. But the function is called MemUsed! Brane hurt! A lot of the problem here is the expectations from the names. - MemUsed() implies we're going to be able to figure out the amount of memory used by a hunk of Perl code without knowing a lot of magic. Instead, we get something about heap sizes. - sbrk() implies dark memory magic. Most people know it's just some memory thing beyond free() and malloc(). When they see it, they get scared, and rightfully so apparently. The problem is when you put those two next to each other, one promising a friendly interface, one a bare-metal interface, it confuses the intent of the module. Is it for Joe End User or is it for Joe Core Hacker? It would seem this module is for Joe Core Hacker, and it's all about sbrk(). So let's make that clear in the names: NAME Devel::sbrk - Access sbrk values SYNOPSIS use Devel::sbrk; my $end = sbrk (); my @array = (1..1); print The creation of this array used , sbrk () - $end, bytes\n; # Easier use Devel::sbrk; my @array = (1..1); sbrkChange creation of array; my @dup = @array; sbrkChange duplication of array; Ironically, I'm now advocating sbrk() as a name. :) Nick is right, if sbrk() is something special, call it that. With this interface, nobody's going to complain that sbrkChange() didn't actually return the amount of memory used by my @dup = @array; You might want to generalize the concept a bit to a more general heap size module. That way on systems where you don't have sbrk but can query the heap size it will still make sense. I don't know if they are quite the same thing, so I just leave it as a suggestion. NAME Devel::Memory::Heap - Access the size of the memory heap SYNOPSIS use Devel::Memory::Heap; my $end = HeapSize (); my @array = (1..1); print The creation of this array used , HeapSize () - $end, bytes\n; # Easier use Devel::Memory::Heap; my @array = (1..1); HeapChange creation of array; my @dup = @array; HeapChange duplication of array; A more broad Devel::Internals or a more user-friendly Devel::Memory can be left for later, and the functionality of Devel::Memory::Heap can be put into it if it makes sense. -- Michael G. Schwern [EMAIL PROTECTED]http://www.pobox.com/~schwern/ Perl Quality Assurance [EMAIL PROTECTED] Kwalitee Is Job One Here's some scholarly-ass opinions...
Re: [ Memory ] Re: Thought
Michael G Schwern wrote:
On Wed, Oct 02, 2002 at 01:50:56PM +0200, H.Merijn Brand wrote:
SYNOPSIS
use Devel::Internals;
A little broad. Perhaps Devel::Memory?
my $end = sbrk ();
my array = (1..1);
print The creation of this array used ,
sbrk () - $end, bytes\n;
# Easier
use Devel::Internals;
my array = (1..1);
MemUsed creation of array;
my dup = array;
MemUsed duplication of array;
DESCRIPTION
sbrk
returns the current memory top. See sbrk(2) manual page
I realize that sbrk() is a familiar concept to deep C programmers on
BSD-style systems, but in order for this to be useful to Perl
programmers, or even users not familiar with BSD, you're going to have
to name it something more descriptive than sbrk() and explain better
what it does.
If what it does is merely return the value of the C sbrk() function,
then IMHO, sbrk() is a perfectly good name.
However, as to other possible names -- how about ProcessSize() ? I'm
not sure if this is really a valid description of what sbrk() returns,
though.
It can still do the same thing as sbrk(), and you might even want to
leave in an alias to sbrk() so BSD core hackers can feel all warm and
fuzzy.
MemUsed ($msg)
Used in void context reports the number of bytes used
since the previous call to MemUsed with the message passed
Global state is a Bad Thing. Consider when you have two different,
unrelated subroutines both calling MemUsed() in the same program.
foo() calls MemUsed() then bar() then foo() calls MemUsed() again
expecting to get the memory used since it's last call but it's
actually getting the memory used since bar() last called it.
Perhaps MemUsed should keep track of where it was called from?
Eg:
sub MemUsed {
my $sbrk = sbrk();
my $prior = \ $sbrks{join ::, (caller(1))[3, 0] };
if( defined $$prior ) {
my $diff = $sbrk - $$prior;
print ;
}
$$prior = $sbrk;
}
This works fine as long as there is no recursion.
If there *is* recursion, then the user need to stuff the return value of
sbrk() in a variable, or somesuch.
--
How many Monks would a Chipmonk chip,
if a Chipmonk could chip Monks?
Re: [ Memory ] Re: Thought
On Wed, Oct 02, 2002 at 05:07:20PM -0400, Michael G Schwern wrote: On Wed, Oct 02, 2002 at 10:26:32PM +0200, H.Merijn Brand wrote: I realize that sbrk() is a familiar concept to deep C programmers on BSD-style systems, but in order for this to be useful to Perl programmers, or even users not familiar with BSD, you're going to have to name it something more descriptive than sbrk() and explain better what it does. Is it going to be useful (see below) to people who don't know this sort of thing? That's why there /is/ an Internals.pm: to hide the gory details and implementation specifics to the user. But you're not. You're just exposing sbrk(), which is a gory detail. My sbrk man page describes sbrk as being used to find the current location of the program break which means nothing to me. Nor does returns the current memory top. It'll be even worse on an OS which doesn't have sbrk, which means no sbrk(2) man page to look at. Does sbrk() just return the current number of bytes allocated to the program? If it only returns the value from sbrk(), damn well call it sbrk. That's what it is; don't invent a new name. It will only cause confusion later for anyone who does migrate from perl to dealing with unix guts, or needs to talk to someone who does know what sbrk() is. (Oh. we're going to call this concept pels. You may hear other people call them pixels...) The reason I'm saying it might not be much use to people unfamiliar with the internals of unix is precisely because it does return a precisely defined but not directly useful value - the highest address on the heap. If you read it, call perl code that creates a known perl structure, and read it again, doesn't go up directly and exactly related to the amount of memory that structure needed. Depending on how much other memory there is free, it may not go up at all, or it may go up more if the allocation system grabs a big block of free space. However, the function returns something that may not be sbrk() on a system without sbrk(), don't call it sbrk, because it ain't. (And do document what it does return on each different system) Nicholas Clark
Re: [ Memory ] Re: Thought
On Thu 03 Oct 2002 11:43, Nicholas Clark [EMAIL PROTECTED] wrote: On Wed, Oct 02, 2002 at 05:07:20PM -0400, Michael G Schwern wrote: On Wed, Oct 02, 2002 at 10:26:32PM +0200, H.Merijn Brand wrote: I realize that sbrk() is a familiar concept to deep C programmers on BSD-style systems, but in order for this to be useful to Perl programmers, or even users not familiar with BSD, you're going to have to name it something more descriptive than sbrk() and explain better what it does. Is it going to be useful (see below) to people who don't know this sort of thing? That's why there /is/ an Internals.pm: to hide the gory details and implementation specifics to the user. But you're not. You're just exposing sbrk(), which is a gory detail. My sbrk man page describes sbrk as being used to find the current location of the program break which means nothing to me. Nor does returns the current memory top. It'll be even worse on an OS which doesn't have sbrk, which means no sbrk(2) man page to look at. Does sbrk() just return the current number of bytes allocated to the program? If it only returns the value from sbrk(), damn well call it sbrk. That's what it is; don't invent a new name. It will only cause confusion later for anyone who does migrate from perl to dealing with unix guts, or needs to talk to someone who does know what sbrk() is. Ahh, someone on /my/ side. I guess most of the discussion boils down to - sbrk () being unknown to people not familiar with unix - Devel::Internals being to wide. [ Gimme another namespace proposal ] - MemUsed interface to polymorphistic So far, all I got was criticism. I asked for it. But no-one said it was useful. (Or I didn't read between the lines enough). If you don't, I'll just use it for myself, with whatever interface *I* like. If there is some support, I'll change it to fit the more common crowd and put it on CPAN. (Oh. we're going to call this concept pels. You may hear other people call them pixels...) The reason I'm saying it might not be much use to people unfamiliar with the internals of unix is precisely because it does return a precisely defined but not directly useful value - the highest address on the heap. If you read it, call perl code that creates a known perl structure, and read it again, doesn't go up directly and exactly related to the amount of memory that structure needed. Depending on how much other memory there is free, it may not go up at all, or it may go up more if the allocation system grabs a big block of free space. However, the function returns something that may not be sbrk() on a system without sbrk(), don't call it sbrk, because it ain't. (And do document what it does return on each different system) *THAT* my dear sir, I don't know :) You, might know. Are there systems where the sbrk () value /decreases/ after mallocs? Top-down stacks and heaps a.o.t. bottom-up. Nicholas Clark -- H.Merijn BrandAmsterdam Perl Mongers (http://amsterdam.pm.org/) using perl-5.6.1, 5.8.0 633 on HP-UX 10.20 11.00, AIX 4.2, AIX 4.3, WinNT 4, Win2K pro WinCE 2.11. Smoking perl CORE: [EMAIL PROTECTED] http:[EMAIL PROTECTED]/ [EMAIL PROTECTED] send smoke reports to: [EMAIL PROTECTED], QA: http://qa.perl.org
Re: [ Memory ] Re: Thought
On Thu, 03 Oct 2002 13:01:52 +0200, H.Merijn Brand [EMAIL PROTECTED] said: If it only returns the value from sbrk(), damn well call it sbrk. Ahh, someone on /my/ side. Mee too. So far, all I got was criticism. I asked for it. But no-one said it was useful. (Or I didn't read between the lines enough). Sure it looks useful to me, but I'm so happy when I do not have to deal with that stuff that I hope I won't have to use it. But should I ever have the need again (and you cannot anticipate when that happens) I definitely will be happy to have it available. You, might know. Are there systems where the sbrk () value /decreases/ after mallocs? Top-down stacks and heaps a.o.t. bottom-up. You know, my all-time-favorite manpage is on the NeXT computer, where you *have* brk and sbrk, but they return just random data. And the *full* manpage reads: BRK(2) UNIX Programmer's Manual BRK(2) NAME brk, sbrk - change data segment size The UNIX system calls brk and sbrk are not supported on the NeXT computer. :-) -- andreas
Re: [ Memory ] Re: Thought
MemUsed ($msg) Used in void context reports the number of bytes used since the previous call to MemUsed with the message passed Used in scalar context returns the current sbrk value Used in list context returns the values saved on every call Ugh, no, seems very inconsistent to me. And I don't like the implicit print in void context, that's too much kitchensinkish. Having the user write the print or warn explicitely isn't too much asked. How about MemUsed $msg Used in scalar (or void) context returns the current sbrk value. Also stores that value (along with the message) on a stack. Used in list context returns the sbrk and message values saved on every previous scalar call as a list of list refs. MemCheckpoint $msg Used in scalar (or void) context returns the number of bytes since the previous invocation. Also stores that value (along with the message) on a stack. Used in list context returns the incremental values and messages saved on every call as a list of list refs. Roland -- [EMAIL PROTECTED]
RE: [ Memory ] Re: Thought
-BEGIN PGP SIGNED MESSAGE- Moin, On 02-Oct-02 H.Merijn Brand carved into stone: As a start for a `normal' interface to the interperters internals, this time no open hack, but a neat interface. Very open to additions. Useful? Is it accurate? Feedback please NAME Devel-Internals - Perl extension for internal interpreter statistics SYNOPSIS use Devel::Internals; my $end = sbrk (); Can you please use a better name? sbrk() doesn't mean anything to me. my array = (1..1); print The creation of this array used , sbrk () - $end, bytes\n; # Easier use Devel::Internals; my array = (1..1); MemUsed creation of array; MemUsed sounds much better (although I displike CamelHumpSpelling but that's just me :) How about just calling it memory? Cheers, Tels - -- perl -MDev::Bollocks -le'print Dev::Bollocks-rand()' elementarily drive advanced technologies http://bloodgate.com/perl My current Perl projects PGP key available on http://bloodgate.com/tels.asc or via email -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.6 (GNU/Linux) Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl. iQEVAwUBPZsLeHcLPEOTuEwVAQHABQf+MzETTn5bnFnzuZfJiMQarUTHzggIUzSK v4/zZHOJ33stVza1VeeJm7tUU8U/t3QiqX9uG5r+DeiNuxhp6OMaoW+CQYSUJ6NS qnJnZPBzfwF6/NTwU4rDv+NX4Xdi/XQqNVg6qVopRIEDlWdqbqq6diLUsZj2qkRO Ze9teSF7bxpn/rzimHn36UI45TrPP1ha5klEIEFXwOiO01bj4/+1rwznkihIBr08 nshzWzN95D6AJk2+q1K5ON6z50J6UtXdGMF68rBb3oDpAJGBOBbWY54OOtIY3dXg QzA4gZVKjql/csqCa8l3wIKb/CAifzeV5kg/RUkD4JXsroFcihTPpg== =sIJ/ -END PGP SIGNATURE-
Re: [ Memory ] Re: Thought
On Wed, Oct 02, 2002 at 05:06:53PM +0200, Tels wrote: On 02-Oct-02 H.Merijn Brand carved into stone: Going to be a bit hard to change, then :-) As a start for a `normal' interface to the interperters internals, this time no open hack, but a neat interface. Very open to additions. Useful? Is it accurate? Feedback please NAME Devel-Internals - Perl extension for internal interpreter statistics SYNOPSIS use Devel::Internals; my $end = sbrk (); Can you please use a better name? sbrk() doesn't mean anything to me. Trouble is that it's entirely accurate. If you know what sbrk() is in Unix, then you know that this would return it. The value is not directly memory used or anything like that. It's the current sbrk() :-) Nicholas Clark
