Smoke db (was perl-current doesn't compil on MinGw-2.0.0-3)
http://www.alianwebserver.com/cgi-bin/smoke_db updated. Reports from 17500 => 17967. My db server on alianwebserver allow me to only fetch 60k per cgi request, so I've limit to 17500 smoke reports. (don't try "All" with all browse list, it's same problem, it's stop before display). If we can find another server else I will install it elsewhere in 1 month. PS: failed tests from Merijn reports are not parsed for now :-( Regards, -- Alain BABRET
RE: Smoke db (was perl-current doesn't compil on MinGw-2.0.0-3)
-BEGIN PGP SIGNED MESSAGE- Moin, On 03-Oct-02 alian carved into stone: > http://www.alianwebserver.com/cgi-bin/smoke_db updated. > > Reports from 17500 => 17967. > My db server on alianwebserver allow me to only fetch 60k per cgi > request, so I've limit to 17500 smoke reports. > (don't try "All" with all browse list, it's same problem, it's stop > before display). > > If we can find another server else I will install it elsewhere in 1 > month. > > PS: failed tests from Merijn reports are not parsed for now :-( > Regards, I can put it up, but there is limited time to update the cgi-scripts (you need to mail them to me and I need to put them up). So if the code is stable, I can donate the page and the bandwidth (up to a limit, though). Cheers, Tels - -- perl -MDev::Bollocks -le'print Dev::Bollocks->rand()' collaboratively initiate leading-edge partnerships http://bloodgate.com/perl My current Perl projects PGP key available on http://bloodgate.com/tels.asc or via email -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.6 (GNU/Linux) Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl. iQEVAwUBPZzJjXcLPEOTuEwVAQFzKAf8DJPtqxPufwPPdGR2cHIkUGQDRdd0ZEEO gOKu+53jlQ2qnt+cYXxEwIiYoQ8JqMiOQTn7HmKhS4mWALRBCZg53K3tKQmbzjdI w1ERwtY+gy+rTdyX+gqgHB+9Jo+WDwOdPDUgHa9tN7DR5zfUBlXaZrlfHexLSFXV KR+adVoc14JECcMJl6lb7xfln70FQN20cCBKj7J1qHri0dLiPVOhBNBVADHkz609 9PjzwJdE/2GtDWRNJXKZAXittJtdei7Ekol5CR91PYRPVdqRT5Leq+KhZl0Tqjnn nToDOb2505s4Y68Bv8hXhlu9rKjI+nqP1GH/uwin5hcz03DtVyBFMQ== =sWfQ -END PGP SIGNATURE-
RE: [ Memory ] Re: Thought
sbrk is very Unixish. It isn't in POSIX at all. Our system is highly POSIX compliant but doesn't have a function that directly maps to sbrk. We do have a way of determining heap max size and heap current usage, but it is a functional (subroutine) interface. If you had a function that just calculated heap usage it would be easier for me to replace it. Thanks PG
Re: [ Memory ] Re: Thought
On Thu 03 Oct 2002 15:19, "Green, Paul" <[EMAIL PROTECTED]> wrote:
> sbrk is very Unixish.
I know. I'm not targetting Win32 (memory shortage is not an option, but a
feature) or OS/2. I was targetting my own problem/curiousity and probing the
usefulness of making it public.
My initial sbrk probe used
--8<---
use Inline C => '
long MemUsed ()
{
return ((long)sbrk (0));
}'
-->8---
Then I thought it was very likely that I would use this more than once, so I
turned it into a module, but before I installed it, well, you all saw it
> It isn't in POSIX at all. Our system is highly POSIX
> compliant but doesn't have a function that directly maps to sbrk. We do
> have a way of determining heap max size and heap current usage, but it is a
> functional (subroutine) interface. If you had a function that just
> calculated heap usage it would be easier for me to replace it.
>
> Thanks
> PG
--
H.Merijn BrandAmsterdam Perl Mongers (http://amsterdam.pm.org/)
using perl-5.6.1, 5.8.0 & 633 on HP-UX 10.20 & 11.00, AIX 4.2, AIX 4.3,
WinNT 4, Win2K pro & WinCE 2.11. Smoking perl CORE: [EMAIL PROTECTED]
http:[EMAIL PROTECTED]/ [EMAIL PROTECTED]
send smoke reports to: [EMAIL PROTECTED], QA: http://qa.perl.org
Re: [ Memory ] Re: Thought
At 01:01 PM 10/3/02 +0200, H.Merijn Brand wrote: >So far, all I got was criticism. I asked for it. But no-one said it was >useful. >(Or I didn't read between the lines enough). Well, since I probably have been the person that set this off, I think I should say something here. Ideally, I think there should be a tool that you give a reference (scalar, list, hash, code or glob) and which will then tell you how many bytes the underlying structure is using within the system. Even better, maybe you should be a able to give it a module stash and it would calculate how much memory is used by that module. Having the ability to find out how much memory the Perl interpreter is using inside the system, is a nice first step towards this. Personally, I would vote against naming it something BSD centric as sbrk(), especially if there are other systems in which you would be able to access similar information, but in a different way. Liz
Re: Smoke db (was perl-current doesn't compil on MinGw-2.0.0-3)
> I can put it up, but there is limited time to update the cgi-scripts (you > need to mail them to me and I need to put them up). So if the code is > stable, I can donate the page and the bandwidth (up to a limit, though). Many thanks for your offer but I don't think if you can support this: Database must be updated daily. Here how I do only with http: - I've a restricted cgi that receive from me via file upload a sql file that contain new insert statement, one request per line. (new report). - this cgi make a list of $dbh->do on each ligne do add them on db. All reports (1250) take 4mo on Mysql. Script do 6/8 Mysql requests / cgi request. cgi request for only last report take 1.8s (user cpu). cgi request for all reports take 9.6s (user cpu). Memory usage ... don't really know, in worst case I think to 10Mo of data. For update scripts, I think that 3 or 4 day is enough. Can you handle this ? Regards, -- Alain BARBET
Re: [ Memory ] Re: Thought
En op 4 oktober 2002 sprak Michael G Schwern: > The problem is when you put those two next to each other, one > promising a friendly interface, one a bare-metal interface, > it confuses the intent of the module. Is it for Joe End User > or is it for Joe Core Hacker? A module to report memory usage of any process (not just yourself), is do-able: on some Unices you might trample thru /proc file system; on Windows, the Performance Registry, and so on. But has it been done? I searched CPAN and could not find such a module. And, if you were to write such a module, what should you call it? I would find such a module generally useful in that it gives a sysadmin view of a Perl process (is it a memory pig compared to a Java process, say). Admittedly, such a module may not be very useful in figuring out how much memory a perl data structure is using, but I don't see how sbrk is any better in that regard. For a more fine-grained view, you need hooks into Perl internals (such as the Perl malloc). /-\
RE: [ Memory ] Re: Thought
H.Merijn Brand [mailto:[EMAIL PROTECTED]] writes: > So far, all I got was criticism. I asked for it. But no-one said it was useful. > (Or I didn't read between the lines enough). In my experience, the phrase "I really liked your idea and I hope you won't mind if I share a few ideas for improving it" needs to be inserted before almost anything critical that an engineer says. I've just taken to assuming that the fact that people are giving narrow criticisms is a sign they like the overall picture. And my life is now better for realizing these two truths. :-) Kind of like you have to insert the elided question "How is the quality, performance, and functional completeness of your source code?" before the usual question "When will you be done?" when your boss shows up in your office. ;-) For the record, I think it is useful to be able to obtain the bytesize of a perl object. > Are there systems where the sbrk () value /decreases/ after > mallocs? Top-down stacks and heaps a.o.t. bottom-up. Our OS runs on 3 hardware architectures. On two of them, heaps grow up and stacks grow down. On the PA-RISC, the heap grows down and the stack grows up. The same heap allocator runs on all of them. The heap allocator has exactly one trick to make 'inverted' heaps work; it allocates and then frees the largest possible block. From then on, this humongous free block is carved up by the normal allocation algorithm. The "size" of the inverted heap is a constant value. Our malloc puts a system header before every allocated block that records its size. So, whether you allocate virgin storage or carve up a free block, we can always tell you how big it is. And when you go to free it up, we can verify that it is a pointer to a block of storage that we actually allocated rather than a pointer to random trash. (We also verify that it is still allocated). As I mentioned before, there is no concept of a break address, but you can presume the existance of a subroutine that, given a pointer to a block of malloc'ed storage, can return the number of bytes it occupies. PG
Re: [ Memory ] Re: Thought
On Thu, Oct 03, 2002 at 11:39:42AM -0400, Green, Paul wrote: > H.Merijn Brand [mailto:[EMAIL PROTECTED]] writes: > > So far, all I got was criticism. I asked for it. But no-one said it was > useful. > > (Or I didn't read between the lines enough). > > In my experience, the phrase "I really liked your idea and I hope you won't > mind if I share a few ideas for improving it" needs to be inserted before > almost anything critical that an engineer says. I've just taken to assuming > that the fact that people are giving narrow criticisms is a sign they like > the overall picture. And my life is now better for realizing these two > truths. :-) We're Perl programmers. Praise is implicit, like everything else in the language. :) -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One The desired effect is what you get when you improve your interplanetary funksmanship.
Re: [ Memory ] Re: Thought
On Thu, Oct 03, 2002 at 10:43:43AM +0100, Nicholas Clark wrote: > The reason I'm saying it might not be much use to people unfamiliar with > the internals of unix is precisely because it does return a precisely defined > but not directly useful value - the highest address on the heap. > If you read it, call perl code that creates a known perl structure, and read > it again, doesn't go up directly and exactly related to the amount of memory > that structure needed. Depending on how much other memory there is free, it > may not go up at all, or it may go up more if the allocation system grabs a > big block of free space. Ok, now I'm really confused. - sbrk() returns, effectively, the size of the heap. - the heap may not go up when you create a perl structure but MemUsed() is shown to be used like this: use Devel::Internals; my @array = (1..1); MemUsed "creation of array"; my @dup = @array; MemUsed "duplication of array"; and MemUsed, in scalar context, returns the sbrk value, the size of the heap. So I presume in the above code it simply returns the difference between the sbrk value between the two calls. But Nick just said this isn't actually the memory used, it's just the different in heap size. But the function is called MemUsed! Brane hurt! A lot of the problem here is the expectations from the names. - MemUsed() implies we're going to be able to figure out the amount of memory used by a hunk of Perl code without knowing a lot of magic. Instead, we get something about heap sizes. - sbrk() implies dark memory magic. Most people know it's just some memory thing beyond free() and malloc(). When they see it, they get scared, and rightfully so apparently. The problem is when you put those two next to each other, one promising a friendly interface, one a bare-metal interface, it confuses the intent of the module. Is it for Joe End User or is it for Joe Core Hacker? It would seem this module is for Joe Core Hacker, and it's all about sbrk(). So let's make that clear in the names: NAME Devel::sbrk - Access sbrk values SYNOPSIS use Devel::sbrk; my $end = sbrk (); my @array = (1..1); print "The creation of this array used ", sbrk () - $end, "bytes\n"; # Easier use Devel::sbrk; my @array = (1..1); sbrkChange "creation of array"; my @dup = @array; sbrkChange "duplication of array"; Ironically, I'm now advocating sbrk() as a name. :) Nick is right, if sbrk() is something special, call it that. With this interface, nobody's going to complain that sbrkChange() didn't actually return the amount of memory used by "my @dup = @array;" You might want to generalize the concept a bit to a more general "heap size" module. That way on systems where you don't have sbrk but can query the heap size it will still make sense. I don't know if they are quite the same thing, so I just leave it as a suggestion. NAME Devel::Memory::Heap - Access the size of the memory heap SYNOPSIS use Devel::Memory::Heap; my $end = HeapSize (); my @array = (1..1); print "The creation of this array used ", HeapSize () - $end, "bytes\n"; # Easier use Devel::Memory::Heap; my @array = (1..1); HeapChange "creation of array"; my @dup = @array; HeapChange "duplication of array"; A more broad Devel::Internals or a more user-friendly Devel::Memory can be left for later, and the functionality of Devel::Memory::Heap can be put into it if it makes sense. -- Michael G. Schwern <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One Here's some scholarly-ass opinions...
Re: [ Memory ] Re: Thought
Michael G Schwern wrote:
>
> On Wed, Oct 02, 2002 at 01:50:56PM +0200, H.Merijn Brand wrote:
> > SYNOPSIS
> > use Devel::Internals;
>
> A little broad. Perhaps Devel::Memory?
>
> > my $end = sbrk ();
> >
> > my @array = (1..1);
> > print "The creation of this array used ",
> > sbrk () - $end, "bytes\n";
> >
> > # Easier
> > use Devel::Internals;
> >
> > my @array = (1..1);
> > MemUsed "creation of array";
> >
> > my @dup = @array;
> > MemUsed "duplication of array";
> >
> > DESCRIPTION
> > sbrk
> > returns the current memory top. See sbrk(2) manual page
>
> I realize that sbrk() is a familiar concept to deep C programmers on
> BSD-style systems, but in order for this to be useful to Perl
> programmers, or even users not familiar with BSD, you're going to have
> to name it something more descriptive than sbrk() and explain better
> what it does.
If "what it does" is merely return the value of the C sbrk() function,
then IMHO, sbrk() is a perfectly good name.
However, as to other possible names -- how about ProcessSize() ? I'm
not sure if this is really a valid description of what sbrk() returns,
though.
> It can still do the same thing as sbrk(), and you might even want to
> leave in an alias to sbrk() so BSD core hackers can feel all warm and
> fuzzy.
>
> > MemUsed ($msg)
> > Used in void context reports the number of bytes used
> > since the previous call to MemUsed with the message passed
>
> Global state is a Bad Thing. Consider when you have two different,
> unrelated subroutines both calling MemUsed() in the same program.
> foo() calls MemUsed() then bar() then foo() calls MemUsed() again
> expecting to get the memory used since it's last call but it's
> actually getting the memory used since bar() last called it.
Perhaps MemUsed should keep track of where it was called from?
Eg:
sub MemUsed {
my $sbrk = sbrk();
my $prior = \ $sbrks{join "::", (caller(1))[3, 0] };
if( defined $$prior ) {
my $diff = $sbrk - $$prior;
print ;
}
$$prior = $sbrk;
}
This works fine as long as there is no recursion.
If there *is* recursion, then the user need to stuff the return value of
sbrk() in a variable, or somesuch.
--
How many Monks would a Chipmonk chip,
if a Chipmonk could chip Monks?
Re: [ Memory ] Re: Thought
On Wed, Oct 02, 2002 at 05:07:20PM -0400, Michael G Schwern wrote: > On Wed, Oct 02, 2002 at 10:26:32PM +0200, H.Merijn Brand wrote: > > > I realize that sbrk() is a familiar concept to deep C programmers on > > > BSD-style systems, but in order for this to be useful to Perl programmers, > > > or even users not familiar with BSD, you're going to have to name it > > > something more descriptive than sbrk() and explain better what it does. Is it going to be useful (see below) to people who don't know this sort of thing? > > That's why there /is/ an Internals.pm: to hide the gory details and > > implementation specifics to the user. > > But you're not. You're just exposing sbrk(), which is a gory detail. My > sbrk man page describes sbrk as being used to find "the current location of > the program break" which means nothing to me. Nor does "returns the current > memory top". > > It'll be even worse on an OS which doesn't have sbrk, which means no sbrk(2) > man page to look at. > > Does sbrk() just return the current number of bytes allocated to the > program? If it only returns the value from sbrk(), damn well call it sbrk. That's what it is; don't invent a new name. It will only cause confusion later for anyone who does migrate from perl to dealing with unix guts, or needs to talk to someone who does know what sbrk() is. (Oh. we're going to call this concept pels. You may hear other people call them pixels...) The reason I'm saying it might not be much use to people unfamiliar with the internals of unix is precisely because it does return a precisely defined but not directly useful value - the highest address on the heap. If you read it, call perl code that creates a known perl structure, and read it again, doesn't go up directly and exactly related to the amount of memory that structure needed. Depending on how much other memory there is free, it may not go up at all, or it may go up more if the allocation system grabs a big block of free space. However, the function returns something that may not be sbrk() on a system without sbrk(), don't call it sbrk, because it ain't. (And do document what it does return on each different system) Nicholas Clark
Re: [ Memory ] Re: Thought
On Thu 03 Oct 2002 11:43, Nicholas Clark <[EMAIL PROTECTED]> wrote: > On Wed, Oct 02, 2002 at 05:07:20PM -0400, Michael G Schwern wrote: > > On Wed, Oct 02, 2002 at 10:26:32PM +0200, H.Merijn Brand wrote: > > > > > I realize that sbrk() is a familiar concept to deep C programmers on > > > > BSD-style systems, but in order for this to be useful to Perl programmers, > > > > or even users not familiar with BSD, you're going to have to name it > > > > something more descriptive than sbrk() and explain better what it does. > > Is it going to be useful (see below) to people who don't know this sort of > thing? > > > > That's why there /is/ an Internals.pm: to hide the gory details and > > > implementation specifics to the user. > > > > But you're not. You're just exposing sbrk(), which is a gory detail. My > > sbrk man page describes sbrk as being used to find "the current location of > > the program break" which means nothing to me. Nor does "returns the current > > memory top". > > > > It'll be even worse on an OS which doesn't have sbrk, which means no sbrk(2) > > man page to look at. > > > > Does sbrk() just return the current number of bytes allocated to the > > program? > > If it only returns the value from sbrk(), damn well call it sbrk. > That's what it is; don't invent a new name. It will only cause confusion > later for anyone who does migrate from perl to dealing with unix guts, or > needs to talk to someone who does know what sbrk() is. Ahh, someone on /my/ side. I guess most of the discussion boils down to - sbrk () being unknown to people not familiar with unix - Devel::Internals being to wide. [ Gimme another namespace proposal ] - MemUsed interface to polymorphistic So far, all I got was criticism. I asked for it. But no-one said it was useful. (Or I didn't read between the lines enough). If you don't, I'll just use it for myself, with whatever interface *I* like. If there is some support, I'll change it to fit the more common crowd and put it on CPAN. > (Oh. we're going to call this concept pels. You may hear other people call > them pixels...) > > The reason I'm saying it might not be much use to people unfamiliar with > the internals of unix is precisely because it does return a precisely defined > but not directly useful value - the highest address on the heap. > If you read it, call perl code that creates a known perl structure, and read > it again, doesn't go up directly and exactly related to the amount of memory > that structure needed. Depending on how much other memory there is free, it > may not go up at all, or it may go up more if the allocation system grabs a > big block of free space. > > > However, the function returns something that may not be sbrk() on a system > without sbrk(), don't call it sbrk, because it ain't. > > (And do document what it does return on each different system) *THAT* my dear sir, I don't know :) You, might know. Are there systems where the sbrk () value /decreases/ after mallocs? Top-down stacks and heaps a.o.t. bottom-up. > Nicholas Clark -- H.Merijn BrandAmsterdam Perl Mongers (http://amsterdam.pm.org/) using perl-5.6.1, 5.8.0 & 633 on HP-UX 10.20 & 11.00, AIX 4.2, AIX 4.3, WinNT 4, Win2K pro & WinCE 2.11. Smoking perl CORE: [EMAIL PROTECTED] http:[EMAIL PROTECTED]/ [EMAIL PROTECTED] send smoke reports to: [EMAIL PROTECTED], QA: http://qa.perl.org
Re: [ Memory ] Re: Thought
> On Thu, 03 Oct 2002 13:01:52 +0200, "H.Merijn Brand" <[EMAIL PROTECTED]> said: >> If it only returns the value from sbrk(), damn well call it sbrk. > Ahh, someone on /my/ side. Mee too. > So far, all I got was criticism. I asked for it. But no-one said it was useful. > (Or I didn't read between the lines enough). Sure it looks useful to me, but I'm so happy when I do not have to deal with that stuff that I hope I won't have to use it. But should I ever have the need again (and you cannot anticipate when that happens) I definitely will be happy to have it available. > You, might know. Are there systems where the sbrk () value /decreases/ after > mallocs? Top-down stacks and heaps a.o.t. bottom-up. You know, my all-time-favorite manpage is on the NeXT computer, where you *have* brk and sbrk, but they return just random data. And the *full* manpage reads: BRK(2) UNIX Programmer's Manual BRK(2) NAME brk, sbrk - change data segment size The UNIX system calls brk and sbrk are not supported on the NeXT computer. :-) -- andreas
Re: [ Memory ] Re: Thought
[EMAIL PROTECTED] wrote: > For a more fine-grained view, you > need hooks into Perl internals (such as the Perl malloc). This sounds like Devel::Peek::mstat(). But I never looked at this before.
