Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
John Indra [EMAIL PROTECTED] writes: Glad to know that there is no problem with malloc() in -CURRENT. But I still think that this must be addressed in Perl. So maybe, the stock Perl should be built against its own malloc library? No! That would break anything that loads system libraries into Perl, like Authen::PAM, because you'd end up calling system malloc() followed by Perl free(), or the other way around. Please stop pretending this is a FreeBSD bug - it's a bug in Perl, which anally tries to conserve microscopic amounts of memory by growing strings in small increments instead of using the traditional (and far more efficient and elegant) 2n + 1 algorithm. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
On Thu, Mar 14, 2002 at 07:40:04AM +0100, Poul-Henning Kamp wrote: But if somebody wants to try to code this optimization, I'll be more than happy to review the result. I just don't expect it to do much in real-life as opposed to silly benchmark situations. Sorry to start this thread at the first place. Well, the reason that I did was because I had this in my head: What? This little snippet of code runs bloddy slower than in Linux? Boy... I use Perl happily in FreeBSD, and I can't lie to myself that in some situations, I felt like my FreeBSD perl scripts run slower than in Linux. Well... If I write to [EMAIL PROTECTED], those genius developers might catch this and maybe discuss on how to overcome the problem. I don't know whether anything's useful, but at least if I post, more people will notice and PROBABLY can improve a thing or two. I hate to run my scripts under Linux and will do everything I can to make my scripts run in FreeBSD at least as fast as in Linux! This is not just talking about benchmarks. This situation will probably make people conclude things that are not beneficial at all for FreeBSD. People in my definition are non-C-coders and those-who-just-like-to-generalize-things. Exclude me from that list, but there are many of others who fit to that group. I am not complaining and bugging FreeBSD developers to do anything at all. Well, I can't help myself, but at least I have done my part to post this. Thanks... /john I go, I fight, and I win! To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
* Poul-Henning Kamp [EMAIL PROTECTED] [020313 22:43] wrote: But if somebody wants to try to code this optimization, I'll be more than happy to review the result. I just don't expect it to do much in real-life as opposed to silly benchmark situations. Have you thought about issuing a madvise(MADV_WILLNEED) after the brk/mmap call in malloc, at least doing it when it's called via realloc, this might get rid of the superfolous (sp?) page faults that David Greenman reported. -Alfred To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
In message [EMAIL PROTECTED], Alfred Perlstein writes: * Poul-Henning Kamp [EMAIL PROTECTED] [020313 22:43] wrote: But if somebody wants to try to code this optimization, I'll be more than happy to review the result. I just don't expect it to do much in real-life as opposed to silly benchmark situations. Have you thought about issuing a madvise(MADV_WILLNEED) after the brk/mmap call in malloc, at least doing it when it's called via realloc, this might get rid of the superfolous (sp?) page faults that David Greenman reported. It would be much more valuable to add a mremap(void *from, void *to, size_t length); since that can _solve_ the problem in _all_ cases, rather than add more or less byzantine workarounds for silly benchmarks. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
* Poul-Henning Kamp [EMAIL PROTECTED] [020314 01:53] wrote: In message [EMAIL PROTECTED], Alfred Perlstein writes: Have you thought about issuing a madvise(MADV_WILLNEED) after the brk/mmap call in malloc, at least doing it when it's called via realloc, this might get rid of the superfolous (sp?) page faults that David Greenman reported. It would be much more valuable to add a mremap(void *from, void *to, size_t length); since that can _solve_ the problem in _all_ cases, rather than add more or less byzantine workarounds for silly benchmarks. You're right that it would be a better optimization, however it's much more code to write than simply passing a flag down to the code responsible for the allocation especially when you _know_ you'll need it. I may be wrong about MADV_WILLNEED making any difference anyhow. :) -- -Alfred Perlstein [[EMAIL PROTECTED]] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
In message [EMAIL PROTECTED], Alfred Perlstein writes: It would be much more valuable to add a mremap(void *from, void *to, size_t length); since that can _solve_ the problem in _all_ cases, rather than add more or less byzantine workarounds for silly benchmarks. You're right that it would be a better optimization, however it's much more code to write than simply passing a flag down to the code responsible for the allocation especially when you _know_ you'll need it. Since when has going that extra mile to do it right been the wrong thing to do in FreeBSD ? :-) And everybody with VM clue I've asked says it would be trivial to flip two page-table entries, so for all I care it can be mexchangemapping(void *from, void *to, size_t length) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
And everybody with VM clue I've asked says it would be trivial to flip two page-table entries, so for all I care it can be It's going to take a fair bit more than just swapping some page table entries, but it's certainly doable. -DG David Greenman Co-founder, The FreeBSD Project - http://www.freebsd.org President, TeraSolutions, Inc. - http://www.terasolutions.com President, Download Technologies, Inc. - http://www.downloadtech.com Pave the road of life with opportunities. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
:Have you thought about issuing a madvise(MADV_WILLNEED) after the :brk/mmap call in malloc, at least doing it when it's called via :realloc, this might get rid of the superfolous (sp?) page faults :that David Greenman reported. : :It would be much more valuable to add a : mremap(void *from, void *to, size_t length); : :since that can _solve_ the problem in _all_ cases, rather than :add more or less byzantine workarounds for silly benchmarks. : :-- :Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 :[EMAIL PROTECTED] | TCP/IP since RFC 956 It would have to be something like: mremap(from, to/NULL, osize, nsize) We have no information on the original size of the mmap() in the system because the original vm_map_entry may (A) get cut up by mprotect() calls and (B) get combined with adjacent vm_map_entry's. Personally speaking I think code that depends on realloc() to the extent that mremap() would be necessary is broken code. - Also, there are other ways of dealing with this problem. A large malloc() or realloc() can certainly mmap() more space then it immediately needs, and then grow into the space (or reuse the extra space for other purposes if UVM gets tight). - MADV_WILLNEED has no effect on pages that are not already in the VM Page cache. A zero-fill page is not so calling madvise() after [s]brk() or mmap(...MAP_ANON...) will have no effect. The mmap manual page describes this in detail. MADV_FREE can be used in liu of munmap() but phkmalloc does not use it quite this way, phkmalloc will still free the page when the cache is blown out. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
:thing to do in FreeBSD ? :-) : :And everybody with VM clue I've asked says it would be trivial to :flip two page-table entries, so for all I care it can be : : mexchangemapping(void *from, void *to, size_t length) : :-- :Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 :[EMAIL PROTECTED] | TCP/IP since RFC 956 :FreeBSD committer | BSD since 4.3-tahoe Or mcopymap(from, to, length, flags) flags: MAP_SHARED share the same content (we back them with the same VM objects). MAP_PRIVATE copy on write, as if you fork()d and the parent was the original area and the child is the new area. But I'm not volunteering. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
On Thu, 14 Mar 2002, Matthew Dillon wrote: :thing to do in FreeBSD ? :-) : :And everybody with VM clue I've asked says it would be trivial to :flip two page-table entries, so for all I care it can be : : mexchangemapping(void *from, void *to, size_t length) : :-- :Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 :[EMAIL PROTECTED] | TCP/IP since RFC 956 :FreeBSD committer | BSD since 4.3-tahoe Or mcopymap(from, to, length, flags) I'm not sure you want to copy it.. I mean you want the dta di disappear from the old address. It's more a movepages() flags: MAP_SHARED share the same content (we back them with the same VM objects). MAP_PRIVATE copy on write, as if you fork()d and the parent was the original area and the child is the new area. But I'm not volunteering. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
: mcopymap(from, to, length, flags) : :I'm not sure you want to copy it.. :I mean you want the dta di disappear from the old address. :It's more a movepages() MAP_MOVE|MAP_SHARED -Matt : : flags: : : MAP_SHARED share the same content (we back them with the : same VM objects). : : MAP_PRIVATE copy on write, as if you fork()d and the parent : was the original area and the child is the new : area. : : But I'm not volunteering. : : -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
malloc() and the stock Perl in -CURRENT (and -STABLE)
Dear all...
This morning I found a very interesting mail. All of you can see it from:
http://docs.freebsd.org/cgi/getmsg.cgi?fetch=1669241+0+current/freebsd-questions
As stated in the mail, a simple Perl script like this:
-- Begin --
#!/usr/bin/perl
$temp = ;
$begin = time;
for ($i = 0; $i 100; $i++) {
$result .= $i\n;
}
print Exec time: , time - $begin, secs\n;
-- End --
can show that there PROBABLY is something wrong with malloc() in -CURRENT
and -STABLE.
I am just bringing this to a wider audience so maybe this could be a
valueable information for FreeBSD developer and Perl maintainer in FreeBSD.
I heard that Perl in -CURRENT would be updated to Perl 5.6.1, maybe the
maintainer can put the default build for this new version of Perl to use its
own malloc (provided from Perl 5.6.1) IF it turns out that there is nothing
wrong with malloc() in -CURRENT and -STABLE (I don't know cause I am not a C
programmer)...
Thanks...
/john
I go, I fight, and I win!
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
* John Indra [EMAIL PROTECTED] [020313 19:47] wrote: Dear all... This morning I found a very interesting mail. All of you can see it from: http://docs.freebsd.org/cgi/getmsg.cgi?fetch=1669241+0+current/freebsd-questions FreeBSD 5.0 has (being a developer release) has special diagnostics turned on in malloc that causes it to take more time to do allocations. see the malloc(3) manpage and make sure to disable these features if doing benchmarks. I'm sorry I can't find out all the other tweaks needed to get the best performance out of 5.x, perhaps the -doc people can point us the way. niether the tuning(7) manpage nor http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/current-stable.html describe this... -Alfred To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
On Wed, Mar 13, 2002 at 09:28:10PM -0800, Alfred Perlstein wrote: FreeBSD 5.0 has (being a developer release) has special diagnostics turned on in malloc that causes it to take more time to do allocations. But... it DOESN'T only happen in -CURRENT. Even Raistlin Majere [EMAIL PROTECTED] is using a -STABLE or a -RELEASE maybe (he doesn't state his uname output in the message). And to confirm, I have just run the script with FreeBSD 4.4-STABLE of Nov 24th 2001: 44 secs! Awfully slow! (Perl 5.005_03 built from buildworld) And... let me tell you more. I run the script in -CURRENT who has: $ ll /etc/malloc.conf lrwxr-xr-x 1 root wheel 2 Feb 8 14:08 /etc/malloc.conf - aj This is a fresh -CURRENT (Mar 13th 2002) with Perl 5.6.0 built from buildworld. And to clarify things... I don't know what's wrong with malloc() in -CURRENT and -STABLE (and I don't even know whether it's even wrong). All I want is to let the Perl maintainer in -CURRENT and -STABLE to compile the stock Perl with its own malloc library, thus Perl in FreeBSD doesn't suffer from this kind of slowness. Thanks... -Alfred /john I go, I fight, and I win! To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
In message [EMAIL PROTECTED], John Indra writes:
Dear all...
This morning I found a very interesting mail. All of you can see it from:
http://docs.freebsd.org/cgi/getmsg.cgi?fetch=1669241+0+current/freebsd-questions
As stated in the mail, a simple Perl script like this:
-- Begin --
#!/usr/bin/perl
$temp = ;
$begin = time;
for ($i = 0; $i 100; $i++) {
$result .= $i\n;
}
print Exec time: , time - $begin, secs\n;
-- End --
can show that there PROBABLY is something wrong with malloc() in -CURRENT
and -STABLE.
No, there is nothing wrong as such, it is a deliberate design-choice
in phkmalloc() which conflicts with perls use of realloc().
Basically I profiled a lot of programs and found that very few programs
used realloc() and decided that consequently it was the least important
of the entries from a performance point of view.
The above perl program results in a loop more or less like:
n = 2
for (i = 0; i 100; i++)
realloc(n++);
Now, if you read _any_ malloc(3) man page, they will tell you that there
is no way it can be guaranteed that this does not result in a lot of
copying.
(insert bikeshed discussion about wisdom of the above code, assume
I maintain the opinion throughout that it is braindamaged to be
that stupid in a so central program as perl)
At the cost of considerable complexity (a mremap(2) implementation amongst
other things), realloc in phkmalloc(3) can be optimised but it is not
on my plate right now.
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
[EMAIL PROTECTED] | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
The above perl program results in a loop more or less like: n = 2 for (i = 0; i 100; i++) realloc(n++); Now, if you read _any_ malloc(3) man page, they will tell you that there is no way it can be guaranteed that this does not result in a lot of copying. Um, except that copying isn't what is causing the problem. The performance problem is apparantly caused by tens of thousands of page faults per second as the memory is freed and immediately reallocated again from the kernel. Doesn't phkmalloc keep a small pool of allocations around to avoid problems like this? -DG David Greenman Co-founder, The FreeBSD Project - http://www.freebsd.org President, TeraSolutions, Inc. - http://www.terasolutions.com President, Download Technologies, Inc. - http://www.downloadtech.com Pave the road of life with opportunities. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
On Thu, Mar 14, 2002 at 07:04:06AM +0100, Poul-Henning Kamp wrote: At the cost of considerable complexity (a mremap(2) implementation amongst other things), realloc in phkmalloc(3) can be optimised but it is not on my plate right now. Glad to know that there is no problem with malloc() in -CURRENT. But I still think that this must be addressed in Perl. So maybe, the stock Perl should be built against its own malloc library? Thank you for the clarification. /john I go, I fight, and I win! To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
On Thu, Mar 14, 2002 at 12:47:29PM +0700, John Indra wrote: And to clarify things... I don't know what's wrong with malloc() in -CURRENT and -STABLE (and I don't even know whether it's even wrong). All I want is to let the Perl maintainer in -CURRENT and -STABLE to compile the stock Perl with its own malloc library, thus Perl in FreeBSD doesn't suffer from this kind of slowness. phkmalloc is generally pretty efficient..how do you know that switching to the perl internal malloc to optimize this particular usage pattern won't severely pessimize others? Kris msg36055/pgp0.pgp Description: PGP signature
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
:The above perl program results in a loop more or less like:
:...
:
:Now, if you read _any_ malloc(3) man page, they will tell you that there
:is no way it can be guaranteed that this does not result in a lot of
:copying.
:
: Um, except that copying isn't what is causing the problem. The performance
:problem is apparantly caused by tens of thousands of page faults per second as
:the memory is freed and immediately reallocated again from the kernel. Doesn't
:phkmalloc keep a small pool of allocations around to avoid problems like
:this?
:
:-DG
:
:David Greenman
:Co-founder, The FreeBSD Project - http://www.freebsd.org
I can significantly reduce page faulting for the above test
(rewritten in C below) with the following:
rm /etc/malloc.conf
ln -s /etc/malloc.conf
That cuts it down by a factor of 15 to 25.
-Matt
Matthew Dillon
[EMAIL PROTECTED]
#include stdio.h
#include stdlib.h
#include unistd.h
int
main(int ac, char **av)
{
char *ptr = NULL;
int i;
for (i = 2; i 100; ++i)
ptr = realloc(ptr, i);
return(0);
}
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
In message [EMAIL PROTECTED], David Greenman writes:
The above perl program results in a loop more or less like:
n = 2
for (i = 0; i 100; i++)
realloc(n++);
Now, if you read _any_ malloc(3) man page, they will tell you that there
is no way it can be guaranteed that this does not result in a lot of
copying.
Um, except that copying isn't what is causing the problem. The performance
problem is apparantly caused by tens of thousands of page faults per second as
the memory is freed and immediately reallocated again from the kernel. Doesn't
phkmalloc keep a small pool of allocations around to avoid problems like
this?
Yes it does, but it doesn't help here. Basically what happens is
that relloc() is called on to extend a string of one megabyte by
another page, so it allocates 1M+1p and copies the contents over.
Now, in this very particular cornercase, we might be able to optimize
for just being able to allocate the next page, but in all real-world
scenarioes I've seen, real usage is more like:
long loop {
realloc(n++);
do some other stuff involving malloc/free/realloc
}
which negates that optimization.
But if somebody wants to try to code this optimization, I'll be more
than happy to review the result. I just don't expect it to do much
in real-life as opposed to silly benchmark situations.
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
[EMAIL PROTECTED] | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
On Thu, Mar 14, 2002 at 12:47:29PM +0700, John Indra wrote: And to clarify things... I don't know what's wrong with malloc() in -CURRENT and -STABLE (and I don't even know whether it's even wrong). All I want is to let the Perl maintainer in -CURRENT and -STABLE to compile the stock Perl with its own malloc library, thus Perl in FreeBSD doesn't suffer from this kind of slowness. phkmalloc is generally pretty efficient..how do you know that switching to the perl internal malloc to optimize this particular usage pattern won't severely pessimize others? If you read the bug report, you'll see that using perl's malloc results in the program running more than 10 times faster. -DG David Greenman Co-founder, The FreeBSD Project - http://www.freebsd.org President, TeraSolutions, Inc. - http://www.terasolutions.com President, Download Technologies, Inc. - http://www.downloadtech.com Pave the road of life with opportunities. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
On Wed, Mar 13, 2002 at 10:42:08PM -0800, David Greenman wrote: On Thu, Mar 14, 2002 at 12:47:29PM +0700, John Indra wrote: And to clarify things... I don't know what's wrong with malloc() in -CURRENT and -STABLE (and I don't even know whether it's even wrong). All I want is to let the Perl maintainer in -CURRENT and -STABLE to compile the stock Perl with its own malloc library, thus Perl in FreeBSD doesn't suffer from this kind of slowness. phkmalloc is generally pretty efficient..how do you know that switching to the perl internal malloc to optimize this particular usage pattern won't severely pessimize others? If you read the bug report, you'll see that using perl's malloc results in the program running more than 10 times faster. Yah, that program..phk seems to be saying that he believes the malloc usage pattern is atypical for applications in general, so it's conceivable that there are also other common usage patterns within perl which would be optimized by phkmalloc and not by the internal malloc. It should be benchmarked more thoroughly before the switch is made; there's only one datapoint at the moment, which isn't enough to decide whether it's a net win. Kris msg36060/pgp0.pgp Description: PGP signature
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
David Greenman wrote: The above perl program results in a loop more or less like: n = 2 for (i = 0; i 100; i++) realloc(n++); Now, if you read _any_ malloc(3) man page, they will tell you that there is no way it can be guaranteed that this does not result in a lot of copying. Um, except that copying isn't what is causing the problem. The performance problem is apparantly caused by tens of thousands of page faults per second as the memory is freed and immediately reallocated again from the kernel. Doesn't phkmalloc keep a small pool of allocations around to avoid problems like this? It's the other order, since it has to copy the prior contents... the new memory is malloc'ed, and the old is freed. Poul is right that a remmap() would be useful, in order to move only the mappings, leaving the pages intact, to grow the mmap'ed region and relocate it at the same time. Another thing that's common in most malloc implementations that are less picky than phkmalloc is growth by power-of-2 on each allocation, causing the allocated region to double, after the first realloc, on each grow. The best performance that this will get, though, is the same performance that the per process open file table grow gets (i.e. pretty bad). The remmap is a better idea. It might be wedged into the exiting mmap for a malloc of the same object with a larger size (e.g. remapping MAP_ANON memory with a larger size and a non-zero address implies a zero address, same pages, larger allocation, since mapping over an existing mapping makes no sense). -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
On Thu, 2002-03-14 at 01:48, Kris Kennaway wrote: It should be benchmarked more thoroughly before the switch is made; there's only one datapoint at the moment, which isn't enough to decide whether it's a net win. Another thing to watch out for: we now force -Uusemymalloc in perl builds because mixing malloc() implementations can lead to core dumps when a chunk of memory is handed to the wrong version of free() (or realloc()). (A test of this is to use Data::Dumper-Dump() (*not* Dumpxs()! that is in fact the workaround...) to print lots of complex hashes; this fairly reliably makes perl dump core (or sometimes just die with a Bizarre copy of ...) on all our supported platforms when perl's malloc() is used. Of course, that might just be a bug in 5.00503, since I never tried 5.6.x with perl's own malloc()...) -- brandon s. allbery [os/2][linux][solaris][japh] [EMAIL PROTECTED] system administrator [WAY too many hats][EMAIL PROTECTED] electrical and computer engineeringKF8NH carnegie mellon university [better check the oblivious first -ke6sls] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
Fyi: perl-5.7.3/pod/perldelta.pod says Incompatible Changes 64-bit platforms and malloc If your pointers are 64 bits wide, the Perl malloc is no longer being used because it does not work well with 8-byte pointers. Also, usually the system mallocs on such platforms are much better optimized for such large memory models than the Perl malloc. Some memory-hungry Perl applications like the PDL don't work well with Perl's malloc. Finally, other applications than Perl (like modperl) tend to prefer the system malloc. Such platforms include Alpha and 64-bit HPPA, MIPS, PPC, and Sparc. -- Jos Backus _/ _/_/_/Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/_/ [EMAIL PROTECTED] _/_/ _/_/_/use Std::Disclaimer; To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: malloc() and the stock Perl in -CURRENT (and -STABLE)
Terry Lambert wrote: The remmap is a better idea. It might be wedged into the exiting mmap for a malloc of the same object with a larger size (e.g. remapping MAP_ANON memory with a larger size and a non-zero address implies a zero address, same pages, larger allocation, since mapping over an existing mapping makes no sense). Actually, there are a lot of options for this. The first thing that mmap does is convert the len from a size_t into an ssize_t -- from unsigned to signed -- and then checks the sign bit, and disallows the request entirely. If it fit the profile, it could have the sign inverted, and the remmap semantics, instead. Similarly, there is a check for an fd != -1 in the MAP_ANON case, that disallows it if it's not -1. Any non -1 value could be interpreted as special. In addition, you could jam the remap semantics into the flags (e.g. adding a MAP_REMAP), or into the prot flag, if the flags bits were too precious (there're plenty available, actually). Validating that the existing mapping is present and anonymous would be a case of looking it up. Actually, vm_mmap uses NULL and 0 compares on a pointer value interchangalby... style violation, that. Probably, you would need to put the old size into the fd field, if you wanted to be sure that it didn't accidently do the wrong thing... it seems that differentiating one ANON mapping region form one contuguous and following it is rather difficult... depending on the implementaiton of phkmalloc, you might have to constrain the caller to even pages, if you want it to be able to work without perhaps taking a combined split region on a realloc of an alloc'ed area, and breaking it. You could get this information from the (noncontiguous) metadata area, but it would not be general, and would still need to be pushed into the kernel somehow. Given that the length is limited effectively to a signed int, even though the man page lists it as a size_t (unsigned int), passing the old size in the fd doesn't seem that abominable... In the worst case of many consecutive reallocations, you would probably want to check the region for the new mapping to see if it was clear, and do a map add, rather than a remap (technically), you would end uprotoring through three adjacent mapping sets, otherwise (looking for the hole in the map). For a very big allocation, this means that for an N-N+1 allocation delta, N could never exceed 1/3 of the user process address space after the process itself not including the realloc in question, was subtracted out.. plus one page. But that's really a micro-optimization that won't be hit (until the first time someone starts testing the speedup claims... 8-)). -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
