Re: malloc+utrace, tracking memory leaks in a running program.
On 1/10/13 2:38 AM, Konstantin Belousov wrote: On Thu, Jan 10, 2013 at 01:56:48AM -0500, Alfred Perlstein wrote: Here are more convenient links that give diffs against FreeBSD and jemalloc for the proposed changes: FreeBSD: https://github.com/alfredperlstein/freebsd/compare/13e7228d5b83c8fcfc63a0803a374212018f6b68~1...utrace2 Why do you need to expedite the records through the ktrace at all ? Wouldn't direct write(2)s to a file allow for better performance due to not stressing kernel memory allocator and single writing thread ? Also, the malloc coupling to the single-system interface would be prevented. I believe that other usermode tracers also behave in the similar way, using writes and not private kernel interface. Also, what /proc issues did you mentioned ? There is sysctl kern.proc.vmmap which is much more convenient than /proc/pid/map and does not require /proc mounted. jemalloc: https://github.com/alfredperlstein/jemalloc/compare/master...utrace2 Konstantin, you are right, it is a strange thing this utrace. I am not sure why it was done this way. You are correct in that much more efficient system could be made using writes gathered into a single write(2). Do you think there is any reason they may have re-used the kernel paths for ktrace even at the cost of efficiency? About kern.proc.vmmap I will look into that. -Alfred ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc+utrace, tracking memory leaks in a running program.
On Thu, Jan 10, 2013 at 10:16:46AM -0500, Alfred Perlstein wrote: On 1/10/13 2:38 AM, Konstantin Belousov wrote: On Thu, Jan 10, 2013 at 01:56:48AM -0500, Alfred Perlstein wrote: Here are more convenient links that give diffs against FreeBSD and jemalloc for the proposed changes: FreeBSD: https://github.com/alfredperlstein/freebsd/compare/13e7228d5b83c8fcfc63a0803a374212018f6b68~1...utrace2 Why do you need to expedite the records through the ktrace at all ? Wouldn't direct write(2)s to a file allow for better performance due to not stressing kernel memory allocator and single writing thread ? Also, the malloc coupling to the single-system interface would be prevented. I believe that other usermode tracers also behave in the similar way, using writes and not private kernel interface. Also, what /proc issues did you mentioned ? There is sysctl kern.proc.vmmap which is much more convenient than /proc/pid/map and does not require /proc mounted. jemalloc: https://github.com/alfredperlstein/jemalloc/compare/master...utrace2 Konstantin, you are right, it is a strange thing this utrace. I am not sure why it was done this way. You are correct in that much more efficient system could be made using writes gathered into a single write(2). Even without writes gathering, non-coalesced writes should be faster than utrace. Do you think there is any reason they may have re-used the kernel paths for ktrace even at the cost of efficiency? I can only speculate. The utracing of the malloc calls in the context of the ktrace stream is useful for the human reading the trace. Instead of seeing the sequence of unexplanaible calls allocating and freeing memory, you would see something more penetrable. For example, you would see accept/malloc/read/write/free, which could be usefully interpreted as network server serving the client. This context is not needed for a leak detector. About kern.proc.vmmap I will look into that. -Alfred pgpMOJFXTDekU.pgp Description: PGP signature
Re: malloc+utrace, tracking memory leaks in a running program.
On 1/10/13 1:05 PM, Konstantin Belousov wrote:
On Thu, Jan 10, 2013 at 10:16:46AM -0500, Alfred Perlstein wrote:
On 1/10/13 2:38 AM, Konstantin Belousov wrote:
On Thu, Jan 10, 2013 at 01:56:48AM -0500, Alfred Perlstein wrote:
Here are more convenient links that give diffs against FreeBSD and
jemalloc for the proposed changes:
FreeBSD:
https://github.com/alfredperlstein/freebsd/compare/13e7228d5b83c8fcfc63a0803a374212018f6b68~1...utrace2
Why do you need to expedite the records through the ktrace at all ?
Wouldn't direct write(2)s to a file allow for better performance
due to not stressing kernel memory allocator and single writing thread ?
Also, the malloc coupling to the single-system interface would be
prevented.
I believe that other usermode tracers also behave in the similar way,
using writes and not private kernel interface.
Also, what /proc issues did you mentioned ? There is
sysctl kern.proc.vmmap which is much more convenient than /proc/pid/map
and does not require /proc mounted.
jemalloc:
https://github.com/alfredperlstein/jemalloc/compare/master...utrace2
Konstantin, you are right, it is a strange thing this utrace. I am not
sure why it was done this way.
You are correct in that much more efficient system could be made using
writes gathered into a single write(2).
Even without writes gathering, non-coalesced writes should be faster than
utrace.
Do you think there is any reason they may have re-used the kernel paths
for ktrace even at the cost of efficiency?
I can only speculate. The utracing of the malloc calls in the context
of the ktrace stream is useful for the human reading the trace. Instead
of seeing the sequence of unexplanaible calls allocating and freeing
memory, you would see something more penetrable. For example, you would
see accept/malloc/read/write/free, which could be usefully interpreted
as network server serving the client.
This context is not needed for a leak detector.
Now I may be wrong here, but I think it's an artifact of someone
noticing how useful fitting this into the ktrace system and leveraging
existing code.
Even though there are significant performance deficiencies, the actual
utility of the existing framework may have been such a stepping stool
towards tracing that it was just used.
Right now the code already exists, however it logs just {operation,
size, ptr}, example:
malloc, 512, - 0xdeadbeef
free, 0, 0xdeadbeef
realloc, 512, 0 - 0xdeadc0de
realloc, 1024, 0xdeadc0de - 0x
free, 0, 0x
What do you think of just adding the address of the caller of
malloc/free/realloc to these already existing tracepoints?
-Alfred
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc+utrace, tracking memory leaks in a running program.
On Thu, Jan 10, 2013 at 01:29:38PM -0500, Alfred Perlstein wrote:
On 1/10/13 1:05 PM, Konstantin Belousov wrote:
On Thu, Jan 10, 2013 at 10:16:46AM -0500, Alfred Perlstein wrote:
On 1/10/13 2:38 AM, Konstantin Belousov wrote:
On Thu, Jan 10, 2013 at 01:56:48AM -0500, Alfred Perlstein wrote:
Here are more convenient links that give diffs against FreeBSD and
jemalloc for the proposed changes:
FreeBSD:
https://github.com/alfredperlstein/freebsd/compare/13e7228d5b83c8fcfc63a0803a374212018f6b68~1...utrace2
Why do you need to expedite the records through the ktrace at all ?
Wouldn't direct write(2)s to a file allow for better performance
due to not stressing kernel memory allocator and single writing thread ?
Also, the malloc coupling to the single-system interface would be
prevented.
I believe that other usermode tracers also behave in the similar way,
using writes and not private kernel interface.
Also, what /proc issues did you mentioned ? There is
sysctl kern.proc.vmmap which is much more convenient than /proc/pid/map
and does not require /proc mounted.
jemalloc:
https://github.com/alfredperlstein/jemalloc/compare/master...utrace2
Konstantin, you are right, it is a strange thing this utrace. I am not
sure why it was done this way.
You are correct in that much more efficient system could be made using
writes gathered into a single write(2).
Even without writes gathering, non-coalesced writes should be faster than
utrace.
Do you think there is any reason they may have re-used the kernel paths
for ktrace even at the cost of efficiency?
I can only speculate. The utracing of the malloc calls in the context
of the ktrace stream is useful for the human reading the trace. Instead
of seeing the sequence of unexplanaible calls allocating and freeing
memory, you would see something more penetrable. For example, you would
see accept/malloc/read/write/free, which could be usefully interpreted
as network server serving the client.
This context is not needed for a leak detector.
Now I may be wrong here, but I think it's an artifact of someone
noticing how useful fitting this into the ktrace system and leveraging
existing code.
Even though there are significant performance deficiencies, the actual
utility of the existing framework may have been such a stepping stool
towards tracing that it was just used.
Right now the code already exists, however it logs just {operation,
size, ptr}, example:
malloc, 512, - 0xdeadbeef
free, 0, 0xdeadbeef
realloc, 512, 0 - 0xdeadc0de
realloc, 1024, 0xdeadc0de - 0x
free, 0, 0x
What do you think of just adding the address of the caller of
malloc/free/realloc to these already existing tracepoints?
In most real-world applications I saw, malloc() was not a function called
to do the allocation. Usually, there is either an app-specific wrapper,
or the language runtime system which calls malloc(), e.g. the new operator
for the C++ code. Than, the caller address becomes constant for the whole
duration of the program run.
What would be useful is the full backtrace of each allocation. The tools
like libunwind are indeed optimized for this usage pattern.
From this POV, the libc malloc(3) might be better offering a set of the
well-defined hooks for a pluggable tracer to utilize. I am on the fence
there, you could override the malloc/free without hooks, by the ELF symbol
interposing technique, but hooks would also offer features not easily
implementable with the interposing.
pgp3bpBpdU_be.pgp
Description: PGP signature
Re: malloc+utrace, tracking memory leaks in a running program.
On 12/23/12 12:28 PM, Jason Evans wrote: On Dec 21, 2012, at 7:37 PM, Alfred Perlstein bri...@mu.org wrote: So the other day in an effort to debug a memory leak I decided to take a look at malloc+utrace(2) and decided to make a tool to debug where leaks are coming from. A few hours later I have: 1) a new version of utrace(2) (utrace2(2)) that uses structured data to prevent overloading of data. (utrace2.diff) 2) changes to ktrace and kdump to decode the new format. (also in utrace2.diff) 3) changes to jemalloc to include the new format AND the function caller so it's easy to get the source of the leaks. (also in utrace2.diff) 4) a program that can take a pipe of kdump(1) and figure out what memory has leaked. (alloctrace.py) 5) simple test program (test_utrace.c) […] Have you looked at the heap profiling functionality built into jemalloc? It's not currently enabled on FreeBSD, but as far as I know, the only issue keeping it from being useful is the absence of a Linux-compatible /proc/pid/maps (and the gperftools folks may already have a solution for that; I haven't looked). I think it makes more sense to get that sorted out than to develop a separate trace-based leak checker. The problem with tracing is that it doesn't scale beyond some relatively small number of allocator events. I have looked at some of this functionality (heap profiling) but alas it is not implemented yet. In addition the dtrace work appears to be quite away from a workable solution with too many performance penalties until some serious hacking is done. I am just not sure how to proceed, on one hand I do not really have the skill to fix the /proc/pid/maps problem, nor figure out how to get dtrace into the system in any time frame that is reasonable. All a few of us need is the addition of the trace back into the existing utrace framework. Is it time to start installing with some form of debug symbols? This would help us also with dtrace. Re: debug symbols, frame pointers, etc. necessary to make userland dtrace work by default, IMO we should strongly prefer such defaults. It's more reasonable to expect people who need every last bit of performance to remove functionality than to expect people who want to figure out what the system is doing to figure out what functionality to turn on. This is very true. I'm going to continue to work towards this end with a few people and get up to speed on it so that hopefully we can get to this point hopefully in the next release cycle or two. If you have a few moments, can you have a look at the utrace2 branches here: https://github.com/alfredperlstein/freebsd/tree/utrace2 This branch contains the addition of the utrace2 system call which is needed to structure data via utrace(2). The point of this is to avoid kdump(1) needing to discern type of ktrace records based on arbitrary size or other parameters and introduces an extensible protocol for new types of utrace data. The utrace2 branch here augments jemalloc to use utrace2 to pass the old utrace records, but in addition to pass the return address along with the type and size of the allocation: https://github.com/alfredperlstein/jemalloc/tree/utrace2 -Alfred ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc+utrace, tracking memory leaks in a running program.
On 1/10/13 1:41 AM, Alfred Perlstein wrote: On 12/23/12 12:28 PM, Jason Evans wrote: On Dec 21, 2012, at 7:37 PM, Alfred Perlstein bri...@mu.org wrote: So the other day in an effort to debug a memory leak I decided to take a look at malloc+utrace(2) and decided to make a tool to debug where leaks are coming from. A few hours later I have: 1) a new version of utrace(2) (utrace2(2)) that uses structured data to prevent overloading of data. (utrace2.diff) 2) changes to ktrace and kdump to decode the new format. (also in utrace2.diff) 3) changes to jemalloc to include the new format AND the function caller so it's easy to get the source of the leaks. (also in utrace2.diff) 4) a program that can take a pipe of kdump(1) and figure out what memory has leaked. (alloctrace.py) 5) simple test program (test_utrace.c) […] Have you looked at the heap profiling functionality built into jemalloc? It's not currently enabled on FreeBSD, but as far as I know, the only issue keeping it from being useful is the absence of a Linux-compatible /proc/pid/maps (and the gperftools folks may already have a solution for that; I haven't looked). I think it makes more sense to get that sorted out than to develop a separate trace-based leak checker. The problem with tracing is that it doesn't scale beyond some relatively small number of allocator events. I have looked at some of this functionality (heap profiling) but alas it is not implemented yet. In addition the dtrace work appears to be quite away from a workable solution with too many performance penalties until some serious hacking is done. I am just not sure how to proceed, on one hand I do not really have the skill to fix the /proc/pid/maps problem, nor figure out how to get dtrace into the system in any time frame that is reasonable. All a few of us need is the addition of the trace back into the existing utrace framework. Is it time to start installing with some form of debug symbols? This would help us also with dtrace. Re: debug symbols, frame pointers, etc. necessary to make userland dtrace work by default, IMO we should strongly prefer such defaults. It's more reasonable to expect people who need every last bit of performance to remove functionality than to expect people who want to figure out what the system is doing to figure out what functionality to turn on. This is very true. I'm going to continue to work towards this end with a few people and get up to speed on it so that hopefully we can get to this point hopefully in the next release cycle or two. If you have a few moments, can you have a look at the utrace2 branches here: https://github.com/alfredperlstein/freebsd/tree/utrace2 This branch contains the addition of the utrace2 system call which is needed to structure data via utrace(2). The point of this is to avoid kdump(1) needing to discern type of ktrace records based on arbitrary size or other parameters and introduces an extensible protocol for new types of utrace data. The utrace2 branch here augments jemalloc to use utrace2 to pass the old utrace records, but in addition to pass the return address along with the type and size of the allocation: https://github.com/alfredperlstein/jemalloc/tree/utrace2 -Alfred Jason, Here are more convenient links that give diffs against FreeBSD and jemalloc for the proposed changes: FreeBSD: https://github.com/alfredperlstein/freebsd/compare/13e7228d5b83c8fcfc63a0803a374212018f6b68~1...utrace2 jemalloc: https://github.com/alfredperlstein/jemalloc/compare/master...utrace2 -Alfred ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc+utrace, tracking memory leaks in a running program.
On Thu, Jan 10, 2013 at 01:56:48AM -0500, Alfred Perlstein wrote: Here are more convenient links that give diffs against FreeBSD and jemalloc for the proposed changes: FreeBSD: https://github.com/alfredperlstein/freebsd/compare/13e7228d5b83c8fcfc63a0803a374212018f6b68~1...utrace2 Why do you need to expedite the records through the ktrace at all ? Wouldn't direct write(2)s to a file allow for better performance due to not stressing kernel memory allocator and single writing thread ? Also, the malloc coupling to the single-system interface would be prevented. I believe that other usermode tracers also behave in the similar way, using writes and not private kernel interface. Also, what /proc issues did you mentioned ? There is sysctl kern.proc.vmmap which is much more convenient than /proc/pid/map and does not require /proc mounted. jemalloc: https://github.com/alfredperlstein/jemalloc/compare/master...utrace2 -Alfred ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org pgp3u3h6Pt05o.pgp Description: PGP signature
Re: malloc+utrace, tracking memory leaks in a running program.
On Dec 21, 2012, at 7:37 PM, Alfred Perlstein bri...@mu.org wrote: So the other day in an effort to debug a memory leak I decided to take a look at malloc+utrace(2) and decided to make a tool to debug where leaks are coming from. A few hours later I have: 1) a new version of utrace(2) (utrace2(2)) that uses structured data to prevent overloading of data. (utrace2.diff) 2) changes to ktrace and kdump to decode the new format. (also in utrace2.diff) 3) changes to jemalloc to include the new format AND the function caller so it's easy to get the source of the leaks. (also in utrace2.diff) 4) a program that can take a pipe of kdump(1) and figure out what memory has leaked. (alloctrace.py) 5) simple test program (test_utrace.c) […] Have you looked at the heap profiling functionality built into jemalloc? It's not currently enabled on FreeBSD, but as far as I know, the only issue keeping it from being useful is the absence of a Linux-compatible /proc/pid/maps (and the gperftools folks may already have a solution for that; I haven't looked). I think it makes more sense to get that sorted out than to develop a separate trace-based leak checker. The problem with tracing is that it doesn't scale beyond some relatively small number of allocator events. Is it time to start installing with some form of debug symbols? This would help us also with dtrace. Re: debug symbols, frame pointers, etc. necessary to make userland dtrace work by default, IMO we should strongly prefer such defaults. It's more reasonable to expect people who need every last bit of performance to remove functionality than to expect people who want to figure out what the system is doing to figure out what functionality to turn on. Thanks, Jason ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc+utrace, tracking memory leaks in a running program.
On 12/23/12 9:28 AM, Jason Evans wrote: On Dec 21, 2012, at 7:37 PM, Alfred Perlstein bri...@mu.org wrote: So the other day in an effort to debug a memory leak I decided to take a look at malloc+utrace(2) and decided to make a tool to debug where leaks are coming from. A few hours later I have: 1) a new version of utrace(2) (utrace2(2)) that uses structured data to prevent overloading of data. (utrace2.diff) 2) changes to ktrace and kdump to decode the new format. (also in utrace2.diff) 3) changes to jemalloc to include the new format AND the function caller so it's easy to get the source of the leaks. (also in utrace2.diff) 4) a program that can take a pipe of kdump(1) and figure out what memory has leaked. (alloctrace.py) 5) simple test program (test_utrace.c) […] Have you looked at the heap profiling functionality built into jemalloc? It's not currently enabled on FreeBSD, but as far as I know, the only issue keeping it from being useful is the absence of a Linux-compatible /proc/pid/maps (and the gperftools folks may already have a solution for that; I haven't looked). I think it makes more sense to get that sorted out than to develop a separate trace-based leak checker. The problem with tracing is that it doesn't scale beyond some relatively small number of allocator events. Ok, we are in agreement on this all. Paul Saab recommended profiling to me, but yes, the problem is that none of this stuff works on FreeBSD out of the box due to missing bits here or there. Augmenting the existing utrace stuff to get what I needed seemed much simpler than figuring out how to get dtrace, pidmaps and whatnot into the system. It's a matter of the requirements to accomplish these higher order things requires X=(skill+time+ability_to_socialize_these_changes) where X alfred-skill_and_time_and_socialize(). :) To be honest, if dtrace just worked, then I could get the same information I'm getting from utrace2(2) from dtrace with no problem. (at least I think so). As far as scaling it, I agree it does not work for long running programs, however there are a few instances of programs leaking large memory in a short while that I can track down by temporarily ktracing for short while. Is it time to start installing with some form of debug symbols? This would help us also with dtrace. Re: debug symbols, frame pointers, etc. necessary to make userland dtrace work by default, IMO we should strongly prefer such defaults. It's more reasonable to expect people who need every last bit of performance to remove functionality than to expect people who want to figure out what the system is doing to figure out what functionality to turn on. Yes!!! :) Is there an easy way to go about this? Rui says it's really a matter of just turning off stripping of shlibs and adding -fno-omit-frame-pointer and WITH_CTF. I'm going to give this a shot, if it works, can you help me refine this? I'll post diffs later today if I don't get completely stuck somehow. -Alfred ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc+utrace, tracking memory leaks in a running program.
On 21 December 2012 22:37, Alfred Perlstein bri...@mu.org wrote: Is it time to start installing with some form of debug symbols? This would help us also with dtrace. I just posted a patch to add a knob to build and install standalone debug files. My intent is that we will build releases with this enabled, and add a base-dbg.txz distribution that contains the debug data for the base system, so that one can install it along with everything else, or add it later on when needed. We could perhaps teach dtrace to read its data from standalone .ctf files, or have it read DWARF directly and use the same debug files. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc+utrace, tracking memory leaks in a running program.
On 12/22/12 8:56 AM, Ed Maste wrote: On 21 December 2012 22:37, Alfred Perlstein bri...@mu.org wrote: Is it time to start installing with some form of debug symbols? This would help us also with dtrace. I just posted a patch to add a knob to build and install standalone debug files. My intent is that we will build releases with this enabled, and add a base-dbg.txz distribution that contains the debug data for the base system, so that one can install it along with everything else, or add it later on when needed. We could perhaps teach dtrace to read its data from standalone .ctf files, or have it read DWARF directly and use the same debug files. Thank you. Added CC'd Rui Paulo. Rui, do you think it's easy to get dtrace to honor these conventions? -Alfred ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc pages map to user space
On Thursday, March 22, 2012 3:57:11 pm Eric Saint-Etienne wrote: If your kernel module creates a device in /dev that implements the mmap method, then you don't need to worry about mucking around with vm_maps and objects and whatnot. Your mmap method just needs to be able to convert offsets into the device into physical memory addresses, Yes I'm aware of this facility, thank you. and the vm infrastructure will do the rest for you. Since this mapping is on the main path of the driver, I'm worried that the overhead on each access of a page fault and a function call (the pager associated with a cdev mmap) is too much to bear. It only does this on the first page fault though, not every access to the page. This can be a bit of a downfall as well as you can't easily invalidate a mapping once you've established it. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc pages map to user space
I've refined the behaviour I observe, which isn't consistent depending on the size one mallocates. (see interleaved comments) In my driver, I need to map some malloc-ed memory, obtained from another module, into userspace. The problem: on the smaller mallocs, as well as on some bigeer ones (8k seems fine, 64k fails): vm_map_lookup() fails finding the underlying vm object. In the current implementation I'm calling vm_map_lookup() against the kmem_map. As a result it either return the kmem_object, or fails at all (for smaller or larger mallocs.) Do somebody know how (or better, have a piece of code!) to retrieve the vm_object associated with malloc-ed memory? (small and big ones) As far as I can see in the vm code, there isn't any object associated with the slabs (the smaller mallocs), it seems that a huge chunk of virtual space is used as is, so I presume the virtual addresses where the SLABs are have some remarkable property, with respect to physical addresses, that could allow creating an object from scratch? By using kernel_map instead of kmem_map, vm_map_lookup() now always return a vm_object. That's a big progress. As expected, when this object is kmem_object, the user mapping works fine (for smaller or larger mallocs.) Otherwise that object doesn't match kernel_object. It's an anonymous object to me. Using that anonymous vm_object for mapping into user map (using vm_map_find()) doesn't directly fail, it does provide a virtual address in the user map. However I read zeros at that address, from within the user process. Any help would be highly appreciated. Thanks! Eric ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc pages map to user space
On Wednesday, March 21, 2012 7:27:58 pm Eric Saint-Etienne wrote: Hi, From within the freeBSD kernel, not all malloc are made equal: * malloc() smaller than KMEM_ZMAX (set to one page size) end up in UMA SLABs, themselves laid out in powers of 2 (from 16 bytes, 32... to 4096 bytes) * bigger malloc() are done through uma_large_malloc() which uses the kmem wired space In my driver, I need to map some malloc-ed memory, obtained from another module, into userspace. The problem: on the smaller mallocs, as well as on some bigeer ones (8k seems fine, 64k fails): vm_map_lookup() fails finding the underlying vm object. Do somebody know how (or better, have a piece of code!) to retrieve the vm_object associated with malloc-ed memory? (small and big ones) As far as I can see in the vm code, there isn't any object associated with the slabs (the smaller mallocs), it seems that a huge chunk of virtual space is used as is, so I presume the virtual addresses where the SLABs are have some remarkable property, with respect to physical addresses, that could allow creating an object from scratch? The usual answer is: use mmap(). It seems mmap() is the solution to everything. But what I dislike with mmap() is the following cost *for each page*: 1/ a page fault 2/ a call to a pager function that will do the on demand mapping. You can prefault each page in userland after you call mmap() if you want to pay the cost up front vs. later during runtime. However, there is another option you can use (though it might require you to rework your interfaces a bit, esp. depending on how you get the malloc'd memory in the first place). Specifically, I recently added an extension to the 'shm_open()' API in the kernel that lets you map a shared memory object into the kernel (shm_map() and shm_unmap(), the change should be easy to MFC to 8 and 9 (I use it in 8 and will MFC it soonish to those stable branches)). The way the workflow works in this case is that userland creates a shared memory object (usually an anonymous one, so using SHM_ANON), and then passes the fd in as part of an ioctl request to something in the kernel. The kernel code uses fget() to convert the fd to a reference to a 'struct file *'. You can then pass that file to shm_map(). Once the kernel code is done with the mapping it should call shm_unmap() to release it as well as using fdrop() to release the reference to the 'fp' obtained from fget(). shm_map() will wire down pages for the mapped region of the shm (and alloc them if needed). As a bonus then, if you mmap() the shm in userland after the ioctl, you can pass MAP_PREFAULT_READ to mmap() and it should remove all the page faults in userland. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc pages map to user space
By using kernel_map instead of kmem_map, vm_map_lookup() now always return a vm_object. That's a big progress. As expected, when this object is kmem_object, the user mapping works fine (for smaller or larger mallocs.) Otherwise that object doesn't match kernel_object. It's an anonymous object to me. Using that anonymous vm_object for mapping into user map (using vm_map_find()) doesn't directly fail, it does provide a virtual address in the user map. However I read zeros at that address, from within the user process. Actually when using kernel_map, the object returned is NULL! However the the vm_entry_t it returns seems a valid address, its 'object' field is NULL too (that's consistent) That's the reason why I didn't find it in any existing 'puclic' map (such as kernel_map, buffers_map, kmem_map, exec_map or pipe_map) But a NULL object isn't good at anything and I'm not sure what to do with a vm_entry_t only... Any idea how to insert it in the process map? ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc pages map to user space
On Thu, Mar 22, 2012 at 10:42 AM, Eric Saint-Etienne eric.saintetie...@gmail.com wrote: Actually when using kernel_map, the object returned is NULL! However the the vm_entry_t it returns seems a valid address, its 'object' field is NULL too (that's consistent) That's the reason why I didn't find it in any existing 'puclic' map (such as kernel_map, buffers_map, kmem_map, exec_map or pipe_map) But a NULL object isn't good at anything and I'm not sure what to do with a vm_entry_t only... Any idea how to insert it in the process map? If your kernel module creates a device in /dev that implements the mmap method, then you don't need to worry about mucking around with vm_maps and objects and whatnot. Your mmap method just needs to be able to convert offsets into the device into physical memory addresses, and the vm infrastructure will do the rest for you. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc pages map to user space
If your kernel module creates a device in /dev that implements the mmap method, then you don't need to worry about mucking around with vm_maps and objects and whatnot. Your mmap method just needs to be able to convert offsets into the device into physical memory addresses, Yes I'm aware of this facility, thank you. and the vm infrastructure will do the rest for you. Since this mapping is on the main path of the driver, I'm worried that the overhead on each access of a page fault and a function call (the pager associated with a cdev mmap) is too much to bear. So I'd like to do it the hard way which I feel is the most optimized. Thanks! ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc pages map to user space
Here is some code which fails with malloc 1 page
and sometimes succeeds with large mallocs ( 16 pages)
What's wrong?
#include sys/param.h
#include sys/proc.h
#include sys/conf.h
#include sys/module.h
#include sys/sysent.h
#include sys/kernel.h
#include sys/systm.h
#include sys/sysproto.h
#include sys/resourcevar.h
// Copyright: skeleton took from an older post on the freebsd list
#include vm/vm.h
#include vm/pmap.h
#include vm/vm_map.h
#include vm/vm_extern.h
#include vm/vm_param.h
#include vm/vm_kern.h
#include vm/vm_object.h
#include vm/vm_page.h
MALLOC_DEFINE(M_FIVEG_SYSC, fiveg_sysc, fiveg_sysc test);
struct args {
unsigned char **p;
};
/* String to be located in maped buffer */
#define SIZE PAGE_SIZE // 1 page always fail
static void initialize_array(char *p) {
int i;
for (i = 0; i 26; i++)
p[i] = i+'a';
p[26] = '!';
p[27] = '\0';
}
static vm_offset_t addr; // allocated/freed at module load/unload
/* Syscall func */
static int syscf(struct thread *td, void *sa) {
vm_offset_t user_addr; /* User space address */
struct args *uap = (struct args*) sa;
struct proc *procp = (struct proc *)td-td_proc;
struct vmspace *vms = procp-p_vmspace;
vm_map_t map;
int result;
vm_object_t object;
vm_ooffset_tobjoffset;
vm_map_entry_t entry;
vm_pindex_t pindex;
vm_prot_t prot;
boolean_t wired;
map = kernel_map; // it always return data within kmeme anyway
uprintf(KERNEL string is '%s' (%p)\n, (char*) addr, (void*) addr);
result = vm_map_lookup(map, addr, VM_PROT_ALL, entry, object,
pindex, prot, wired);
if (result != KERN_SUCCESS) {
uprintf(KERNEL vm_map_lookup failed (%d)\n, result);
return ENOMEM;
}
vm_map_lookup_done(map, entry);
if (object == kernel_object) uprintf(object is kernel_object\n);
else if (object == kmem_object) uprintf(object is kmem_object\n);
else uprintf(object=%p (not kmem, not kernel)\n, object);
uprintf(entry=%p\n, entry);
/* Offset in vm_object */
objoffset = addr - entry-start + entry-offset;
user_addr = 0;
result = vm_map_find(vms-vm_map, object, objoffset, (vm_offset_t
*) user_addr, SIZE, VMFS_ANY_SPACE, VM_PROT_RW, VM_PROT_RW, 0);
if (result != KERN_SUCCESS)
uprintf(vm_map_find failed: %d\n, result);
else {
*uap-p = (char*) user_addr;
uprintf(KERNEL --- Syscall: user_addr for allocating space =
0x%lx\n, user_addr);
}
return result;
}
/* Sysent entity for syscall */
static struct sysent sc_sysent = {
1, /* Number of arguments */
syscf /* Syscall function*/
};
//static struct sysent *old_sysent;
/* Offset in sysent[] */
static int offset = NO_SYSCALL;
/* Loader */
static int
load (struct module *m, int cmd, void *something)
{
int error = 0;
switch(cmd){
case MOD_LOAD:
//MALLOC(addr, vm_offset_t, SIZE, M_FIVEG_SYSC, M_WAITOK | M_ZERO);
addr = (vm_offset_t) malloc(SIZE, M_FIVEG_SYSC, M_WAITOK);
initialize_array((char*) addr);
uprintf(KERNEL Module with sysc loaded. Offset = %d \n, offset);
break;
case MOD_UNLOAD:
free((void*) addr, M_FIVEG_SYSC);
uprintf(KERNEL Module with sysc unloaded. Offset = %d \n, offset);
break;
default:
error = EOPNOTSUPP;
break;
}
return (error);
/* Syscall macro*/
SYSCALL_MODULE(fiveg_sysc, offset, sc_sysent, load, NULL);
/* eof */
-- USERLAND PROGRAM
#include stdio.h
#include stdlib.h
#include limits.h
#include sys/syscall.h
#include sys/types.h
#include sys/module.h
int main(int argc, char *argv[])
{
int sysc_num, error;
struct module_stat mstat;
/* This Variable will save the addres of remapped buffer */
unsigned char *some_var = NULL;
/* Pointer to pointer to remapped buffer */
unsigned char **p = some_var;
/* Search module with system call */
mstat.version = sizeof(mstat);
if (!(modstat(modfind(sys/fiveg_sysc), mstat))){
/* Our module */
sysc_num = mstat.data.intval;
printf(USER: Module found, Syscall number = %d \n, sysc_num);
/* make system call */
error = syscall(sysc_num, p);
if (error != 0) {
printf(USER: an error occured: %d\n, error);
return -1;
}
/* Read the string from remapped buffer */
printf(USER: p = %p\n, p);
printf(USER: *p = %p\n, *p);
printf(USER: String = %s\n, *p);
} else
printf(USER: Module seems to be not loaded! \n);
return 0;
}
/* eof */
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: malloc() in kernel and increasing mbuf and cluster size
sys/malloc.h has function prototypes for malloc() kern/kern_malloc.c defines the malloc() the malloc() definition is void * malloc(size, type, flags) unsigned long size; struct malloc_type *type; int flags; i understand the size and flags but what shall i do with the malloc_type. man 9 malloc :-) i saw the man pages. it says to use malloc_type via MALLOC_DEFINE(type,shortdesc,longdesc) MALLOC_DECLARE(type) the man pages use M_FOOBUF(where did it come from ??) in the field type. Now how should i code it. struct malloc_type mytype; mytype=MALLOC_DEFINE(.,mybuffers,mybuffers); what should i put in the type field ?? thanks in advance, kamal __ Yahoo! Music Unlimited Access over 1 million songs. Try it free. http://music.yahoo.com/unlimited/ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc() in kernel and increasing mbuf and cluster size
On Mon, 17 Oct 2005 21:19, kamal kc wrote: void * malloc(size, type, flags) unsigned long size; struct malloc_type *type; int flags; i understand the size and flags but what shall i do with the malloc_type. man 9 malloc It is used to do basic sanity checking and for statistics. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C pgp1THOobuOBO.pgp Description: PGP signature
Re: malloc() in kernel and increasing mbuf and cluster size
Hi, Kamal On 17.10.2005 04:49:01, kamal kc wrote: this may be a trivial question for many of you but i am confused in doing memory allocation in the kernel. sys/malloc.h has function prototypes for malloc() kern/kern_malloc.c defines the malloc() the malloc() definition is void * malloc(size, type, flags) unsigned long size; struct malloc_type *type; int flags; i understand the size and flags but what shall i do with the malloc_type. man 9 malloc :-) -- Nick Strebkov Public key: http://humgat.org/~nick/pubkey.txt fpr: 552C 88D6 895B 6E64 F277 D367 8A70 8132 47F5 C1B6 ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc() in kernel and increasing mbuf and cluster size
On Mon, 17 Oct 2005 21:56, kamal kc wrote: the man pages use M_FOOBUF(where did it come from ??) in the field type. 'foo' is a generic term for a random variable name. Now how should i code it. struct malloc_type mytype; mytype=MALLOC_DEFINE(.,mybuffers,mybuffers); what should i put in the type field ?? Read the man page!! /* sys/something/foo_extern.h */ MALLOC_DECLARE(M_FOOBUF); /* sys/something/foo_main.c */ MALLOC_DEFINE(M_FOOBUF, foobuffers, Buffers to foo data into the ether); /* sys/something/foo_subr.c */ ... MALLOC(buf, struct foo_buf *, sizeof *buf, M_FOOBUF, M_NOWAIT); Read other code, there are plenty of examples in the tree. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C pgpwPfcG0B6N3.pgp Description: PGP signature
Re: malloc() in kernel and increasing mbuf and cluster size
2005/10/17, kamal kc [EMAIL PROTECTED]: sys/malloc.h has function prototypes for malloc() kern/kern_malloc.c defines the malloc() the malloc() definition is void * malloc(size, type, flags) unsigned long size; struct malloc_type *type; int flags; i understand the size and flags but what shall i do with the malloc_type. man 9 malloc :-) i saw the man pages. it says to use malloc_type via MALLOC_DEFINE(type,shortdesc,longdesc) MALLOC_DECLARE(type) the man pages use M_FOOBUF(where did it come from ??) in the field type. Now how should i code it. struct malloc_type mytype; mytype=MALLOC_DEFINE(.,mybuffers,mybuffers); what should i put in the type field ?? thanks in advance, kamal malloc type is defined for some kind of statistics/trackings. If you define some specific malloc types for your module/driver etc., you can track the memory usage more accurately. For example, if you do vmstat -m you can see how much memories are used for specific allocations by looking at type field. Regards, Sangwoo Shim __ Yahoo! Music Unlimited Access over 1 million songs. Try it free. http://music.yahoo.com/unlimited/ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Malloc while freeing some object
On 2005-06-30 22:27, Roman Kurakin [EMAIL PROTECTED] wrote: Dmitry Mityugov wrote: On 6/30/05, Roman Kurakin [EMAIL PROTECTED] wrote: One simple question on programming style. Is it ok to call malloc while we releasing some object? Are you releasing it in another thread? Is it a C++ object? How are you releasing it? This was general question. In fact I've found that ttyfree() function from kernel code allocates tempory variables to free tty. I didn't expect such behaviour. I always thought that deallocation of smth should need other alloc. Not necessarily. This could be a problem if you are implementing the allocator itself, and start an infinite recursion or end up deadlocking against the allocator which waits the allocator to finish, which waits the allocator... etc. Users of a general purpose memory allocator don't generally have to worry about the same thing though ;-) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Malloc while freeing some object
On 6/30/05, Roman Kurakin [EMAIL PROTECTED] wrote: Hi, One simple question on programming style. Is it ok to call malloc while we releasing some object? Are you releasing it in another thread? Is it a C++ object? How are you releasing it? -- Dmitry We live less by imagination than despite it - Rockwell Kent, N by E ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Malloc while freeing some object
Dmitry Mityugov wrote: On 6/30/05, Roman Kurakin [EMAIL PROTECTED] wrote: Hi, One simple question on programming style. Is it ok to call malloc while we releasing some object? Are you releasing it in another thread? Is it a C++ object? How are you releasing it? This was general question. In fact I've found that ttyfree() function from kernel code allocates tempory variables to free tty. I didn't expect such behaviour. I always thought that deallocation of smth should need other alloc. rik ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc vs ptmalloc2
On Mon, 14 Feb 2005 08:04:50 -0500, David Schultz [EMAIL PROTECTED] wrote: Right, databases, language runtimes, and the small set of other applications for which it really matters usually have their own special-purpose allocators. I was counting on that when I said that replacing malloc() is unlikely to make a big difference. (One could argue, of course, that it's unfortunate that applications need to do so.) I've worked on several heavily threaded applications that have benefitted greatly from either the introduction of a threads-conscious allocator, especially on MP machines, or sacrificing space for time and simplicity by having per-thread queues for commonly sized blocks of memory (ie, a specialised allocator.) They have included language runtimes and databases, but also networking apps, and other server systems. An application can't assume it's master of the machine you run on, and dynamically sizing the app over time can be important. Long-running server applications may also have similar design goals the kernel deals with: by providing a generally fast allocator, different subsystems don't have to all have their own independent specialised heap allocators, causing fragmentation of memory in the long term. (But instead of having mbufs and other subsystem-specific structures, you've to deal with sharing your address space with third party libraries (A JVM and XML parsers, for example) I'm just saying don't be too quick to see heavy (as opposed to ridiculous) malloc/free use in applications as a design problem in userland apps when use of similar services in the kernel is encouraged over rolling your own allocators. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc vs ptmalloc2
On Mon, Feb 14, 2005 at 02:08:07PM -0800, John-Mark Gurney wrote: David Schultz wrote this message on Mon, Feb 14, 2005 at 08:04 -0500: Right, databases, language runtimes, and the small set of other applications for which it really matters usually have their own special-purpose allocators. I was counting on that when I said that replacing malloc() is unlikely to make a big difference. (One could argue, of course, that it's unfortunate that applications need to do so.) Hmmm, maybe we should bring the zone API into userland? :) It'd be nice to have some of the features, like the ctor/dtor's... Why not just make UMA the userland implementation, too? It should only cost us one kernel trap instruction per allocation/deallocation operation. With fine-grained kernel locking, why we wouldn't want to trust the same code to both tasks, and possibly optimize more because it can do that? -- Brian Fundakowski Feldman \'[ FreeBSD ]''\ [EMAIL PROTECTED] \ The Power to Serve! \ Opinions expressed are my own. \,,\ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc vs ptmalloc2
Jason Henson wrote: On 02/13/05 03:21:29, David Schultz wrote: [...] With a little bit of work, you should be able to replace src/lib/libc/stdlib/malloc.c. ptmalloc is much more heavyweight, but it would probably do better in cases where you have a large number of threads doing a massive number of malloc/free operations on a multiprocessor system. Other than that, I don't know enough details about ptmalloc to speculate, except to say that for most real-world workloads on modern systems, the impact of the malloc implementation is likely to be negligible. Of course, test results would be interesting... I see what you mean by heavy weight! Looking through the sources. The gains looked promising in this thread http://docs.freebsd.org/cgi/mid.cgi?420BB1FF.11156.68F6CEC I might find the time for it, and if I do I hope it is not too difficult. Just from memory, doesn't Linux' malloc require kernel support for re-mapping memory regions, which is not available in FreeBSD? This issue came up in the discussion about FreeBSD's anemic realloc performance. Or has this kernel functionality been added to recent versions of FreeBSD? You may want to investigate this before you invest too much time into your porting effort. Uwe -- Uwe Doering | EscapeBox - Managed On-Demand UNIX Servers [EMAIL PROTECTED] | http://www.escapebox.net ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc vs ptmalloc2
David Schultz wrote: Other than that, I don't know enough details about ptmalloc to speculate, except to say that for most real-world workloads on modern systems, the impact of the malloc implementation is likely to be negligible. Of course, test results would be interesting... Some language interpreters by design malloc()/realloc()/free() memory constantly. Python being a well known example of such an interpreter. Because the issues with memory allocators are legion in the context of a multitude of platforms, Python eventually gained a highly specialised allocator geared to its usage patterns (which brought some other benefits with it too). I think I've seen references to Perl doing something similar. The performance degradations due to corner cases in allocator design exposed by Python (when using native platform allocators rather than its own) is often painful. Nothing comes for free though, and Python's allocator has a limitation that a small percentage of users find painfull - it doesn't return vacant arenas to the OS, which with some usage patterns gives rise to large amounts of wasted swap space. - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc vs ptmalloc2
On Mon, Feb 14, 2005, Andrew MacIntyre wrote: David Schultz wrote: Other than that, I don't know enough details about ptmalloc to speculate, except to say that for most real-world workloads on modern systems, the impact of the malloc implementation is likely to be negligible. Of course, test results would be interesting... Some language interpreters by design malloc()/realloc()/free() memory constantly. Python being a well known example of such an interpreter. Because the issues with memory allocators are legion in the context of a multitude of platforms, Python eventually gained a highly specialised allocator geared to its usage patterns (which brought some other benefits with it too). I think I've seen references to Perl doing something similar. Right, databases, language runtimes, and the small set of other applications for which it really matters usually have their own special-purpose allocators. I was counting on that when I said that replacing malloc() is unlikely to make a big difference. (One could argue, of course, that it's unfortunate that applications need to do so.) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc vs ptmalloc2
On Mon, 14 Feb 2005, Uwe Doering wrote: Just from memory, doesn't Linux' malloc require kernel support for re-mapping memory regions, which is not available in FreeBSD? This issue came up in the discussion about FreeBSD's anemic realloc performance. Or has this kernel functionality been added to recent versions of FreeBSD? You may want to investigate this before you invest too much time into your porting effort. Uwe -- Uwe Doering | EscapeBox - Managed On-Demand UNIX Servers I thought that thread was speculative, with someone saying it would be nice if realloc had kernel help, or something like that. However, if that feature is used by some malloc library which might be portable to FreeBSD (so that the feature's use can be shown), I'd suggest that someone contact Alan Cox ([EMAIL PROTECTED]) - he seems to be working on optimizing the VM system right now, so he'd be the person who could code up support for this feature quickest. Mike Silby Silbersack ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc vs ptmalloc2
David Schultz wrote this message on Mon, Feb 14, 2005 at 08:04 -0500: Right, databases, language runtimes, and the small set of other applications for which it really matters usually have their own special-purpose allocators. I was counting on that when I said that replacing malloc() is unlikely to make a big difference. (One could argue, of course, that it's unfortunate that applications need to do so.) Hmmm, maybe we should bring the zone API into userland? :) It'd be nice to have some of the features, like the ctor/dtor's... -- John-Mark Gurney Voice: +1 415 225 5579 All that I will do, has been done, All that I have, has not. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc vs ptmalloc2
On Mon, Feb 14, 2005, John-Mark Gurney wrote: David Schultz wrote this message on Mon, Feb 14, 2005 at 08:04 -0500: Right, databases, language runtimes, and the small set of other applications for which it really matters usually have their own special-purpose allocators. I was counting on that when I said that replacing malloc() is unlikely to make a big difference. (One could argue, of course, that it's unfortunate that applications need to do so.) Hmmm, maybe we should bring the zone API into userland? :) It'd be nice to have some of the features, like the ctor/dtor's... That's essentially what Solaris did: http://www.usenix.org/event/usenix01/bonwick.html ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc vs ptmalloc2
On Sun, Feb 13, 2005, Jason Henson wrote: I saw on a few of the lists here how linux uses ptmalloc2 and it outperforms bsd's malloc. I tried to do some research into it and found PHK's pdf on it and it seems bsd's malloc was ment to be ok in most every situation. Because of this it shines when primary storage is seriously over committed. So here is my question, I use FreeBSD as a desktop and never ever use swap(I just don't stress my system enough?), can I use ptmalloc in stead of malloc? Like defining SCHED_ULE instead of SCHED_4BSD. Can the system malloc be switched out? With a little bit of work, you should be able to replace src/lib/libc/stdlib/malloc.c. ptmalloc is much more heavyweight, but it would probably do better in cases where you have a large number of threads doing a massive number of malloc/free operations on a multiprocessor system. Other than that, I don't know enough details about ptmalloc to speculate, except to say that for most real-world workloads on modern systems, the impact of the malloc implementation is likely to be negligible. Of course, test results would be interesting... ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc vs ptmalloc2
On 02/13/05 03:21:29, David Schultz wrote: On Sun, Feb 13, 2005, Jason Henson wrote: I saw on a few of the lists here how linux uses ptmalloc2 and it outperforms bsd's malloc. I tried to do some research into it and found PHK's pdf on it and it seems bsd's malloc was ment to be ok in most every situation. Because of this it shines when primary storage is seriously over committed. So here is my question, I use FreeBSD as a desktop and never ever use swap(I just don't stress my system enough?), can I use ptmalloc in stead of malloc? Like defining SCHED_ULE instead of SCHED_4BSD. Can the system malloc be switched out? With a little bit of work, you should be able to replace src/lib/libc/stdlib/malloc.c. ptmalloc is much more heavyweight, but it would probably do better in cases where you have a large number of threads doing a massive number of malloc/free operations on a multiprocessor system. Other than that, I don't know enough details about ptmalloc to speculate, except to say that for most real-world workloads on modern systems, the impact of the malloc implementation is likely to be negligible. Of course, test results would be interesting... I see what you mean by heavy weight! Looking through the sources. The gains looked promising in this thread http://docs.freebsd.org/cgi/mid.cgi?420BB1FF.11156.68F6CEC I might find the time for it, and if I do I hope it is not too difficult. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc calls and ioctl calls to soundcard cause segfault
On Tue, 12 Oct 2004, Shawn Webb wrote: (attached is the source code to the segfaulting application) Doesn't appear to be -- if it was a large attachment, maybe the mailing list stripped it. Could you give a URL for the source? Robert N M Watson FreeBSD Core Team, TrustedBSD Projects [EMAIL PROTECTED] Principal Research Scientist, McAfee Research ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: malloc calls and ioctl calls to soundcard cause segfault
I've got to rewrite the source due to hard disk problems, so I'll just put
it in this email:
/* dsp_seg.c */
/* If there are any compile errors, forgive me; I'm not on a FreeBSD box */
#include stdio.h
#include stdlib.h
#include string.h
#include unistd.h
#include sys/types.h
#include sys/soundcard.h
#include fcntl.h
#define FORMAT AFMT_S16_LE
#define RATE 8000
#define CHANNELS 1
char *buf;
int fd;
int buflen;
void this_segfaults(void);
int main(void)
{
fd = open(/dev/dsp, O_RDWR);
if (fd 0)
{
perror(open);
exit(1);
}
this_segfaults();
printf(m00\n);
exit(0);
}
void this_segfaults(void)
{
int arg;
arg = FORMAT;
if (ioctl(fd, SNDCTL_DSP_SETFMT, arg) 0)
{
perror(ioctl setfmt);
exit(1);
}
if (ioctl(fd, SNDCTL_DSP_GETOSPACE, arg) 0)
{
perror(ioctl getospace);
exit(1);
}
printf(arg: %d\n, arg);
buf = malloc(arg);
if (!buf)
{
perror(malloc);
exit(1);
}
arg = CHANNELS;
if (ioctl(fd, SNDCTL_DSP_CHANNELS, arg) 0)
{
perror(ioctl channels);
exit(1);
}
arg = RATE;
if (ioctl(fd, SNDCTL_DSP_SETRATE, arg) 0)
{
perror(ioctl setrate);
exit(1);
}
}
/* End of source */
- Original Message -
From: Robert Watson [EMAIL PROTECTED]
To: Shawn Webb [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Sent: Wednesday, October 13, 2004 1:47 AM
Subject: Re: malloc calls and ioctl calls to soundcard cause segfault
On Tue, 12 Oct 2004, Shawn Webb wrote:
(attached is the source code to the segfaulting application)
Doesn't appear to be -- if it was a large attachment, maybe the mailing
list stripped it. Could you give a URL for the source?
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[EMAIL PROTECTED] Principal Research Scientist, McAfee
Research
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc calls and ioctl calls to soundcard cause segfault
In the last episode (Oct 14), Shawn Webb said:
I've got to rewrite the source due to hard disk problems, so I'll just put
it in this email:
arg = FORMAT;
if (ioctl(fd, SNDCTL_DSP_SETFMT, arg) 0)
{
perror(ioctl setfmt);
exit(1);
}
if (ioctl(fd, SNDCTL_DSP_GETOSPACE, arg) 0)
{
perror(ioctl getospace);
exit(1);
}
SNDCTL_DSP_GETOSPACE takes a pointer to an audio_buf_info type, so
you actually asked it to write sizeof(audio_buf_info) bytes to the
location of the arg variable, which is... (drumroll)
on the stack :)
Create another variable audio_buf_info info; above main, and change
that call to if (ioctl(fd, SNDCTL_DSP_GETOSPACE, info) 0) and your
program will run fine.
--
Dan Nelson
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc backed md/mfs filesystem swapped?
On Fri, 13 Feb 2004, Andrew J Caines wrote: After Ring the various FMs including, but not limited to, mdmfs(8), mdconfig(8) malloc(9), I am unclear whether of not the memory used by md of type MD_MALLOC is kernel memory which will not be swapped, or not. On the same subject, does the the MD_SWAP backed device simply use swapable userland VM or does it specifically use a piece of the (presumably) disk backed swap partition? FYI, the relevant fstab entries for a malloc backed disk having a UFS2 with softupdates and async would look like: Malloc-backed md devices will be backed by unpageable kernel address space, and doing this with anything but a very small virtual disk will result in a kernel panic once the pages are allocated and the rest of the kernel runs out of address space and memory. Swap-backed md devices will be backed by pageable memory, but I'm not sure what the practical limits (if any) are for address space concerns. In general, I use malloc-backed disks only for diskless systems, and then, only in a sparing way. If you have swap available, you pretty much always want to use swap-backing for memory disks -- if there's room in memory they will run as fast as malloc-backed, but you don't have to be as worried about the Oh shoot, I'm out of room case. I use a pretty large swap-backed file system for /tmp on almost all of my production systems, since swap is cheap, and most of the time so is memory. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects [EMAIL PROTECTED] Senior Research Scientist, McAfee Research md/tmpmfs rw,-M,-s128m,async 2 0 md/var/runmfs rw,-M,-s1m,async2 0 -Andrew- -- ___ | -Andrew J. Caines- Unix Systems Engineer [EMAIL PROTECTED] | | They that can give up essential liberty to obtain a little temporary | | safety deserve neither liberty nor safety - Benjamin Franklin, 1759 | ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc backed md/mfs filesystem swapped?
At 00:56 14/02/2004, Robert Watson wrote: If you have swap available, you pretty much always want to use swap-backing for memory disks -- if there's room in memory they will run as fast as malloc-backed, but you don't have to be as worried about the Oh shoot, I'm out of room case. Actually, there is one consideration: swap-backed memory disks have a sector size equal to the machine page size. This will result in some inflation in memory usage, and can confuse program which expect a sector size of 512 bytes (for example, dd, which I plan on fixing but I haven't gotten around to yet). Colin Percival ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc backed md/mfs filesystem swapped?
On Sat, 14 Feb 2004, Colin Percival wrote: At 00:56 14/02/2004, Robert Watson wrote: If you have swap available, you pretty much always want to use swap-backing for memory disks -- if there's room in memory they will run as fast as malloc-backed, but you don't have to be as worried about the Oh shoot, I'm out of room case. Actually, there is one consideration: swap-backed memory disks have a sector size equal to the machine page size. This will result in some inflation in memory usage, and can confuse program which expect a sector size of 512 bytes (for example, dd, which I plan on fixing but I haven't gotten around to yet). One such application is Vinum, actually, which does not like using swap-backed storage nodes, although maybe I fixed that. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects [EMAIL PROTECTED] Senior Research Scientist, McAfee Research ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: malloc
Thus spake Danny Braniss [EMAIL PROTECTED]: What a lame program... If this program is indicative of your real-world work-load, you can optimize a lot by getting better programmers. If it is not indicative, then forget about it. i wish i could :-) A non-technical answer to your question is that GNU malloc has special hacks to speed up allocations of 64 bytes or less. FreeBSD malloc does not. The reason people are getting annoyed at you is that allocating memory one word at a time is bad practice anyway, and you shouldn't expect the general-purpose memory allocator to be efficient in this special case. Any microbenchmark is going to expose differences, but that doesn't indicate anything about real-world behavior. For example, you might have noticed in your own test that GNU malloc makes a *huge* waste of space; for every one-word chunk of memory you allocate, it allocates another word to keep track of the allocation. PHKMalloc allocates just one bit in the same case (plus space to keep track of the page directory). You can find a somewhat more thorough comparison of malloc implementations at http://citeseer.nj.nec.com/440671.html . Note that the implementation referred to as GNU malloc in the paper has been replaced by Doug Lea's malloc in glibc (and good riddance!) You'll notice that both PHKMalloc and Lea Malloc do reasonably well in all of the tests, and that they both excel in different conditions. Disclaimer: The paper isn't all that great. The authors incorrectly assume that all of the overhead in their tests is due to fragmentation, but much of it is actually due to the size of the malloc metadata. Moreover, a good analysis is lacking. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
In message [EMAIL PROTECTED], David Schultz writes: You can find a somewhat more thorough comparison of malloc implementations at http://citeseer.nj.nec.com/440671.html . There are many problems with this paper, and my feeling is that it was written with a very specific purpose in mind, although I havn't been able to figure out just what that purpose was. My own research while writing phkmalloc indicated that applications are their own worst enemy when it comes to memory allocation, and therefore the focus for phkmalloc was not to optimize for the individual application, but rather for the system as a whole. The philosphy being that if the entire system gets more work done, the individual applications must ipso facto on average also benefit. I was never able to get the authors of this paper into a dialogue, in particular I tried to find out if they just measured where sbrk() was or if they had tried to examine the real memory load of the application. The reference to correspondence with me in the conclusion can at best me interpreted as we didn't understand his answer and/or didn't have/wanted to spend more time. It is to some extent remarkable that people still, ten years after I pointed out the fact in my paper on phkmalloc, still hasn't realized that VM systems don't behave like swapping systems did and that memory allocators need to be aware of this. The standing of sbrk() is only a very weak indicator of the live set of active pages needed for an application to run in a VM system, yet people still keep measuring it as a performance parameter. Imagine a C-compiler: read in source for function foo() check on it a lot (allocating 4M) allocate the resulting info, 64 bytes. generate the ass'y output free all temp memory read in source for function bar() ... This will usually result in sbrk() sitting just above that 64 bytes but there may be 4MB of untouched memory beneath it. If you run phkmalloc with the 'H' option, this will be madvise(2)'ed to the system, but even without that, the fact that it is _truly_ untouched means that it will soon become a candidate for pageout, and it will stay out until needed. Many mallocs make the mistake of storing the free-list in the actual free memory as a linked list. This means that to free a bit of memory you have to page in all the otherwise unused space, just to traverse the list. If physical RAM is limited, this is a bad plan. The _real_ way to benchmark a malloc implementatio is therefore to measure how it performs with different amounts of physical RAM available, because then a bad malloc will suffer a terrible hit in paging activity where a good malloc may not result in any significant amount of paging. In informal tests along those lines I have seen differences of up to a factor five in wall-clock time. Of course, with 512MB RAM in a workstation, people will never notice this and I'm just a old foghorn who doesn't know anything about performance, but run a server where you actually have RAM pressure and you might notice the difference. Progress may be overrated, but it seldom goes too far... Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Thus spake Poul-Henning Kamp [EMAIL PROTECTED]: In message [EMAIL PROTECTED], David Schultz writes: You can find a somewhat more thorough comparison of malloc implementations at http://citeseer.nj.nec.com/440671.html . There are many problems with this paper, and my feeling is that it was written with a very specific purpose in mind, although I havn't been able to figure out just what that purpose was. I did say `somewhat', didn't I? ;-) As I mentioned in the part of my message that you didn't quote, I don't much care for the paper either, but it's the only half-reasonable comparison I know of. I don't think the authors know what they're talking about, but they did collect extensive data for some real world programs, which I assume is valid. I agree that the behavior of the program from the point of view of the VM system is the most important metric. But internal and external fragmentation are also significant issues. Often, these are a result of programmers not understanding how their malloc works. For example, programs that make numerous 2K allocations in phkmalloc will get twice the amount of memory they asked for, and since each chunk is page-aligned, it will be twice as bad for the VM system. A harder problem to solve is fragmentation for long-running servers, where the RSS tends to creep upwards over time as virtual memory fills with holes. There seems to be a fair amount of research on the subject, although I'm not well read on it. The phkmalloc bucket approach seems to work quite well, as the aforementioned paper claims. I'm guessing that one of these days it will have to be modified for threaded applications to reduce false sharing on SMP systems. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
In message [EMAIL PROTECTED], David Schultz writes: Thus spake Poul-Henning Kamp [EMAIL PROTECTED]: In message [EMAIL PROTECTED], David Schultz writes: You can find a somewhat more thorough comparison of malloc implementations at http://citeseer.nj.nec.com/440671.html . There are many problems with this paper, and my feeling is that it was written with a very specific purpose in mind, although I havn't been able to figure out just what that purpose was. I did say `somewhat', didn't I? ;-) As I mentioned in the part of my message that you didn't quote, I don't much care for the paper either, but it's the only half-reasonable comparison I know of. I don't think the authors know what they're talking about, but they did collect extensive data for some real world programs, which I assume is valid. Right, and their study of the correlation between number of bugs on the windshield versus price-tag is scientifically rigorous and highly commendable. You just can't conclude anything useful from it :-) I agree that the behavior of the program from the point of view of the VM system is the most important metric. But internal and external fragmentation are also significant issues. Often, these are a result of programmers not understanding how their malloc works. For example, programs that make numerous 2K allocations in phkmalloc will get twice the amount of memory they asked for, and since each chunk is page-aligned, it will be twice as bad for the VM system. Not if they are 2k allocations, but if they are 2k+1 allocations: yes. A harder problem to solve is fragmentation for long-running servers, where the RSS tends to creep upwards over time as virtual memory fills with holes. This is where you want to run phkmalloc with the 'H' option. It practically makes it a non-issue last I tried. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Thus spake Poul-Henning Kamp [EMAIL PROTECTED]: A harder problem to solve is fragmentation for long-running servers, where the RSS tends to creep upwards over time as virtual memory fills with holes. This is where you want to run phkmalloc with the 'H' option. It practically makes it a non-issue last I tried. Perhaps phkmalloc could be made self-tuning with regards to 'H'; I doubt many people know when to use that feature. For example, you might have a heuristic where phkmalloc detects that the program has been running for a long time or has called malloc() and free() many times, so it starts using madvise() on some free pages. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
What a lame program... If this program is indicative of your real-world work-load, you can optimize a lot by getting better programmers. If it is not indicative, then forget about it. i wish i could :-) danny To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Danny Braniss wrote: What a lame program... If this program is indicative of your real-world work-load, you can optimize a lot by getting better programmers. If it is not indicative, then forget about it. i wish i could :-) This is a memory overcommit architecture. If you want to avoid the problem, set process limits, such that the programs you run will not hit hard system limits, and will hit administrative limitations instead. This will prevnt the core dumps you were seeing. I implied this in my last post; I'm being very explicit now. If you want GNU malloc behaviour, then you should install the port for the GNU allocator, and use it instead of the system allocator, and you will end up with the same behaviour that your application has on Linux. As far as the speed of your FP programs, I suggest looking at man fpsetmask, and realize that the Linux defaults fail to comply with IEEE floating point standards. Specifically, there was a recent discussion on the -current list about FP compliance testing, which demonstrated Linux non-compliance. I believe the code in that thread was downloadable from NIST and from UC Berkeley. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Your code is not efficient; try this instead: He, the code is not mine, and the programmer is being invited for some coffee and indocrination. The program showed at least two things, 1- the linux emulation ignores the datasize limit, 2- is faster for this particular case of bad programing. btw, the host has 4gb of main memory, so i don't think it swapped. actually i just rebooted the machine with no swap to make sure. I'm not passing judgement as to what is better - though im biased to *BSD - but the fact is that i have some 1500 users, and some 600 computers, and (i hope none of my users are reading this) the users are getting less sophisticated - and when i get complains that it works on Linux, but not under FreeBSD i try very hard to prove them wrong, but im loosing. danny To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
[...] If you want GNU malloc behaviour, then you should install the port for the GNU allocator, and use it instead of the system allocator, and you will end up with the same behaviour that your application has on Linux. what ticked my curiosity was that the linux binary did work, while the fbsd binary did the right thing with respect to the admin limits and coredumped when the datasize limit was exeeded. danny To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Danny Braniss wrote: If you want GNU malloc behaviour, then you should install the port for the GNU allocator, and use it instead of the system allocator, and you will end up with the same behaviour that your application has on Linux. what ticked my curiosity was that the linux binary did work, while the fbsd binary did the right thing with respect to the admin limits and coredumped when the datasize limit was exeeded. The FreeBSD malloc uses anonymous pages mmap'ed off of /dev/zero. The Linux malloc uses pages added to the process address space via a call to sbrk. The FreeBSD malloc guarantees that the pages are zeroed before being obtained from the system; this is probably the majority of the cost. It is a security measure, so that you do not leak data from one process to another through anonymous pages. The Linux malloc does not. The FreeBSD malloc does not impose a size limit based on the (potential) collision of the stack with the heap. The Linux Malloc must impose a memory limit based on the possibility of such a collision. The FreeBSD malloc allocates untouched pages: the pages are faulted in when they are accessed. The Linux malloc pays the penalty up front, and so doesn't run into a usage shortage, if you use a thrashing access pattern, like your example application used, to intentionally thrash memory (twice; once on allocation, once again on checking of the reference, after). The FreeBSD malloc references an environment variable and a readlink() of a potentially non-existant symbolic link containing configuration data for the malloc. The Linux malloc does not have this overhead. The FreeBSD malloc would be lower performance than the Linux malloc, if you allocate space in teeny, tiny chunks; it has much higher performance for large allocations. Good programmers allocate their resources up front, once, instead of doing the allocations in time critical internal loops. The FreeBSD allocation is an overcommit allocation; this means that the malloc could succeed, obtain a memory mapping for the pages, but run out of pages to use as backing store for the address space so mapped. This *only* happens if the administrative limits on the applications are larger than the available physical resources: don't do that: man login.conf, man limit. If you want to have the Linux malloc behaviour on FreeBSD, then link your application with the Linux malloc library. It is available as a port for you to install on your FreeBSD system: cd /usr/ports/devel/libmalloc make all install ...and them link your applications with the malloc library, instead of using the system malloc. Can't really make it any more clear than that. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Danny Braniss wrote: Your code is not efficient; try this instead: He, the code is not mine, and the programmer is being invited for some coffee and indocrination. Good. The program showed at least two things, 1- the linux emulation ignores the datasize limit, Possibly. One would expect it to core, then, when you ran out of memory; you said (in your original posting) that it was the FreeBSD version of the code that cored?!? 2- is faster for this particular case of bad programing. Don't expect FreeBSD to ever optimize system code to make bad programmers look like good programmers. It's not going to happen. btw, the host has 4gb of main memory, so i don't think it swapped. actually i just rebooted the machine with no swap to make sure. Then you aren't getting the core dump you said you were. 8-). I'm not passing judgement as to what is better - though im biased to *BSD - but the fact is that i have some 1500 users, and some 600 computers, and (i hope none of my users are reading this) the users are getting less sophisticated - and when i get complains that it works on Linux, but not under FreeBSD i try very hard to prove them wrong, but im loosing. If you are fighting that fight, you might as well switch to Windows, and be done with it, because people can make the same argument with Windows vs. Linux instead of Linux vs. FreeBSD, since the issues are exactly analogous: the Windows environment does a much better job than Linux of allowing bad programmers to write code that runs. And Linux does this better than FreeBSD (there are even Linux libraries that will treat NULL pointers as if they were pointers to strings with a value of -- and don't think there isn't runtime overhead associated with the extra compares again NULL for each and every strcpy(), strlen(), etc.. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
[...] Possibly. One would expect it to core, then, when you ran out of memory; you said (in your original posting) that it was the FreeBSD version of the code that cored?!? true, but i also mentioned how i fixed it, by increasing the MAXDSIZ option. danny To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Terry Lambert [EMAIL PROTECTED] wrote: The FreeBSD malloc guarantees that the pages are zeroed before being obtained from the system; this is probably the majority of the cost. It is a security measure, so that you do not leak data from one process to another through anonymous pages. The Linux malloc does not. Utter bollocks. FreeBSD malloc can be configured to re-initialize memory on every allocation, but this is designed to assist with buggy programs, it is *not* a security measure. Memory obtained from the kernel on *all* unices (including Linux) is zeroed; that is when security matters, not in malloc. This will not affect the relative performance of phk and gnu malloc. The FreeBSD malloc references an environment variable and a readlink() of a potentially non-existant symbolic link containing configuration data for the malloc. Once at program startup. This is not a significant cost. The FreeBSD allocation is an overcommit allocation True for Linux too, by default. Tony. -- f.a.n.finch [EMAIL PROTECTED] http://dotat.at/ NORTH UTSIRE: EAST 4 OR 5 INCREASING 6 TO GALE 8. RAIN. MODERATE. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
On Tue, Oct 22, 2002 at 01:20:42AM -0700, Terry Lambert wrote: The FreeBSD malloc uses anonymous pages mmap'ed off of /dev/zero. The Linux malloc uses pages added to the process address space via a call to sbrk. There is no Linux malloc, neither does Linux have a sbrk syscall :) But glibc malloc indeed uses both brk and mmap. The FreeBSD malloc guarantees that the pages are zeroed before being obtained from the system; this is probably the majority of the cost. It is a security measure, so that you do not leak data from one process to another through anonymous pages. The Linux malloc does not. glibc malloc doesn not guarantee you zeroed memory if you reuse memory that was freed in your process. Otherwise it's of course zeroed. once on allocation, once again on checking of the reference, after). The FreeBSD malloc references an environment variable and a readlink() of a potentially non-existant symbolic link containing configuration data for the malloc. The Linux malloc does not have this overhead. glibc malloc reads a bunch of enviroment variables, though to allow stuff like memleak debuging and such The FreeBSD malloc would be lower performance than the Linux malloc, if you allocate space in teeny, tiny chunks; it has much higher performance for large allocations. Good programmers allocate their resources up front, once, instead of doing the allocations in time critical internal loops. It's certainly debatable wehther the whole point of a userlevel memory allocator like malloc() isn't avoiding the reimlementation of a better allocator in each an every program :) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
On Tue, Oct 22, 2002 at 01:52:34PM -0500, Stephen Montgomery-Smith wrote: Brooks Davis wrote: The user may also see a performance gain on Linux if they use a less stupid allocation scheme. I ran into some code once that read strings one character at a time via getc() and did a realloc for each read. Needless to say, performance was truly awful since a typical run required parsing over 600MB of text. I saw a better then 50% speedup on Alpha Linux when I fixed that mess. As an amateur programmer, I wrote a program something like this. But it allocates memory 1000 bytes at a time in order to save some of this overhead. I would be interested in getting feedback about this little function I wrote as to whether it could be greatly improved. (It works great for me, but probably I haven't really pushed it very hard.) If it performs well enough for your application, there's nothing wrong with it. If you want to speed it up, I'd suggest using the profiler (gprof) to see where you are spending the most time. What makes sense, depends on the layout of the file and how you want to process it. -- Brooks -- Any statement of the form X is the one, true Y is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4 msg37540/pgp0.pgp Description: PGP signature
Re: malloc
On Tue, Oct 22, 2002 at 12:01:19PM -0700, Terry Lambert wrote: Tony Finch wrote: Terry Lambert [EMAIL PROTECTED] wrote: The FreeBSD malloc guarantees that the pages are zeroed before being obtained from the system; this is probably the majority of the cost. It is a security measure, so that you do not leak data from one process to another through anonymous pages. The Linux malloc does not. Utter bollocks. FreeBSD malloc can be configured to re-initialize memory on every allocation, but this is designed to assist with buggy programs, it is *not* a security measure. Memory obtained from the kernel on *all* unices (including Linux) is zeroed; that is when security matters, not in malloc. This will not affect the relative performance of phk and gnu malloc. *before being obtained from the system*. Linux does that too, and you appeared to be saying that it doesn't which is clearly wrong for the security reasons that you stated. It therefore won't affect the relative performance. And I didn't say that. I only said that the pages were zeroed *before being obtained from the system*. This is what you would expect, with anonymous memory accessed off /dev/zero. PHK malloc uses MAP_ANON on FreeBSD, not /dev/zero -- it uses the latter only if compiled for Solaris. Tony. -- f.a.n.finch [EMAIL PROTECTED] http://dotat.at/ FAIR ISLE: NORTHEAST BACKING NORTH 5 TO 7 INCREASING 7 TO SEVERE GALE 9, OCCASIONALLY STORM 10 IN EAST. RAIN. MODERATE OR POOR. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Dan Nelson wrote: The FreeBSD malloc uses anonymous pages mmap'ed off of /dev/zero. The Linux malloc uses pages added to the process address space via a call to sbrk. Actually, on FreeBSD only the page directory is mmap'ed. Data returned to the user is allocated via sbrk. Please see: /usr/src/lib/libc/stdlib/malloc.c The only calls to sbrk have a 0 argument. This is only used to find the segment end, so that the mmap's do not occur over top of anything important. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Brooks Davis wrote: On Tue, Oct 22, 2002 at 01:20:42AM -0700, Terry Lambert wrote: The FreeBSD malloc would be lower performance than the Linux malloc, if you allocate space in teeny, tiny chunks; it has much higher performance for large allocations. Good programmers allocate their resources up front, once, instead of doing the allocations in time critical internal loops. The user may also see a performance gain on Linux if they use a less stupid allocation scheme. I ran into some code once that read strings one character at a time via getc() and did a realloc for each read. Needless to say, performance was truly awful since a typical run required parsing over 600MB of text. I saw a better then 50% speedup on Alpha Linux when I fixed that mess. This goes without saying. It's the difference between programmers and software engineers. Nevertheless, there will continue to be a performance differential between FreeBSD and Linux. I personally don't think it's a problem; optimizing that area is optimizing noise. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Tony Finch wrote: The FreeBSD malloc guarantees that the pages are zeroed before being obtained from the system; this is probably the majority of the cost. It is a security measure, so that you do not leak data from one process to another through anonymous pages. The Linux malloc does not. [ ... ] *before being obtained from the system*. Linux does that too, and you appeared to be saying that it doesn't which is clearly wrong for the security reasons that you stated. It therefore won't affect the relative performance. Yes, it will. It has to do with the use of anonymous memory being different between the systems. You are arguing that there is nothing that can account for the performance difference, when in fact there is a measured performance difference. And I didn't say that. I only said that the pages were zeroed *before being obtained from the system*. This is what you would expect, with anonymous memory accessed off /dev/zero. PHK malloc uses MAP_ANON on FreeBSD, not /dev/zero -- it uses the latter only if compiled for Solaris. And tell me, what does the Linux malloc use? -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Dan Nelson wrote: The only calls to sbrk have a 0 argument. This is only used to find the segment end, so that the mmap's do not occur over top of anything important. Ah, but take a look at the calls to brk, especially in map_pages() and free_pages(). How the anonymous pages (which I have already made reference to) are explcitly obtained is irrelevent to the current discussion. If you want me to be exact in terms of implementation details, on both Linux and FreeBSD, I can do so. That isn't the real purpose of the current discussion. The real purpose of this discussion is to defend the choice of FreeBSD, rather than Linux, to users who are strong advocates of Linux. Any answers that don't provide *only* general details on the differences between the systems which lead to the differences in performance behaviour aren't really useful... in fact, they are counter-productive. Yes, this is really nothing more than a Linux shortage discussion, where someone has written some bogus code to demonstrate how terrible the world is, without Linux everywhere. ... I have a bad idea insert Bad Idea Dance here. Why don't you explain how malloc works in excruciating implementation detail, instead of why his program is failing on FreeBSD and not on Linux, so that he gets no real useful comparative information out of the resulting discussion, and everyone reading the archives thinks we bury valid Linux vs. FreeBSD questions in technobabble, thus affirming the validity of the original complaint to any third parties who are later referred to the thread as an example of Why FreeBSD sucks. Then he can throw up his hands in surrender, because no one is willing to speak in the abstract, instead of in terms of implementation details, and switch over to Linux, as his users are demanding he do, since they appear to spend their entire lives looking for degenerate cases that they can use to advocate the the use of Linux in place of whatever other software their administration happens to be using, because they are NOT in fact users being prevented from doing their jobs, they are Linux advocates *pretending* to be users being prevented from doing their jobs by a severe lack of Linux, God's One True Blessed OS. ... Yes, technically, this discussion should be taking place on -advocacy, but that's not where the question was asked, and it's not where the discussion will be read, and it's not where the results will be archived. And it's not the archive that will be referred to by future advocates who want to make FreeBSD look bad. Moving it will only serve the purpose of placing a question about FreeBSD for which there is apparently no good answer, for lack of an answer in the archives of the forum (-hackers) in which the question was asked. Thanks, -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
On Tue, Oct 22, 2002 at 12:48:03PM -0700, Terry Lambert wrote: Tony Finch wrote: Linux [clears memory in the kernel before handing it over to userland], and you appeared to be saying that it doesn't which is clearly wrong for the security reasons that you stated. It therefore won't affect the relative performance. Yes, it will. It has to do with the use of anonymous memory being different between the systems. The only significant difference I can see is that large callocs are mmapped by gnu malloc which doesn't then re-clear them, whereas phk malloc only mmaps its page table (not memory returned to the user) so calloc always clears memory, which happens to be redundant if the pages have just come from brk (which isn't always the case). However calloc isn't relevant to this benchmark. [If you are talking about a different difference, perhaps you should say what it is so that this thread doesn't get even more unnecessarily long.] You are arguing that there is nothing that can account for the performance difference, when in fact there is a measured performance difference. No, I'm saying that some of what you said is either wrong or misleading, and the comment about security was especially stupid. PHK malloc uses MAP_ANON on FreeBSD, not /dev/zero -- it uses the latter only if compiled for Solaris. And tell me, what does the Linux malloc use? Exactly the same, and it uses MAP_ANONYMOUS on Linux. It uses mmap for large allocations whereas phk malloc does not. Since both mmap and sbrk get zero-filled pages from the kernel this shoudld make little difference, except in the calloc case explained above. Tony. -- f.a.n.finch [EMAIL PROTECTED] http://dotat.at/ FORTIES CROMARTY: EAST BECOMING CYCLONIC 7 TO SEVERE GALE 9, OCCASIONALLY STORM 10 LATER. RAIN. MODERATE OR POOR. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Tony Finch wrote: You are arguing that there is nothing that can account for the performance difference, when in fact there is a measured performance difference. No, I'm saying that some of what you said is either wrong or misleading, and the comment about security was especially stupid. Yes, it was misleading, in that I didn't provide full implementation details, so a naieve reader looking to reimplement malloc() as it was implemented on FreeBSD, would get it wrong. The security comment had to do with the fact that zeroing occurs in the kernel in the idle loop, and can account for a large latency in the case of a big demand in user space. It's a philosophy issue that led to the implementation, and it has a performance impact that's higher in FreeBSD than in Linux, under some usage patterns. I don't think it's necessary to discuss the implementation details to the level of documenting them for people wanting to reimplement the code. The context of the current discussion is a FreeBSD admin with Linux users bitching at him about core dumps in an overcommit case, where he's hitting an administrative limit, and then trying to dereference a pointer to a page that has an established mapping, but for which there is no page available to act as backing store. Saying this, though, really doesn't help, when you have a user that's pretending to not understand your explanations, and wants an explanation in layman's terms, which is what the original poster is facing. If it weren't for the fact that we were dealing with a programmer that was pretending to be dumb enough to put allocations into an interior loop in order to give FreeBSD a bad performance number relative to Linux, this wouldn't have ever been an issue in the first place: for the perfectly correct, we would just refer the person to the malloc() source code. Instead, we have to deal with a fictional microbenchmark, and discuss why decisions were made. There's perfectly correct, and then there's speaking to the level of your audience. PHK malloc uses MAP_ANON on FreeBSD, not /dev/zero -- it uses the latter only if compiled for Solaris. And tell me, what does the Linux malloc use? Exactly the same, and it uses MAP_ANONYMOUS on Linux. It uses mmap for large allocations whereas phk malloc does not. Since both mmap and sbrk get zero-filled pages from the kernel this shoudld make little difference, except in the calloc case explained above. The allocations in this case are lots of very small allocations; it makes no real difference, unless there is a preallocation of space by the malloc() subsystem in order to refill a free pool, to deal with expected future malloc() requests. FreeBSD's overallocation pool size is one 4K page, which is relatively small for modern systems with 4G of memory. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Terry Lambert [EMAIL PROTECTED] wrote: The security comment had to do with the fact that zeroing occurs in the kernel in the idle loop, and can account for a large latency in the case of a big demand in user space. It's a philosophy issue that led to the implementation, and it has a performance impact that's higher in FreeBSD than in Linux, under some usage patterns. You said that Linux doesn't guarantee to zero pages handed from the system to userland, which is wrong. You've also mentioned the in- kernel page-zeroing strategy which is irrelevant when comparing different userland malloc implementations on the same OS. Hand-waving about the peanut gallery when I am trying to get you to be more specific about your vague assertions is not helpful. The context of the current discussion is a FreeBSD admin with Linux users bitching at him about core dumps in an overcommit case, where he's hitting an administrative limit, and then trying to dereference a pointer to a page that has an established mapping, but for which there is no page available to act as backing store. I'm slightly perplexed about this: his program shouldn't have dumped core when hitting an administrative limit because it was correctly checking the return value from malloc(), and he's unlikely to be in an overcommit situation on a machine with 4GB of RAM when allocating only 800MB especially when it works with the administrative limit removed. Perhaps the core came from a different version of the program what didn't check for errors properly. Tony. -- f.a.n.finch [EMAIL PROTECTED] http://dotat.at/ IRISH SEA: NORTHWEST BACKING WEST 6 TO GALE 8, OCCASIONALLY SEVERE GALE 9. SQUALLY SHOWERS. GOOD. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
Tony Finch wrote: You said that Linux doesn't guarantee to zero pages handed from the system to userland, which is wrong. You've also mentioned the in- kernel page-zeroing strategy which is irrelevant when comparing different userland malloc implementations on the same OS. No. I said: | The FreeBSD malloc guarantees that the pages are zeroed before being | obtained from the system; this is probably the majority of the cost. | It is a security measure, so that you do not leak data from one process | to another through anonymous pages. | | The Linux malloc does not. It has to do with *when the pages are zeroed... *before* being obtained from the system or *at the time they are obtained from the system*. Hand-waving about the peanut gallery when I am trying to get you to be more specific about your vague assertions is not helpful. Specificity is counterproductive in the current context. The person was not asking how does FreeBSD malloc work?. He was asking about relative performance of FreeBSD and Linux. See the original post: http://docs.freebsd.org/cgi/getmsg.cgi?fetch=73677+0+current/freebsd-hackers ...it was Yet Another Bogus MicroBenchmark. The context of the current discussion is a FreeBSD admin with Linux users bitching at him about core dumps in an overcommit case, where he's hitting an administrative limit, and then trying to dereference a pointer to a page that has an established mapping, but for which there is no page available to act as backing store. I'm slightly perplexed about this: his program shouldn't have dumped core when hitting an administrative limit because it was correctly checking the return value from malloc(), and he's unlikely to be in an overcommit situation on a machine with 4GB of RAM when allocating only 800MB especially when it works with the administrative limit removed. Perhaps the core came from a different version of the program what didn't check for errors properly. The alloc succeeded because there was page mapping space available. The pages to back the pages in that space were not allocated until they were referenced. The core dump may have occurred as a result of any demand for memory at all (including a copy on write of a dirty data page or a stack grow). Running the bad program from the original posting on my local machine, however, indicates, that it happened when there was an attempt to dereference it in the compare in the second for. On a system with different limits, where the limits were enforced, the mapping could be established, with no room left over to obtain backing pages for the region mapped. If this happened, then it would occur in the array pointer dereference for the first element on a new page boundary in the initial for loop. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
In the last episode (Oct 22), Terry Lambert said: Danny Braniss wrote: If you want GNU malloc behaviour, then you should install the port for the GNU allocator, and use it instead of the system allocator, and you will end up with the same behaviour that your application has on Linux. what ticked my curiosity was that the linux binary did work, while the fbsd binary did the right thing with respect to the admin limits and coredumped when the datasize limit was exeeded. The FreeBSD malloc uses anonymous pages mmap'ed off of /dev/zero. The Linux malloc uses pages added to the process address space via a call to sbrk. Actually, on FreeBSD only the page directory is mmap'ed. Data returned to the user is allocated via sbrk. -- Dan Nelson [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
In the last episode (Oct 22), Terry Lambert said: Dan Nelson wrote: The FreeBSD malloc uses anonymous pages mmap'ed off of /dev/zero. The Linux malloc uses pages added to the process address space via a call to sbrk. Actually, on FreeBSD only the page directory is mmap'ed. Data returned to the user is allocated via sbrk. Please see: /usr/src/lib/libc/stdlib/malloc.c The only calls to sbrk have a 0 argument. This is only used to find the segment end, so that the mmap's do not occur over top of anything important. Ah, but take a look at the calls to brk, especially in map_pages() and free_pages(). -- Dan Nelson [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc
On Mon, Oct 21, 2002 at 11:25:43AM +0200, Danny Braniss wrote: comments? That code is a REALLY inefficient use of malloc(). You can always write bizarre code that exaggerates the differences between different algorithms (e.g. Linux malloc vs FreeBSD malloc). Kris msg37495/pgp0.pgp Description: PGP signature
Re: malloc
On Fri, 3 Aug 2001, [iso-8859-1] vishwanath pargaonkar wrote: Hi, can anybody tell me in malloc what does third parameter DONTWAIT ,NOWAIT and WAITOK mean? Bcoz i have function being called using timeout.in that function i need to malloc a buffer. can i use WAITOK? please tell me abt this. you must use NOWAIT, and you must handle the case where it may fail and return NULL. TIA To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: malloc to arrays?
Chris Costello writes:
On Wednesday, August 02, 2000, Mike Smith wrote:
tcpcash_addr = (typeof(tcpcash_addr))
malloc(sizeof(*tcpcash_addr) *
TCPCASH_ROWSIZE *
TCPCASH_COOLSIZE);
Just as a note on coding style, this will only hide warnings
caused by not including stdlib.h. malloc(3) returns void * and
does not need to be cast.
So there is an error in gcc in base system?
See:
1cicuta~/w/ra-tools(6)cat tmp.c
#include stdlib.h
int main()
{ int *(i[100]);
i = malloc(400);
}
0cicuta~/w/ra-tools(7)cc tmp.c
tmp.c: In function `main':
tmp.c:6: incompatible types in assignment
1cicuta~/w/ra-tools(8)uname -a
FreeBSD cicuta.babolo.ru 4.0-STABLE FreeBSD 4.0-STABLE #0: Fri Jun 9 14:01:52 MSD
2000 [EMAIL PROTECTED]:/tmp/obj/usr/src/sys/cicuta i386
0cicuta~/w/ra-tools(9)
--
@BABOLO http://links.ru/
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: malloc to arrays?
On Fri, 4 Aug 2000, Aleksandr A.Babaylov wrote:
Chris Costello writes:
On Wednesday, August 02, 2000, Mike Smith wrote:
tcpcash_addr = (typeof(tcpcash_addr))
malloc(sizeof(*tcpcash_addr) *
TCPCASH_ROWSIZE *
TCPCASH_COOLSIZE);
Just as a note on coding style, this will only hide warnings
caused by not including stdlib.h. malloc(3) returns void * and
does not need to be cast.
So there is an error in gcc in base system?
See:
1cicuta~/w/ra-tools(6)cat tmp.c
#include stdlib.h
int main()
{ int *(i[100]);
i = malloc(400);
}
0cicuta~/w/ra-tools(7)cc tmp.c
tmp.c: In function `main':
tmp.c:6: incompatible types in assignment
1cicuta~/w/ra-tools(8)uname -a
FreeBSD cicuta.babolo.ru 4.0-STABLE FreeBSD 4.0-STABLE #0: Fri Jun 9 14:01:52 MSD
2000 [EMAIL PROTECTED]:/tmp/obj/usr/src/sys/cicuta i386
0cicuta~/w/ra-tools(9)
There's no error at all. You've declared i to be an array of 100
pointers to int, but then you've tried to assign a pointer to that
array, and gcc quite naturally complains.
--
Chris BeHanna
Software Engineer (at yourfit.com)
[EMAIL PROTECTED]
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: malloc to arrays?
Chris Costello wrote:
On Wednesday, August 02, 2000, [EMAIL PROTECTED] wrote:
Sorry I have no other direction to ask this:
I have declaration:
u_int32_t *(tcpcash_addr[256]);
and want malloc some memory for tcpcash_addr:
tcpcash_addr = (typeof(tcpcash_addr))malloc(u_int32_t * 256 * n);
and have an error:
ra-sum.c:386: cast specifies array type
*** Error code 1
Try this:
u_int32_t *tcphash_haddr;
tcphash_addr = malloc(sizeof(*tcphash_addr) * 256);
You shouldn't be casting malloc and I don't see any reason
to use *(foo[nmemb]) syntax.
u_int32_t **tcphash_haddr;
From the code, it's a variable-sized array of fixed-sized arrays.
Yes, and whan I use u_int32_t *(tcpcash_addr[256]) declaration,
I can use tcpcash_addr[x][y], and I must use
tcpcash_addr[x * 256 + y] if declaration is u_int32_t *tcphash_haddr,
I cant use if u_int32_t **tcphash_haddr,
because of need of array of pointers in addition to array of arrays,
and the only clean but not nice way I know is
struct tcphash_haddr {u_int32_t tcpcash_addr[256];};
struct tcphash_haddr *tcphash_haddr;
but tcphash_haddr[x].tcpcash_addr[y] is not so clear
as tcpcash_addr[x][y]...
Is the reason to use *(foo[nmemb]) syntax clear enough?
And more, in the original code not malloc but mmap...
tcpcash_addr = (typeof(tcpcash_addr))mmap( NULL, ...
so using of pointers is dirty enougth.
and more, assign without a cast impossible:
ra-sum.c:385: incompatible types in assignment
And even
u_int32_t *(tcpcash_addr[256]);
tcpcash_addr = (void*)mmap( NULL, ...
does not translated
--
áÌÅËÓÁÎÄÒ á. âÁÂÁÊÌÏ× mailto:[EMAIL PROTECTED]/
http://links.ru/
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: malloc to arrays?
Yes, and whan I use u_int32_t *(tcpcash_addr[256]) declaration,
I can use tcpcash_addr[x][y], and I must use
tcpcash_addr[x * 256 + y] if declaration is u_int32_t *tcphash_haddr,
I cant use if u_int32_t **tcphash_haddr,
because of need of array of pointers in addition to array of arrays,
and the only clean but not nice way I know is
struct tcphash_haddr {u_int32_t tcpcash_addr[256];};
struct tcphash_haddr *tcphash_haddr;
but tcphash_haddr[x].tcpcash_addr[y] is not so clear
as tcpcash_addr[x][y]...
Is the reason to use *(foo[nmemb]) syntax clear enough?
It's only "clear" insofar as it's "clear" that the code is terrible.
#define TCPCASH_ROWSIZE 256
#define TCPCASH_COLSIZE 256
#define TCPCASH_ADDR(x) ((x) * TCPCASH_ROWSIZE)
u_int32_t *tcpcash_addr
tcpcash_addr = (typeof(tcpcash_addr))
malloc(sizeof(*tcpcash_addr) *
TCPCASH_ROWSIZE *
TCPCASH_COOLSIZE);
...
foo = tcpcash[TCPCASH_ADDR(x)];
This is fast and unambiguous, as well as keeping all the magic numbers
away from the actual code.
Oh, and you spell "cache" like that. "cash" is money.
--
... every activity meets with opposition, everyone who acts has his
rivals and unfortunately opponents also. But not because people want
to be opponents, rather because the tasks and relationships force
people to take different points of view. [Dr. Fritz Todt]
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: malloc to arrays?
It's only "clear" insofar as it's "clear" that the code is terrible. Coffee first, critique second. 8) #define TCPCASH_ROWSIZE 256 #define TCPCASH_COLSIZE 256 #define TCPCASH_ADDR(x) ((x) * TCPCASH_ROWSIZE) #define TCPCASH_ADDR(x,y) ((y) * TCPCASH_ROWSIZE + (x)) -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: malloc to arrays?
[Charset iso-8859-1 unsupported, filtering to ASCII...] It's only "clear" insofar as it's "clear" that the code is terrible. Sorry I know English bad, do you want to say that my code is terrible? Coffee first, critique second. 8) #define TCPCASH_ROWSIZE 256 #define TCPCASH_COLSIZE 256 #define TCPCASH_ADDR(x) ((x) * TCPCASH_ROWSIZE) #define TCPCASH_ADDR(x,y) ((y) * TCPCASH_ROWSIZE + (x)) Yes I know that I will use tcpcashe_addr(x,y) instead of [x][y], (thank for cash - cashe) with some macro declarations, but it is something artificial for array of array representation: [0,0][0,1]...[0,255][1,0][1,1]...[1,255]... with statically unknown first index range. -- áÌÅËÓÁÎÄÒ á. âÁÂÁÊÌÏ× mailto:[EMAIL PROTECTED]/ http://links.ru/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: malloc to arrays?
On Wednesday, August 02, 2000, Mike Smith wrote: tcpcash_addr = (typeof(tcpcash_addr)) malloc(sizeof(*tcpcash_addr) * TCPCASH_ROWSIZE * TCPCASH_COOLSIZE); Just as a note on coding style, this will only hide warnings caused by not including stdlib.h. malloc(3) returns void * and does not need to be cast. -- |Chris Costello [EMAIL PROTECTED] |Last one out, turn off the computer! ` To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: malloc to arrays?
On Wednesday, August 02, 2000, [EMAIL PROTECTED] wrote: Sorry I have no other direction to ask this: I have declaration: u_int32_t *(tcpcash_addr[256]); and want malloc some memory for tcpcash_addr: tcpcash_addr = (typeof(tcpcash_addr))malloc(u_int32_t * 256 * n); and have an error: ra-sum.c:386: cast specifies array type *** Error code 1 Try this: u_int32_t *tcphash_haddr; tcphash_addr = malloc(sizeof(*tcphash_addr) * 256); You shouldn't be casting malloc and I don't see any reason to use *(foo[nmemb]) syntax. -- |Chris Costello [EMAIL PROTECTED] |Performance proven: It works through beta test. `--- To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: malloc to arrays?
Chris Costello wrote: On Wednesday, August 02, 2000, [EMAIL PROTECTED] wrote: Sorry I have no other direction to ask this: I have declaration: u_int32_t *(tcpcash_addr[256]); and want malloc some memory for tcpcash_addr: tcpcash_addr = (typeof(tcpcash_addr))malloc(u_int32_t * 256 * n); and have an error: ra-sum.c:386: cast specifies array type *** Error code 1 Try this: u_int32_t *tcphash_haddr; tcphash_addr = malloc(sizeof(*tcphash_addr) * 256); You shouldn't be casting malloc and I don't see any reason to use *(foo[nmemb]) syntax. u_int32_t **tcphash_haddr; From the code, it's a variable-sized array of fixed-sized arrays. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] Satan was once an angel, Gates started by writing a BASIC interpreter. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
