Re: [RFC] how to get the size of a malloc(9) block ?
jb gmail.com> writes: > ... > So far, the options could be as follows: > - realloc_flags(p, s, option) > where option is one or a combination (where appropriate) of: > REALLOCF_ANY- default (move or no-move; regular > realloc()) Actually, the REALLOCF_ANY flag is not needed to be able to call default behavior, like in realloc_flags(p, s). > REALLOCF_NO_MOVE- no-move > REALLOCF_ELASTIC- combined with REALLOCF_NO_MOVE > REALLOCF_FORCE - combined with REALLOCF_NO_MOVE > REALLOCF_FALLBACK_ANY - combined with REALLOCF_NO_MOVE or its > derivatives like REALLOCF_ELASTIC, etc So, the REALLOCF_FALLBACK_ANY could be just renamed to REALLOCF_DEFAULT. This way we reduced the number of option tags to four :-) jb ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On Friday, November 29, 2013 6:16:01 am David Chisnall wrote: > > On 28 Nov 2013, at 15:13, jb wrote: > > > Luigi Rizzo iet.unipi.it> writes: > > > >> ... > >> But I don't understand why you find ksize()/malloc_usable_size() dangerous. > >> ... > > > > The original crime is commited when *usable size* (an implementation detail) > > is exported (leaked) to the caller. > > To be blunt, when a caller requests memory of certain size, and its request > > is > > satisfied, then it is not its business to learn details beyond that (and > > they > > should not be offered as well). > > The API should be sanitized, in kernel and user space. > > Otherwise, all kind of charlatans will try to play hair-raising games with > > it. > > If the caller wants to track the *requested size* programmatically, it is > > its > > business to do it and it can be done very easily. > > > > Some of these guys got it perfectly right: > > http://stackoverflow.com/questions/5813078/is-it-possible-to-find-the-memory-allocated-to-the-pointer-without-searching-fo > > I disagree. I've encountered several occasions where either locality > doesn't matter so much or I know the pointer is aliased, and I'd like > increase the size of a relatively large allocation. I have two choices: > > - Call realloc(), potentially copying a lot of data > - Call malloc(), and chain two (or more) allocations together. > > What I'd like to do is call realloc() if it's effectively free, or call > malloc() in other cases. > > The malloc_useable_size() API is wrong though. In the kernel, realloc() > already takes a flag and a M_DONTALLOCATE would make more sense, enlarging > the allocation if it can be done without doing the allocate-copy-free dance, > but returning NULL and leaving the allocation unmodified if not. This sounds sensible to me. There might be cases where you'd like to know how much you can grow an allocation by "for free", and M_DONTALLOCATE doesn't help you with that. In general, I don't like malloc_usable_size(). OTOH, this is C, not C# or Python. Foot-shooting is permitted. -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
Luigi Rizzo iet.unipi.it> writes: > ... > > So far, the options could be as follows: > > - realloc_flags(p, s, option) > > where option is one or a combination (where appropriate) of: > > REALLOCF_ANY- default (move or no-move; regular > > realloc()) > > REALLOCF_NO_MOVE- no-move > > REALLOCF_ELASTIC- combined with REALLOCF_NO_MOVE > > REALLOCF_FORCE - combined with REALLOCF_NO_MOVE > > REALLOCF_FALLBACK_ANY - combined with REALLOCF_NO_MOVE or its > > derivatives like REALLOCF_ELASTIC, etc > > > > just five ? for a (quote) "clean, safe and maintainable API", > I'd probably also add a few more, such as > REALLOCF_ALWAYS to trigger bugs on bad assumptions in the code, > REALLOCF_I_AM_FEELING_LUCKY for the newbies, and > REALLOCF_REAL_PROGRAMMERS_NEVER_DO_THAT_I_WILL_PANIC > for skilled folks. > ... These are more or less part of current implementation of realloc() :-) But seriously, the new API takes advantage of current logic - the no-move is already implemented as part of default. It will not (and should not) interfere with current implementation-specific details; it will just be smarter and useful thru its options by asking specific requests, some of which could be already be partially satisified now (think of that extra, unused allocated space, if present), thus giving a programmer more power in one call. Adding some if-else logic and perhaps limited code restructuring will yield a really powerful, functional API. Some of the hidden bugs in current realloc() & co, if any, will be discovered during testing of new implementation (which will be more specific and thus less forgiving). Think of it as being presented with a chance to become part of history, as a co-creator of a new-and-improved memory management function, admittedly being many OS' and libraries' Achilles' heel :-) jb ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On Mon, Dec 2, 2013 at 4:36 AM, jb wrote: > gmx.com> writes: > > > > > So new flags could be [1]: > > - realloc_flags(p, s, REALLOCF_NO_MOVE) > > ... > > - realloc_flags(p, s, REALLOCF_NO_MOVE | REALLOCF_ELASTIC) > > ... > > For this, there could be a REALLOCF_FORCE flag > > In case of realloc_flags() failing the request, to avoid unnecessary > followups with regular realloc() when desired, we should include an option > like REALLOCF_FALLBACK_ANY that would allow to fallback on the regular > realloc() logic, in one call. > > In addition, because I have an impression that realloc_flags() may have > a future as a replacement for regular realloc() and we should aim for it, > we should include an option like REALLOCF_ANY for that purpose. > > So far, the options could be as follows: > - realloc_flags(p, s, option) > where option is one or a combination (where appropriate) of: > REALLOCF_ANY- default (move or no-move; regular > realloc()) > REALLOCF_NO_MOVE- no-move > REALLOCF_ELASTIC- combined with REALLOCF_NO_MOVE > REALLOCF_FORCE - combined with REALLOCF_NO_MOVE > REALLOCF_FALLBACK_ANY - combined with REALLOCF_NO_MOVE or its > derivatives like REALLOCF_ELASTIC, etc > just five ? for a (quote) "clean, safe and maintainable API", I'd probably also add a few more, such as REALLOCF_ALWAYS to trigger bugs on bad assumptions in the code, REALLOCF_I_AM_FEELING_LUCKY for the newbies, and REALLOCF_REAL_PROGRAMMERS_NEVER_DO_THAT_I_WILL_PANIC for skilled folks. I am not sure they are enough to cover the spectrum of possible options, but there are at least 32 bits in the flags so we have a few more left. cheers luigi ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
gmx.com> writes: > > So new flags could be [1]: > - realloc_flags(p, s, REALLOCF_NO_MOVE) > ... > - realloc_flags(p, s, REALLOCF_NO_MOVE | REALLOCF_ELASTIC) > ... > For this, there could be a REALLOCF_FORCE flag In case of realloc_flags() failing the request, to avoid unnecessary followups with regular realloc() when desired, we should include an option like REALLOCF_FALLBACK_ANY that would allow to fallback on the regular realloc() logic, in one call. In addition, because I have an impression that realloc_flags() may have a future as a replacement for regular realloc() and we should aim for it, we should include an option like REALLOCF_ANY for that purpose. So far, the options could be as follows: - realloc_flags(p, s, option) where option is one or a combination (where appropriate) of: REALLOCF_ANY- default (move or no-move; regular realloc()) REALLOCF_NO_MOVE- no-move REALLOCF_ELASTIC- combined with REALLOCF_NO_MOVE REALLOCF_FORCE - combined with REALLOCF_NO_MOVE REALLOCF_FALLBACK_ANY - combined with REALLOCF_NO_MOVE or its derivatives like REALLOCF_ELASTIC, etc jb ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
Daniel Nebdal gmail.com> writes: > ... > That could alternatively be solved by having an "if I ask for N bytes right > now, how large would the block be" - API that doesn't promise too much. > Call it something like "malloc_suggest_size(size_t minsize) ", and make the > description something like ... "return the largest number of bytes that > would not allocate a larger block of memory than the provided minsize, in > the current memory situation", plus some veiled threats about not using > this value to do anything fancy with pointers to already-allocated memory. Yeap. It is like asking teenagers to be abstinent ... http://www.youtube.com/watch?v=SWlbN2b1PGg Good luck ! jb ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On Sun, Dec 1, 2013 at 4:04 AM, wrote: > John-Mark Gurney wrote, On 12/01/2013 03:20: > > Either it happens rarely, and always doing a realloc won't hurt >> performance, or it happens often, and then you should be using a larger >> buffer in the first place.. >> > > What if a size-elastic implementation of a dynamic data structure would be > able to adjust to the malloc implementation, such as agreeing to allocate > regions of size (2^k - 8)? Much like the use of getpagesize() (yes, I know > it's not part of a technical standard). > > That could alternatively be solved by having an "if I ask for N bytes right now, how large would the block be" - API that doesn't promise too much. Call it something like "malloc_suggest_size(size_t minsize) ", and make the description something like ... "return the largest number of bytes that would not allocate a larger block of memory than the provided minsize, in the current memory situation", plus some veiled threats about not using this value to do anything fancy with pointers to already-allocated memory. -- Daniel Nebdal ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
John-Mark Gurney wrote, On 12/01/2013 03:20: Either it happens rarely, and always doing a realloc won't hurt performance, or it happens often, and then you should be using a larger buffer in the first place.. What if a size-elastic implementation of a dynamic data structure would be able to adjust to the malloc implementation, such as agreeing to allocate regions of size (2^k - 8)? Much like the use of getpagesize() (yes, I know it's not part of a technical standard). ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
Luigi Rizzo wrote this message on Fri, Nov 29, 2013 at 17:11 -0800: > On Fri, Nov 29, 2013 at 4:49 PM, Adrian Chadd wrote: > > > The reason I wouldn't implement this is to avoid having code that > > _relies_ on this behaviour in order to function or perform well. > > > nobody ever said (or could reasonably expect to do) that. > > Applications don't know if the allocator overallocates, > so they have no hope unless they work even without the feature. > > This is only about giving them an option to improve > performance in those (rare ?) cases where, as i showed, > knowing the underlying allocation size may lead to > better usage of memory. This sounds like optimizing before measuring if your putting in code like that... Either it happens rarely, and always doing a realloc won't hurt performance, or it happens often, and then you should be using a larger buffer in the first place.. Do I need to quote the optimization rules? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
gmx.com> writes: > ... > So new flags could be [1]: > - realloc_flags(p, s, REALLOCF_NO_MOVE): Resize object p, without moving > it, to size s. With this restriction, when requesting more memory, and > the specified amount isn't available, don't do anything (when requesting > less memory, always succeed). > - realloc_flags(p, s, REALLOCF_NO_MOVE | REALLOCF_ELASTIC): Resize object > p, without moving it, to size s. With this restriction, when requesting > more memory, and the specified amount isn't available, reserve as much as > possible (when requesting less memory, always succeed). > ... The realloc_flags(), having different behavior from realloc(), should state what happens if: If the pointer is a null pointer, the function does not change anything. If the new size is zero, the function does not change anything. If the new size is the same as the old size, the function does not change anything. The return values have to be reviewed also. The function returns: - a pointer to the object specifid on entry - a null pointer if the object could not be modified - a null pointer if there was insufficient free memory available to extend the size of the object Reference to realloc(): http://www.cplusplus.com/reference/cstdlib/realloc/ jb ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On Fri, Nov 29, 2013 at 08:37:28PM -0800, Tim Kientzle wrote: > > On Nov 29, 2013, at 3:44 PM, jb wrote: > > > Luigi Rizzo iet.unipi.it> writes: > > > >> ... > >> There is a difference between applications peeking into > >> implementation details that should be hidden, and providing > >> instead limited and specific information through a well defined API. > >> ... > > > > Right. > > > > If you want to improve memory management, that is, have the system (kernel > > or user space) handle memory reallocation intelligently and transparently > > to the user, then aim at a well defined API: > > Don?t forget: > > * Request a block of ?at least N bytes? and have the > allocator tell you what it *really* allocated. > > This allows applications to use memory more efficiently > by taking advantage of over-allocation when it happens. Taking random message in the thread. IMO, more important point of this API is not to allow the code to micro-optimize. By the nature of malloc-like API, consumer code usually pass only the allocated pointed around, without attached notion of the size. Just note the signature of free(9) or free(3). As consequence, any interceptor-like functionality, e.g. performing accounting, leak detection, consistency analysis or anything else somebody could imagine, has to either track correspondence between allocation address and size on its own, or dive deep into the allocator implementation to obtain the size from address. When I am working with user-space and doing any trick with malloc, this functionality is in fact absolutely required. pgpDj7Oy2TAjR.pgp Description: PGP signature
Re: [RFC] how to get the size of a malloc(9) block ?
gmx.com> writes: > ... > It appears that it's not possible to make a proper API with > malloc_usable_size() included, at least when > multi-threading is involved (ie., in the modern world). > > However, it is still useful to create an API that supports the following cases: > ... Well, this is a step forward toward achieving a well defined API for memory reallocation. But can we arrive at this goal without consideration for leaked implementation details via malloc_usable_size() & co ? We want to get rid of that leak and associated hacks. We want to induce reallocation function to do "the right thing" thru one API call with clear and smart options. If it does 90% of what we would ideally want, then the job is done. jb ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On Nov 29, 2013, at 3:44 PM, jb wrote: > Luigi Rizzo iet.unipi.it> writes: > >> ... >> There is a difference between applications peeking into >> implementation details that should be hidden, and providing >> instead limited and specific information through a well defined API. >> ... > > Right. > > If you want to improve memory management, that is, have the system (kernel > or user space) handle memory reallocation intelligently and transparently > to the user, then aim at a well defined API: Don’t forget: * Request a block of “at least N bytes” and have the allocator tell you what it *really* allocated. This allows applications to use memory more efficiently by taking advantage of over-allocation when it happens. Tim ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
What is the supposed definition of malloc_usable_size(p) in a hypothetical, upcoming C standard? With the rest of the C standard remaining the same, one could try: Definition: The value of malloc_usable_size(p) is the amount of space allocated for object p, plus the amount of space after object p that can currently be written without messing up other objects or memory-management areas. This definition is practically useless, because data written to the slack space after the object could be claimed by calls to alloc()ish function -- perhaps by other threads. Another attempt at the defining something useful: Definition: The value of malloc_usable_size(p) is the amount of space allocated for object p, plus the amount of space after object p that can be written without messing up other objects or memory-management areas, while alloc()ish functions are not called on p. With this, one asks the question: How much is usually overallocated? In some implementations, usually just a few bytes (say, when the minimal allocation unit is 8 bytes); where not, it can be said that the memory manager is quite space-leaky. It appears that it's not possible to make a proper API with malloc_usable_size() included, at least when multi-threading is involved (ie., in the modern world). However, it is still useful to create an API that supports the following cases: - A program knows how to adapt to memory fragmentation without moving an ever-growing, but chainable array of data. - A program would become faster, if it knew when moving is required; then, the program could update various pointer-based (as opposed to arrayindex-based) references to the object being moved. (Just like when memory is defragmentated in a garbage-collected programming language.) - A program requires more memory in real-time, which means to either receive more memory immediately and do something, or to signal a real-time failure. So new flags could be [1]: - realloc_flags(p, s, REALLOCF_NO_MOVE): Resize object p, without moving it, to size s. With this restriction, when requesting more memory, and the specified amount isn't available, don't do anything (when requesting less memory, always succeed). - realloc_flags(p, s, REALLOCF_NO_MOVE | REALLOCF_ELASTIC): Resize object p, without moving it, to size s. With this restriction, when requesting more memory, and the specified amount isn't available, reserve as much as possible (when requesting less memory, always succeed). On the other hand, be advised of a hypothetical scenario, in which realloc() would like to jump at the opportunity to move the object to a different space, say, for the purpose of condensing slack space, when statistics show that allocated areas have plenty of holes. This means that the design of the new API can have more goals: - The allocator implementation should be able to shape the workings of a program at quick-realloc points, for example, by coaxing it to call realloc() when memory is very scattered. - The program should always be able to take advantage of a quick-realloc functionality, for example, to support certain real-time requirements of applications, to the extent reasonably possible within the implementation. For this, there could be a REALLOCF_FORCE flag, to be used in real-time scenarios. Without the flag, the call can be expected to be rejected on the basis of some implementation-specific preference, such as anti-fragmentation. Is there any insufficiency in this API, in anyone's mind? [1] When such a distinction makes sense and is supported (not stubbed) in the current architecture, environment, implementation, etc. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On Fri, Nov 29, 2013 at 5:02 PM, jb wrote: > Luigi Rizzo iet.unipi.it> writes: > > > ... > > > If you want to improve memory management, that is, have the system > (kernel > > > or user space) handle memory reallocation intelligently and > transparently > > > to the user, then aim at a well defined API: > > > - reallocate "with no copy", which means new space appended (taking > into > > > account *usable size*, a hidden-to-user implementation detail), if > > > possible > > > - otherwise fail, and let the user decide about reallocation "with > copy" > > > or allocation of a new space > > > > > > > i respectfully disagree :) but am not pushing to add ksize. > > Just note that both mine and your "well defined API" leak details: > > > > yours is (A) "I may be overallocating but won't tell you how much"; > > mine is (B) "I may be overallocating and here is exactly how much". > > > > Now if I may make a comparison with going shopping, > > I'd rather hear the final price from the seller (case B), > > than having to guess by repeated trial and error, > > which is what case A leads to if i really want to figure out. > > ... > > This is not necessarily true - I omitted the details of reallocation > implementation on purpose. > From the caller's point of view, if it requested allocation of memory > size, then that's what it wanted in the first place. If it got it, then > there is no other info needed. > This is not what we are discussing. We are discussing the case where the caller, _before_ requesting extra memory, would like to know how much space is available to make different decisions such as 1. realloc unconditionally 2. give up 3. allocate a separate block and chain to it 4. reduce its requirements and live with what extra space is available (if any). Your suggested flags support #1 and #2 directly, #3 can be simulated with realloc(NO_ALLOC) + malloc(), but prevent #4. cheers luigi Next, if the caller came to the conclusion that more would be needed, then > it should ask for memory reallocation, trusting that the system will do it > in the most efficient way. > If the caller wants to influence that process, then proper option(s) are > needed in reallocation API, e.g.: > - with no copy > - with copy > That means one call with options, with a specific (wanted by user) result. > Of course, thinking thru the options (default, mutual exclusion, etc) is > an important process and subject to RFC. > A user-empowering API. No magic, no hacks. > > So, how about Request-for-Enhancement to GNU C lib, and the ugly hacks > will disappear quickly. > > jb > > > > ___ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" > -- -+--- Prof. Luigi RIZZO, ri...@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/. Universita` di Pisa TEL +39-050-2211611 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -+--- ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On Fri, Nov 29, 2013 at 4:49 PM, Adrian Chadd wrote: > The reason I wouldn't implement this is to avoid having code that > _relies_ on this behaviour in order to function or perform well. > nobody ever said (or could reasonably expect to do) that. Applications don't know if the allocator overallocates, so they have no hope unless they work even without the feature. This is only about giving them an option to improve performance in those (rare ?) cases where, as i showed, knowing the underlying allocation size may lead to better usage of memory. > > Heck, it may not even be portable to other operating systems. Except, > Linux, I guess. > > in userspace, as jb commented, all major OSes have it (malloc_usable_size() on FreeBSD and Linux, _msize() on Windows, malloc_size() on OSX). In the kernel, I have no idea, but porting kernel code across systems is a nightmare anyways... cheers luigi ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
Luigi Rizzo iet.unipi.it> writes: > ... > > If you want to improve memory management, that is, have the system (kernel > > or user space) handle memory reallocation intelligently and transparently > > to the user, then aim at a well defined API: > > - reallocate "with no copy", which means new space appended (taking into > > account *usable size*, a hidden-to-user implementation detail), if > > possible > > - otherwise fail, and let the user decide about reallocation "with copy" > > or allocation of a new space > > > > i respectfully disagree :) but am not pushing to add ksize. > Just note that both mine and your "well defined API" leak details: > > yours is (A) "I may be overallocating but won't tell you how much"; > mine is (B) "I may be overallocating and here is exactly how much". > > Now if I may make a comparison with going shopping, > I'd rather hear the final price from the seller (case B), > than having to guess by repeated trial and error, > which is what case A leads to if i really want to figure out. > ... This is not necessarily true - I omitted the details of reallocation implementation on purpose. >From the caller's point of view, if it requested allocation of memory size, then that's what it wanted in the first place. If it got it, then there is no other info needed. Next, if the caller came to the conclusion that more would be needed, then it should ask for memory reallocation, trusting that the system will do it in the most efficient way. If the caller wants to influence that process, then proper option(s) are needed in reallocation API, e.g.: - with no copy - with copy That means one call with options, with a specific (wanted by user) result. Of course, thinking thru the options (default, mutual exclusion, etc) is an important process and subject to RFC. A user-empowering API. No magic, no hacks. So, how about Request-for-Enhancement to GNU C lib, and the ugly hacks will disappear quickly. jb ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
The reason I wouldn't implement this is to avoid having code that _relies_ on this behaviour in order to function or perform well. Heck, it may not even be portable to other operating systems. Except, Linux, I guess. -adrian ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On Fri, Nov 29, 2013 at 3:44 PM, jb wrote: > Luigi Rizzo iet.unipi.it> writes: > > > ... > > There is a difference between applications peeking into > > implementation details that should be hidden, and providing > > instead limited and specific information through a well defined API. > > ... > > Right. > > If you want to improve memory management, that is, have the system (kernel > or user space) handle memory reallocation intelligently and transparently > to the user, then aim at a well defined API: > - reallocate "with no copy", which means new space appended (taking into > account *usable size*, a hidden-to-user implementation detail), if > possible > - otherwise fail, and let the user decide about reallocation "with copy" > or allocation of a new space > i respectfully disagree :) but am not pushing to add ksize. Just note that both mine and your "well defined API" leak details: yours is (A) "I may be overallocating but won't tell you how much"; mine is (B) "I may be overallocating and here is exactly how much". Now if I may make a comparison with going shopping, I'd rather hear the final price from the seller (case B), than having to guess by repeated trial and error, which is what case A leads to if i really want to figure out. > The malloc_usable_size() is a hack. > The extra space allocated or not due to fragmentation, alignment, etc, is > an internal by-product, irrelevant to original memory alloc request, and it > should not be leaked, also because its details may change in future API > implementations. > So, these memory allocation functions leaking implementation details, and > the two derived functions, ksize() and malloc_usable_size() (and other > derivatives like malloc_size() in Mac OS X), are a violations of a clean, > safe, and maintainable API. > > Note that malloc_usable_size() is a GNU C Library extension, not part of > Single UNIX Specification. > Honestly i did not even know they existed until a few days ago; but the fact that many different systems have come out with similar extensions at least make me wonder whether the SUS missed it. cheers luigi ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
Luigi Rizzo iet.unipi.it> writes: > ... > There is a difference between applications peeking into > implementation details that should be hidden, and providing > instead limited and specific information through a well defined API. > ... Right. If you want to improve memory management, that is, have the system (kernel or user space) handle memory reallocation intelligently and transparently to the user, then aim at a well defined API: - reallocate "with no copy", which means new space appended (taking into account *usable size*, a hidden-to-user implementation detail), if possible - otherwise fail, and let the user decide about reallocation "with copy" or allocation of a new space The malloc_usable_size() is a hack. The extra space allocated or not due to fragmentation, alignment, etc, is an internal by-product, irrelevant to original memory alloc request, and it should not be leaked, also because its details may change in future API implementations. So, these memory allocation functions leaking implementation details, and the two derived functions, ksize() and malloc_usable_size() (and other derivatives like malloc_size() in Mac OS X), are a violations of a clean, safe, and maintainable API. Note that malloc_usable_size() is a GNU C Library extension, not part of Single UNIX Specification. jb ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On Thu, Nov 28, 2013 at 7:13 AM, jb wrote: > Luigi Rizzo iet.unipi.it> writes: > > > ... > > But I don't understand why you find ksize()/malloc_usable_size() > dangerous. > > ... > > The original crime is commited when *usable size* (an implementation > detail) > is exported (leaked) to the caller. > To be blunt, when a caller requests memory of certain size, and its > request is > satisfied, then it is not its business to learn details beyond that (and > they > should not be offered as well). > The API should be sanitized, in kernel and user space. > Otherwise, all kind of charlatans will try to play hair-raising games with > it. > If the caller wants to track the *requested size* programmatically, it is > its > business to do it and it can be done very easily. > There is a difference between applications peeking into implementation details that should be hidden, and providing instead limited and specific information through a well defined API. In general (not in the specific code I am handling and not something I personally need), what the caller might want to do is optimize its requests according to how system behaves, and it cannot do that without some help from the below. I have seen the following types of comments in this thread: - "you should get it right the first time and never realloc" Maybe, but then the offending api is realloc() not ksize() - "build your own allocator" Yes i do it when it makes sense, but sometimes it is either overkill or a bad idea (as it loses opportunities for global optimizations, duplicates code, takes memory in subsystem-specific freelists...) - "what if ksize()/malloc_usable_size() lies ?" Well, that would be a bug in the allocator: if it says the memory is usable, it must be usable, period. - "rather than ksize() i'll give you a fix for one use case" (the NO_REALLOC flag to realloc()). This i think would be a mistake -- it acknowledges the need for exposing some information but then only provides a specific fix for one use case. I'll just restate that there are multiple situations where an application might use some information on actual allocation sizes: - when it needs to extend memory and has a choice between a cheap realloc() (if extra space is available), chaining blocks (when the memcpy would be too expensive), give up and live with whatever space is available. - when it has freedom in picking the block size and so it wants to optimize its requests basing on what the underlying allocator does. As an example, long ago FreeBSD was really suboptimal when you allocated blocks whose size was a power of 2, because the metadata was inline. These days, there is a different issue: powers of 2 are ok but blocks 2049 bytes and above seem to be padded to a multiple of 2048, leading to a huge overhead in some cases. cheers luigi ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On 11/29/13, 7:26 PM, Daniel Nebdal wrote: On Fri, Nov 29, 2013 at 11:59 AM, Gleb Smirnoff wrote: On Thu, Nov 28, 2013 at 03:13:53PM +, jb wrote: j> > But I don't understand why you find ksize()/malloc_usable_size() dangerous. j> > ... j> j> The original crime is commited when *usable size* (an implementation detail) j> is exported (leaked) to the caller. j> To be blunt, when a caller requests memory of certain size, and its request is j> satisfied, then it is not its business to learn details beyond that (and they j> should not be offered as well). j> The API should be sanitized, in kernel and user space. j> Otherwise, all kind of charlatans will try to play hair-raising games with it. j> If the caller wants to track the *requested size* programmatically, it is its j> business to do it and it can be done very easily. +1 This is kind of APIs that just shouldn't exist. -- Totus tuus, Glebius. Then again: Using the "overflow" memory is only going to bite them if the API lies : If the return value is exactly "the size of the block you got allocated and can safely use until you free it", using it will per definition be safe. If the allocator later changes to, say, always allocate exact byte ranges, or to allocating blocks but having the option to fragment them later - then the return value would have to shrink to match, and any program using it would still DTRT. I'm completely ambivalent about adding it, though - it's not something I need, it's more stuff that needs to be handled if you change/rewrite the allocator, and it's not my decision. I think that if you want to play games with expanding buffers etc, then you should write your own allocator. You asked for X bytes. you should expect that you get X bytes and nothing more... either that or you should have asked for more in the first place. -- Daniel Nebdal ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On Fri, Nov 29, 2013 at 11:59 AM, Gleb Smirnoff wrote: > On Thu, Nov 28, 2013 at 03:13:53PM +, jb wrote: > j> > But I don't understand why you find ksize()/malloc_usable_size() > dangerous. > j> > ... > j> > j> The original crime is commited when *usable size* (an implementation > detail) > j> is exported (leaked) to the caller. > j> To be blunt, when a caller requests memory of certain size, and its > request is > j> satisfied, then it is not its business to learn details beyond that > (and they > j> should not be offered as well). > j> The API should be sanitized, in kernel and user space. > j> Otherwise, all kind of charlatans will try to play hair-raising games > with it. > j> If the caller wants to track the *requested size* programmatically, it > is its > j> business to do it and it can be done very easily. > > +1 > > This is kind of APIs that just shouldn't exist. > > -- > Totus tuus, Glebius. > Then again: Using the "overflow" memory is only going to bite them if the API lies : If the return value is exactly "the size of the block you got allocated and can safely use until you free it", using it will per definition be safe. If the allocator later changes to, say, always allocate exact byte ranges, or to allocating blocks but having the option to fragment them later - then the return value would have to shrink to match, and any program using it would still DTRT. I'm completely ambivalent about adding it, though - it's not something I need, it's more stuff that needs to be handled if you change/rewrite the allocator, and it's not my decision. -- Daniel Nebdal ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On 28 Nov 2013, at 15:13, jb wrote: > Luigi Rizzo iet.unipi.it> writes: > >> ... >> But I don't understand why you find ksize()/malloc_usable_size() dangerous. >> ... > > The original crime is commited when *usable size* (an implementation detail) > is exported (leaked) to the caller. > To be blunt, when a caller requests memory of certain size, and its request is > satisfied, then it is not its business to learn details beyond that (and they > should not be offered as well). > The API should be sanitized, in kernel and user space. > Otherwise, all kind of charlatans will try to play hair-raising games with it. > If the caller wants to track the *requested size* programmatically, it is its > business to do it and it can be done very easily. > > Some of these guys got it perfectly right: > http://stackoverflow.com/questions/5813078/is-it-possible-to-find-the-memory-allocated-to-the-pointer-without-searching-fo I disagree. I've encountered several occasions where either locality doesn't matter so much or I know the pointer is aliased, and I'd like increase the size of a relatively large allocation. I have two choices: - Call realloc(), potentially copying a lot of data - Call malloc(), and chain two (or more) allocations together. What I'd like to do is call realloc() if it's effectively free, or call malloc() in other cases. The malloc_useable_size() API is wrong though. In the kernel, realloc() already takes a flag and a M_DONTALLOCATE would make more sense, enlarging the allocation if it can be done without doing the allocate-copy-free dance, but returning NULL and leaving the allocation unmodified if not. David ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On Thu, Nov 28, 2013 at 03:13:53PM +, jb wrote: j> > But I don't understand why you find ksize()/malloc_usable_size() dangerous. j> > ... j> j> The original crime is commited when *usable size* (an implementation detail) j> is exported (leaked) to the caller. j> To be blunt, when a caller requests memory of certain size, and its request is j> satisfied, then it is not its business to learn details beyond that (and they j> should not be offered as well). j> The API should be sanitized, in kernel and user space. j> Otherwise, all kind of charlatans will try to play hair-raising games with it. j> If the caller wants to track the *requested size* programmatically, it is its j> business to do it and it can be done very easily. +1 This is kind of APIs that just shouldn't exist. -- Totus tuus, Glebius. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
Luigi Rizzo iet.unipi.it> writes: > ... > But I don't understand why you find ksize()/malloc_usable_size() dangerous. > ... The original crime is commited when *usable size* (an implementation detail) is exported (leaked) to the caller. To be blunt, when a caller requests memory of certain size, and its request is satisfied, then it is not its business to learn details beyond that (and they should not be offered as well). The API should be sanitized, in kernel and user space. Otherwise, all kind of charlatans will try to play hair-raising games with it. If the caller wants to track the *requested size* programmatically, it is its business to do it and it can be done very easily. Some of these guys got it perfectly right: http://stackoverflow.com/questions/5813078/is-it-possible-to-find-the-memory-allocated-to-the-pointer-without-searching-fo jb ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
On Thu, Nov 28, 2013 at 01:33:41PM +, jb wrote: > Luigi Rizzo iet.unipi.it> writes: > > > > > in porting some linux kernel code to FreeBSD we > > stumbled upon ksize(), which returns the > > actual size of a kmalloc() block. > > > > We could easily implement it as the first part > > of realloc(9) -- see kern/kern_malloc.c > > > > Would it make sense to add this to the malloc(9) API ? > > The userspace equivalent seems to be > > malloc_usable_size(3) which according to the > > manpage appeared in FreeBSD 7.0 > > Hi, > > The implementation of ksize() depends on (has non-standard behavior across) > architectures, memory allocators, page sizes, C libs, etc. > > It means, ksize() exports its implementation details to the caller, and this > disqualifies it, regardless whether in kernel or user spaces. > > This leads to dangerous and conflicting assertions: > > malloc_usable_size(3) on Linux: > NOTES >The value returned by malloc_usable_size() may be greater than the >requested size of the allocation because of alignment and minimum size >constraints. Although the excess bytes can be overwritten by the >application without ill effects, this is not good programming practice: >the number of excess bytes in an allocation depends on the underlying >implementation. > >The main use of this function is for debugging and introspection. This is the same exact text we have in the FreeBSD manpage, and exactly the behaviour of ksize() in the linux kernel (and exactly the semantics that our realloc(9) relies upon). While I personally prefer that applications call realloc() directly, there might be cases where the decision is best left to the application (or kernel code): e.g. say you would like to get some extra space, but would rather do without it than risk a malloc() failure or having to sleep for the allocation. And there is also a (minor) issue of portability of code. But I don't understand why you find ksize()/malloc_usable_size() dangerous. Of course the result depends on the underlying implementation, but this happens in a ton of places including compiler behaviour and system calls. The danger is when the programmer makes assumptions that do not match reality, but the purpose of this call seems to be exactly the opposite: avoid making assumptions. cheers luigi ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] how to get the size of a malloc(9) block ?
Luigi Rizzo iet.unipi.it> writes: > > in porting some linux kernel code to FreeBSD we > stumbled upon ksize(), which returns the > actual size of a kmalloc() block. > > We could easily implement it as the first part > of realloc(9) -- see kern/kern_malloc.c > > Would it make sense to add this to the malloc(9) API ? > The userspace equivalent seems to be > malloc_usable_size(3) which according to the > manpage appeared in FreeBSD 7.0 Hi, The implementation of ksize() depends on (has non-standard behavior across) architectures, memory allocators, page sizes, C libs, etc. It means, ksize() exports its implementation details to the caller, and this disqualifies it, regardless whether in kernel or user spaces. This leads to dangerous and conflicting assertions: malloc_usable_size(3) on Linux: NOTES The value returned by malloc_usable_size() may be greater than the requested size of the allocation because of alignment and minimum size constraints. Although the excess bytes can be overwritten by the application without ill effects, this is not good programming practice: the number of excess bytes in an allocation depends on the underlying implementation. The main use of this function is for debugging and introspection. Other sources: ... The caller may use this additional memory, even though a smaller amount of memory was initially specified with the kmalloc call. ... This is hair-raising: http://lwn.net/Articles/319686/ Result ? The same code works here, but may not elsewhere. It follows, you should remove malloc_usable_size() from user space instead. jb ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
