Re: std.allocator ready for some abuse
On Saturday, 12 September 2015 at 06:45:16 UTC, bitwise wrote: [...] Alternatively, GC.addRange() could return a value indicating whether or not the range had actually been added(for the first time) and should be removed. Bit Maybe the solution is as simple as specifying the state of memory that should be received from an untyped allocator. It could be explicitly stated that an untyped allocator should give out raw memory, and should not initialize it's content or add any ranges to the GC. While it may seem obvious for a C++ allocator not to initialize it's contents, I think this makes sense in the presence of a GC. I would appreciate some feedback on this. Bit
Re: std.allocator ready for some abuse
On 09/14/2015 10:46 AM, bitwise wrote: On Saturday, 12 September 2015 at 06:45:16 UTC, bitwise wrote: [...] Alternatively, GC.addRange() could return a value indicating whether or not the range had actually been added(for the first time) and should be removed. Bit Maybe the solution is as simple as specifying the state of memory that should be received from an untyped allocator. It could be explicitly stated that an untyped allocator should give out raw memory, and should not initialize it's content or add any ranges to the GC. While it may seem obvious for a C++ allocator not to initialize it's contents, I think this makes sense in the presence of a GC. I would appreciate some feedback on this. Bit I think what we need here is a GCConnectedAllocator that can be inserted at the bottom of the allocation foodchain to insert calls to addRange and removeRange appropriately. -- Andrei
Re: std.allocator ready for some abuse
On Saturday, 12 September 2015 at 06:30:42 UTC, bitwise wrote: On Saturday, 12 September 2015 at 00:31:27 UTC, Andrei Alexandrescu wrote: [...] This sounds like the default setup I'll be using. The containers will use a Mallocator by default, so I will have to add the ranges when the contained type is found to have indirections. [...] Alternatively, GC.addRange() could return a value indicating whether or not the range had actually been added(for the first time) and should be removed. Bit
Re: std.allocator ready for some abuse
On Saturday, 12 September 2015 at 00:31:27 UTC, Andrei Alexandrescu wrote: On 09/11/2015 07:46 PM, bitwise wrote: [...] Say you have a container that uses its own allocator inside, yet offers the user to store objects with indirections that use the GC. Then indeed the container would need to call addRange and removeRange on its own internal structures. This sounds like the default setup I'll be using. The containers will use a Mallocator by default, so I will have to add the ranges when the contained type is found to have indirections. If, on the contrary, the container imposes that its own held objects use the container's allocator as well (a rare but not implausible design), it wouldn't need to do that. I don't understand exactly what you mean, but here is a more verbose version of my concern: Lets say someone's allocator calls GC.addRange on the memory it allocates before a container gets it. The container would call GC.addRange when it gets it, but then call GC.removeRange before calling allocator.deallocate(). The allocator which think's the block has already been registered with the GC could possibly reuse it, thinking it's registered with the GC alreadybut that allocator may pass the memory to a container which chooses not to call GC.addRange, because it thinks it's using an allocator that does that for it. That second container may now store types with indirections, which would be unreachable, and get freed, and..chaos. I suppose that as a general rule, a container could always add ranges it gets from an allocator to the GC, which would prevent the above problem...but only for containers that abide by the rule. It sounds like you're saying that although it may not be needed sometimes, that no harm would be done. I think a better solution though would be for the GC to have something like GC.hasRange(). The container could check the state of the memory it receives and take appropriate action. I don't see anything in the current GC docs which would allow you to tell if a range had already been added to the GC. Bit
Re: std.allocator ready for some abuse
Am 11.09.2015 um 04:35 schrieb bitwise: On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote: Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html Am I the only one seeing dead links? The code has moved to std.experimental: https://github.com/andralex/phobos/tree/allocator/std/experimental/allocator http://erdani.com/d/phobos-prerelease/std_experimental_allocator.html
Re: std.allocator ready for some abuse
On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu
wrote:
On 10/24/13 2:38 PM, Namespace wrote:
On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
Awesome! Will Appender get an option to use a suitable
allocator?
A dream of me, that will probably never come true, would be
also
something like this:
with (Mallocator) {
int[] arr;
arr ~= 42; /// will use Mallocator.it.allocate internal
}
Oddly enough this can be actually done.
with (setAllocator!Mallocator)
{
...
}
setAllcator returns an rvalue that changes the global allocator
to the Mallocator in the constructor, and restores it to
whatever it was in the destructor.
Andrei
Doesn't this race because the allocator instance is shared?
I couldn't find 'setAllocator' in the source code.
Bit
Re: std.allocator ready for some abuse
On 09/11/2015 06:32 PM, bitwise wrote:
On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu wrote:
On 10/24/13 2:38 PM, Namespace wrote:
On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
Awesome! Will Appender get an option to use a suitable allocator?
A dream of me, that will probably never come true, would be also
something like this:
with (Mallocator) {
int[] arr;
arr ~= 42; /// will use Mallocator.it.allocate internal
}
Oddly enough this can be actually done.
with (setAllocator!Mallocator)
{
...
}
setAllcator returns an rvalue that changes the global allocator to the
Mallocator in the constructor, and restores it to whatever it was in
the destructor.
Andrei
Doesn't this race because the allocator instance is shared?
I couldn't find 'setAllocator' in the source code.
Yah, that was a rumination, not something already implemented. -- Andrei
Re: std.allocator ready for some abuse
On Friday, 11 September 2015 at 23:13:16 UTC, Andrei Alexandrescu
wrote:
On 09/11/2015 06:32 PM, bitwise wrote:
On Friday, 25 October 2013 at 00:00:36 UTC, Andrei
Alexandrescu wrote:
On 10/24/13 2:38 PM, Namespace wrote:
On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace
wrote:
Awesome! Will Appender get an option to use a suitable
allocator?
A dream of me, that will probably never come true, would be
also
something like this:
with (Mallocator) {
int[] arr;
arr ~= 42; /// will use Mallocator.it.allocate internal
}
Oddly enough this can be actually done.
with (setAllocator!Mallocator)
{
...
}
setAllcator returns an rvalue that changes the global
allocator to the
Mallocator in the constructor, and restores it to whatever it
was in
the destructor.
Andrei
Doesn't this race because the allocator instance is shared?
I couldn't find 'setAllocator' in the source code.
Yah, that was a rumination, not something already implemented.
-- Andrei
Ok, thanks. I thought that may be the case.
One more question:
I'd like to integrate these into the containers I'm building, but
I'm not clear about how to determine if GC.addRange() should be
called.
I've thought through this, and I'm pretty sure that I can call
GC.addRange and GC.removeRange indiscriminately on any memory my
container gets from an allocator, as long as the container finds
that (hasIndirections!T == true).
My reasoning is that if an allocator calls GC.addRange on it's
own memory, then it should also reinitialize that memory when it
gets deallocated, which would include calling GC.addRange again
if it had to. Also, calling GC.addRange or GC.removeRange on GC
allocated memory should have no effect.
Does this sound right, or am I crazy?
Bit
Re: std.allocator ready for some abuse
On 09/11/2015 07:46 PM, bitwise wrote:
On Friday, 11 September 2015 at 23:13:16 UTC, Andrei Alexandrescu wrote:
On 09/11/2015 06:32 PM, bitwise wrote:
On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu wrote:
On 10/24/13 2:38 PM, Namespace wrote:
On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
Awesome! Will Appender get an option to use a suitable allocator?
A dream of me, that will probably never come true, would be also
something like this:
with (Mallocator) {
int[] arr;
arr ~= 42; /// will use Mallocator.it.allocate internal
}
Oddly enough this can be actually done.
with (setAllocator!Mallocator)
{
...
}
setAllcator returns an rvalue that changes the global allocator to the
Mallocator in the constructor, and restores it to whatever it was in
the destructor.
Andrei
Doesn't this race because the allocator instance is shared?
I couldn't find 'setAllocator' in the source code.
Yah, that was a rumination, not something already implemented. -- Andrei
Ok, thanks. I thought that may be the case.
One more question:
I'd like to integrate these into the containers I'm building, but I'm
not clear about how to determine if GC.addRange() should be called.
I've thought through this, and I'm pretty sure that I can call
GC.addRange and GC.removeRange indiscriminately on any memory my
container gets from an allocator, as long as the container finds that
(hasIndirections!T == true).
My reasoning is that if an allocator calls GC.addRange on it's own
memory, then it should also reinitialize that memory when it gets
deallocated, which would include calling GC.addRange again if it had to.
Also, calling GC.addRange or GC.removeRange on GC allocated memory
should have no effect.
Does this sound right, or am I crazy?
Say you have a container that uses its own allocator inside, yet offers
the user to store objects with indirections that use the GC. Then indeed
the container would need to call addRange and removeRange on its own
internal structures.
If, on the contrary, the container imposes that its own held objects use
the container's allocator as well (a rare but not implausible design),
it wouldn't need to do that.
Andrei
Re: std.allocator ready for some abuse
On Friday, 1 November 2013 at 02:33:57 UTC, Andrei Alexandrescu wrote: Added SharedFreelist, a lock-free freelist. http://erdani.com/d/phobos-prerelease/std_allocator.html#.SharedFreelist Andrei Hi Andrei, Please check this bug fix for SharedFreelist https://github.com/andralex/phobos/pull/21 . I have found that source code for bounded `maxNodes` version of SharedFreelist is commented out. I understand that `maxNodes` can be only approximate bound for _shared_ free list. However, approximate `maxNodes` bound is very useful too. Can I create PR for this feature? Best Regards, --Ilya
Re: std.allocator ready for some abuse
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote: Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html Am I the only one seeing dead links?
Re: std.allocator ready for some abuse
On Friday, 27 February 2015 at 08:18:53 UTC, ANtlord wrote: I think, that if use this project https://github.com/andralex/std_allocator/, than you can post the issue to related issue tracker. Oh, I must be blind. I thought the issue tracker was disables on the repository in the same way as for Phobos. Thanks for checking. I submitted the issue. And I see, that types in traceback are different from source https://github.com/andralex/std_allocator/blob/master/source/std/allocator.d#L857. Maybe you need to upgrade package. If you mean size_t and uint difference it's because size_t is an alias for uint on 32-bit os. roundUpToMultipleOf(size_t s, uint base) becomes roundUpToMultipleOf (uint s, uint base) Piotrek
Re: std.allocator ready for some abuse
On Thursday, 26 February 2015 at 21:01:27 UTC, Piotrek wrote: Hi, Sorry for putting it here but I don't know where to file a bug report for the allocator project. On 32-bit linux with the latest dmd beta I get errors for ulong - uint (size_t) conversions. dmd -main -unittest allocator.d allocator.d(2015): Error: cannot implicitly convert expression (i * 4096LU) of type ulong to uint allocator.d(2015): Error: cannot implicitly convert expression ((i + cast(ulong)blocks) * 4096LU) of type ulong to uint allocator.d(1743): Error: template instance std.allocator.HeapBlock!(4096u, 4u) cut the long line allocator.d(331):instantiated from here: HeapBlock!(4096u, 4u) allocator.d(334): Error: template instance Segregator! cut the long line allocator.d(2015): Error: cannot implicitly convert expression (i * 128LU) of type ulong to uint allocator.d(2015): Error: cannot implicitly convert expression ((i + cast(ulong)blocks) * 128LU) of type ulong to uint allocator.d(1743): Error: template instance std.allocator.HeapBlock!(128u, 4u) cut the long line , __ctmp2303).this(m)) error instantiating allocator.d(1342):instantiated from here: HeapBlock!(128u, 4u) allocator.d(1493): Error: cannot implicitly convert expression (x / 64LU) of type ulong to immutable(uint) allocator.d(1495): Error: cannot implicitly convert expression (y / 64LU) of type ulong to immutable(uint) allocator.d(1520): Error: cannot implicitly convert expression (x / 64LU) of type ulong to uint allocator.d(1526): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(1527): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(1544): Error: cannot implicitly convert expression (w) of type ulong to uint allocator.d(1553): Error: cannot implicitly convert expression (w) of type ulong to uint allocator.d(1572): Error: cannot implicitly convert expression (w) of type ulong to uint allocator.d(1582): Error: cannot implicitly convert expression (w) of type ulong to uint allocator.d(1607): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(1615): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(1627): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(1633): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(4143): Error: function std.allocator.roundUpToMultipleOf (uint s, uint base) is not callable using argument types (ulong, uint) Is it a known issue? Or are there currently only 64-bit OSes supported by the allocator project? Piotrek I think, that if use this project https://github.com/andralex/std_allocator/, than you can post the issue to related issue tracker. And I see, that types in traceback are different from source https://github.com/andralex/std_allocator/blob/master/source/std/allocator.d#L857. Maybe you need to upgrade package. Sorry, if my english isn't clear.
Re: std.allocator ready for some abuse
Hi, Sorry for putting it here but I don't know where to file a bug report for the allocator project. On 32-bit linux with the latest dmd beta I get errors for ulong - uint (size_t) conversions. dmd -main -unittest allocator.d allocator.d(2015): Error: cannot implicitly convert expression (i * 4096LU) of type ulong to uint allocator.d(2015): Error: cannot implicitly convert expression ((i + cast(ulong)blocks) * 4096LU) of type ulong to uint allocator.d(1743): Error: template instance std.allocator.HeapBlock!(4096u, 4u) cut the long line allocator.d(331):instantiated from here: HeapBlock!(4096u, 4u) allocator.d(334): Error: template instance Segregator! cut the long line allocator.d(2015): Error: cannot implicitly convert expression (i * 128LU) of type ulong to uint allocator.d(2015): Error: cannot implicitly convert expression ((i + cast(ulong)blocks) * 128LU) of type ulong to uint allocator.d(1743): Error: template instance std.allocator.HeapBlock!(128u, 4u) cut the long line , __ctmp2303).this(m)) error instantiating allocator.d(1342):instantiated from here: HeapBlock!(128u, 4u) allocator.d(1493): Error: cannot implicitly convert expression (x / 64LU) of type ulong to immutable(uint) allocator.d(1495): Error: cannot implicitly convert expression (y / 64LU) of type ulong to immutable(uint) allocator.d(1520): Error: cannot implicitly convert expression (x / 64LU) of type ulong to uint allocator.d(1526): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(1527): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(1544): Error: cannot implicitly convert expression (w) of type ulong to uint allocator.d(1553): Error: cannot implicitly convert expression (w) of type ulong to uint allocator.d(1572): Error: cannot implicitly convert expression (w) of type ulong to uint allocator.d(1582): Error: cannot implicitly convert expression (w) of type ulong to uint allocator.d(1607): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(1615): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(1627): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(1633): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(4143): Error: function std.allocator.roundUpToMultipleOf (uint s, uint base) is not callable using argument types (ulong, uint) Is it a known issue? Or are there currently only 64-bit OSes supported by the allocator project? Piotrek
Re: std.allocator ready for some abuse
On Thursday, 12 February 2015 at 11:10:28 UTC, Paulo Pinto wrote: On Thursday, 12 February 2015 at 08:56:53 UTC, ANtlord wrote: On Wednesday, 11 February 2015 at 15:56:07 UTC, Andrei Alexandrescu wrote: On 2/11/15 4:55 AM, ANtlord wrote: On Monday, 17 February 2014 at 15:49:27 UTC, Andrei Alexandrescu wrote: On 2/17/14, 5:55 AM, Dicebot wrote: Andrei, what is current state of std.allocator? I am asking this in context of recent Walter Phobos proposal (ScopeBuffer) to evaluate how feasible is to define any relations between two at current stage. Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei Mr. Alexandrescu. What about std allocator? Is feature valuable? I've took a look in Wish List in wiki and Allocator still there. If this feature is valuable. What is left in currently implementation of your project? Can you tell about state, maybe show To Do list? Sorry, if my english is not clear. Thank you. Interface with garbage collector is to be done. The rest is usable right now. -- Andrei I thought, that general purpose of allocator it is memory management without garbage collector. Because allocator will be used in containers, and they must work without garbage collector. If no, what is difference between containers and arrays in current implementation of D? I assumed, that containers are need for increased performance within saving safety code. But a using GC will not increase performance. Or Am I dead wrong? You still need to cooperate with the GC and let it know which memory should be hands off, for example. Or just use it for the initial allocation and then ask it politely to release ownership of it. -- Paulo Aaa, thank you for explanation. I've got it. If the rest is usable right now and interface with GC. Why is not this module in std? Is it in queue for review?
Re: std.allocator ready for some abuse
On Thursday, 12 February 2015 at 08:56:53 UTC, ANtlord wrote: On Wednesday, 11 February 2015 at 15:56:07 UTC, Andrei Alexandrescu wrote: On 2/11/15 4:55 AM, ANtlord wrote: On Monday, 17 February 2014 at 15:49:27 UTC, Andrei Alexandrescu wrote: On 2/17/14, 5:55 AM, Dicebot wrote: Andrei, what is current state of std.allocator? I am asking this in context of recent Walter Phobos proposal (ScopeBuffer) to evaluate how feasible is to define any relations between two at current stage. Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei Mr. Alexandrescu. What about std allocator? Is feature valuable? I've took a look in Wish List in wiki and Allocator still there. If this feature is valuable. What is left in currently implementation of your project? Can you tell about state, maybe show To Do list? Sorry, if my english is not clear. Thank you. Interface with garbage collector is to be done. The rest is usable right now. -- Andrei I thought, that general purpose of allocator it is memory management without garbage collector. Because allocator will be used in containers, and they must work without garbage collector. If no, what is difference between containers and arrays in current implementation of D? I assumed, that containers are need for increased performance within saving safety code. But a using GC will not increase performance. Or Am I dead wrong? You still need to cooperate with the GC and let it know which memory should be hands off, for example. Or just use it for the initial allocation and then ask it politely to release ownership of it. -- Paulo
Re: std.allocator ready for some abuse
On Wednesday, 11 February 2015 at 15:56:07 UTC, Andrei Alexandrescu wrote: On 2/11/15 4:55 AM, ANtlord wrote: On Monday, 17 February 2014 at 15:49:27 UTC, Andrei Alexandrescu wrote: On 2/17/14, 5:55 AM, Dicebot wrote: Andrei, what is current state of std.allocator? I am asking this in context of recent Walter Phobos proposal (ScopeBuffer) to evaluate how feasible is to define any relations between two at current stage. Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei Mr. Alexandrescu. What about std allocator? Is feature valuable? I've took a look in Wish List in wiki and Allocator still there. If this feature is valuable. What is left in currently implementation of your project? Can you tell about state, maybe show To Do list? Sorry, if my english is not clear. Thank you. Interface with garbage collector is to be done. The rest is usable right now. -- Andrei I thought, that general purpose of allocator it is memory management without garbage collector. Because allocator will be used in containers, and they must work without garbage collector. If no, what is difference between containers and arrays in current implementation of D? I assumed, that containers are need for increased performance within saving safety code. But a using GC will not increase performance. Or Am I dead wrong?
Re: std.allocator ready for some abuse
On 2/11/15 4:55 AM, ANtlord wrote: On Monday, 17 February 2014 at 15:49:27 UTC, Andrei Alexandrescu wrote: On 2/17/14, 5:55 AM, Dicebot wrote: Andrei, what is current state of std.allocator? I am asking this in context of recent Walter Phobos proposal (ScopeBuffer) to evaluate how feasible is to define any relations between two at current stage. Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei Mr. Alexandrescu. What about std allocator? Is feature valuable? I've took a look in Wish List in wiki and Allocator still there. If this feature is valuable. What is left in currently implementation of your project? Can you tell about state, maybe show To Do list? Sorry, if my english is not clear. Thank you. Interface with garbage collector is to be done. The rest is usable right now. -- Andrei
Re: std.allocator ready for some abuse
On Wednesday, 11 February 2015 at 15:57:26 UTC, John Colvin wrote: Should it be in std.experimental? Or at least on code.dlang.org? Yeah, dub package would be really nice!
Re: std.allocator ready for some abuse
On Wednesday, 11 February 2015 at 15:56:07 UTC, Andrei Alexandrescu wrote: On 2/11/15 4:55 AM, ANtlord wrote: On Monday, 17 February 2014 at 15:49:27 UTC, Andrei Alexandrescu wrote: On 2/17/14, 5:55 AM, Dicebot wrote: Andrei, what is current state of std.allocator? I am asking this in context of recent Walter Phobos proposal (ScopeBuffer) to evaluate how feasible is to define any relations between two at current stage. Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei Mr. Alexandrescu. What about std allocator? Is feature valuable? I've took a look in Wish List in wiki and Allocator still there. If this feature is valuable. What is left in currently implementation of your project? Can you tell about state, maybe show To Do list? Sorry, if my english is not clear. Thank you. Interface with garbage collector is to be done. The rest is usable right now. -- Andrei Should it be in std.experimental? Or at least on code.dlang.org?
Re: std.allocator ready for some abuse
On Monday, 17 February 2014 at 15:49:27 UTC, Andrei Alexandrescu wrote: On 2/17/14, 5:55 AM, Dicebot wrote: Andrei, what is current state of std.allocator? I am asking this in context of recent Walter Phobos proposal (ScopeBuffer) to evaluate how feasible is to define any relations between two at current stage. Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei Mr. Alexandrescu. What about std allocator? Is feature valuable? I've took a look in Wish List in wiki and Allocator still there. If this feature is valuable. What is left in currently implementation of your project? Can you tell about state, maybe show To Do list? Sorry, if my english is not clear. Thank you.
Re: std.allocator ready for some abuse
Andrei, what is current state of std.allocator? I am asking this in context of recent Walter Phobos proposal (ScopeBuffer) to evaluate how feasible is to define any relations between two at current stage.
Re: std.allocator ready for some abuse
On 2/17/14, 5:55 AM, Dicebot wrote: Andrei, what is current state of std.allocator? I am asking this in context of recent Walter Phobos proposal (ScopeBuffer) to evaluate how feasible is to define any relations between two at current stage. Progress on std.allocator is slow but I do have a plan. I don't think we should pull ScopeBuffer yet. Andrei
Re: std.allocator ready for some abuse
Passing -g flag to the linker causes a linker error on win32.
Very simple program:
import std.allocator;
void main() {}
Compilation commands:
dmd.exe -g -debug -c std/allocator.d -ofallocator.obj
dmd.exe -g -debug -c main.d -ofmain.obj
dmd.exe out.exe -g allocator.obj main.obj
Output:
dmd.exe out.exe -g allocator.obj main.obj
OPTLINK (R) for Win32 Release 8.00.13
Copyright (C) Digital Mars 1989-2010 All rights reserved.
http://www.digitalmars.com/ctg/optlink.html
allocator.obj(allocator)
Error 42: Symbol Undefined __aligned_malloc
allocator.obj(allocator)
Error 42: Symbol Undefined __aligned_free
allocator.obj(allocator)
Error 42: Symbol Undefined __aligned_realloc
--- errorlevel 3
Without -g flag on the last command (dmd.exe out.exe
allocator.obj main.obj) no linker error appears.
Re: std.allocator ready for some abuse
On 11/16/13 2:50 AM, Jack Applegame wrote: Passing -g flag to the linker causes a linker error on win32. [snip] Maybe this would help? http://community.rti.com/kb/why-does-visual-studio-complain-about-alignedmalloc-and-alignedfree Andrei
Re: std.allocator ready for some abuse
Tried to register github package andralex/phobos, and: 500 - Internal Server Error Internal Server Error Now fixed https://github.com/rejectedsoftware/dub-registry/issues/31.
Re: std.allocator ready for some abuse
On 11/01/2013 10:00 PM, Andrei Alexandrescu wrote: On 10/30/13 1:02 PM, Martin Nowak wrote: This looks really promising. There are a lot of building blocks and the way different capabilities are modelled by optional methods nicely solves the biggest difficulty with allocators. I think it's important to put this in it's own github repo and add a dub package for it on code.dlang.org so that it's easy to test the implementation and to contribute improvements. Tried to register github package andralex/phobos, and: I transferred you a separate repo with all std.allocator commits. https://github.com/dawgfoto/std_allocator You can build the library with dub and run unittests with dub -build=unittest -config=unittest. Creating docs is a problem, maybe some expert can chime in. This is obviously because package.json is absent from the repo, but I'd say it shouldn't cause such an error. https://github.com/rejectedsoftware/dub-registry/issues/31 https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L3723 The newly-added allocator should come to the front of the list. My suspicion, however, is that you won't be able to measure a difference. A well-dimensioned cascade of regions will have a high ration of allocations within a regions to number of regions. You're probably right.
Re: std.allocator ready for some abuse
How about a stack allocator like this:
enum StackSize = 8192;
struct Stack {
static Stack it;
void[StackSize] _buffer = void;
size_t _bufUsage;
void[] take(size_t N) {
if ((this._bufUsage + N) = StackSize) {
scope(exit) this._bufUsage += N;
return _buffer[this._bufUsage .. this._bufUsage + N];
}
return null;
}
void reset() {
this._bufUsage = 0;
}
}
Would that fit in std.allocator?
Re: std.allocator ready for some abuse
Awesome! Since we have SIMD instructions in core it will be nice to have AlignedGCallocator. Best Regards, Ilya
Re: std.allocator ready for some abuse
Am 07.11.2013 11:32, schrieb Namespace:
How about a stack allocator like this:
enum StackSize = 8192;
struct Stack {
static Stack it;
void[StackSize] _buffer = void;
size_t _bufUsage;
void[] take(size_t N) {
if ((this._bufUsage + N) = StackSize) {
scope(exit) this._bufUsage += N;
return _buffer[this._bufUsage .. this._bufUsage + N];
}
return null;
}
void reset() {
this._bufUsage = 0;
}
}
Would that fit in std.allocator?
That's std.allocator.InSituRegion, just that is misses the reset() method.
Re: std.allocator ready for some abuse
On Thursday, 7 November 2013 at 13:15:03 UTC, Sönke Ludwig wrote:
Am 07.11.2013 11:32, schrieb Namespace:
How about a stack allocator like this:
enum StackSize = 8192;
struct Stack {
static Stack it;
void[StackSize] _buffer = void;
size_t _bufUsage;
void[] take(size_t N) {
if ((this._bufUsage + N) = StackSize) {
scope(exit) this._bufUsage += N;
return _buffer[this._bufUsage .. this._bufUsage +
N];
}
return null;
}
void reset() {
this._bufUsage = 0;
}
}
Would that fit in std.allocator?
That's std.allocator.InSituRegion, just that is misses the
reset() method.
Nice! But I suggest two things:
1. Add a reset function for reusing the same storage
2. Should that:
https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L2907
not be replaced with ubyte[size] _store = void; ?
Re: std.allocator ready for some abuse
07-Nov-2013 17:30, Namespace пишет:
On Thursday, 7 November 2013 at 13:15:03 UTC, Sönke Ludwig wrote:
Am 07.11.2013 11:32, schrieb Namespace:
How about a stack allocator like this:
enum StackSize = 8192;
struct Stack {
static Stack it;
void[StackSize] _buffer = void;
size_t _bufUsage;
void[] take(size_t N) {
if ((this._bufUsage + N) = StackSize) {
scope(exit) this._bufUsage += N;
return _buffer[this._bufUsage .. this._bufUsage + N];
}
return null;
}
void reset() {
this._bufUsage = 0;
}
}
Would that fit in std.allocator?
That's std.allocator.InSituRegion, just that is misses the reset()
method.
Nice! But I suggest two things:
1. Add a reset function for reusing the same storage
2. Should that:
https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L2907
not be replaced with ubyte[size] _store = void; ?
Hm... Andrei has just been caught using internal pointers in structs :o)
--
Dmitry Olshansky
Re: std.allocator ready for some abuse
On 11/7/13 2:32 AM, Namespace wrote:
How about a stack allocator like this:
enum StackSize = 8192;
struct Stack {
static Stack it;
void[StackSize] _buffer = void;
size_t _bufUsage;
void[] take(size_t N) {
if ((this._bufUsage + N) = StackSize) {
scope(exit) this._bufUsage += N;
return _buffer[this._bufUsage .. this._bufUsage + N];
}
return null;
}
void reset() {
this._bufUsage = 0;
}
}
Would that fit in std.allocator?
It's there!
Andrei
Re: std.allocator ready for some abuse
On 11/7/13 5:15 AM, Sönke Ludwig wrote: That's std.allocator.InSituRegion, just that is misses the reset() method. freeAll should take care of it. Andrei
Re: std.allocator ready for some abuse
On 11/7/13 5:30 AM, Namespace wrote: Nice! But I suggest two things: 1. Add a reset function for reusing the same storage freeAll(). 2. Should that: https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L2907 not be replaced with ubyte[size] _store = void; ? Made the change and pushed, thanks. Unfortunately there's a performance bug in the front-end (if at least one field has initialization, all fields are initialized by bitblasting .init over the object). https://d.puremagic.com/issues/show_bug.cgi?id=11331 Andrei
Re: std.allocator ready for some abuse
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote: Hello, I know it's been a long wait. Hopefully it was worth it. The alpha release of untyped allocators is ready for tire-kicking and a test drive. Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html Warning: this is alpha quality. Unit tests are thin, and there are no benchmarks. Both would be appreciated, particularly benchmarks to validate the gains (which I speculate can be very sizable) of custom-built, special-purpose allocators compared to traditional allocators. I acknowledge I'm clearly in no position to evaluate this design. I have been knocking around it for long enough to have no idea how easy it is to get into it from the outside, or how good it is. By all signs I could gather this feels like good design, and one of the best I've ever put together. The allocators defined have an archetypal feeling, are flexible both statically and dynamically, and morph and combine in infinite ways. CAllocator and CAllocatorImpl make the link between the static and dynamic worlds. Once an allocator is assembled out of pieces and finely tuned, wrapping it in a dynamic API is a snap. Please destroy! I've literally sweat as I'm sending this :o). Andrei All I can say is - I can't wait to see it in the run-time library! I read about the jmalloc last year, and your work makes it possible to have something similar in D. Kudos! System developers will definitely like std.allocator, no doubt! :)
Re: std.allocator ready for some abuse
On Thursday, 7 November 2013 at 16:28:42 UTC, Andrei Alexandrescu
wrote:
On 11/7/13 5:30 AM, Namespace wrote:
Nice! But I suggest two things:
1. Add a reset function for reusing the same storage
freeAll().
There is no freeAll. Did you mean deallocateAll?
void deallocateAll()
{
_crt = null;
}
_crt is null and the same storage can't be used again.
Re: std.allocator ready for some abuse
On 11/7/13 9:02 AM, Namespace wrote:
On Thursday, 7 November 2013 at 16:28:42 UTC, Andrei Alexandrescu wrote:
On 11/7/13 5:30 AM, Namespace wrote:
Nice! But I suggest two things:
1. Add a reset function for reusing the same storage
freeAll().
There is no freeAll. Did you mean deallocateAll?
void deallocateAll()
{
_crt = null;
}
_crt is null and the same storage can't be used again.
Sorry that's indeed the name and there's a bug in it!
Andrei
Re: std.allocator ready for some abuse
On Thursday, 24 October 2013 at 21:17:45 UTC, Andrei Alexandrescu wrote: On 10/24/13 2:08 PM, Walter Bright wrote: On 10/24/2013 12:54 PM, Andrei Alexandrescu wrote: I know it's been a long wait. Hopefully it was worth it. The alpha release of untyped allocators is ready for tire-kicking and a test drive. Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html I'm impressed. I looked for 3 or 4 must haves, and they were there! It covers all the cases I've written custom allocators for. big sigh of relief Andrei So, is the idea to add an optional allocator template parameter to various objects in the phobos library, or is this only intended to be used standalone via the with statement? -Chris
Re: std.allocator ready for some abuse
Andrei Alexandrescu seewebsiteforem...@erdani.org writes: On 10/26/13 8:00 AM, Andrei Alexandrescu wrote: Just implemented AlignedMallocator and pushed. http://erdani.com/d/phobos-prerelease/std_allocator.html#.AlignedMallocator I'm having a vision of a perverse version called MalignedAllocator...
Re: std.allocator ready for some abuse
Andrei Alexandrescu seewebsiteforem...@erdani.org writes: On 10/25/13 7:23 AM, Manu wrote: Woke up this morning with the following idea. 1. We add two optional API functions: void[] alignedAllocate(size_t, uint); bool alignedReallocate(ref void[], size_t, uint); A bit of bikeshedding: Since you have allocate() allocateAll() I would use allocateAligned() reallocateAligned() to be more consistent. Jerry
Re: std.allocator ready for some abuse
Hi, Andrei. I cannot compile the allocator. allocator.d(799): Error: undefined identifier returned At 499 line: returned alignedReallocate(b, newSize, alignment); Ur mistake?
Re: std.allocator ready for some abuse
On 11/3/13 4:14 AM, Temtaime wrote: Hi, Andrei. I cannot compile the allocator. allocator.d(799): Error: undefined identifier returned At 499 line: returned alignedReallocate(b, newSize, alignment); Ur mistake? Yes, mine. I don't have access to Windows so I expect a bunch of typos in the Windows-specific code. Fixed and pushed this one. Thanks, Andrei
Re: std.allocator ready for some abuse
On 10/31/13 7:34 PM, Andrei Alexandrescu wrote: On 10/24/13 12:54 PM, Andrei Alexandrescu wrote: Hello, I know it's been a long wait. Hopefully it was worth it. The alpha release of untyped allocators is ready for tire-kicking and a test drive. Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html Added SharedFreelist, a lock-free freelist. http://erdani.com/d/phobos-prerelease/std_allocator.html#.SharedFreelist Added documentation clarifying the allocators' approach to shared memory. See section Allocating memory for sharing across threads near the top of http://erdani.com/d/phobos-prerelease/std_allocator.html Andrei
Re: std.allocator ready for some abuse
01-Nov-2013 06:34, Andrei Alexandrescu пишет: On 10/24/13 12:54 PM, Andrei Alexandrescu wrote: Hello, I know it's been a long wait. Hopefully it was worth it. The alpha release of untyped allocators is ready for tire-kicking and a test drive. Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html Added SharedFreelist, a lock-free freelist. http://erdani.com/d/phobos-prerelease/std_allocator.html#.SharedFreelist Neat. However it's missing from Synopsis of predefined allocator building blocks table. -- Dmitry Olshansky
Re: std.allocator ready for some abuse
On 11/2/13 1:15 PM, Dmitry Olshansky wrote: 01-Nov-2013 06:34, Andrei Alexandrescu пишет: On 10/24/13 12:54 PM, Andrei Alexandrescu wrote: Hello, I know it's been a long wait. Hopefully it was worth it. The alpha release of untyped allocators is ready for tire-kicking and a test drive. Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html Added SharedFreelist, a lock-free freelist. http://erdani.com/d/phobos-prerelease/std_allocator.html#.SharedFreelist Neat. However it's missing from Synopsis of predefined allocator building blocks table. Thanks, fixed and pushed. Andrei
Re: std.allocator ready for some abuse
From the Allocating memory for sharing across threads section, this paragraph: All allocators in this module accept and return void[] **(as opposed to )**. This is because at the time of allocation, deallocation, or reallocation, the memory is effectively not shared (**it** it were, it would be a bug at the application level). Starred items appear to be typos or missing a piece.
Re: std.allocator ready for some abuse
On 11/2/13 1:39 PM, Meta wrote: From the Allocating memory for sharing across threads section, this paragraph: All allocators in this module accept and return void[] **(as opposed to )**. This is because at the time of allocation, deallocation, or reallocation, the memory is effectively not shared (**it** it were, it would be a bug at the application level). Starred items appear to be typos or missing a piece. Ew. Thanks, fixed and pushed. Andrei
Re: std.allocator ready for some abuse
On 10/31/13 8:37 PM, safety0ff wrote: On Friday, 1 November 2013 at 02:43:00 UTC, Andrei Alexandrescu wrote: On 10/31/13 7:26 PM, safety0ff wrote: I noticed that the GCAllocator provides no way of controlling the memory block attributes (http://dlang.org/phobos/core_memory.html#.GC.BlkAttr ,) all allocations get the default (no attributes.) This is a leaky abstraction, a data structure or composed allocators may desire to control the attributes to reduce GC pressure. These attributes seem to be informed by the types stored, which would be above the charter of untyped allocator. Andrei The attributes are informed by whatever code is calling the GC, the GC interface deals in void*'s. Consider an AA implementation that wishes to use FancyAllocator with fallback GCAllocator with block attributes NO_INTERIOR and NO_SCAN. With your proposed GCAllocator you either need to rewrite GCAllocator, or you need to add some nasty code to set the attributes depending on whether the primary allocator or secondary allocator own the memory. By fixing the leaky abstraction this use case can be coded as follows: FallbackAllocator!(FancyAllocator, GCAllocator!(GC.BLkAttr.NO_INTERIOR | GC.BLkAttr.NO_SCAN)) a; Migrating the flags into the type is a possibility but maybe it's easiest to add flags as runtime parameters. Allocators can always define additional nonstandard routines. The standard routines concern mostly composition. Of course, it is also possible to make such flags standard (it may be the case that typed allocators require such). Andrei
Re: std.allocator ready for some abuse
On Friday, 1 November 2013 at 15:15:10 UTC, Andrei Alexandrescu wrote: Migrating the flags into the type is a possibility but maybe it's easiest to add flags as runtime parameters. I was using that as an example to argue for its inclusion because it was concise. I'm not sure what the best interface would be, I'd have to think about it for a while.
Re: std.allocator ready for some abuse
On 11/01/2013 03:34 AM, Andrei Alexandrescu wrote: Added SharedFreelist, a lock-free freelist. http://erdani.com/d/phobos-prerelease/std_allocator.html#.SharedFreelist Andrei Nice, that reminds me that I still want to polish my implementation of a lock free doubly-linked list in D. It would be great to collect these efforts in a library. I remember the request of Adam Wilson for a System.Collections.Concurrent in D. I put my stuff on github https://github.com/dawgfoto/lock-free. The doubly-linked list is based on this paper. IIRC the paper had a bug. http://dx.doi.org/10.1016/j.jpdc.2008.03.001 Recently I also found a C++ implementation. It's much simpler in D due to the GC. https://github.com/Kometes/Honeycomb/blob/master/src/common/Honey/Thread/LockFree/List.h
Re: std.allocator ready for some abuse
I have another request despite putting this in it's own repo. Often one want an exponentially (power of 2) growing step size for Bucketizer. Currently only equally spaced buckets are possible which isn't very practical to scale from 16b to 2Mb. http://erdani.com/d/phobos-prerelease/std_allocator.html#.Bucketizer
Re: std.allocator ready for some abuse
On 10/30/13 1:02 PM, Martin Nowak wrote: This looks really promising. There are a lot of building blocks and the way different capabilities are modelled by optional methods nicely solves the biggest difficulty with allocators. I think it's important to put this in it's own github repo and add a dub package for it on code.dlang.org so that it's easy to test the implementation and to contribute improvements. Tried to register github package andralex/phobos, and: 500 - Internal Server Error Internal Server Error Internal error information: object.Exception@source/dubregistry/repositories/repository.d(43): Failed to read JSON from https://raw.github.com/andralex/phobos/master/package.json: Unexpected reply for 'https://raw.github.com/andralex/phobos/master/package.json': Not Found ./dub-registry(dubregistry.repositories.repository.PackageVersionInfo dubregistry.repositories.github.GithubRepository.getVersionInfo(immutable(char)[])+0x3b5) [0x6e8ec1] ./dub-registry(void dubregistry.registry.DubRegistry.addPackage(vibe.data.json.Json, vibe.data.bson.BsonObjectID)+0xa6) [0x62d372] ./dub-registry(void dubregistry.web.DubRegistryWebFrontend.addPackage(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse, userman.controller.User)+0x222) [0x6e4406] ./dub-registry(void delegate(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse) userman.web.UserManWebInterface.auth(void delegate(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse, userman.controller.User)).void requestHandler(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)+0x10d) [0x800181] ./dub-registry(void vibe.http.router.URLRouter.handleRequest(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)+0x179) [0x6fb8b5] ./dub-registry(bool vibe.http.server.handleRequest(vibe.core.stream.Stream, immutable(char)[], vibe.http.server.HTTPServerListener, ref vibe.http.server.HTTPServerSettings, ref bool)+0x16c8) [0x6f0344] ./dub-registry(void vibe.http.server.handleHTTPConnection(vibe.core.net.TCPConnection, vibe.http.server.HTTPServerListener)+0x143) [0x6eebb7] ./dub-registry(void vibe.http.server.listenHTTPPlain(vibe.http.server.HTTPServerSettings, void delegate(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse)).void doListen(vibe.http.server.HTTPServerSettings, vibe.http.server.HTTPServerListener, immutable(char)[]).void __lambda54(vibe.core.net.TCPConnection)+0x2c) [0x6eb160] ./dub-registry(extern (C) nothrow void vibe.core.drivers.libevent2_tcp.onConnect(int, short, void*).void ClientTask.execute()+0x2d6) [0x70939a] ./dub-registry(void vibe.core.core.CoreTask.run()+0xf2) [0x7172fe] ./dub-registry(void core.thread.Fiber.run()+0x2a) [0x83eae2] ./dub-registry(fiber_entryPoint+0x61) [0x83e9ed] [(nil)] This is obviously because package.json is absent from the repo, but I'd say it shouldn't cause such an error. That makes me think probably Phobos should have a package.json so people can install updates via code.dlang.org. That said it failed my litmus test. I previously used David Simcha's RegionAllocator https://github.com/dsimcha/TempAlloc/blob/master/std/allocators/region.d. Let's see! The pattern is to allocate some metadata followed by allocating many fixed size tree nodes. When the tree is constructed it is used to render an image which is the result of that operation. The tree and all metadata is freed and the region allocator is reused for the next method invocation (it keeps the memory). I think the closest would be to use CascadingAllocator with Region but there are two issues. CascadingAllocator successively tries all allocators and if that fails creates a new region. So this runs in O(N) complexity even though most of the time only the last allocator will have memory available. Yah, I'd left a TODO in there when I first wrote the code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L3723 The newly-added allocator should come to the front of the list. My suspicion, however, is that you won't be able to measure a difference. A well-dimensioned cascade of regions will have a high ration of allocations within a regions to number of regions. There is no simple way to deallocateAll without freeing the regions. What I need is something similar to clear in appender. Hm, interesting. But then again - do you think it makes a difference? Allocation of regions is a very small fraction of the work done on using the regions. I also can't relinquish the memory from the inner regions because they are private. How do we formalize that? So for my use-case the only way that I found to use this module is to compute the upper bound of memory needed when the renderer is invoked. Then I have to relinquish the buffer from a region, reallocate it using Mallocator.it and construct a new region
Re: std.allocator ready for some abuse
On 11/1/13 1:36 PM, Martin Nowak wrote: I have another request despite putting this in it's own repo. I assume s/despite/in addition to/ :o). Often one want an exponentially (power of 2) growing step size for Bucketizer. Currently only equally spaced buckets are possible which isn't very practical to scale from 16b to 2Mb. http://erdani.com/d/phobos-prerelease/std_allocator.html#.Bucketizer I considered the growth strategy as a policy. My personal favorite is choose an approximate exponential growth strategy that keeps maximum internal fragmentation less than x%. That's how jemalloc is dimensioned. I decided to stick with linear at least for now, for a simple reason: it's easy enough to simply enumerate the strategy by hand by using Segregator. Exponentials quickly grow to span a bunch of memory, so there aren't a lot of terms involved. Nevertheless it would be a nice illustration of D's generative powers. Andrei
Re: std.allocator ready for some abuse
That makes me think probably Phobos should have a package.json so people can install updates via code.dlang.org. How would you imagine that? Not-yet-in-Phobos packages are expected to be submitted as standalone ones, not as whole Phobos repo.
Re: std.allocator ready for some abuse
On 11/1/13 2:12 PM, Dicebot wrote: That makes me think probably Phobos should have a package.json so people can install updates via code.dlang.org. How would you imagine that? Not-yet-in-Phobos packages are expected to be submitted as standalone ones, not as whole Phobos repo. I meant people who'd want to just install Phobos from master. Probably not a good idea because Phobos changes often are in sync with compiler changes. Andrei
Re: std.allocator ready for some abuse
On 10/27/13, 7:10, Andrei Alexandrescu wrote:
The second problem is that the logged file/line is always in
std.allocator.d. It's probably not easy to get this working in all
cases (especially regarding polymorphism).
Uhm, that /is/ a problem.
This is supported by using default values:
1:import std.stdio;
2:void w(int l = __LINE__){writeln(l);}
3:void main(){w();}
outputs:
3
By design :)
Re: std.allocator ready for some abuse
On 10/31/13 8:57 AM, Lionello Lunesu wrote:
On 10/27/13, 7:10, Andrei Alexandrescu wrote:
The second problem is that the logged file/line is always in
std.allocator.d. It's probably not easy to get this working in all
cases (especially regarding polymorphism).
Uhm, that /is/ a problem.
This is supported by using default values:
1:import std.stdio;
2:void w(int l = __LINE__){writeln(l);}
3:void main(){w();}
outputs:
3
By design :)
The problem is only in CAllocator.
Andrei
Re: std.allocator ready for some abuse
I noticed that the GCAllocator provides no way of controlling the memory block attributes (http://dlang.org/phobos/core_memory.html#.GC.BlkAttr ,) all allocations get the default (no attributes.) This is a leaky abstraction, a data structure or composed allocators may desire to control the attributes to reduce GC pressure.
Re: std.allocator ready for some abuse
On 10/24/13 12:54 PM, Andrei Alexandrescu wrote: Hello, I know it's been a long wait. Hopefully it was worth it. The alpha release of untyped allocators is ready for tire-kicking and a test drive. Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html Added SharedFreelist, a lock-free freelist. http://erdani.com/d/phobos-prerelease/std_allocator.html#.SharedFreelist Andrei
Re: std.allocator ready for some abuse
On 10/31/13 7:26 PM, safety0ff wrote: I noticed that the GCAllocator provides no way of controlling the memory block attributes (http://dlang.org/phobos/core_memory.html#.GC.BlkAttr ,) all allocations get the default (no attributes.) This is a leaky abstraction, a data structure or composed allocators may desire to control the attributes to reduce GC pressure. These attributes seem to be informed by the types stored, which would be above the charter of untyped allocator. Andrei
Re: std.allocator ready for some abuse
On Friday, 1 November 2013 at 02:43:00 UTC, Andrei Alexandrescu wrote: On 10/31/13 7:26 PM, safety0ff wrote: I noticed that the GCAllocator provides no way of controlling the memory block attributes (http://dlang.org/phobos/core_memory.html#.GC.BlkAttr ,) all allocations get the default (no attributes.) This is a leaky abstraction, a data structure or composed allocators may desire to control the attributes to reduce GC pressure. These attributes seem to be informed by the types stored, which would be above the charter of untyped allocator. Andrei The choice is for the typed allocator. That mean that untyped allocator should either provide the choice, or assume the worse (it may contain pointer).
Re: std.allocator ready for some abuse
On Friday, 1 November 2013 at 02:43:00 UTC, Andrei Alexandrescu wrote: On 10/31/13 7:26 PM, safety0ff wrote: I noticed that the GCAllocator provides no way of controlling the memory block attributes (http://dlang.org/phobos/core_memory.html#.GC.BlkAttr ,) all allocations get the default (no attributes.) This is a leaky abstraction, a data structure or composed allocators may desire to control the attributes to reduce GC pressure. These attributes seem to be informed by the types stored, which would be above the charter of untyped allocator. Andrei The attributes are informed by whatever code is calling the GC, the GC interface deals in void*'s. Consider an AA implementation that wishes to use FancyAllocator with fallback GCAllocator with block attributes NO_INTERIOR and NO_SCAN. With your proposed GCAllocator you either need to rewrite GCAllocator, or you need to add some nasty code to set the attributes depending on whether the primary allocator or secondary allocator own the memory. By fixing the leaky abstraction this use case can be coded as follows: FallbackAllocator!(FancyAllocator, GCAllocator!(GC.BLkAttr.NO_INTERIOR | GC.BLkAttr.NO_SCAN)) a;
Re: std.allocator ready for some abuse
On 10/25/2013 03:08 AM, Walter Bright wrote: Would it be possible that this use the package idea with one allocator per file instead of the all-in-one-file setup? I'd like to see that as well.
Re: std.allocator ready for some abuse
On 10/24/2013 09:54 PM, Andrei Alexandrescu wrote: Hello, I know it's been a long wait. Hopefully it was worth it. The alpha release of untyped allocators is ready for tire-kicking and a test drive. Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html This looks really promising. There are a lot of building blocks and the way different capabilities are modelled by optional methods nicely solves the biggest difficulty with allocators. I think it's important to put this in it's own github repo and add a dub package for it on code.dlang.org so that it's easy to test the implementation and to contribute improvements. That said it failed my litmus test. I previously used David Simcha's RegionAllocator https://github.com/dsimcha/TempAlloc/blob/master/std/allocators/region.d. The pattern is to allocate some metadata followed by allocating many fixed size tree nodes. When the tree is constructed it is used to render an image which is the result of that operation. The tree and all metadata is freed and the region allocator is reused for the next method invocation (it keeps the memory). I think the closest would be to use CascadingAllocator with Region but there are two issues. CascadingAllocator successively tries all allocators and if that fails creates a new region. So this runs in O(N) complexity even though most of the time only the last allocator will have memory available. There is no simple way to deallocateAll without freeing the regions. What I need is something similar to clear in appender. I also can't relinquish the memory from the inner regions because they are private. So for my use-case the only way that I found to use this module is to compute the upper bound of memory needed when the renderer is invoked. Then I have to relinquish the buffer from a region, reallocate it using Mallocator.it and construct a new region with the reallocated buffer. This works only because I can cheaply compute the upper bound of required memory. This wouldn't work in other scenarios. I think this a very important use-case, e.g. using an auto-growing thread local region is what I would use to serve HTTP requests. But for this one might also want to use nested regions.
Re: std.allocator ready for some abuse
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote: Hello, I know it's been a long wait. Hopefully it was worth it. The alpha release of untyped allocators is ready for tire-kicking and a test drive. About time! ;) But it was definitely worth waiting for. I don't have that much experience with using custom allocators, but API-wise, this looks great. Not overburdened, but not underpowered either. I would suggest a different name for goodAllocSize(), though. Maybe actualAllocSize() or just allocSize(). This may be a stupid question, but is expand() useful outside the allocator infrastructure itself? Would you ever use that instead of reallocate() in a container, for instance? Lars
Re: std.allocator ready for some abuse
On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote: Just implemented AlignedMallocator and pushed. http://erdani.com/d/phobos-prerelease/std_allocator.html#.AlignedMallocator Untested on Windows. It seems inconsistent that allocate() forwards to alignedAllocate(), while reallocate() does not forward to alignedReallocate(). Why is this? Lars
Re: std.allocator ready for some abuse
On 10/28/13 1:03 AM, Lars T. Kyllingstad wrote: On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote: Just implemented AlignedMallocator and pushed. http://erdani.com/d/phobos-prerelease/std_allocator.html#.AlignedMallocator Untested on Windows. It seems inconsistent that allocate() forwards to alignedAllocate(), while reallocate() does not forward to alignedReallocate(). Why is this? Lars Two reasons: (1) Posix does not support aligned reallocation. That would make forwarding a pessimization on that OS; (2) On Windows, realloc() cannot be applied to memory gotten with _aligned_malloc and vice versa. Andrei
Re: std.allocator ready for some abuse
On 10/28/13 8:15 AM, Andrei Alexandrescu wrote: On 10/28/13 1:03 AM, Lars T. Kyllingstad wrote: On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote: Just implemented AlignedMallocator and pushed. http://erdani.com/d/phobos-prerelease/std_allocator.html#.AlignedMallocator Untested on Windows. It seems inconsistent that allocate() forwards to alignedAllocate(), while reallocate() does not forward to alignedReallocate(). Why is this? Lars Two reasons: (1) Posix does not support aligned reallocation. That would make forwarding a pessimization on that OS; (2) On Windows, realloc() cannot be applied to memory gotten with _aligned_malloc and vice versa. Andrei Oh, my mistake. On Windows reallocate() should forward to alignedReallocate. Andrei
Re: std.allocator ready for some abuse
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote: Please destroy! I've literally sweat as I'm sending this :o). Andrei Going through the code today after reading the thread for the past few days I was blown away by the depth of the concepts involved. Now the information in the article Memory Allocation: Either Love It or Hate It (or just think it’s okay) makes a lot more sense. The composability and customizability of the design look good. This is not an area of coding that I am that familiar with but I am interested in trying out a couple of ideas using the code in the coming weeks. I had a good laugh when I saw the ASCII art before the definition of NullAllocator. Pretty cool! Thanks for this work. Joseph
Re: std.allocator ready for some abuse
On 10/26/13 5:23 AM, Johannes Pfau wrote: Some small nitpicks: * byAllocation is not accessible from outside std.allocator Fixed. * Is it intentional that AllocatorWithStats can't log the __FUNCTION__ / __PRETTY_FUNCTION__ ? Just an omission, good idea! Fixed. Let's just record __FUNCTION__. * The documentation for byAllocation should probably state that it only lists 'alive' allocations and that this is especially useful for finding memory leaks. Fixed. And one bigger problem: CAllocatorImpl doesn't work well with AllocatorWithStats. AllocatorWithStats special members (byAllocation, etc) are not directly accessible. Although we can access them by using .impl it'd be nice to have a special AllocatorWithStats class which directly exposes those members. That shouldn't be a problem - just downcast to CAllocatorImpl!AllocatorWithStats and you got access. The second problem is that the logged file/line is always in std.allocator.d. It's probably not easy to get this working in all cases (especially regarding polymorphism). Uhm, that /is/ a problem. Andrei
Re: std.allocator ready for some abuse
On 10/25/13 12:19 PM, Dmitry Olshansky wrote: - I don't like the name CAllocator. Whatever that C stands for it's ambiguous (Class, Cee, Caramel?). Class. Better names welcome. - It may be only me but I _think_ it could be more useful to have a few specific interfaces then 1 fat CAllocator. I've come to dislike fat-interfaces with isSupported kludges but that's IMHO. Yah it's a good idea. I'll think of it. - I see that shrink primitive didn't make it... Well, on the upside the primitives count is really low. Yah. Getting away with as little as you can is my mantra :o). - In description of chooseAtRuntime: HeapBlock!chooseAtRuntime -- HeapBlock!(Allocator, chooseAtRuntime) Fixed and pushed. https://github.com/andralex/phobos/blob/allocator/std/allocator.d Andrei
Re: std.allocator ready for some abuse
On Sunday, 27 October 2013 at 06:15:52 UTC, Andrei Alexandrescu wrote: Class. Better names welcome. Allocator. Or we could call CAllocator AllocatorBase and call CAllocatorImpl Allocator. It depends on which of these two classes is most likely to be used in API's. I think the one most likely to be used should be called Allocator. Other names could be DynamicAllocator or RuntimeAllocator. -- /Jacob Carlborg
Re: std.allocator ready for some abuse
On Sunday, 27 October 2013 at 06:15:52 UTC, Andrei Alexandrescu wrote: On 10/25/13 12:19 PM, Dmitry Olshansky wrote: - I don't like the name CAllocator. Whatever that C stands for it's ambiguous (Class, Cee, Caramel?). Class. Better names welcome. I think DynamicAllocator would be a better name than CAllocator, but regardless, I think they're both pretty bad. However, I think it's indicative of a deeper problem with the interface chosen here: CAllocator is probably a kind of god object. Like Dmitry, I think using `interface`s (maybe `wrap` can be leveraged for the adaptation role[1]) is an approach worth investigating. It lets the library statically enforce that it gets the functionality it needs as opposed to having to raise a runtime error, while still allowing the caller to implement that functionality dynamically (urgh, reaching for words!). [1] Assuming `wrap` works on structs.
Re: std.allocator ready for some abuse
On 10/26/13 8:00 AM, Andrei Alexandrescu wrote: 3. Mallocator could also implement these on Posix: http://man7.org/linux/man-pages/man3/posix_memalign.3.html. However, Windows requires a specific call for deallocating aligned memory. To accommodate both portably, we leave Mallocator as is and create AlignedMallocator that uses the _aligned_* family on Windows and the respective functions on Posix. On Windows, allocate() requests would pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc. Just implemented AlignedMallocator and pushed. http://erdani.com/d/phobos-prerelease/std_allocator.html#.AlignedMallocator Untested on Windows. Andrei
Re: std.allocator ready for some abuse
On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote: On 10/26/13 8:00 AM, Andrei Alexandrescu wrote: 3. Mallocator could also implement these on Posix: http://man7.org/linux/man-pages/man3/posix_memalign.3.html. However, Windows requires a specific call for deallocating aligned memory. To accommodate both portably, we leave Mallocator as is and create AlignedMallocator that uses the _aligned_* family on Windows and the respective functions on Posix. On Windows, allocate() requests would pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc. Just implemented AlignedMallocator and pushed. http://erdani.com/d/phobos-prerelease/std_allocator.html#.AlignedMallocator Untested on Windows. Andrei In the following line: -- On Posix, forwards to realloc. On Windows, calls _aligned_realloc(b.ptr, newSize, platformAlignment). -- Link is incorrect (http//, colon is missing).
Re: std.allocator ready for some abuse
On 2013-10-26 17:01, Andrei Alexandrescu wrote: I think it's a commonly-used convention. Any book, covering the topic, will say that mangling types in names is bad practice. We don't use this naming convention anywhere in Phobos. -- /Jacob Carlborg
Re: std.allocator ready for some abuse
On Sunday, 27 October 2013 at 10:52:06 UTC, Tourist wrote: On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote: On 10/26/13 8:00 AM, Andrei Alexandrescu wrote: 3. Mallocator could also implement these on Posix: http://man7.org/linux/man-pages/man3/posix_memalign.3.html. However, Windows requires a specific call for deallocating aligned memory. To accommodate both portably, we leave Mallocator as is and create AlignedMallocator that uses the _aligned_* family on Windows and the respective functions on Posix. On Windows, allocate() requests would pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc. Just implemented AlignedMallocator and pushed. http://erdani.com/d/phobos-prerelease/std_allocator.html#.AlignedMallocator Untested on Windows. Andrei In the following line: -- On Posix, forwards to realloc. On Windows, calls _aligned_realloc(b.ptr, newSize, platformAlignment). -- Link is incorrect (http//, colon is missing). That's how Firefox interprets it. In source code it's: http://http//msdn.microsoft.com/en-US/library/y69db7sx(v=vs.80).aspx
Re: std.allocator ready for some abuse
On 10/27/13 11:26 AM, safety0ff wrote: On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote: On 10/26/13 8:00 AM, Andrei Alexandrescu wrote: AlignedMallocator that uses the _aligned_* family on Windows and the respective functions on Posix. On Windows, allocate() requests would pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc. Just implemented AlignedMallocator and pushed. http://erdani.com/d/phobos-prerelease/std_allocator.html#.AlignedMallocator The constraints on the alignment parameter are neither documented nor checked. i.e. Alignment must be a power of two, and for posix_memalign it must additionally be greater than sizeof(void*). Fixed, will push soon. I can only think of one use case for needing runtime specified alignment: allocating operating system page sized chunks. Are there any other use cases? Cache-line aligned. http://stackoverflow.com/questions/794632/programmatically-get-the-cache-line-size/1900464 Also some I/O requires aligned buffers. http://goo.gl/ni860U Andrei
Re: std.allocator ready for some abuse
On 10/27/13 3:52 AM, Tourist wrote: In the following line: -- On Posix, forwards to realloc. On Windows, calls _aligned_realloc(b.ptr, newSize, platformAlignment). -- Link is incorrect (http//, colon is missing). Thanks, ouch, there were a few more bugs around there as well in the untested code. Andrei
Re: std.allocator ready for some abuse
On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote: On 10/26/13 8:00 AM, Andrei Alexandrescu wrote: AlignedMallocator that uses the _aligned_* family on Windows and the respective functions on Posix. On Windows, allocate() requests would pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc. Just implemented AlignedMallocator and pushed. http://erdani.com/d/phobos-prerelease/std_allocator.html#.AlignedMallocator The constraints on the alignment parameter are neither documented nor checked. i.e. Alignment must be a power of two, and for posix_memalign it must additionally be greater than sizeof(void*). I can only think of one use case for needing runtime specified alignment: allocating operating system page sized chunks. Are there any other use cases?
Re: std.allocator ready for some abuse
i.e. Alignment must be a power of two, and for posix_memalign it must additionally be greater than sizeof(void*). It must be a power of two and a multiple of sizeof(void*), which means that it must be either zero or greater or equal to sizeof(void*).
Re: std.allocator ready for some abuse
On 10/25/13 7:23 AM, Manu wrote: 1. I'm still sad there are no allocAligned() functions or something of that type to request explicit alignment with allocations. I'm not sure there is sufficient support for requesting alignment with allocations. The set-able alignment property approach seems a little weird (and only seemed to be supported on one allocator?). I guess experience will tell if this is sufficient and/or convenient. I'd still like to see an allocWithAlignment() method or something, which may be implemented efficiently by allocators that can support it. Code: http://erdani.com/d/phobos-prerelease/std_allocator.html Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html I've made alignedAllocate() and alignedReallocate() parts of the official API, added alignedAlloc to all regions, added AlignedMallocator which taps into the aligned system APIs, and eliminated all that dynamic alignment setting stuff. I think we're in better shape now. Thanks for the suggestion! I still need to: 1. add stats for aligned calls to AllocatorWithStats 2. define HeapBlock.alignedXxx (which should be interesting) 3. separate IOwns, IAlignedAllocate, IDeallocate from CAllocator and have CAllocatorImpl!alloc conditionally implement them depending on whether alloc implements the respective primitives. Andrei
Re: std.allocator ready for some abuse
On 28 October 2013 04:36, Andrei Alexandrescu seewebsiteforem...@erdani.org wrote: On 10/27/13 11:26 AM, safety0ff wrote: On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote: On 10/26/13 8:00 AM, Andrei Alexandrescu wrote: AlignedMallocator that uses the _aligned_* family on Windows and the respective functions on Posix. On Windows, allocate() requests would pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc. Just implemented AlignedMallocator and pushed. http://erdani.com/d/phobos-**prerelease/std_allocator.html#** .AlignedMallocatorhttp://erdani.com/d/phobos-prerelease/std_allocator.html#.AlignedMallocator The constraints on the alignment parameter are neither documented nor checked. i.e. Alignment must be a power of two, and for posix_memalign it must additionally be greater than sizeof(void*). Fixed, will push soon. I can only think of one use case for needing runtime specified alignment: allocating operating system page sized chunks. Are there any other use cases? Cache-line aligned. http://stackoverflow.com/**questions/794632/**programmatically-get-the-** cache-line-size/1900464http://stackoverflow.com/questions/794632/programmatically-get-the-cache-line-size/1900464 Also some I/O requires aligned buffers. http://goo.gl/ni860U GPU's also tend to deal with strict alignments of various buffers. SIMD requires = 16byte alignment, which is higher than the default 4-8 byte alignment of many allocators.
Re: std.allocator ready for some abuse
On Monday, 28 October 2013 at 02:30:53 UTC, Manu wrote: GPU's also tend to deal with strict alignments of various buffers. SIMD requires = 16byte alignment, which is higher than the default 4-8 byte alignment of many allocators. AFAIK the alignments for those cases are known a priori, the question was regarding alignments that are only known at runtime.
Re: std.allocator ready for some abuse
On 28 October 2013 12:47, safety0ff safety0ff@gmail.com wrote: On Monday, 28 October 2013 at 02:30:53 UTC, Manu wrote: GPU's also tend to deal with strict alignments of various buffers. SIMD requires = 16byte alignment, which is higher than the default 4-8 byte alignment of many allocators. AFAIK the alignments for those cases are known a priori, the question was regarding alignments that are only known at runtime. You don't know the runtime GPU at compile time.
Re: std.allocator ready for some abuse
On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote: Hello, ... awesome stuff ... Please destroy! I've literally sweat as I'm sending this :o). Andrei I like it a lot so far. I was really worried about being able to dynamically dispatch to an allocator determined at a previous place in the call stack, and it seems you're all over it with CAllocator. Hell yeah! I have an editing suggestion for the CAllocator comment: Instead of Implementation of CAllocator using Allocator. [...] I suggest Implements CAllocator using the given Allocator. [...] The current one read strangely to me at first, and I had to re-read it several times and notice that Allocator referred to the template parameter. I agree with others that say that the name CAllocator is too ambiguous or vague. When I scanned through the allocators, I initially dismissed it because I though it was a proxy for the system's underlying C allocator, with Mallocator being a D-based optimized reimplementation of the C allocator. Reading further clarified this, but it does probably harm skimming and searching. I suggest an alternative name for CAllocator: DispatchingAllocator. I believe this may represent what it does: dispatch allocation to another allocator that is behind a curtain. Something like AbstractAllocator might work too, but still seems slightly ambiguous to me (i.e. abstract in what sense?). I just hope that the future top allocator that handles language-builtin allocations will be one that can maintain a stack of allocators and push/pop the current default allocator, as well as prevent or redirect allocator choice made within calls to 3rd party libraries (assuming the libraries are written in D, of course).
Re: std.allocator ready for some abuse
Am Sat, 26 Oct 2013 00:11:13 +0200 schrieb Timon Gehr timon.g...@gmx.ch: - CAllocator. The name is non-descriptive/misleading. Class Allocator :o) ... Then its purpose must be allocating classes? :o) I'm actually fine with that name, probably because I tend to prefix my D reference types in that manner, too. 'I' for interfaces, 'C' for classes and maybe 'A' for abstract classes. -- Marco
Re: std.allocator ready for some abuse
On 2013-10-26 11:03, Marco Leise wrote: I'm actually fine with that name, probably because I tend to prefix my D reference types in that manner, too. 'I' for interfaces, 'C' for classes and maybe 'A' for abstract classes. That's just horrible. -- /Jacob Carlborg
Re: std.allocator ready for some abuse
Am Thu, 24 Oct 2013 12:54:41 -0700 schrieb Andrei Alexandrescu seewebsiteforem...@erdani.org: Hello, I know it's been a long wait. Hopefully it was worth it. The alpha release of untyped allocators is ready for tire-kicking and a test drive. Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html Warning: this is alpha quality. Unit tests are thin, and there are no benchmarks. Both would be appreciated, particularly benchmarks to validate the gains (which I speculate can be very sizable) of custom-built, special-purpose allocators compared to traditional allocators. I acknowledge I'm clearly in no position to evaluate this design. I have been knocking around it for long enough to have no idea how easy it is to get into it from the outside, or how good it is. By all signs I could gather this feels like good design, and one of the best I've ever put together. The allocators defined have an archetypal feeling, are flexible both statically and dynamically, and morph and combine in infinite ways. CAllocator and CAllocatorImpl make the link between the static and dynamic worlds. Once an allocator is assembled out of pieces and finely tuned, wrapping it in a dynamic API is a snap. Please destroy! I've literally sweat as I'm sending this :o). Andrei The overall design looks great! Some small nitpicks: * byAllocation is not accessible from outside std.allocator * Is it intentional that AllocatorWithStats can't log the __FUNCTION__ / __PRETTY_FUNCTION__ ? * The documentation for byAllocation should probably state that it only lists 'alive' allocations and that this is especially useful for finding memory leaks. And one bigger problem: CAllocatorImpl doesn't work well with AllocatorWithStats. AllocatorWithStats special members (byAllocation, etc) are not directly accessible. Although we can access them by using .impl it'd be nice to have a special AllocatorWithStats class which directly exposes those members. The second problem is that the logged file/line is always in std.allocator.d. It's probably not easy to get this working in all cases (especially regarding polymorphism).
Re: std.allocator ready for some abuse
On 10/25/13 7:23 AM, Manu wrote: 1. I'm still sad there are no allocAligned() functions or something of that type to request explicit alignment with allocations. I'm not sure there is sufficient support for requesting alignment with allocations. The set-able alignment property approach seems a little weird (and only seemed to be supported on one allocator?). Woke up this morning with the following idea. 1. We add two optional API functions: void[] alignedAllocate(size_t, uint); bool alignedReallocate(ref void[], size_t, uint); These would satisfy aligned allocation requests. The pointer thus allocated should be deallocated the usual way (there is no alignedDeallocate). 2. We make alignment a statically-known enum and delete all code that allows it to be get and set at runtime. Consequences: 1. The API gets larger due to the two new APIs. However, that is offset (probably more than enough) by the removal the option to set the alignment at runtime, which currently is causing severe ripple effects. I think the result will be a simpler overall design and implementation. 2. HeapBlock should be able to implement the two functions. All regions should be able to implement the first. 3. Mallocator could also implement these on Posix: http://man7.org/linux/man-pages/man3/posix_memalign.3.html. However, Windows requires a specific call for deallocating aligned memory. To accommodate both portably, we leave Mallocator as is and create AlignedMallocator that uses the _aligned_* family on Windows and the respective functions on Posix. On Windows, allocate() requests would pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc. Is this satisfactory for everyone? Andrei
Re: std.allocator ready for some abuse
On 10/26/13 4:02 AM, Jacob Carlborg wrote: On 2013-10-26 11:03, Marco Leise wrote: I'm actually fine with that name, probably because I tend to prefix my D reference types in that manner, too. 'I' for interfaces, 'C' for classes and maybe 'A' for abstract classes. That's just horrible. I think it's a commonly-used convention. Andrei
Re: std.allocator ready for some abuse
On 24/10/13 21:54, Andrei Alexandrescu wrote: I know it's been a long wait. Hopefully it was worth it. The alpha release of untyped allocators is ready for tire-kicking and a test drive. ... Please destroy! I've literally sweat as I'm sending this :o). Hmm, seems like I'm coming late to quite a party ... :-) Like John Colvin, I feel that much of this is over my head -- I don't have the experience/expertise to judge requirements or solutions. But it looks like very nice code indeed and I'm encouraged by the fact that most Can it do ...? questions seem to have a ready answer. My own interests have always been less in allocators per se and more in the functionality they unlock -- stuff like containers, for example. It may be unfamiliarity -- my own code has either used new/delete (the latter only in C++, not D...) or malloc/dealloc -- but looking at this module as it stands I don't really see how to practically make use of it; which makes me wonder whether the people calling for it to be in core rather than std have a point. Maybe it would help to have code examples that really show these allocators really being used to do something -- the existing examples are comprehensive and show what to do, but not really what to do _with_ the constructs thus created. Or, maybe I should just educate myself more ... ;-) Suffice to say that my overwhelming feeling is of gratitude and admiration for all the obvious thought and hard work put into creating this module. Thanks and congratulations, Andrei! :-) Best wishes, -- Joe
Re: std.allocator ready for some abuse
On 25/10/13 11:37, Namespace wrote: We would have then the possibility to manage our memory by ourself. One of D's promises is, that the GC can be disabled. Yes, it can, but then we have many many things which do not work. For example built-in arrays. With the ability of allocators the promise could come true. That's something I'd really like to know more about. My D code almost invariably works with the natural way to handle memory in D, which is to use new where needed, plus stuff like array appending ~ and alterations to array lengths, with all allocations handled behind the scenes by the GC. I've always felt bad about the fact that this therefore imposes use of the GC on anyone who uses my code. It would be great if one could just write idiomatic D code and know that others using it could dictate different memory-management strategies and have them just work.
Re: std.allocator ready for some abuse
On Saturday, 26 October 2013 at 15:42:48 UTC, Joseph Rushton Wakeling wrote: On 25/10/13 11:37, Namespace wrote: We would have then the possibility to manage our memory by ourself. One of D's promises is, that the GC can be disabled. Yes, it can, but then we have many many things which do not work. For example built-in arrays. With the ability of allocators the promise could come true. That's something I'd really like to know more about. My D code almost invariably works with the natural way to handle memory in D, which is to use new where needed, plus stuff like array appending ~ and alterations to array lengths, with all allocations handled behind the scenes by the GC. I've always felt bad about the fact that this therefore imposes use of the GC on anyone who uses my code. It would be great if one could just write idiomatic D code and know that others using it could dictate different memory-management strategies and have them just work. Read on from here to find out more: http://forum.dlang.org/thread/l4btsk$5u8$1...@digitalmars.com?page=5#post-uqolhuqqygquxnaxahkz:40forum.dlang.org :)
Re: std.allocator ready for some abuse
On 26/10/13 17:46, Namespace wrote: Read on from here to find out more: http://forum.dlang.org/thread/l4btsk$5u8$1...@digitalmars.com?page=5#post-uqolhuqqygquxnaxahkz:40forum.dlang.org I already did, and if I understand right, things are looking good ... but I wanted to be sure whether or not I understand :-)
Re: std.allocator ready for some abuse
On 25/10/13 20:41, Namespace wrote: Did you mean to get rid of built-in arrays / kill int[] and replace it with Array!T? Array!T is problematic as things stand -- e.g. you can't foreach over one. So, forgetting syntax preferences, there needs to be some work on containers before they can just work like the builtins. If it's possible, I'd rather see the converse -- that code that assumes the GC will just work with other allocation strategies, so one can use the builtins without worrying. But am I being naively hopeful in seeking that? :-)
Re: std.allocator ready for some abuse
26-Oct-2013 20:01, Joseph Rushton Wakeling пишет:
On 25/10/13 20:41, Namespace wrote:
Did you mean to get rid of built-in arrays / kill int[] and replace it
with
Array!T?
Array!T is problematic as things stand -- e.g. you can't foreach over
one.
Sure you can. Try it and rejoice:
void main()
{
import std.container, std.stdio;
Array!int a = make!(Array!int)(1,2,3,4);
//the rule is: if a can be sliced then slice it and use that slice
foreach(v; a)
{
writeln(v);
}
}
So, forgetting syntax preferences, there needs to be some work on
containers before they can just work like the builtins.
Depends on what's required for you to consider it just works.
But in general they can't be complete replica of built-ins for many
reasons, built-ins being designed with GC in mind is one.
Other problems include having no user-land analog of implicit tail-const
of arrays.
If it's possible, I'd rather see the converse -- that code that assumes
the GC will just work with other allocation strategies, so one can use
the builtins without worrying.
Only if you are switching to from one GC kind to another. There is no
way out of automatic memory management.
But am I being naively hopeful in
seeking that? :-)
Yes.
--
Dmitry Olshansky
Re: std.allocator ready for some abuse
On Saturday, 26 October 2013 at 16:10:46 UTC, Dmitry Olshansky wrote: If it's possible, I'd rather see the converse -- that code that assumes the GC will just work with other allocation strategies, so one can use the builtins without worrying. Only if you are switching to from one GC kind to another. There is no way out of automatic memory management. Waiting for ARC :)
Re: std.allocator ready for some abuse
On Saturday, 26 October 2013 at 16:01:18 UTC, Joseph Rushton Wakeling wrote: On 25/10/13 20:41, Namespace wrote: Did you mean to get rid of built-in arrays / kill int[] and replace it with Array!T? Array!T is problematic as things stand -- e.g. you can't foreach over one. So, forgetting syntax preferences, there needs to be some work on containers before they can just work like the builtins. If it's possible, I'd rather see the converse -- that code that assumes the GC will just work with other allocation strategies, so one can use the builtins without worrying. But am I being naively hopeful in seeking that? :-) I would never vote to replace the built-in arrays with something ugly as Array!T. If D would switch to Array!T and Map!(T, U) it would be the same hell as with C++. But I hope allocators enable the possibility that built-in arrays could use other memory mangaement besides the GC. That would be awesome.
