Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Sunday, 9 April 2017 at 15:52:50 UTC, Basile B. wrote: On Sunday, 9 April 2017 at 08:56:52 UTC, Atila Neves wrote: Using std.experimental.allocator? Tired of writing `scope(exit) allocator.dispose(foo);` in a language with RAII? Me too: http://code.dlang.org/packages/automem I think that the Array misses - a reservation strategy, something like reserve() and allocBy(). reserve is done. What would allocBy be? - dup / idup that return new distinct and deep copies. dup is done. I'm trying to figure out how one would use .idup. - maybe .ptr at least for reading with pointer arithmetic. Done. - opBinary for "~" . Also you have bugs with operators: Done. ```d import std.experimental.allocator.mallocator; UniqueArray!(int, Mallocator) a; a ~= [0,1]; ``` crashes directly. Fixed. Atila
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Monday, 17 April 2017 at 13:21:50 UTC, Kagamin wrote:
If we can control memory layout, we can do what shared_ptr does
and couple the reference counter with the object, then we can
have just one pointer:
struct RefCounted(T)
{
struct Wrapper
{
int count;
T payload;
}
Wrapper* payload;
}
I'm not sure I follow your comment. Indeed, that is how
shared_ptr, or, in this case, RefCounted, is implemented. My
point was that there is no practical sense in having a
shared(RefCounted).
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Wednesday, 12 April 2017 at 13:32:36 UTC, Stanislav Blinov
wrote:
Syntax is not the core of the issue, it's not about just
marking a destructor as shared. Making RefCounted itself shared
would require implementing some form of synchronization of all
the 'dereference' operations, including assignments. I.e. if we
have some shared(RefCounted!T) ptr, what should happen when two
threads simultaneously attempt to do ptr =
shared(RefCounted!T)(someNewValue) ? Should a library
implementation even consider this? Or should such
synchronization be left to client's care? It seems like in this
regard shared(RefCounted!T) would be no different from
shared(T*), which brings me to the next point.
If we can control memory layout, we can do what shared_ptr does
and couple the reference counter with the object, then we can
have just one pointer:
struct RefCounted(T)
{
struct Wrapper
{
int count;
T payload;
}
Wrapper* payload;
}
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Monday, 10 April 2017 at 08:11:37 UTC, Atila Neves wrote: On Sunday, 9 April 2017 at 13:59:14 UTC, Andrei Alexandrescu wrote: Great. Can RefCounted itself be shared? I learned this is important for composition, i.e. you want to make a RefCounted a field in another object that is itself shared, immutable etc. Since it has a destructor, no: http://forum.dlang.org/post/sqazguejrcdtjimtj...@forum.dlang.org The only way to do that would be to split it into two. Which I guess I could with a template mixin implementing the guts. Syntax is not the core of the issue, it's not about just marking a destructor as shared. Making RefCounted itself shared would require implementing some form of synchronization of all the 'dereference' operations, including assignments. I.e. if we have some shared(RefCounted!T) ptr, what should happen when two threads simultaneously attempt to do ptr = shared(RefCounted!T)(someNewValue) ? Should a library implementation even consider this? Or should such synchronization be left to client's care? It seems like in this regard shared(RefCounted!T) would be no different from shared(T*), which brings me to the next point. On Andrei's comment regarding composition: if we're thinking about the design of shared data, IMHO, copying and shared are (or at least, should be) mutually exclusive. The only exception being references (pointers), and even for those care should be taken (which is one of the reasons of even having something like a RefCounted). If you can make a copy, there is no reason to share, or, conversely, if you intend to share, why would you copy? From that follows that a shared object should not ever have a RefCounted field. A Unique field, perhaps, but not a RefCounted. RefCounted's purpose is to be copied, not shared between threads. Unless I'm missing something.
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Tuesday, 11 April 2017 at 22:32:51 UTC, Martin Nowak wrote: On Monday, 10 April 2017 at 08:31:28 UTC, Atila Neves wrote: ```d import std.experimental.allocator.mallocator; UniqueArray!(int, Mallocator) a; a ~= [0,1]; ``` So the difference between std.container.Array and UniqueArray is that the latter supports allocators? That's the general idea, but I confess I didn't even look at Array. Atila
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Monday, 10 April 2017 at 08:31:28 UTC, Atila Neves wrote: ```d import std.experimental.allocator.mallocator; UniqueArray!(int, Mallocator) a; a ~= [0,1]; ``` So the difference between std.container.Array and UniqueArray is that the latter supports allocators?
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Tuesday, 11 April 2017 at 10:24:08 UTC, Nicholas Wilson wrote: In LDC we have an attribute for that `allocSize` (https://github.com/ldc-developers/druntime/blob/ldc/src/ldc/attributes.d#L16) perhaps this attribute should be used across compilers and be in druntime? Nice, if pure required strong purity, it would be quite a huge hack, so a specific attribute seems friendlier.
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Tuesday, 11 April 2017 at 08:09:15 UTC, Martin Nowak wrote: On Sunday, 9 April 2017 at 10:22:49 UTC, Atila Neves wrote: I did not. Thanks for telling me! The way I wrote it RefCounted!(shared T) works - RefCounted doesn't have to be shared itself, but I guess it could be. I think the other design is slightly more correct, having a single thread own a shared value => RefCounted!(shared T), having multiple threads own a value (which is transitively shared) => shared(RefCounted!T). The latter is also neede for `static shared RC!T rc;`. Unfortunately I later remembered that because it has a destructor it can't be shared and not shared with the sample implementation. And I'd really like to avoid having to have two different names like Rust does with Rc and Arc. Atila
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Tuesday, 11 April 2017 at 09:53:46 UTC, Martin Nowak wrote: I think we might be able to solve this problem in D by making IAllocator.allocate pure, which tells the compiler that this function returns a fresh piece of memory without any side-effect, i.e. enough information to optimize away allocations. Pure might be too restrictive for some allocators, but maybe this can be solved with a little type system hack (or at worse a compiler exemption). In LDC we have an attribute for that `allocSize` (https://github.com/ldc-developers/druntime/blob/ldc/src/ldc/attributes.d#L16) perhaps this attribute should be used across compilers and be in druntime?
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Sunday, 9 April 2017 at 13:59:14 UTC, Andrei Alexandrescu wrote: . The allocator has to be specified as part of the type: this means the user can choose how to store it in the smart pointer, which for singletons (e.g. Mallocator) or stateless allocators means they can take up zero space. If a singleton (or the default theAllocator), the allocator doesn't need to be passed in to the constructor, otherwise it does. Specifying, e.g. Mallocator also means the relevant code can be marked @nogc. After extensively studying how C++ allocator framework works, I got to the notion that making the allocator part of the type is an antipattern. Just repeating an older argument of mine which didn't make it during the std.allocator discussions (http://forum.dlang.org/post/ubithltzbtdypaegn...@forum.dlang.org). Yes, we clearly want type erasure for the Allocator, because having the Allocator as part of the type tends to push Allocator choices/forwarding everywhere into the APIs. It also prevents conversion/copying values with different allocators. The obvious solution is to use an Allocator interface and/or a custom deleter function (deleter was needed for attribute correct destruction with polymorphic RC!Klass, see [¹]). Now as Chandler Carruth mentions, https://www.youtube.com/watch?v=fHNmRkzxHWs=3950 https://www.youtube.com/watch?v=fHNmRkzxHWs=4037 , an interface with dynamic dispatch would prevent optimizing away redundant allocations. Allocations are complex enough that I'm not too worried about the virtual call overhead itself. I think we might be able to solve this problem in D by making IAllocator.allocate pure, which tells the compiler that this function returns a fresh piece of memory without any side-effect, i.e. enough information to optimize away allocations. Pure might be too restrictive for some allocators, but maybe this can be solved with a little type system hack (or at worse a compiler exemption). [¹]: https://github.com/MartinNowak/phobos/commit/8cf0ec29ad65ac2a13bd6917b4ff3da0fdea5ab0#diff-4e008aedb3026d4a84f58323e53bf017R4896
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Sunday, 9 April 2017 at 08:56:52 UTC, Atila Neves wrote: http://code.dlang.org/packages/automem You might find my own containers interesting, especially https://github.com/nordlow/phobos-next/blob/master/src/array_ex.d Supports all the different ways I could think an array needs to work: https://github.com/nordlow/phobos-next/blob/master/src/array_ex.d#L64 https://github.com/nordlow/phobos-next/blob/master/src/array_ex.d#L90 https://github.com/nordlow/phobos-next/blob/master/src/array_ex.d#L1677 Does not (yet) support custom allocators, though. But that can be added, if requested. Are now in extensive use in my (non-public) applications.
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Sunday, 9 April 2017 at 10:22:49 UTC, Atila Neves wrote: I did not. Thanks for telling me! The way I wrote it RefCounted!(shared T) works - RefCounted doesn't have to be shared itself, but I guess it could be. I think the other design is slightly more correct, having a single thread own a shared value => RefCounted!(shared T), having multiple threads own a value (which is transitively shared) => shared(RefCounted!T). The latter is also neede for `static shared RC!T rc;`.
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Sunday, 9 April 2017 at 19:04:22 UTC, mogu wrote: On Sunday, 9 April 2017 at 08:56:52 UTC, Atila Neves wrote: Using std.experimental.allocator? Tired of writing `scope(exit) allocator.dispose(foo);` in a language with RAII? Me too: [...] Nice! Should UniqueArray be implemented as a overloaded version of Unique? Unique!(Object[]) instead of UniqueArray!(Object). I started like that, but after many a `static if` realised they had very little in common. Atila
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Sunday, 9 April 2017 at 15:52:50 UTC, Basile B. wrote: On Sunday, 9 April 2017 at 08:56:52 UTC, Atila Neves wrote: Using std.experimental.allocator? Tired of writing `scope(exit) allocator.dispose(foo);` in a language with RAII? Me too: http://code.dlang.org/packages/automem I think that the Array misses - a reservation strategy, something like reserve() and allocBy(). - dup / idup that return new distinct and deep copies. - maybe .ptr at least for reading with pointer arithmetic. - opBinary for "~" . Also you have bugs with operators: Thanks for the suggestions. ```d import std.experimental.allocator.mallocator; UniqueArray!(int, Mallocator) a; a ~= [0,1]; ``` crashes directly. Fixed now, thanks. Atila
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Sunday, 9 April 2017 at 13:59:14 UTC, Andrei Alexandrescu wrote: On 4/9/17 4:56 AM, Atila Neves wrote: Using std.experimental.allocator? Tired of writing `scope(exit) allocator.dispose(foo);` in a language with RAII? Me too: http://code.dlang.org/packages/automem Example: I think the code in the README should be enough to understand what's going on. Alpha stuff here but I think the main things missing are weak pointers and a ref counted array. Given that I've never had to use std::weak_ptr in C++, I'm not in a hurry to implement the former. Nice work! Notable design decisions / features: . The smart pointers are responsible for allocating the memory for the container object using the allocator of choice. This is to guarantee that one can't allocate and deallocate using different allocators. Nice! . The allocator has to be specified as part of the type: this means the user can choose how to store it in the smart pointer, which for singletons (e.g. Mallocator) or stateless allocators means they can take up zero space. If a singleton (or the default theAllocator), the allocator doesn't need to be passed in to the constructor, otherwise it does. Specifying, e.g. Mallocator also means the relevant code can be marked @nogc. After extensively studying how C++ allocator framework works, I got to the notion that making the allocator part of the type is an antipattern. I was aware of this, but here we have a crucial workaround - theAllocator, which is the default anyway. It's probably the best of both worlds, since you can still specify the type if needed, which also means the guarantee of @nogc if needed. . RefCounted only increments/decrements the ref count atomically if the contained type is `shared` Great. Can RefCounted itself be shared? I learned this is important for composition, i.e. you want to make a RefCounted a field in another object that is itself shared, immutable etc. Since it has a destructor, no: http://forum.dlang.org/post/sqazguejrcdtjimtj...@forum.dlang.org The only way to do that would be to split it into two. Which I guess I could with a template mixin implementing the guts. . RefCounted!(shared T) can be sent to other threads. Awes. . UniqueArray behaves nearly like a normal array. You can even append to it, but it won't use GC memory (unless, of course, you chose to use GCAllocator)! This may be a great candidate for the standard library. I think this needs to be used in production first, and having it as a dub package makes it easy for people to do so. Atila
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Sunday, 9 April 2017 at 08:56:52 UTC, Atila Neves wrote: Using std.experimental.allocator? Tired of writing `scope(exit) allocator.dispose(foo);` in a language with RAII? Me too: [...] Nice! Should UniqueArray be implemented as a overloaded version of Unique? Unique!(Object[]) instead of UniqueArray!(Object).
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Sunday, 9 April 2017 at 08:56:52 UTC, Atila Neves wrote: Using std.experimental.allocator? Tired of writing `scope(exit) allocator.dispose(foo);` in a language with RAII? Me too: http://code.dlang.org/packages/automem I think that the Array misses - a reservation strategy, something like reserve() and allocBy(). - dup / idup that return new distinct and deep copies. - maybe .ptr at least for reading with pointer arithmetic. - opBinary for "~" . Also you have bugs with operators: ```d import std.experimental.allocator.mallocator; UniqueArray!(int, Mallocator) a; a ~= [0,1]; ``` crashes directly.
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On 09/04/2017 2:59 PM, Andrei Alexandrescu wrote: On 4/9/17 4:56 AM, Atila Neves wrote: snip . UniqueArray behaves nearly like a normal array. You can even append to it, but it won't use GC memory (unless, of course, you chose to use GCAllocator)! This may be a great candidate for the standard library. To further this, I should rewrite my managed memory concept, it would be very useful for e.g. collections if at the same quality as Atila's as it gives the creator (of the memory) a heck a lot of control over it. [0] https://github.com/rikkimax/alphaPhobos/blob/master/source/std/experimental/memory/managed.d
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On 4/9/17 4:56 AM, Atila Neves wrote: Using std.experimental.allocator? Tired of writing `scope(exit) allocator.dispose(foo);` in a language with RAII? Me too: http://code.dlang.org/packages/automem Example: I think the code in the README should be enough to understand what's going on. Alpha stuff here but I think the main things missing are weak pointers and a ref counted array. Given that I've never had to use std::weak_ptr in C++, I'm not in a hurry to implement the former. Nice work! Notable design decisions / features: . The smart pointers are responsible for allocating the memory for the container object using the allocator of choice. This is to guarantee that one can't allocate and deallocate using different allocators. Nice! . The allocator has to be specified as part of the type: this means the user can choose how to store it in the smart pointer, which for singletons (e.g. Mallocator) or stateless allocators means they can take up zero space. If a singleton (or the default theAllocator), the allocator doesn't need to be passed in to the constructor, otherwise it does. Specifying, e.g. Mallocator also means the relevant code can be marked @nogc. After extensively studying how C++ allocator framework works, I got to the notion that making the allocator part of the type is an antipattern. . RefCounted only increments/decrements the ref count atomically if the contained type is `shared` Great. Can RefCounted itself be shared? I learned this is important for composition, i.e. you want to make a RefCounted a field in another object that is itself shared, immutable etc. . RefCounted!(shared T) can be sent to other threads. Awes. . UniqueArray behaves nearly like a normal array. You can even append to it, but it won't use GC memory (unless, of course, you chose to use GCAllocator)! This may be a great candidate for the standard library. Andrei
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Sunday, 9 April 2017 at 09:36:53 UTC, Martin Nowak wrote: On Sunday, 9 April 2017 at 08:56:52 UTC, Atila Neves wrote: I benchmarked RefCounted against C++'s std::shared_ptr comparing ldc to clang using both shared and non-shared payloads in D. std::shared_ptr is faster (I've never written a smart pointer before), but the advantage of non-atomic operations makes my D implementation just a bit faster when non-shared. I think with some work it can be significantly faster without any loss of thread safety. Nice! You know you can overload `this(this) shared` do you? My plan was to use that for atomic RC, so that ppl. can use `shared(RefCounted)` when necessary. I did not. Thanks for telling me! The way I wrote it RefCounted!(shared T) works - RefCounted doesn't have to be shared itself, but I guess it could be. Atila
Re: automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
On Sunday, 9 April 2017 at 08:56:52 UTC, Atila Neves wrote: I benchmarked RefCounted against C++'s std::shared_ptr comparing ldc to clang using both shared and non-shared payloads in D. std::shared_ptr is faster (I've never written a smart pointer before), but the advantage of non-atomic operations makes my D implementation just a bit faster when non-shared. I think with some work it can be significantly faster without any loss of thread safety. Nice! You know you can overload `this(this) shared` do you? My plan was to use that for atomic RC, so that ppl. can use `shared(RefCounted)` when necessary.
automem v0.0.7 - C++ style smart pointers using std.experimental.allocator
Using std.experimental.allocator? Tired of writing `scope(exit) allocator.dispose(foo);` in a language with RAII? Me too: http://code.dlang.org/packages/automem Example: I think the code in the README should be enough to understand what's going on. Alpha stuff here but I think the main things missing are weak pointers and a ref counted array. Given that I've never had to use std::weak_ptr in C++, I'm not in a hurry to implement the former. Notable design decisions / features: . The smart pointers are responsible for allocating the memory for the container object using the allocator of choice. This is to guarantee that one can't allocate and deallocate using different allocators. . The allocator has to be specified as part of the type: this means the user can choose how to store it in the smart pointer, which for singletons (e.g. Mallocator) or stateless allocators means they can take up zero space. If a singleton (or the default theAllocator), the allocator doesn't need to be passed in to the constructor, otherwise it does. Specifying, e.g. Mallocator also means the relevant code can be marked @nogc. . RefCounted only increments/decrements the ref count atomically if the contained type is `shared` . RefCounted!(shared T) can be sent to other threads. . UniqueArray behaves nearly like a normal array. You can even append to it, but it won't use GC memory (unless, of course, you chose to use GCAllocator)! I benchmarked RefCounted against C++'s std::shared_ptr comparing ldc to clang using both shared and non-shared payloads in D. std::shared_ptr is faster (I've never written a smart pointer before), but the advantage of non-atomic operations makes my D implementation just a bit faster when non-shared. I think with some work it can be significantly faster without any loss of thread safety. Atila
