Re: An important pull request: accessing shared affix for immutable data

Am Sat, 13 Feb 2016 19:16:43 +0100 schrieb Timon Gehr : > Not necessarily. shared is transitive and prefix/suffix are arbitrary > types which might contain mutable indirections. I don't want to disturb your conversation, but how about designing a working "shared" first? My

Re: An important pull request: accessing shared affix for immutable data

On 02/12/2016 02:12 PM, Andrei Alexandrescu wrote: https://github.com/D-Programming-Language/phobos/pull/3991 Finally I got around to update this. Please find holes in my logic. Updated documentation at:

Re: An important pull request: accessing shared affix for immutable data

On Sunday, 14 February 2016 at 11:14:59 UTC, Andrei Alexandrescu wrote: We can do the same, but we also have a better alternative. Most of our allocators support shrink-in-place. For now I haven't exposed it as a primitive but that's short work. When the object goes away we can shrink memory

Re: An important pull request: accessing shared affix for immutable data

One thing that I really miss from most of the discussions about RC is support for weak references, which IMO should definitely be supported in the standard RC implementation. Event if they are only needed rarely, there are some things that simply do not work without them. And I don't think

Re: An important pull request: accessing shared affix for immutable data

On 02/14/2016 03:08 AM, Sönke Ludwig wrote: For them to work natively, the lifetime of the allocated memory block and that of the reference count must be separate. Not necessarily. C++ makes this work for make_shared by keeping the memory allocated around (but not the object) until the last

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 08:16:04 UTC, Ola Fosheim Grøstad wrote: On Saturday, 13 February 2016 at 03:03:05 UTC, tsbockman wrote: I was actually suggesting encoding the shared-ness into the memory address itself. (No tricks - just allocate shared data in a separate memory range. This

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 01:42:08 UTC, Dicebot wrote: On 02/13/2016 03:35 AM, Andrei Alexandrescu wrote: Folks who defined refcounted types use the special AffixAllocator internally in an encapsulated manner. They define it, they use it - outside intervention is not possible. Those

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 00:30:58 UTC, Andrei Alexandrescu wrote: On 02/12/2016 06:52 PM, deadalnix wrote: [...] I think we're good there. -- Andrei Is there somewhere where I / others can see an explanation of how "we're good"? Those sound like genuine problems.

Re: An important pull request: accessing shared affix for immutable data

On 2/13/16 7:40 AM, John Colvin wrote: On Saturday, 13 February 2016 at 00:30:58 UTC, Andrei Alexandrescu wrote: On 02/12/2016 06:52 PM, deadalnix wrote: [...] I think we're good there. -- Andrei Is there somewhere where I / others can see an explanation of how "we're good"? Those sound

Re: An important pull request: accessing shared affix for immutable data

On 2/12/16 10:10 PM, tsbockman wrote: Cool. Then this technique could also be used to address Timon Gehr's concerns about violating the type system, if necessary. Regarding that. Not sure what he meant, but it dawned on me we've never mentioned in the language spec that using synchronization

Re: An important pull request: accessing shared affix for immutable data

On 13.02.2016 03:35, Andrei Alexandrescu wrote: On 02/12/2016 09:21 PM, Timon Gehr wrote: Const could also mean mutable. This can hence reference the same data as both shared and unshared, which violates the type system. If const comes from mutable, then shared is superfluous leading to extra

Re: An important pull request: accessing shared affix for immutable data

On 02/13/2016 11:55 AM, Timon Gehr wrote: If you alias the same data as both shared and unshared, the shared version can be sent to another thread which will then modify it, violating the guarantees on the unshared reference. Oh, I understand. So you're thinking of someone taking the address

Re: An important pull request: accessing shared affix for immutable data

On 13.02.2016 18:42, Andrei Alexandrescu wrote: On 02/13/2016 11:55 AM, Timon Gehr wrote: If you alias the same data as both shared and unshared, the shared version can be sent to another thread which will then modify it, violating the guarantees on the unshared reference. Oh, I understand.

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 18:16:43 UTC, Timon Gehr wrote: On 13.02.2016 18:42, Andrei Alexandrescu wrote: On 02/13/2016 11:55 AM, Timon Gehr wrote: If you alias the same data as both shared and unshared, the shared version can be sent to another thread which will then modify it,

Re: An important pull request: accessing shared affix for immutable data

On Friday, 12 February 2016 at 23:29:58 UTC, Timon Gehr wrote: The first thing that comes to mind is that accessing a global associative array is not 'pure'. Not necessarily. Purity depends on the allocator and how you access it (via parameter or through a static/shared instance). I have

Re: An important pull request: accessing shared affix for immutable data

2016-02-12 20:12 GMT+01:00 Andrei Alexandrescu via Digitalmars-d < digitalmars-d@puremagic.com>: > https://github.com/D-Programming-Language/phobos/pull/3991 > The only difference between an approach based on an associative array and > AffixAllocator is that the latter is faster (the association

Re: An important pull request: accessing shared affix for immutable data

On Friday, 12 February 2016 at 19:12:48 UTC, Andrei Alexandrescu wrote: https://github.com/D-Programming-Language/phobos/pull/3991 A short while ago Dicebot discussed the notion of using the allocator to store the reference count of objects (and generally metadata). The allocator seems to be

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 03:03:05 UTC, tsbockman wrote: On Saturday, 13 February 2016 at 01:27:49 UTC, Andrei Alexandrescu wrote: On 02/12/2016 08:02 PM, tsbockman wrote: What about providing a way to mark a piece of data as thread-local at allocation time? Yah, could be part of the

Re: An important pull request: accessing shared affix for immutable data

On 02/13/2016 03:07 PM, Iakh wrote: So you can use metadata only with global allocators, until you don't need to save ref to the allocator. Well you can use other allocators if you save them so you have them available for deallocation. -- Andrei

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 21:49:48 UTC, ZombineDev wrote: On Saturday, 13 February 2016 at 02:35:43 UTC, Andrei Alexandrescu wrote: On 02/12/2016 09:21 PM, Timon Gehr wrote: Const could also mean mutable. This can hence reference the same data as both shared and unshared, which

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 13:10:19 UTC, Andrei Alexandrescu wrote: On 2/13/16 7:40 AM, John Colvin wrote: On Saturday, 13 February 2016 at 00:30:58 UTC, Andrei Alexandrescu wrote: On 02/12/2016 06:52 PM, deadalnix wrote: [...] I think we're good there. -- Andrei Is there somewhere

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 22:16:02 UTC, rsw0x wrote: On Saturday, 13 February 2016 at 21:10:50 UTC, Andrei Alexandrescu wrote: On 02/13/2016 01:50 PM, Mathias Lang via Digitalmars-d wrote: [...] There's no need. I'll do the implementation with the prefix, and if you do it with a

Re: An important pull request: accessing shared affix for immutable data

On 02/13/2016 01:50 PM, Mathias Lang via Digitalmars-d wrote: 2016-02-12 20:12 GMT+01:00 Andrei Alexandrescu via Digitalmars-d >: https://github.com/D-Programming-Language/phobos/pull/3991 The only difference between an

Re: An important pull request: accessing shared affix for immutable data

On Friday, 12 February 2016 at 19:12:48 UTC, Andrei Alexandrescu wrote: https://github.com/D-Programming-Language/phobos/pull/3991 A short while ago Dicebot discussed the notion of using the allocator to store the reference count of objects (and generally metadata). The allocator seems to be

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 02:35:43 UTC, Andrei Alexandrescu wrote: On 02/12/2016 09:21 PM, Timon Gehr wrote: Const could also mean mutable. This can hence reference the same data as both shared and unshared, which violates the type system. If const comes from mutable, then shared is

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 21:38:30 UTC, ZombineDev wrote: ... get non-shared references in to it (and vice-versa - you should have shared references in a non-shared objects). ... you should ***not be able to*** have shared references in a non-shared objects

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 21:10:50 UTC, Andrei Alexandrescu wrote: There's no need. I'll do the implementation with the prefix, and if you do it with a global hashtable within the same or better speed, my hat is off to you. That is false dichotomy. What about storing the metadata at

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 21:10:50 UTC, Andrei Alexandrescu wrote: On 02/13/2016 01:50 PM, Mathias Lang via Digitalmars-d wrote: 2016-02-12 20:12 GMT+01:00 Andrei Alexandrescu via Digitalmars-d >:

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 21:12:10 UTC, Andrei Alexandrescu wrote: On 02/13/2016 03:07 PM, Iakh wrote: So you can use metadata only with global allocators, until you don't need to save ref to the allocator. Well you can use other allocators if you save them so you have them available

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 22:01:45 UTC, deadalnix wrote: On Saturday, 13 February 2016 at 21:10:50 UTC, Andrei Alexandrescu wrote: There's no need. I'll do the implementation with the prefix, and if you do it with a global hashtable within the same or better speed, my hat is off to you.

Re: An important pull request: accessing shared affix for immutable data

On 02/13/2016 04:34 AM, Andrei Alexandrescu wrote: > On 02/12/2016 08:42 PM, Dicebot wrote: >> So you >> envision this kind of metadata support to be only available via specific >> type of allocator, AffixAllocator, and not being supported by most of >> them? > > That is correct. You want

Re: An important pull request: accessing shared affix for immutable data

On 2/13/16 5:01 PM, deadalnix wrote: What about storing the metadata at an address that is computable from from the object's address, while not being contiguous with the object allocated ? Is substracting a constant really the only option here ? (hint, it is not) I'd say, you have at your

Re: An important pull request: accessing shared affix for immutable data

On Sunday, 14 February 2016 at 01:27:40 UTC, Andrei Alexandrescu wrote: On 2/13/16 5:01 PM, deadalnix wrote: What about storing the metadata at an address that is computable from from the object's address, while not being contiguous with the object allocated ? Is substracting a constant really

An important pull request: accessing shared affix for immutable data

https://github.com/D-Programming-Language/phobos/pull/3991 A short while ago Dicebot discussed the notion of using the allocator to store the reference count of objects (and generally metadata). The allocator seems to be a good place because in a way it's a source of "ground truth" - no

Re: An important pull request: accessing shared affix for immutable data

On Friday, 12 February 2016 at 19:12:48 UTC, Andrei Alexandrescu wrote: https://github.com/D-Programming-Language/phobos/pull/3991 A short while ago Dicebot discussed the notion of using the allocator to store the reference count of objects (and generally metadata). The allocator seems to be

Re: An important pull request: accessing shared affix for immutable data

On 02/12/2016 06:29 PM, Timon Gehr wrote: The first thing that comes to mind is that accessing a global associative array is not 'pure'. The weird thing is in this case is. The analogy is limited. -- Andrei

Re: An important pull request: accessing shared affix for immutable data

On Friday, 12 February 2016 at 19:12:48 UTC, Andrei Alexandrescu wrote: * If the buffer is const, then the allocator must conservatively assume it might have been immutable and subsequently shared among threads. Therefore, several threads may request the affix of the same buffer

Re: An important pull request: accessing shared affix for immutable data

On Friday, 12 February 2016 at 19:12:48 UTC, Andrei Alexandrescu wrote: * If the buffer is const, then the allocator must conservatively assume it might have been immutable and subsequently shared among threads. Therefore, several threads may request the affix of the same buffer

Re: An important pull request: accessing shared affix for immutable data

On 02/12/2016 09:12 PM, Andrei Alexandrescu wrote: > So after thinking a bit I managed to convince myself that the affixes in > an AffixAllocator can be accessed with removing immutable, but without > actually breaking any guarantee made by the type system. (Affixes are > extra bytes allocated

Re: An important pull request: accessing shared affix for immutable data

On 02/13/2016 03:35 AM, Andrei Alexandrescu wrote: > Folks who defined refcounted types use the special AffixAllocator > internally in an encapsulated manner. They define it, they use it - > outside intervention is not possible. Those who use their own allocation > calls and then use assumeUnique

Re: An important pull request: accessing shared affix for immutable data

On 02/12/2016 08:42 PM, Dicebot wrote: So you envision this kind of metadata support to be only available via specific type of allocator, AffixAllocator, and not being supported by most of them? That is correct. You want metadata, you put an AffixAllocator together. -- Andrei

Re: An important pull request: accessing shared affix for immutable data

On 12.02.2016 20:12, Andrei Alexandrescu wrote: https://github.com/D-Programming-Language/phobos/pull/3991 A short while ago Dicebot discussed the notion of using the allocator to store the reference count of objects (and generally metadata). The allocator seems to be a good place because in a

Re: An important pull request: accessing shared affix for immutable data

On 02/12/2016 06:52 PM, deadalnix wrote: Providing some metadata in the allocate is in itself a good idea. Locating these data with the object is usually not : - Mutating the metadata will create sharing overhead on the whole cache line. Sharing of immutable would become inefficient. - It

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 01:27:49 UTC, Andrei Alexandrescu wrote: On 02/12/2016 08:02 PM, tsbockman wrote: What about providing a way to mark a piece of data as thread-local at allocation time? Yah, could be part of the metadata. Then after inspecting that bit, shared can be casted

Re: An important pull request: accessing shared affix for immutable data

On 02/12/2016 09:21 PM, Timon Gehr wrote: Const could also mean mutable. This can hence reference the same data as both shared and unshared, which violates the type system. If const comes from mutable, then shared is superfluous leading to extra synchronization. That's suboptimal, but how

Re: An important pull request: accessing shared affix for immutable data

On Saturday, 13 February 2016 at 03:05:08 UTC, Andrei Alexandrescu wrote: I was actually suggesting encoding the shared-ness into the memory address itself. (No tricks - just allocate shared data in a separate memory range. This may only be feasible on 64-bit though, with its over-abundance

Re: An important pull request: accessing shared affix for immutable data

On 02/12/2016 08:02 PM, tsbockman wrote: What about providing a way to mark a piece of data as thread-local at allocation time? Yah, could be part of the metadata. Then after inspecting that bit, shared can be casted away. I consider that an optimization that doesn't add or remove

Re: An important pull request: accessing shared affix for immutable data

On 13.02.2016 01:30, Andrei Alexandrescu wrote: On 02/12/2016 06:29 PM, Timon Gehr wrote: The first thing that comes to mind is that accessing a global associative array is not 'pure'. The weird thing is in this case is. The analogy is limited. -- Andrei Why is it limited? Accessing the

Re: An important pull request: accessing shared affix for immutable data

On 12.02.2016 20:12, Andrei Alexandrescu wrote: * If the buffer is mutable, then the allocator assumes it hasn't been shared across threads, so it returns a reference to a mutable affix. * If the buffer is const, then the allocator must conservatively assume it might have been immutable and

Re: An important pull request: accessing shared affix for immutable data

On 02/12/2016 08:24 PM, Dicebot wrote: - considering this API is going to be dangerously @system, possible corruption of metadata is indeed very scary - though not a blocker within my own value list I don't think this is an argument worth minding, either. - I wonder if would make for a

Re: An important pull request: accessing shared affix for immutable data

On 02/12/2016 10:03 PM, tsbockman wrote: On Saturday, 13 February 2016 at 01:27:49 UTC, Andrei Alexandrescu wrote: On 02/12/2016 08:02 PM, tsbockman wrote: What about providing a way to mark a piece of data as thread-local at allocation time? Yah, could be part of the metadata. Then after