Re: getOpt with shared
On Friday, 11 May 2018 at 17:25:44 UTC, Danny Arends wrote: Hey all, Is getopt not supposed to be used with shared structs ? I don't know but if you are opened to alternative i have just tested my getopt-like function at it works with shared, i don't know why... The design is different tho, it's not based on pointer, it rather generates the code to write the data without indirection. links: - https://github.com/BBasile/iz/commit/ac8b2e23214ec8c2d3a1000895b374f9c73f8ca7 - http://bbasile.github.io/iz/iz/options.html#handleArguments Baz.
Re: getOpt with shared
On 5/11/18 2:49 PM, Jonathan M Davis wrote:
On Friday, May 11, 2018 14:31:17 Steven Schveighoffer via Digitalmars-d-
learn wrote:
On 5/11/18 1:25 PM, Danny Arends wrote:
Hey all,
I have been working on creating a multi-threaded application, so I have
a shared configuration object which hold several command line parameters
(which I fill using getopt).
The problem is that I get deprecation warnings when trying to set
numerical values:
/usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation:
read-modify-write operations are not allowed for shared variables. Use
core.atomic.atomicOp!"+="(*receiver, 1) instead.
using getopt with a shared object and boolean values seems completely
broken:
/usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation not
allowed on bool *receiver += 1
/usr/include/dmd/phobos/std/getopt.d(751,46): Error: template instance
`std.getopt.handleOption!(shared(bool)*)` error instantiating
/usr/include/dmd/phobos/std/getopt.d(435,15):6 recursive
instantiations from here: getoptImpl!(string, shared(string)*, string,
shared(string)*, string, shared(VSync)*, string, shared(ulong)*, string,
shared(bool)*, string, shared(Verbose)*)
Is getopt not supposed to be used with shared structs ?
No, just fill in a local copy, and then copy it to the shared location.
In the case where all you want is read-only access to the data, build it
once and then cast to immutable:
immutable Config config;
void main(string[] args)
{
Config localConfig;
getopt(...); // fill in localConfig
// VERY IMPORTANT, only do this ONCE, and don't use it before this
line *(cast()) = localConfig;
}
It should now be accessible from all threads.
Normally, this is not encouraged (casting away immutable), but the
compiler gets that an immutable global is allowed to be initialized once
(in fact, you can do this without casting inside shared static
constructors).
Except that because this is done outside of a shared static constructor,
you're technically mutating immutable data. This _should_ work, but it is
violating the type system, and technically, the compiler is allowed to
assume that config is Config.init everywhere. In practice, I wouldn't expect
a problem, but because it's violating the type system, all bets are off.
It's not a problem. The compiler cannot assume the value is .init
because it doesn't have access to the whole codebase to see if some
static ctor has changed it. Effectively, you ARE changing it in a static
ctor, but just after all the other static ctors have run.
Crucially noted as well: don't use this value inside a static ctor!
-Steve
Re: getOpt with shared
On Friday, May 11, 2018 14:31:17 Steven Schveighoffer via Digitalmars-d-
learn wrote:
> On 5/11/18 1:25 PM, Danny Arends wrote:
> > Hey all,
> >
> > I have been working on creating a multi-threaded application, so I have
> > a shared configuration object which hold several command line parameters
> > (which I fill using getopt).
> >
> > The problem is that I get deprecation warnings when trying to set
> > numerical values:
> >
> > /usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation:
> > read-modify-write operations are not allowed for shared variables. Use
> > core.atomic.atomicOp!"+="(*receiver, 1) instead.
> >
> > using getopt with a shared object and boolean values seems completely
> > broken:
> >
> > /usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation not
> > allowed on bool *receiver += 1
> > /usr/include/dmd/phobos/std/getopt.d(751,46): Error: template instance
> > `std.getopt.handleOption!(shared(bool)*)` error instantiating
> > /usr/include/dmd/phobos/std/getopt.d(435,15):6 recursive
> > instantiations from here: getoptImpl!(string, shared(string)*, string,
> > shared(string)*, string, shared(VSync)*, string, shared(ulong)*, string,
> > shared(bool)*, string, shared(Verbose)*)
> >
> > Is getopt not supposed to be used with shared structs ?
>
> No, just fill in a local copy, and then copy it to the shared location.
>
> In the case where all you want is read-only access to the data, build it
> once and then cast to immutable:
>
> immutable Config config;
>
> void main(string[] args)
> {
> Config localConfig;
> getopt(...); // fill in localConfig
> // VERY IMPORTANT, only do this ONCE, and don't use it before this
> line *(cast()) = localConfig;
> }
>
> It should now be accessible from all threads.
>
> Normally, this is not encouraged (casting away immutable), but the
> compiler gets that an immutable global is allowed to be initialized once
> (in fact, you can do this without casting inside shared static
> constructors).
Except that because this is done outside of a shared static constructor,
you're technically mutating immutable data. This _should_ work, but it is
violating the type system, and technically, the compiler is allowed to
assume that config is Config.init everywhere. In practice, I wouldn't expect
a problem, but because it's violating the type system, all bets are off.
- Jonathan M Davis
Re: getOpt with shared
On 5/11/18 1:25 PM, Danny Arends wrote:
Hey all,
I have been working on creating a multi-threaded application, so I have
a shared configuration object which hold several command line parameters
(which I fill using getopt).
The problem is that I get deprecation warnings when trying to set
numerical values:
/usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation:
read-modify-write operations are not allowed for shared variables. Use
core.atomic.atomicOp!"+="(*receiver, 1) instead.
using getopt with a shared object and boolean values seems completely
broken:
/usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation not
allowed on bool *receiver += 1
/usr/include/dmd/phobos/std/getopt.d(751,46): Error: template instance
`std.getopt.handleOption!(shared(bool)*)` error instantiating
/usr/include/dmd/phobos/std/getopt.d(435,15): 6 recursive
instantiations from here: getoptImpl!(string, shared(string)*, string,
shared(string)*, string, shared(VSync)*, string, shared(ulong)*, string,
shared(bool)*, string, shared(Verbose)*)
Is getopt not supposed to be used with shared structs ?
No, just fill in a local copy, and then copy it to the shared location.
In the case where all you want is read-only access to the data, build it
once and then cast to immutable:
immutable Config config;
void main(string[] args)
{
Config localConfig;
getopt(...); // fill in localConfig
// VERY IMPORTANT, only do this ONCE, and don't use it before this line
*(cast()) = localConfig;
}
It should now be accessible from all threads.
Normally, this is not encouraged (casting away immutable), but the
compiler gets that an immutable global is allowed to be initialized once
(in fact, you can do this without casting inside shared static
constructors).
-Steve
Re: getOpt with shared
On Friday, May 11, 2018 18:01:18 Danny Arends via Digitalmars-d-learn wrote: > On Friday, 11 May 2018 at 17:49:17 UTC, Jonathan M Davis wrote: > > On Friday, May 11, 2018 17:25:44 Danny Arends via > > > > Digitalmars-d-learn wrote: > >> [...] > > > > getopt is designed to be single-threaded. The keyword shared is > > not used a single type in that module. If you want to use > > shared with anything in D, you have three options: > > > > [...] > > Hey Jonathan, > > Thanks for the long and insightful answer. > The object is indeed constructed from the main thread, but > afterwards multiple threads need to read the values given via the > command line. since everything in the object is read only I was > hoping to get away with making it shared. > > I should just define tls variables to use with getopt and then > set the corresponding variables in the shared object. If you want to operate on that data as shared, then yes. But if you're really just looking for each thread to have its own copy, I'd suggest that you either give each thread its own copy on thread creation or pass it using std.concurrency rather than trying to deal with a shared variable - especially since the type system has no way to know that you're just planning to read from the shared variable after that, and it will scream at you for various operations - and once shared is fully locked down, it will probably scream if you do much of _anything_ with shared, since at that point, the compiler would only allow operations that were either marked with shared or where it could guarantee that they were thread-safe. - Jonathan M Davis
Re: getOpt with shared
On Friday, 11 May 2018 at 17:49:17 UTC, Jonathan M Davis wrote: On Friday, May 11, 2018 17:25:44 Danny Arends via Digitalmars-d-learn wrote: [...] getopt is designed to be single-threaded. The keyword shared is not used a single type in that module. If you want to use shared with anything in D, you have three options: [...] Hey Jonathan, Thanks for the long and insightful answer. The object is indeed constructed from the main thread, but afterwards multiple threads need to read the values given via the command line. since everything in the object is read only I was hoping to get away with making it shared. I should just define tls variables to use with getopt and then set the corresponding variables in the shared object. Danny
Re: getOpt with shared
On Friday, May 11, 2018 17:25:44 Danny Arends via Digitalmars-d-learn wrote:
> Hey all,
>
> I have been working on creating a multi-threaded application, so
> I have a shared configuration object which hold several command
> line parameters (which I fill using getopt).
>
> The problem is that I get deprecation warnings when trying to set
> numerical values:
>
> /usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation:
> read-modify-write operations are not allowed for shared
> variables. Use core.atomic.atomicOp!"+="(*receiver, 1) instead.
>
> using getopt with a shared object and boolean values seems
> completely broken:
>
> /usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation
> not allowed on bool *receiver += 1
> /usr/include/dmd/phobos/std/getopt.d(751,46): Error: template
> instance `std.getopt.handleOption!(shared(bool)*)` error
> instantiating
> /usr/include/dmd/phobos/std/getopt.d(435,15):6 recursive
> instantiations from here: getoptImpl!(string, shared(string)*,
> string, shared(string)*, string, shared(VSync)*, string,
> shared(ulong)*, string, shared(bool)*, string, shared(Verbose)*)
>
> Is getopt not supposed to be used with shared structs ?
getopt is designed to be single-threaded. The keyword shared is not used a
single type in that module. If you want to use shared with anything in D,
you have three options:
1. Use core.atomic to atomically do stuff like increment a integer.
2. Use a mutex (or synchronized block) to protect access to a shared object.
You lock the mutex before accessing the object (and before _all_ accesses to
that object). Within that section of code, you cast away shared and operate
on the object as thread-local. Then at the end of that section, before
releasing the lock, you make sure that no thread-local references to the
shared object remain, and then free the lock. e.g. something like
synchronized(mutex)
{
auto tls = cast(MyObject)mySharedObject.
//... do stuff...
} // at this point, no thread-local references to mySharedObject
// should // exist
In this scenario, the types in question are not designed to be used with
shared at all. They're designed to be thread-local. So, if they're marked as
shared, they're basically useless except when you protect them with a mutex
and correctly cast away shared in order to operate on the object while it's
protected by the mutex and therefore is thread-safe.
3. An object is designed to be used as shared. In this case, it can have
shared member functions, and they can be called on shared objects, but then
internally, the object has to deal with properly managing shared. It either
uses atomics and/or uses mutexes as in #2 - it's just that in this case,
it's done internally by the object rather than the programmer using the
object.
synchronized classes are supposed to help with this particular case, but
unfortunately, they're not fully implemented, so they don't currently help.
But either way, the concept is still the same. You have on object that is
designed to work with shared and deals with all of the appropriate
protections internally.
So, those are your three options. In the case of getopt, if you want to use
shared, you basically have to use #2. It's not dealing with a basic type
like an int or pointer, so atomics aren't going to work, and it's not
designed to work with shared. So, if you want to do anything with getopt and
shared, you're going to have to protect it with a mutex.
That being said, I have to say that getopt seems like a really weird choice
for wanting to do anything with shared. You normally call it immediately at
the beginning of the program before doing anything with threads. The results
might then be passed on to other threads via std.concurrency or through
shared variables, but I wouldn't think that it would make a lot of sense to
try and used getopt from more than one thread.
- Jonathan M Davis
Re: getOpt with shared
On Friday, 11 May 2018 at 17:25:44 UTC, Danny Arends wrote:
Hey all,
I have been working on creating a multi-threaded application,
so I have a shared configuration object which hold several
command line parameters (which I fill using getopt).
The problem is that I get deprecation warnings when trying to
set numerical values:
/usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation:
read-modify-write operations are not allowed for shared
variables. Use core.atomic.atomicOp!"+="(*receiver, 1) instead.
using getopt with a shared object and boolean values seems
completely broken:
/usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation
not allowed on bool *receiver += 1
/usr/include/dmd/phobos/std/getopt.d(751,46): Error: template
instance `std.getopt.handleOption!(shared(bool)*)` error
instantiating
/usr/include/dmd/phobos/std/getopt.d(435,15):6
recursive instantiations from here: getoptImpl!(string,
shared(string)*, string, shared(string)*, string,
shared(VSync)*, string, shared(ulong)*, string, shared(bool)*,
string, shared(Verbose)*)
Is getopt not supposed to be used with shared structs ?
small example:
import std.getopt : getopt;
struct Settings {
size_t myvalue = 0;
}
shared(Settings) config;
int main(string[] args) {
getopt(args, "m|myvalue", &(config.myvalue));
return(0);
}
Changing size_t to bool, results in the compilation error
mentioned in the previous post
