Re: shared - i need it to be useful
On Fri, Oct 19, 2018 at 9:45 AM Steven Schveighoffer via Digitalmars-d
wrote:
>
> On 10/18/18 9:09 PM, Manu wrote:
> > On Thu, Oct 18, 2018 at 5:30 PM Timon Gehr via Digitalmars-d
> > wrote:
> >>
> >> On 18.10.18 23:34, Erik van Velzen wrote:
> >>> If you have an object which can be used in both a thread-safe and a
> >>> thread-unsafe way that's a bug or code smell.
> >>
> >> Then why do you not just make all members shared? Because with Manu's
> >> proposal, as soon as you have a shared method, all members effectively
> >> become shared.
> >
> > No they don't, only facets that overlap with the shared method.
> > I tried to present an example before:
> >
> > struct Threadsafe
> > {
> >int x;
> >Atomic!int y;
> >void foo() shared { ++y; } // <- shared interaction only affects 'y'
> >void bar() { ++x; ++y; } // <- not threadsafe, but does not violate
> > foo's commitment; only interaction with 'y' has any commitment
> > associated with it
> >void unrelated() { ++x; } // <- no responsibilities are transposed
> > here, you can continue to do whatever you like throughout the class
> > where 'y' is not concerned
> > }
> >
> > In practise, and in my direct experience, classes tend to have exactly
> > one 'y', and either zero (pure utility), or many such 'x' members.
> > Threadsafe API interacts with 'y', and the rest is just normal
> > thread-local methods which interact with all members thread-locally,
> > and may also interact with 'y' while not violating any threadsafety
> > commitments.
>
> I promised I wouldn't respond, I'm going to break that (obviously).
>
> But that's because after reading this description I ACTUALLY understand
> what you are looking for.
>
> I'm going to write a fuller post later, but I can't right now. But the
> critical thing here is, you want a system where you can divvy up a type
> into pieces you share and pieces you don't. But then you *don't* want to
> have to share only the shared pieces. You want to share the whole thing
> and be sure that it can't access your unshared pieces.
>
> This critical requirement makes things a bit more interesting. For the
> record, the most difficult thing to reaching this understanding was that
> whenever I proposed anything, your answer was something like 'I just
> can't work with that', and when I asked why, you said 'because it's
> useless', etc. Fully explaining this point is very key to understanding
> your thinking.
>
> To be continued...
I'm glad that there's movement here... but I'm still not 100%
convinced you understood me; perhaps getting close though.
I only say that because your description above has a whole lot more
words and complexity than is required to express my proposal. If you
perceive that complexity in structural terms, then I am still not
clearly understood.
> "divvy up a type into pieces"
This is an odd mental model of what I'm saying, and I can't sympathise
with those words, but if they work for you, and we agree on the
semantics, then sure...
If you write an object with some const methods and some non-const
methods, then take a const instance of the object... you can only call
the const methods. Have you 'divvied up the type' into a const portion
and a non-const portion? If the answer is yes, then I can accept your
description.
I would talk in terms of restriction:
An object has 4 functions, 2 are mutable, 2 are const... you apply
const to the type and you are *restricted* to only calling the 2 const
functions.
An object has 4 functions, 2 are unsahred, 2 are shared... you apply
shared to the type and you are *restricted* to only calling the 2
shared (threadsafe) functions.
I haven't 'broken the type up', I'm just restricting what you can do
to it from within a particular context. In the const context, you
can't mutate it. In the shared context, you can't do un-threadsafe to
it, and the guarantee of that is embedded in the rules:
1. shared data can not be read or written
2. shared methods must be threadsafe
a. this may require that they be @trusted at the low-level, like the
methods of `Atomic(T)`
b. no other method may violate the shared method's promise,
otherwise it does not actually deliver its promise
i. I have extensive experience with this and it's just not a
problem in practise, but compiler technology to assist would be
welcome!
ii. This does NOT mean the not-shared methods are somehow shared;
they just need to be careful when interacting with some (usually
small) subset of members
This design implies strong encapsulation, but that's a naturally
occurring tendency implementing anything that's threadsafe.
As a helpful best-practise to assure that non-shared methods don't
undermine a shared method's commitment; prefer to interact with
volatile members via accessors or properties that are themselves
shared (can be private if you like).
You will find that it's not actually hard to deliver on the object's commitment.
If you write tooling that is at the level one-up from the bott
Re: shared - i need it to be useful
On Fri., 19 Oct. 2018, 3:10 am Walter Bright via Digitalmars-d, < digitalmars-d@puremagic.com> wrote: > On 10/17/2018 12:20 AM, Manu wrote: > > What does it mean 'aliased' precisely? > > Aliasing means there are two paths to the same piece of data. That could > be two > pointers pointing to the same data, or one pointer to a variable that is > accessible by name. > The reason I ask is because, by my definition, if you have: int* a; shared(int)* b = a; While you have 2 numbers that address the same data, it is not actually aliased because only `a` can access it. It is not aliased in any practical sense. > It doesn't really give us > > anything in practice that we don't have in C++. > > It provides a standard, enforced way to distinguish shared data from > unshared > data, and no way to bypass it in @safe code. There's no way to do that in > C++. > Right, but we can do so much better. I want shared to model "what is thread-@safe to do", because that models what you are able to do, and what API's should encourage when operating on `shared` things. Exclusively distinguishing shared and unshared data is not an interesting distinction if shared data has no access. I've been trying to say over and over; ignore what you think you know about that definition, accept my rules strictly as given (they're very simple and concise, there's only 2 rules), such that shared will mean "is threadsafe to call with this data" when applied to function args... Build the thought experiment outward from there. That's an interesting and useful definition for shared, and it leads to a framework where shared is useful in a fully @safe SMP program, and even models @safe transitions across unshared -> shared boundaries (parallel for, map/reduce, etc, fork and join style workloads), which are typical lock-free patterns. Lock-and-cast semantics are preserved unchanged for those that interact in the way shared is prescribed today, but strictly modeling that workflow is uninteresting, because it's unsafe by definition. I'm not losing that, but I'm trying to introduce a safe workflow that exists in complement, and my model works. I don't even know if we have a mutex defined in our codebase, we don't use them... but we can max out a 64core thread ripper. >
Re: Shared - Another Thread
On Fri, Oct 19, 2018 at 5:05 PM Dominikus Dittes Scherkl via Digitalmars-d wrote: > > Therefore it is possible to implicitly cast from mutable or > immutable to const but not in any other direction. > > I think for unshared, shared and threadsave it should be the same: > The second is a declaration attribute, the third a parameter > attribute. I certainly had this thought... if you take a purist point of view, it sounds reasonable. BUT, I think it's a bad choice for practical reasons: 1. Adding a new attribute is heavy handed. 2. It's also unnecessary; the one that would be analogous to immutable is not actually very interesting. At best, it offers the potential for a small number of optimisations, which can be accessed without it. 3. Threadsafety carries a VERY small surface area in the language and in code in general... we don't need to add a pile of machinery for this. Sure, immutable is kinda nice, but in strict terms, it's unnecessary... we could all get on without it. If immutable were removed from the language completely, it would barely affect me. We couldn't get on without const. The same applies here, we must have the const analogous one. Can't live without that. Consider the frequency of immutable to const in any code you've ever seen. This is a very real view of the relative usefulness of the 2 constructs. Now imagine the same comparison to shared... you hardly see shared at all to begin with; and the one analogous to immutable would be relatively so much more rare still. How can you find that such a construct carries its weight with respect to its rare-ness, when its usefulness is very limited to begin with?
Re: Shared - Another Thread
On Fri, Oct 19, 2018 at 4:45 PM Dominikus Dittes Scherkl via Digitalmars-d wrote: > > On Friday, 19 October 2018 at 18:11:50 UTC, Manu wrote: > > On Fri, Oct 19, 2018 at 6:45 AM Dominikus Dittes Scherkl via > > Digitalmars-d wrote: > >> > >> On Thursday, 18 October 2018 at 16:24:39 UTC, Manu wrote: > >> > [...] What issues am I failing to address? > >> [...] Another point is the part of "how can the compiler > >> support the expert in writing threadsave methods" - which you > >> answered with "not a little bit at the moment, but we may > >> improve this in the future" - and that is not at all > >> satisfying. > > > > I think you've misunderstood. > > My proposal is @safe... if you stay @safe, you will receive > > guarantee that your code is threadsafe. > On user side, yes. > > If you want to implement a low-level device, you must implement > > a @trusted function, and I don't know what the compiler can do > > to help you. > Yes, but that's seldom. More often the "expert" will write new > shared types using the low level trusted functions like anybody > else. But that > still requires special care - he has to consider tread-safety in > every > method of a new type, even the non-shared ones. And he has to > fill any > possible gap like construction and assignment so that the > end-user is really sure to not accidentally misusing the type! > And I think a serious proposal need to address this - I think the > compiler could really help here (e.g. prescribe what operators > need to be overloaded and check that all methods use the proper > mechanisms to lock the shared members before operating on them > etc.) > > > So saying that my response that "there is @trusted code at the > > bottom of the stack" is not satisfying is really just a comment > > on your opinion about @trusted code in general. > That just comes on top of it. > > > My proposal is designed to be useful and @safe for *users* as > > primary goal. > I agree with you, but others seem not so convinced (yet?). > [...] > > The user has manually cast to unshared inside their > > unsafe/(@trusted?) function, what more signal do they need that > > they've engaged in an unsafe operation? > Some hints what to do to be able to trust them? > You asked what issues you were failing to address. That was just > some ideas of mine what you may address in addition. I understand. I don't have good ideas to add mechanical guarantees, other than something extremely likely to flag false-positives like "any `shared` piece of data involved in an unsafe cast in any function should look suspicious when accessed elsewhere within this module"...? I think this is an area for further development, but I don't think it's a barrier to making shared a useful @safe language construct with respect to the type safety. It is possible to define the typesafety rules correctly and then start writing experimental code. > More often the "expert" will write new > shared types using the low level trusted functions like anybody > else. But that > still requires special care - he has to consider tread-safety in > every > method of a new type, even the non-shared ones. This shouldn't be true unless they're writing new @trusted functions. If they're staying @safe, then there is nothing you can do to the lower-level utility that's not threadsafe, so you can freely use it throughout your new type. If this is true, then the @trusted function is not threadsafe as it promises. One thing I know is, your proposition above is unusual. In my experience among hundreds, perhaps thousands of colleagues, people don't just start presuming to write threadsafe tooling. We have 4-5 such threadsafe tools in our library, they are at the bottom of the stack, and would contain @trusted functions under my proposal... all other code just makes use of them (and quite extensive use of them!). People don't write new thredsafe machinery for fun, it's a very serious endeavour. The countless uses of those tools and the structure built on top should be @safely interactible, which is possible under my proposal.
Re: Need help with setting up LDC to cross-compile to Android/ARM
On Fri, Oct 19, 2018 at 08:50:36PM +, Joakim via Digitalmars-d wrote: > On Wednesday, 17 October 2018 at 21:23:21 UTC, H. S. Teoh wrote: > > I'm trying to follow the instructions on this page: > > > > https://wiki.dlang.org/Build_D_for_Android [...] On a side note, the last section on that page mentions not knowing how to create a keystore from scratch; actually, it's very simple with the `keytool` utility that comes with the Oracle JRE. I added the steps on the talk page. The only thing I'm unsure about is whether keytool is available natively on Android. If not, then you'll have to generate the keystore on a PC and copy it over to Android afterwards. T -- Computers shouldn't beep through the keyhole.
Re: Shared - Another Thread
On Saturday, 20 October 2018 at 00:46:36 UTC, Nicholas Wilson wrote: Mutable = value may change const = I will not change the value immutable = the value will not change unshared = I (well the current thread) owns the reference shared = reference not owned, no unordered access, no (unordered) writes threadsafe = ??? unshared = the current thread owns the reference threadsafe = I guarantee no race conditions or deadlocks will occur shared = every thread may have references
Re: Truly @nogc Exceptions?
On Friday, 19 October 2018 at 23:46:29 UTC, Nicholas Wilson wrote: On Friday, 19 October 2018 at 23:34:01 UTC, Atila Neves wrote: On Thursday, 20 September 2018 at 12:48:13 UTC, Steven Schveighoffer wrote: How is the exception destroyed when dip1008 is enabled? Apparently, it isn't. Which renders dip1008 pretty much useless since we could already use static immutable exceptions before. Wow, that is useless! Is that an implementation bug or was that specified by the DIP? I hope it's a bug - I opened an issue: https://issues.dlang.org/show_bug.cgi?id=19317
Re: Shared - Another Thread
On Saturday, 20 October 2018 at 00:00:49 UTC, Dominikus Dittes Scherkl wrote: Hmm. mutable, immutable and const form a triple, the second is a declaration attribute, the last an parameter attribute, indicating that you don't want to modify the parameter, may it be because you can't (as it is immutable) or you only don't need to despite it would be possible (if it was mutable). The later is your responsibility to guarantee (with the help from the compiler). Therefore it is possible to implicitly cast from mutable or immutable to const but not in any other direction. I think for unshared, shared and threadsave it should be the same: The second is a declaration attribute, the third a parameter attribute. The first two can implicitly be cast to threadsave, may be because it is thread-local and therefore no race condition is possible, or may be because you take special care in your type to guarantee the thread safety by using atomic operations or locking or whatever. That make it possible, that the implicit cast from shared to unshared can be avoided while still providing functions that can take both kinds of arguments. Yes, that would add a little to the attribute bloat (new keyword) but not to the number of attributes per type or parameter. Mutable = value may change const = I will not change the value immutable = the value will not change unshared = I (well the current thread) owns the reference shared = reference not owned, no unordered access, no (unordered) writes threadsafe = ???
Re: Shared - Another Thread
On Friday, 19 October 2018 at 15:46:20 UTC, Stanislav Blinov wrote: On Friday, 19 October 2018 at 13:40:54 UTC, Dominikus Dittes Scherkl wrote: Conflating "shared" and "threadsave" in that manner was, I think, the biggest mistake of your proposal. He talked about it in a previous thread, and generally I would agree with him that such conflation is indeed beneficial provided that some concessions are made for `shared`. Moreover, yet another attribute? Please no... Hmm. mutable, immutable and const form a triple, the second is a declaration attribute, the last an parameter attribute, indicating that you don't want to modify the parameter, may it be because you can't (as it is immutable) or you only don't need to despite it would be possible (if it was mutable). The later is your responsibility to guarantee (with the help from the compiler). Therefore it is possible to implicitly cast from mutable or immutable to const but not in any other direction. I think for unshared, shared and threadsave it should be the same: The second is a declaration attribute, the third a parameter attribute. The first two can implicitly be cast to threadsave, may be because it is thread-local and therefore no race condition is possible, or may be because you take special care in your type to guarantee the thread safety by using atomic operations or locking or whatever. That make it possible, that the implicit cast from shared to unshared can be avoided while still providing functions that can take both kinds of arguments. Yes, that would add a little to the attribute bloat (new keyword) but not to the number of attributes per type or parameter.
Re: DMD Linker Issue on Windows
On Thursday, 18 October 2018 at 16:21:00 UTC, Kai wrote: On Thursday, 18 October 2018 at 07:51:07 UTC, Andre Pany wrote: On Thursday, 18 October 2018 at 00:24:29 UTC, Kai wrote: On Wednesday, 17 October 2018 at 17:44:34 UTC, Adam D. Ruppe wrote: [...] Hmm - wish it was so. When architecture not specified, the linker crashes. When it's given, this happens (seems to be a vibe issue?): [...] As far as I can see, there are some Windows libraries missing. These libraries are part of the Windows sdk (You can use the vs build tools installer). Maybe we can include at least the libraries needed for vibe.d into the dmd Windows package? Kind regards Andre I have multiple incarnations of both libs on my machine. I copied the latest version of each into my the lib folder of the DMD install path and it still fails with neither of them being found. What am I doing wrong? Where do they need to go? Thanks for any help once more... Who knows what path it is using for libs, I don't know if it does for lld-link and the config file was removed almost entirely. It determines these things on it's own and I'm not sure if there is any way to display what it is actually using without looking through the source. Try using the "-v" argument with DMD and look for the command it uses to run the linker. There might be a parameter passed for library paths there. Those two files were removed since VS 2015 I think, so odds are that's why it can't find it. If it is using a newer VS install path.
Re: Truly @nogc Exceptions?
On Friday, 19 October 2018 at 23:34:01 UTC, Atila Neves wrote: On Thursday, 20 September 2018 at 12:48:13 UTC, Steven Schveighoffer wrote: How is the exception destroyed when dip1008 is enabled? Apparently, it isn't. Which renders dip1008 pretty much useless since we could already use static immutable exceptions before. Wow, that is useless! Is that an implementation bug or was that specified by the DIP?
Re: Shared - Another Thread
On Friday, 19 October 2018 at 18:11:50 UTC, Manu wrote: On Fri, Oct 19, 2018 at 6:45 AM Dominikus Dittes Scherkl via Digitalmars-d wrote: On Thursday, 18 October 2018 at 16:24:39 UTC, Manu wrote: > [...] What issues am I failing to address? [...] Another point is the part of "how can the compiler support the expert in writing threadsave methods" - which you answered with "not a little bit at the moment, but we may improve this in the future" - and that is not at all satisfying. I think you've misunderstood. My proposal is @safe... if you stay @safe, you will receive guarantee that your code is threadsafe. On user side, yes. If you want to implement a low-level device, you must implement a @trusted function, and I don't know what the compiler can do to help you. Yes, but that's seldom. More often the "expert" will write new shared types using the low level trusted functions like anybody else. But that still requires special care - he has to consider tread-safety in every method of a new type, even the non-shared ones. And he has to fill any possible gap like construction and assignment so that the end-user is really sure to not accidentally misusing the type! And I think a serious proposal need to address this - I think the compiler could really help here (e.g. prescribe what operators need to be overloaded and check that all methods use the proper mechanisms to lock the shared members before operating on them etc.) So saying that my response that "there is @trusted code at the bottom of the stack" is not satisfying is really just a comment on your opinion about @trusted code in general. That just comes on top of it. My proposal is designed to be useful and @safe for *users* as primary goal. I agree with you, but others seem not so convinced (yet?). [...] The user has manually cast to unshared inside their unsafe/(@trusted?) function, what more signal do they need that they've engaged in an unsafe operation? Some hints what to do to be able to trust them? You asked what issues you were failing to address. That was just some ideas of mine what you may address in addition.
Re: Truly @nogc Exceptions?
On Thursday, 20 September 2018 at 12:48:13 UTC, Steven
Schveighoffer wrote:
On 9/20/18 6:48 AM, Atila Neves wrote:
On Wednesday, 19 September 2018 at 21:16:00 UTC, Steven
Schveighoffer wrote:
Given dip1008, we now can throw exceptions inside @nogc code!
This is really cool, and helps make code that uses exceptions
or errors @nogc. Except...
The mechanism to report what actually went wrong for an
exception is a string passed to the exception during
*construction*. Given that you likely want to make such an
exception inside a @nogc function, you are limited to passing
a compile-time-generated string (either a literal or one
generated via CTFE).
I expressed my concern for DIP1008 and the `msg` field when it
was first announced. I think the fix is easy and a one line
change to dmd. I also expressed this on that thread but was
apparently ignored. What's the fix? Have the compiler insert a
call to the exception's destructor at the end of the
`catch(scope Exception)` block. That's it. The `msg` field is
just a slice, point it to RAII managed memory and you're good
to go.
Give me deterministic destruction of exceptions caught by
scope when using dip1008 and I'll give you @nogc exception
throwing immediately. I've even already written the code!
I thought it already did that?
Nope:
---
class MyException: Exception {
static int numInstances;
this(string msg) {
super(msg);
++numInstances;
}
~this() {
--numInstances;
}
}
void main() {
assert(MyException.numInstances == 0);
try
throw new MyException("oops");
catch(MyException _)
assert(MyException.numInstances == 1);
assert(MyException.numInstances == 0);
}
---
% dmd -dip1008 -run exception.d
core.exception.AssertError@exception.d(21): Assertion failure
How is the exception destroyed when dip1008 is enabled?
Apparently, it isn't. Which renders dip1008 pretty much useless
since we could already use static immutable exceptions before.
But this means you still have to build msg when throwing the
error/exception. It's not needed until you print it, and
there's no reason anyway to make it allocate, even with RAII.
For some reason D forces msg to be built, but it does't e.g.
build the entire stack trace string before hand, or build the
string that shows the exception class name or the file/line
beforehand.
Allocating and building the string doesn't bother me - in all of
my uses it's eventually going to get printed (which means the
string needed to be built), and the exceptional path can be slow,
I don't mind.
But, one could always store a tuple of members in an exception
class instead and only build the string on demand.
I just think it's easier with an RAII string.
Re: Need help with setting up LDC to cross-compile to Android/ARM
On Fri, Oct 19, 2018 at 02:41:48PM -0700, H. S. Teoh via Digitalmars-d wrote: [...] > In the meantime, is there a particular version of the NDK that I > should use? Currently I have android-ndk-r13b-linux-x86_64.zip > installed. Will it work? [...] Haha, I feel so silly now. NDK r13b does not seem to have the sysroot subdir required by the clang build command, that's why it couldn't find the system headers. So I ditched r13b and installed r17b instead, and now I can build the runtime successfully! T -- Questions are the beginning of intelligence, but the fear of God is the beginning of wisdom.
Re: Need help with setting up LDC to cross-compile to Android/ARM
On Fri, Oct 19, 2018 at 02:41:48PM -0700, H. S. Teoh via Digitalmars-d wrote: [...] > I tried ldc-build-runtime with --ninja and it came back with a bunch of > errors about "cortex-a8" being an unsupported target, and then > segfaulted. So I'm going to try the "official" LDC release instead... [...] I'm getting the same error with the official LDC 12.0. Running with --ninja revealed that it's not just sys/types.h, but seems that a bunch of standard C headers (possibly all?) that are mysteriously missing: $ ldc-build-runtime --targetPreset=Android-arm --dFlags=-w;-mcpu=cortex-a8 --buildDir=droid32 --ninja -- Configuring done -- Generating done CMake Warning (dev): Policy CMP0058 is not set: Ninja requires custom command byproducts to be explicit. Run "cmake --help-policy CMP0058" for policy details. Use the cmake_policy command to set the policy and suppress this warning. This project specifies custom command DEPENDS on files in the build tree that are not specified as the OUTPUT or BYPRODUCTS of any add_custom_command or add_custom_target: ldc-src/runtime/druntime/src/core/atomic.d ldc-src/runtime/druntime/src/core/attribute.d ldc-src/runtime/druntime/src/core/bitop.d ldc-src/runtime/druntime/src/core/checkedint.d ldc-src/runtime/druntime/src/core/cpuid.d ldc-src/runtime/druntime/src/core/demangle.d ldc-src/runtime/druntime/src/core/exception.d ldc-src/runtime/druntime/src/core/internal/abort.d ldc-src/runtime/druntime/src/core/internal/arrayop.d ldc-src/runtime/druntime/src/core/internal/convert.d For compatibility with versions of CMake that did not have the BYPRODUCTS option, CMake is generating phony rules for such files to convince 'ninja' to build. Project authors should add the missing BYPRODUCTS or OUTPUT options to the custom commands that produce these files. This warning is for project developers. Use -Wno-dev to suppress it. -- Build files have been written to: /mnt/3/usr/src/d/android/droid32 [1/40] Building C object CMakeFiles/phobos2-ldc.dir/phobos/etc/c/zlib/gzwrite.c.o FAILED: CMakeFiles/phobos2-ldc.dir/phobos/etc/c/zlib/gzwrite.c.o /usr/lib/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -DHAVE_UNISTD_H -fPIC -ffunction-sections -funwind-tables -fstack-protector-strong -Wno-invalid-command-line-argument -Wno-unused-command-line-argument -no-canonical-prefixes -g -DNDEBUG -DANDROID -D__ANDROID_API__=21 -Wa,--noexecstack -Wformat -Werror=format-security -fpie -target armv7-none-linux-androideabi21 -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -mthumb -Os -gcc-toolchain /usr/lib/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 --sysroot /usr/lib/android-ndk/sysroot -isystem /usr/lib/android-ndk/sysroot/usr/include/arm-linux-androideabi -fno-integrated-as -MD -MT CMakeFiles/phobos2-ldc.dir/phobos/etc/c/zlib/gzwrite.c.o -MF CMakeFiles/phobos2-ldc.dir/phobos/etc/c/zlib/gzwrite.c.o.d -o CMakeFiles/phobos2-ldc.dir/phobos/etc/c/zlib/gzwrite.c.o -c ldc-src/runtime/phobos/etc/c/zlib/gzwrite.c In file included from ldc-src/runtime/phobos/etc/c/zlib/gzwrite.c:6: ldc-src/runtime/phobos/etc/c/zlib/gzguts.h:21:10: fatal error: 'stdio.h' file not found #include ^ 1 error generated. [2/40] Building C object CMakeFiles/phobos2-ldc-debug.dir/phobos/etc/c/zlib/compress.c.o FAILED: CMakeFiles/phobos2-ldc-debug.dir/phobos/etc/c/zlib/compress.c.o /usr/lib/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -DHAVE_UNISTD_H -fPIC -ffunction-sections -funwind-tables -fstack-protector-strong -Wno-invalid-command-line-argument -Wno-unused-command-line-argument -no-canonical-prefixes -g -DNDEBUG -DANDROID -D__ANDROID_API__=21 -Wa,--noexecstack -Wformat -Werror=format-security -fpie -target armv7-none-linux-androideabi21 -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -mthumb -Os -gcc-toolchain /usr/lib/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 --sysroot /usr/lib/android-ndk/sysroot -isystem /usr/lib/android-ndk/sysroot/usr/include/arm-linux-androideabi -fno-integrated-as -MD -MT CMakeFiles/phobos2-ldc-debug.dir/phobos/etc/c/zlib/compress.c.o -MF CMakeFiles/phobos2-ldc-debug.dir/phobos/etc/c/zlib/compress.c.o.d -o CMakeFiles/phobos2-ldc-debug.dir/phobos/etc/c/zlib/compress.c.o -c ldc-src/runtime/phobos/etc/c/zlib/compress.c In file included from ldc-src/runtime/phobos/etc/c/zlib/compress.c:9: In file included from ldc-src/runtime/phobos/etc/c/zlib/zlib.h:34: ldc-src/runtime/phobos/etc/c/zlib/zconf.h:444:14: fatal error: 'sys/types.h' file not found #include /* for off_t */
Re: Need help with setting up LDC to cross-compile to Android/ARM
On Fri, Oct 19, 2018 at 08:54:25PM +, Joakim via Digitalmars-d wrote: [...] > Also, if you're using a system-provided LDC, it may not support Android, if > it wasn't built against our slightly tweaked llvm: > > https://github.com/ldc-developers/llvm > > In that case, use the LDC download from github instead: > > https://github.com/ldc-developers/ldc/releases I tried ldc-build-runtime with --ninja and it came back with a bunch of errors about "cortex-a8" being an unsupported target, and then segfaulted. So I'm going to try the "official" LDC release instead... In the meantime, is there a particular version of the NDK that I should use? Currently I have android-ndk-r13b-linux-x86_64.zip installed. Will it work? T -- I'm still trying to find a pun for "punishment"...
More zero-initialization optimizations pending in std.experimental.allocator?
Now that https://github.com/dlang/phobos/pull/6411 has been merged and DMD stable soon has the new __traits(isZeroInit, T) found here https://dlang.org/changelog/2.083.0.html#isZeroInit are there more zero-initializations that can be optimized in std.experimental.allocator?
Re: Need help with setting up LDC to cross-compile to Android/ARM
On Friday, 19 October 2018 at 20:50:36 UTC, Joakim wrote: On Wednesday, 17 October 2018 at 21:23:21 UTC, H. S. Teoh wrote: I'm trying to follow the instructions on this page: https://wiki.dlang.org/Build_D_for_Android [...] Hmm, that's weird: can you extract the full compiler command for that file? For example, if you use Ninja, by appending --ninja to ldc-build-runtime, it will tell you the full command that failed. Not sure if Make has a way to get that too. Also, if you're using a system-provided LDC, it may not support Android, if it wasn't built against our slightly tweaked llvm: https://github.com/ldc-developers/llvm In that case, use the LDC download from github instead: https://github.com/ldc-developers/ldc/releases
Re: Need help with setting up LDC to cross-compile to Android/ARM
On Wednesday, 17 October 2018 at 21:23:21 UTC, H. S. Teoh wrote: I'm trying to follow the instructions on this page: https://wiki.dlang.org/Build_D_for_Android [...] Hmm, that's weird: can you extract the full compiler command for that file? For example, if you use Ninja, by appending --ninja to ldc-build-runtime, it will tell you the full command that failed. Not sure if Make has a way to get that too.
Re: shared - i need it to be useful
On Friday, 19 October 2018 at 18:00:47 UTC, Atila Neves wrote: Because int or int* does not have threadsafe member functions. https://dlang.org/phobos/core_atomic.html Atomic and thread-safe are two very different concepts. Thread-safe is more of an ecosystem thing - if there are ways to do non-thread-safe operations on something, atomic operations are not enough to make things thread-safe. This is what core.atomic does: https://i.imgur.com/PnKMigl.jpg If you close the other openings, atomics are fantastic building blocks to making something thread-safe, but on their own they are not enough. -- Simen
Re: shared - i need it to be useful
On 20/10/2018 2:07 AM, Dominikus Dittes Scherkl wrote: This document provide no reasoning about what usecases it supports: It was a basic idea of mine... It was never meant to be PR'd.
Re: [OT] Android
On Fri, Oct 19, 2018 at 06:34:50PM +, Joakim via Digitalmars-d wrote: > On Thursday, 18 October 2018 at 19:37:24 UTC, H. S. Teoh wrote: [...] > > Eventually I resorted to generating Java code from D for some fo the > > most painful repetitive parts, and the way things are looking, I'm > > likely to be doing a lot more of that. I fear the way things are going > > will have be essentially writing a D to Java compiler at some point! > > Why not just use the Android port of D? I want to. But I couldn't get the cross-compiler setup properly: https://forum.dlang.org/post/mailman.4361.1539811552.29801.digitalmar...@puremagic.com If I can get past that hurdle, Java is going out the window pronto. :-D T -- Everybody talks about it, but nobody does anything about it! -- Mark Twain
Re: shared - i need it to be useful
On Thursday, 18 October 2018 at 21:24:53 UTC, jmh530 wrote: On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves wrote: [snip] Assuming this world... how do you use shared? https://github.com/atilaneves/fearless I had posted your library before to no response... I had two questions, if you'll indulge me. The first is perhaps more wrt automem. I noticed that I couldn't use automem's Unique with @safe currently. Is there any way to make it @safe, perhaps with dip1000? Yeah, I punted on making anything there @safe. I have to go back and fix it. At least I wrote Vector from scratch to be @safe. Second, Rust's borrow checker is that you can only have one mutable borrow. This is kind of like Exclusive, but it does it at compile-time, rather than with GC/RC. Is this something that can be incorporated into fearless? Well, Rust's version of Exclusive is Mutex, and that's pretty much always used with Arc. The closest we have to a borrow checker is DIP1000 and that's what fearless relies on.
[OT] Android
On Thursday, 18 October 2018 at 19:37:24 UTC, H. S. Teoh wrote: On Thu, Oct 18, 2018 at 07:09:42PM +, Patrick Schluter via Digitalmars-d wrote: [...] I often have the impression that a lot of things are going slower than necessary because a mentality where the perfect is in the way of good. That is indeed an all-too-frequent malady around these parts, sad to say. Which has the sad consequence that despite all efforts, there are still unfinished areas in D, and promises that haven't materialized in years (like multiple alias this). Still, the parts of D that are working well form a very powerful and comfortable-to-use language. Not quite the ideal we wish it to be, but IMO much closer than any other language I've seen yet. Recently I began dabbling in Android programming, and the one thing that keeps sticking out to me is how painful writing Java is. Almost every day of writing Java code has me wishing for this or that feature in D. Slices. Closures. Meta-programming. I found most of my time spent fighting with language limitations rather than make progress with the problem domain. Yes, this is why I began the Android port: I couldn't imagine writing Java. Eventually I resorted to generating Java code from D for some fo the most painful repetitive parts, and the way things are looking, I'm likely to be doing a lot more of that. I fear the way things are going will have be essentially writing a D to Java compiler at some point! Why not just use the Android port of D?
Re: Shared - Another Thread
On Fri, Oct 19, 2018 at 6:45 AM Dominikus Dittes Scherkl via Digitalmars-d wrote: > > On Thursday, 18 October 2018 at 16:24:39 UTC, Manu wrote: > > On Wednesday, 17 October 2018 at 22:56:26 UTC, H. S. Teoh wrote: > >> What cracks me up with Manu's proposal is that it is its > >> simplicity and lack of ambition that is criticized the most. > >> shared is a clusterfuck, according to what I gathered from the > >> forum, I never had yet to use it in my code. Manu's idea makes > >> it a little less of a clusterfuck, and people attack the idea > >> because it doesn't solve all and everything that's wrong with > >> shared. Funny. > >> > > > > Elaborate on this... It's clearly over-ambitious if anything. > > What issues am I failing to address? > > First of all, you called it "shared", but what your concept > describes is "theadsave". > If you had called it the later, it would have been clear to > everybody that thread local data is indeed automatically > threadsave, because only one thread has access to it (that > "implicit conversion"). But if something is "shared" (in the > common-world sense), it is of course no more "threadsave" - you > have to implement special methods to treat it. > > Conflating "shared" and "threadsave" in that manner was, I think, > the biggest mistake of your proposal. > > Another point is the part of "how can the compiler support the > expert in writing threadsave methods" - which you answered with > "not a little bit at the moment, but we may improve this in the > future" - and that is not at all satisfying. I think you've misunderstood. My proposal is @safe... if you stay @safe, you will receive guarantee that your code is threadsafe. If you want to implement a low-level device, you must implement a @trusted function, and I don't know what the compiler can do to help you. You will have to produce unsafe casts, so at least it will be completely clear every operation you perform that is unsafe. Users of your @trusted library though will experience @safe code upward through the stack. So saying that my response that "there is @trusted code at the bottom of the stack" is not satisfying is really just a comment on your opinion about @trusted code in general. My proposal is designed to be useful and @safe for *users* as primary goal. Authors that are composing low-level tools/libs will enjoy @safe guarantees. Only authors implementing ground-level machinery would need to write @trusted code... and that's the nature of the whole @safe stack. @safe can't practically exist without @trusted at the bottom of the stack. > Are there really no > ideas? I have some ideas, but I don't think they're practical to implement. Some cross-referencing of access would be required for the compiler to suspect foul-play. Functions would need to be analysed in conjunction; that sounds hard to implement. > No check that the proper atomic funtions are used or the > cast to "unshared" is ok at where it is used? The user has manually cast to unshared inside their unsafe/(@trusted?) function, what more signal do they need that they've engaged in an unsafe operation? > Even the expert > needs a little help to find the upcomming and well hidden bugs in > their oh so threadsave API... I think a lot of people probably over-estimate the number of tooling libs that live at the bottom of the stack. The list is fairly short: Atomic, Mutex/Semaphore, and a couple of lock-free-queue/list type structures. I don't know what other useful constructs exist... they will all be in libraries. Users would almost never come in contact with unsafe code, and again, that's the whole point of my design!
Re: shared - i need it to be useful
On Thursday, 18 October 2018 at 19:04:58 UTC, Erik van Velzen wrote: On Thursday, 18 October 2018 at 17:47:29 UTC, Stanislav Blinov wrote: On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves wrote: On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: Assuming the rules above: "can't read or write to members", and the understanding that `shared` methods are expected to have threadsafe implementations (because that's the whole point), what are the risks from allowing T* -> shared(T)* conversion? int i; tid.send(&i); ++i; // oops, data race Doesn't work. No matter what you show Manu or Simen here they think it's just a bad contrived example. You can't sway them by the fact that the compiler currently *prevents* this from happening. Manu said clearly that the receiving thread won't be able to read or write the pointer. Not directly - but obviously there must be *some* way to using it, in this case since it's an int with one of the core.atomic functions. At that point the spawned thread will be accessing it correctly, but the parent thread can modify the int in a non-atomic fashion. Because int or int* does not have threadsafe member functions. https://dlang.org/phobos/core_atomic.html
Re: shared - i need it to be useful
On Fri., 19 Oct. 2018, 6:10 am Dominikus Dittes Scherkl via Digitalmars-d, < digitalmars-d@puremagic.com> wrote: > On Friday, 19 October 2018 at 06:25:00 UTC, rikki cattermole > wrote: > > On 19/10/2018 7:09 PM, Norm wrote: > > > [0] > > https://github.com/rikkimax/DIPs/blob/shared/DIPs/DIP1xxx-RC2.md > > This document provide no reasoning about what usecases it > supports: > > Is it possible to create objects that are shared just for short > periods during their livetime and guarantee that they can be used > threadsave like Manu want it to be? > > Does it prohibit misuse any better than Manus proposal (that > requires the "Expert" to implement all theadsave API)? > No, a key misunderstanding. My proposal is @safe. The only thing an expert must do is write the few @trusted implementations that live at the very bottom of the stack. That would always be in a lib. When was the last time you rewrote std::map because you thought you could do better? The whole stack from there on up (the user stack) is safe, and you can have confidence in the @safe-ty. My goal is to make it safe, clearly communicate how a user interact with the API, and mechanically confirm that users do the right stuff. My proposal is specifically structured to not require *any* unsafe interactions at the user level. Only core machinery that is @trusted needs expert attention. I don't think it's possible to invent a proposal with a higher degree of verifiable safety.
Re: shared - i need it to be useful
On 10/18/18 9:09 PM, Manu wrote:
On Thu, Oct 18, 2018 at 5:30 PM Timon Gehr via Digitalmars-d
wrote:
On 18.10.18 23:34, Erik van Velzen wrote:
If you have an object which can be used in both a thread-safe and a
thread-unsafe way that's a bug or code smell.
Then why do you not just make all members shared? Because with Manu's
proposal, as soon as you have a shared method, all members effectively
become shared.
No they don't, only facets that overlap with the shared method.
I tried to present an example before:
struct Threadsafe
{
int x;
Atomic!int y;
void foo() shared { ++y; } // <- shared interaction only affects 'y'
void bar() { ++x; ++y; } // <- not threadsafe, but does not violate
foo's commitment; only interaction with 'y' has any commitment
associated with it
void unrelated() { ++x; } // <- no responsibilities are transposed
here, you can continue to do whatever you like throughout the class
where 'y' is not concerned
}
In practise, and in my direct experience, classes tend to have exactly
one 'y', and either zero (pure utility), or many such 'x' members.
Threadsafe API interacts with 'y', and the rest is just normal
thread-local methods which interact with all members thread-locally,
and may also interact with 'y' while not violating any threadsafety
commitments.
I promised I wouldn't respond, I'm going to break that (obviously).
But that's because after reading this description I ACTUALLY understand
what you are looking for.
I'm going to write a fuller post later, but I can't right now. But the
critical thing here is, you want a system where you can divvy up a type
into pieces you share and pieces you don't. But then you *don't* want to
have to share only the shared pieces. You want to share the whole thing
and be sure that it can't access your unshared pieces.
This critical requirement makes things a bit more interesting. For the
record, the most difficult thing to reaching this understanding was that
whenever I proposed anything, your answer was something like 'I just
can't work with that', and when I asked why, you said 'because it's
useless', etc. Fully explaining this point is very key to understanding
your thinking.
To be continued...
-Steve
Re: Shared - Another Thread
On Friday, 19 October 2018 at 13:40:54 UTC, Dominikus Dittes
Scherkl wrote:
On Thursday, 18 October 2018 at 16:24:39 UTC, Manu wrote:
On Wednesday, 17 October 2018 at 22:56:26 UTC, H. S. Teoh
wrote:
What cracks me up with Manu's proposal is that it is its
simplicity and lack of ambition that is criticized the most.
shared is a clusterfuck, according to what I gathered from
the forum, I never had yet to use it in my code. Manu's idea
makes it a little less of a clusterfuck, and people attack
the idea because it doesn't solve all and everything that's
wrong with shared. Funny.
Elaborate on this... It's clearly over-ambitious if anything.
What issues am I failing to address?
First of all, you called it "shared", but what your concept
describes is "theadsave".
If you had called it the later, it would have been clear to
everybody that thread local data is indeed automatically
threadsave, because only one thread has access to it (that
"implicit conversion"). But if something is "shared" (in the
common-world sense), it is of course no more "threadsave" - you
have to implement special methods to treat it.
Conflating "shared" and "threadsave" in that manner was, I
think, the biggest mistake of your proposal.
He talked about it in a previous thread, and generally I would
agree with him that such conflation is indeed beneficial provided
that some concessions are made for `shared`. Moreover, yet
another attribute? Please no...
struct X {
void foo(threadsafe const shared Bar* bar) @nogc @trusted
notrhow pure const shared threadsafe;
}
Attribute explosion is bad enough already.
Re: Shared - Another Thread
On Thursday, 18 October 2018 at 16:24:39 UTC, Manu wrote: On Wednesday, 17 October 2018 at 22:56:26 UTC, H. S. Teoh wrote: What cracks me up with Manu's proposal is that it is its simplicity and lack of ambition that is criticized the most. shared is a clusterfuck, according to what I gathered from the forum, I never had yet to use it in my code. Manu's idea makes it a little less of a clusterfuck, and people attack the idea because it doesn't solve all and everything that's wrong with shared. Funny. Elaborate on this... It's clearly over-ambitious if anything. What issues am I failing to address? First of all, you called it "shared", but what your concept describes is "theadsave". If you had called it the later, it would have been clear to everybody that thread local data is indeed automatically threadsave, because only one thread has access to it (that "implicit conversion"). But if something is "shared" (in the common-world sense), it is of course no more "threadsave" - you have to implement special methods to treat it. Conflating "shared" and "threadsave" in that manner was, I think, the biggest mistake of your proposal. Another point is the part of "how can the compiler support the expert in writing threadsave methods" - which you answered with "not a little bit at the moment, but we may improve this in the future" - and that is not at all satisfying. Are there really no ideas? No check that the proper atomic funtions are used or the cast to "unshared" is ok at where it is used? Even the expert needs a little help to find the upcomming and well hidden bugs in their oh so threadsave API...
Buildkite and the Project Tester
Hi, So for those of you who have contributed to D on GitHub in the last few months, you might have noticed the new Buildkite CI status checks. tl;dr: - it's the replacement for the Jenkins project tester (which has been deactivated ~ three months ago) - it allows us to add our own agents to the build fleet [1, 2] - it's intended to unify all CI systems on the long run (currently not possible due to a lack of build resources) So what are the other CIs doing? auto-tester: normal testing pipeline DAutotest: documentation testing (diff + preview) CircleCi: code coverage builds + style checks SemaphoreCI: self-bootstrapping (build dmd with LDC, GDC or a freshly built dmd) AppVeyor: 64-bit Windows builds + Visual Studio builds Travis: only used for tools, dlang.org and dub (test steps) CodeCov: code coverage diff (install the browser extension to see uncovered lines in the diff) Why can't I access the Buildkite build logs? Unfortunately Buildkite doesn't allow public access to their logs yet (though they are working on it). This is why I have just sent out a few more invitations. If I haven't gotten you, I'm sorry, but please send me a ping [3]. Otherwise you can also use the read-only dummy account below. Having an account has the advantage that you can restart builds, the dummy account is read-only. My build is failing and I think the failure isn't related to my changes --- Report this on #ci in Slack or https://github.com/dlang/ci The Auto-tester has its own GitHub repository: https://github.com/braddr/d-tester/issues Dlang-Bot has its own GitHub repository too: https://github.com/dlang-bots/dlang-bot I want my project to be tested on the Project Tester Your project's testsuite will be run on every PR which should prevent any regressions with a new DMD release. Open an issue (or a PR) here: https://github.com/dlang/ci I have resources I can spare for more testing agents Awesome! Talk to us in #ci or https://github.com/dlang/ci Dummy Account - user: du...@dlang.io password: dlangrocks Happy hacking! [1] https://buildkite.com/docs/agent/v3 [2] https://buildkite.com/docs/agent/v3/ubuntu [3] https://github.com/wilzbach
Re: shared - i need it to be useful
On Friday, 19 October 2018 at 06:25:00 UTC, rikki cattermole wrote: On 19/10/2018 7:09 PM, Norm wrote: [0] https://github.com/rikkimax/DIPs/blob/shared/DIPs/DIP1xxx-RC2.md This document provide no reasoning about what usecases it supports: Is it possible to create objects that are shared just for short periods during their livetime and guarantee that they can be used threadsave like Manu want it to be? Does it prohibit misuse any better than Manus proposal (that requires the "Expert" to implement all theadsave API)? Is the "normal" User still enforced to do some unsave casts? Has the normal User to have high knownledge of how a threadsave API is to be used or can the compiler provide any guarantees that using them can only fail if the implementation behind the API has bugs (e.g. provide some encapsulation)? Or any other usecases why and how this design is better than what we have now? And also some ideas how to implement some useacases (examples) are completely missing.
Re: Norwich 2018-11-07
On Wednesday, 17 October 2018 at 13:15:44 UTC, bachmeier wrote: I can definitely see that. I wanted to write a GUI program some time ago and looked at GtkD. It wasn't easy to see where to start with GtkD, and I eventually ended up running a local web server and creating the GUI in the browser. This is a good example of the non-technical things holding D back, largely because of the small community. Same here. I once started a GUI program using JavaFX + D (JNI) and then switched to GtkD + Glade. I could find my way around (with some difficulty), reading the docs / examples in C and translating them to GtkD in my head. It worked, but I always had the feeling of "Is that best practice?". When I re-opened the project after 2 years or so, I just couldn't be bothered to go through all that again. I just used my JavaFX code again and extended the app in no time. It also made me realize a lot of things about D and its "small community". Something is rotten in the state of D... PS No, I won't write GtkD docs myself (may the Universe forgive me!), because by the eternal DIY logic I would have had to implement D for ARM, write GtkD docs and whatnot...instead of using D to actually write programs.
Re: shared - i need it to be useful
On 10/17/2018 12:20 AM, Manu wrote: What does it mean 'aliased' precisely? Aliasing means there are two paths to the same piece of data. That could be two pointers pointing to the same data, or one pointer to a variable that is accessible by name. It doesn't really give us anything in practice that we don't have in C++. It provides a standard, enforced way to distinguish shared data from unshared data, and no way to bypass it in @safe code. There's no way to do that in C++.
Re: shared - i need it to be useful
On 19/10/2018 9:02 PM, Walter Bright wrote: On 10/17/2018 4:29 AM, jmh530 wrote: Isn't that also true for isolated data (data that only allows one alias)? That's colloquially called "unique" data. And yes, it is also true for that. That's why casting the return value of malloc() to 'shared' is safe. It's just that the language has no way to semantically identify unique data with its current type system. Actually we kind of have a way to do this (I think). Scope. It can't cross the thread boundary and it has a pretty clear owner as far as the stack is concerned. Given a bit of time and empowering of scope, we could do a @scopeonly type attribute for functions.
Re: shared - i need it to be useful
On 10/17/2018 4:29 AM, jmh530 wrote: Isn't that also true for isolated data (data that only allows one alias)? That's colloquially called "unique" data. And yes, it is also true for that. That's why casting the return value of malloc() to 'shared' is safe. It's just that the language has no way to semantically identify unique data with its current type system.
