Re: Initializing D runtime and executing module and TLS ctors for D libraries
Thank you very much for your answers. I think I've been on the right track and the following bug that I've mentioned has been messing up by hitting me randomly: https://issues.dlang.org/show_bug.cgi?id=11736 On 1/23/21 5:18 PM, IGotD- wrote: > During rt_init in the main thread, thread_attachThis is performed what I > have seen. That explains why everything just works on most cases. > Actual ctor code that runs for each TLS thread is language specific and > not part of the ELF standard therefore no such TLS ctor code are being > run in the lower level API. The initialization (only copy and zeroing) > of TLS data is being done when each thread starts. That must be the case for threads started by D runtime, right? It sounds like I must call rt_moduleTlsCtor explicitly for foreign threads. It's still not clear to me which modules' TLS variables are initialized (copied over). Only this module's or all modules that are in the program? I don't know whether it's possible to initialize one module; rt_moduleTlsCtor does not take any parameter. > This can even be done > in a lazy manner when the first TLS variable is being accessed. I hope that's the case. > I have to make a generic API and then a D > API on top of that. Did you mean a generic API, which makes calls to D? That's how I have it: an extern(C) API function calling proper D code. > In practice this means there is a trampoline > function involved where and thread_attachThis and thread_detachThis is > being called. Also this is where I call TLS ctors/dtors. That's what I will be doing. > It is an effect > that delegates is language specific and it falls natural that way. Avoid > extern(C) calls directly into D code. I hope I am misunderstanding you there. All I have are extern(C) function on the library API. > In practice you can do this for any thread even if there are several > delegates during the thread lifetime. You can simply have a TLS bool > variable telling if the thread_attachThis and rt_moduleTlsCtor have > already been run. I've already experimented with it but it didn't work likely because of the bug mentioned above. > In general the main thread that goes into main must also be the last one > returning the entire line of functions that was called during entry of > the process. Main entry belongs to another language, so I have to document that this library can only work in such "well behaved" cases. > What will happen is that you possibly do a > thread_detachThis twice. Sounds like I can track that with a bool variable as well, no? > Short answer is just park the main thread while the bulk is being done > by other threads. Unfortunately that's how many libraries work today. Agreed. That's for me to specify in the library documentation. I should revive my old PR and see whether it is needed at all: https://github.com/dlang/druntime/pull/1989 I am surprised how much I had learned at that time and how much I've already forgotten. :/ For example, my PR involves thread_setThis, which seems to be history now: https://docarchives.dlang.io/v2.068.0/phobos/core_thread.html#.thread_setThis And thread_detachThis seems to be missing now: https://dlang.org/phobos/core_thread.html https://dlang.org/phobos/core_thread_osthread.html Documentation issue or is it not needed anymore? Ali
Re: Deduct and return class type
On Saturday, 23 January 2021 at 19:19:29 UTC, Paul Backus wrote: On Saturday, 23 January 2021 at 07:17:38 UTC, Виталий Фадеев wrote: Where source ? Where deduction implementation ? Here: https://github.com/dlang/dmd/blob/v2.095.0/src/dmd/dtemplate.d#L1308 Thank!
Re: Nested functions and their linkage type
On Sunday, 24 January 2021 at 02:30:37 UTC, Mike Parker wrote: On Saturday, 23 January 2021 at 20:37:49 UTC, kdevel wrote: In § 19.18.7 [1] it is said that Nested functions always have the D function linkage type. Why and for what application is that important? I assume because of mangling. The outer function is mangled into the nested function's name. And also that nested functions needs to reference the stack of the enclosing function.
Re: Nested functions and their linkage type
On Saturday, 23 January 2021 at 20:37:49 UTC, kdevel wrote: In § 19.18.7 [1] it is said that Nested functions always have the D function linkage type. Why and for what application is that important? I assume because of mangling. The outer function is mangled into the nested function's name. shows from the object with D function linkage type named "nested" a delegate is created as soon as its address is taken. (function pointers are only generated from static nested functions) Yes. That's documented here: https://dlang.org/spec/function.html#closures
Re: How can I create a Standalone Bundle Portable file application using Dlang?
On Monday, 18 January 2021 at 19:42:22 UTC, Marcone wrote: How can I create a Standalone Bundle Portable file application using Dlang? What are the dependencies that you would like to merge into executable? dlls? resources?
Re: Initializing D runtime and executing module and TLS ctors for D libraries
On Sunday, 24 January 2021 at 00:24:55 UTC, Ali Çehreli wrote: One question I have is, does rt_init already do thread_attachThis? I ask because I have a library that is loaded by Python and things work even *without* calling thread_attachThis. During rt_init in the main thread, thread_attachThis is performed what I have seen. Another question: Are TLS ctors executed when I do loadLibrary? And when they are executed, which modules are involved? The module that is calling rt_moduleTlsCtor or all modules? What are "all modules"? The TLS standard (at least the ELF standard) does not have ctors. Only simple initialization are allowed meaning the initial data is stored as .tdata which is copied to the specific memory area for each thread. There is also a .tbss which is zero memory just like the .bss section. Actual ctor code that runs for each TLS thread is language specific and not part of the ELF standard therefore no such TLS ctor code are being run in the lower level API. The initialization (only copy and zeroing) of TLS data is being done when each thread starts. This can even be done in a lazy manner when the first TLS variable is being accessed. I have questions regarding thread_attachThis and thread_detachThis: When should they be called? Should the library expose a function that the users must call from *each thread* that they will be using? This may not be easy because a user may not know what thread they are running on. For example, the user of our library may be on a framework where threads may come and go, where the user may not have an opportunity to call thread_detachThis when a thread goes away. For example, the user may provide callback functions (which call us) to a framework that is running on a thread pool. I call thread_attachThis as soon the thread is supposed to call a D function. For example a callback from a thread in a thread pool. This usually happens when there is a function or delegate involved as any jump to D code would use them. I have to make a generic API and then a D API on top of that. In practice this means there is a trampoline function involved where and thread_attachThis and thread_detachThis is being called. Also this is where I call TLS ctors/dtors. It is an effect that delegates is language specific and it falls natural that way. Avoid extern(C) calls directly into D code. In practice you can do this for any thread even if there are several delegates during the thread lifetime. You can simply have a TLS bool variable telling if the thread_attachThis and rt_moduleTlsCtor have already been run. More questions: Can I thread_detachThis the thread that called rt_init? Can I call rt_moduleTlsCtor more than once? I guess it depends on each module. It will be troubling if a TLS ctor reinitializes an module state. :/ I have brought up this question before because like it is right now I haven't seen any "rt_uninit" or "rt_close" function. This is bit limiting for me as the main thread can exit while the process lives on. In general the main thread that goes into main must also be the last one returning the entire line of functions that was called during entry of the process. What will happen is that you possibly do a thread_detachThis twice. Short answer is just park the main thread while the bulk is being done by other threads. Unfortunately that's how many libraries work today.
Initializing D runtime and executing module and TLS ctors for D libraries
tl;dr I know enough to sense there are important stuff that I don't know.
Even though I sometimes act[1] like someone who knows stuff, there are
many fuzzy areas for me especially in the runtime.
Things work great when D code is inside a D program. The runtime and
module states are magically initialized and everything works. It is not
clear when it comes to writing a D library and especially when that
library may be used by other language runtimes, necessarily on foreign
threads.
Here are the essential points that I do and don't understand.
- Initialize the runtime: This is automatically done for a D program as
described on the wiki[2]. This must be done by calling rt_init[3] for a
D shared library. I handle this by calling rt_init from a
pragma(crt_constructor) function[4]. Luckily, this is easy and works for
all cases that I have.
- Execute module constructors ("ctor" for short, i.e. 'shared static
this' blocks). This is done automatically for a D program and when the D
library is loaded by other language code like C++ and Python. However,
I've encountered a case[5] where module ctors were not being called.
This could be due to runtime bugs or something that I don't understand
with loading shared libraries. (My workaround is very involved: I grep
the output of 'nm' to determine the symbol for the module ctor, call it
after dlsym'ing, and because 'nm | grep' is a slow process, I cache this
information in a file along with the ~2K libraries that I may load
conditionally.)
- Loading D libraries from D code: I call loadLibrary[6] to load a D
library so that "[its] D runtime [...] will be integrated with the
current runtime". Sounds promising; assuming that rt_init is already
called for the calling library, I assume loadLibrary will handle
everything, and all code will use a single runtime and things will work
fine. This works flawlessly for my D and C++ programs that load my D
library that loads the other D libraries.
- Attaching foreign threads: D runtime needs to know about all threads
that are running D code so that it will know what threads consist of
"the world" for it to "stop the world" when performing garbage
collection. The function to do this is thread_attachThis[7].
One question I have is, does rt_init already do thread_attachThis? I ask
because I have a library that is loaded by Python and things work even
*without* calling thread_attachThis.
- Execute thread local storage (TLS) ctors: Again, this happens
automatically for most cases. However, thread_attachThis says "[if] full
functionality as a D thread is desired, [rt_moduleTlsCtor] must be
called after thread_attachThis". Ok. When would I not want "full
functionality" anyway?
Another question: Are TLS ctors executed when I do loadLibrary?
And when they are executed, which modules are involved? The module that
is calling rt_moduleTlsCtor or all modules? What are "all modules"?
- Detaching foreign threads: Probably even more important than
thread_attachThis is thread_detachThis[8]. As its documentation says,
one should call rt_moduleTlsDtor as well for "full functionality".
This is very important because when the GC collection kick in, it will
stop all threads that makes up its world. If one of those threads has
already been terminated, we will crash. (Related, I have an abandoned
PR[9] that tried to fix issues with thread_detachThis, which stalled due
to failing unit tests for the 32-bit Apple operating system, which D
stopped supporting since then.) (And I stopped working on that issue
mostly because the company I used to work for stopped using D and
rewrote their library in C++.)
I have questions regarding thread_attachThis and thread_detachThis: When
should they be called? Should the library expose a function that the
users must call from *each thread* that they will be using? This may not
be easy because a user may not know what thread they are running on. For
example, the user of our library may be on a framework where threads may
come and go, where the user may not have an opportunity to call
thread_detachThis when a thread goes away. For example, the user may
provide callback functions (which call us) to a framework that is
running on a thread pool.
For that reason, my belief has been to call thread_attachThis upon
entering an API function and calling thread_detachThis upon leaving it
because I may not know whether this thread will survive or die soon.
(thread_detachThis is so important because the next GC cycle will try to
stop this thread and may crash.)
More questions: Can I thread_detachThis the thread that called rt_init?
Can I call rt_moduleTlsCtor more than once? I guess it depends on each
module. It will be troubling if a TLS ctor reinitializes an module state. :/
While trying to sort all of these out, I am facing a bug[10], which will
force me to move away from std.parallelism and perhaps use
std.concurrency. Even though that bug is report
Re: How can I create a Standalone Bundle Portable file application using Dlang?
On 1/20/21 6:50 AM, Marcone wrote: On Tuesday, 19 January 2021 at 14:20:06 UTC, Imperatorn wrote: On Tuesday, 19 January 2021 at 11:10:25 UTC, Marcone wrote: On Tuesday, 19 January 2021 at 06:25:31 UTC, Imperatorn wrote: On Monday, 18 January 2021 at 19:42:22 UTC, Marcone wrote: How can I create a Standalone Bundle Portable file application using Dlang? Could you describe what you mean with "Bundle portable file application"? All dependencies inside an exe file. Like Python Pyinstaller. Do you with "dependencies" mean "resources"? In that case, yeah import is an option someone mentioned. I do not mean resources .res, except if is possible use files inside resources without copy to hard disc and make accessible as it is in local path. I am afraid we are not speaking the same language. Because it sounds like you may not be using "dependencies" as it is conventionally understood in most programming communities, you'll need to give examples. For the record, dependencies are typically either compile-time dependencies or run-time dependencies, and in both cases I think the commonest example would be a library.
Nested functions and their linkage type
In § 19.18.7 [1] it is said that
Nested functions always have the D function linkage type.
Why and for what application is that important? As this example
~~~znested.d
void foo () { }
unittest {
void nested () { }
import std.stdio;
writeln (typeof (nested).stringof);
writeln (typeof (&nested).stringof);
writeln (typeof (foo).stringof);
writeln (typeof (&foo).stringof);
}
~~~
$ dmd -unittest -checkaction=context -main -run znested.d
pure nothrow @nogc @safe void()
void delegate() pure nothrow @nogc @safe
void()
void function()
1 unittests passed
shows from the object with D function linkage type named "nested"
a delegate is created as soon as its address is taken. (function
pointers are only generated from static nested functions)
[1] https://dlang.org/spec/function.html#nested
Re: Deduct and return class type
On Saturday, 23 January 2021 at 07:17:38 UTC, Виталий Фадеев wrote: Where source ? Where deduction implementation ? Here: https://github.com/dlang/dmd/blob/v2.095.0/src/dmd/dtemplate.d#L1308
Re: std.algorithm.splitter on a string not always bidirectional
On 1/22/21 2:13 PM, Jon Degenhardt wrote:
On Friday, 22 January 2021 at 17:29:08 UTC, Steven Schveighoffer wrote:
On 1/22/21 11:57 AM, Jon Degenhardt wrote:
I think the idea is that if a construct like 'xyz.splitter(args)'
produces a range with the sequence of elements {"a", "bc", "def"},
then 'xyz.splitter(args).back' should produce "def". But, if finding
the split points starting from the back results in something like
{"f", "de", "abc"} then that relationship hasn't held, and the
results are unexpected.
But that is possible with all 3 splitter variants. Why is one allowed
to be bidirectional and the others are not?
I'm not defending it, just explaining what I believe the thinking was
based on the examination I did. It wasn't just looking at the code,
there was a discussion somewhere. A forum discussion, PR discussion, bug
or code comments. Something somewhere, but I don't remember exactly.
However, to answer your question - The relationship described is
guaranteed if the basis for the split is a single element. If the range
is a string, that's a single 'char'. If the range is composed of
integers, then a single integer. Note that if the basis for the split is
itself a range, then the relationship described is not guaranteed.
Personally, I can see a good argument that bidirectionality should not
be supported in any of these cases, and instead force the user to choose
between eager splitting or reversing the range via retro. For the common
case of strings, the further argument could be made that the distinction
between char and dchar is another point of inconsistency.
I would not want that. My use case is splitting a string on punctuation,
and using the lazy result for testing equality of something. But I have
some special suffix items that I want to handle first (and pop off).
dchar/char inconsistency isn't a problem, because they are both dchar
ranges (and both are bidirectional).
Regardless whether the choices made were the best choices, there was
some thinking that went into it, and it is worth understanding the
thinking when considering changes.
I believe there was that thinking. It's why I posted, because before I
filed a bug, I wanted to make sure there wasn't a good reason.
It looks like there is NOT a good reason for the single-item based
splitting as you say to prevent bidirectional access. But there IS a
good reason (thanks for the example H.S. Teoh) to prevent it for
multi-element delimiters.
-Steve
Re: std.algorithm.splitter on a string not always bidirectional
On 1/22/21 2:56 PM, H. S. Teoh wrote:
On Friday, 22 January 2021 at 17:29:08 UTC, Steven Schveighoffer wrote:
On 1/22/21 11:57 AM, Jon Degenhardt wrote:
[...]
Another way to look at it: If split (eager) took a predicate, that
'xyz.splitter(args).back' and 'xyz.split(args).back' should produce
the same result. But they will not with the example given.
With what example given? The example you gave is incomplete (what are
args?)
[...]
Here is a case for which iterating forwards yields a different sequence
from iterating backwards (if we were to allow the latter):
"bbcbcba".splitter("bcb")
Iterating forwards gives us the subranges: "b", "cba".
Iterating backwards gives us: "a", "bbc".
So it cannot be a bidirectional range, at least not in the expected sense
that iterating from the back ought to give us the same subranges as
iterating from the front, only in a reverse order. Here iterating
backwards yields a completely different decomposition.
Yes thank you! That makes sense, and I wasn't thinking of that.
I still believe that any splitter based on individual elements should be
bidirectional.
-Steve
Re: which free operating systems have a gtkd package?
On Sat, Jan 23, 2021 at 7:15 AM dan via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: > ... > So debian 10 and ubuntu 20.4 are candidates, but i'm wondering if > there are others. (I tried to find gtkd on linux mint but did > not see a package for it, but man i could sure be wrong.) > > Thanks in advance for any info! > > dan > Anything archlinux based (Manjaro, Arch linux, EndeavourOS, RebornOS, Artix ...), Alpine linux, Fedora and many others. https://pkgs.org/search/?q=gtkd
