Re: Building several dynamic libraries with one shared GC
On 9/12/21 1:25 PM, NonNull wrote: On Sunday, 12 September 2021 at 18:56:50 UTC, Ali Çehreli wrote: All initialization functions of the plugins were called automatically in my D test environment and all plugins were usable. The trouble started when the main library was being used in a foreign environment (something like Python loading Python loading C++ library loading our D library): Although the initialization function of the main library was being called, the 'shared static this' functions of the plugins were not being called. So here, your main dynamic library in turn dynamically loads plugins. Did you try simply calling a function exported by a plugin from the static constructor in the main library after it had made the call to initialize druntime to see if that stimulated running the plugin's static constructors first? The problem you linked to suggests that might do the job. I haven't run into this problem yet myself. But I'm interested. I did not try it because I learned about that potential cause after we worked around the issue with the daemons. Ali
Re: Building several dynamic libraries with one shared GC
On Sunday, 12 September 2021 at 18:56:50 UTC, Ali Çehreli wrote: All initialization functions of the plugins were called automatically in my D test environment and all plugins were usable. The trouble started when the main library was being used in a foreign environment (something like Python loading Python loading C++ library loading our D library): Although the initialization function of the main library was being called, the 'shared static this' functions of the plugins were not being called. So here, your main dynamic library in turn dynamically loads plugins. Did you try simply calling a function exported by a plugin from the static constructor in the main library after it had made the call to initialize druntime to see if that stimulated running the plugin's static constructors first? The problem you linked to suggests that might do the job. I haven't run into this problem yet myself. But I'm interested.
Re: Building several dynamic libraries with one shared GC
On Sunday, 12 September 2021 at 18:56:50 UTC, Ali Çehreli wrote: All initialization functions of the plugins were called automatically in my D test environment and all plugins were usable. The trouble started when the main library was being used in a foreign environment (something like Python loading Python loading C++ library loading our D library): Although the initialization function of the main library was being called, the 'shared static this' functions of the plugins were not being called. (I tried dlopen after guessing intelligently the name of the 'shared static this' function (not obvious); it was not satisfactory and I don't remember exactly why not.) Later I learned, this could be because merely loading a plugin might not be enough, and perhaps the main library might have to use a feature of the library as well: https://forum.dlang.org/post/sdb5jb$2rk3$1...@digitalmars.com [...] > If several plugins are built by different third parties, each dynamic > library will have its own GC and copy of druntime right now. Like the user 'frame', I don't think that's the case. I hope you're right about this last. Not sure how static versus dynamic linking affects it. ldc2 seems to default to dynamic linking for phobos and have druntime in a dynamic library too. [Not clear what DMD does with druntime when it dynamically links to phobos.] In that ideal situation you seem to be right. Not sure how a plugin can be distributed as an executable only (to those without a D compiler installed) without static linking and then what? I have a mess to sort out. Any info or suggestions?
Re: Building several dynamic libraries with one shared GC
On 9/12/21 7:31 AM, NonNull wrote: > I am making a plug-in development system for a high performance Linux > application that already exists and is written in C and will not be > modified for this purpose. I've done something similar but in my case the main application is in D and the plugins are code-generated D (hand edited by non-D developers). However, as is common in software, the requirements changed :) and we wanted to use it as a library as well; so, plugins became part of a main D library (which exposed C functions). All initialization functions of the plugins were called automatically in my D test environment and all plugins were usable. The trouble started when the main library was being used in a foreign environment (something like Python loading Python loading C++ library loading our D library): Although the initialization function of the main library was being called, the 'shared static this' functions of the plugins were not being called. (I tried dlopen after guessing intelligently the name of the 'shared static this' function (not obvious); it was not satisfactory and I don't remember exactly why not.) Later I learned, this could be because merely loading a plugin might not be enough, and perhaps the main library might have to use a feature of the library as well: https://forum.dlang.org/post/sdb5jb$2rk3$1...@digitalmars.com If that link indeed explains the issue, I did not know about it when we worked around the initialization problem by running a D daemon per thread behind this main library. Each thread of the library passes information to its daemon through shared memory and the daeman does it's thing and returns the result back. (The difference is, the plugins are linked to the daemon; avoiding the problem initialization scenario; 'shared static this' are called from a D environment.) > If several plugins are built by different third parties, each dynamic > library will have its own GC and copy of druntime right now. Like the user 'frame', I don't think that's the case. Going off topic, I started thinking about this "one daemon per library thread" idea: becaues I am under the impression that running multiple daemons does not put much stress on the system under Linux, this idea can automatically translate to "independent GCs for each thread". For example, the main library has 10 threads using 10 separate daemons on their backgrounds; each daemon handles its own GC needs without even knowing about the other daemons that are effectively working for the same main library. I haven't experimented with this yet. Ali
Re: Building several dynamic libraries with one shared GC
On Sunday, 12 September 2021 at 16:23:13 UTC, frame wrote: Shouldn't the runtime not already be shared on Linux? The `Runtime.loadLibrary` specs say `If the library contains a D runtime it will be integrated with the current runtime.` This should be true for the GC too. At least the memory is shared because as I remember I could access __gshared variables, whereas on Windows nothing like this works and any DLL will spawn a new thread (for each thread you use too). The plugin libraries expose a C API and are dynamically loaded by the application which is written in C, so presumably using dlopen. No D runtime there. Still your reply does suggest a way to proceed...
Re: Building several dynamic libraries with one shared GC
On Sunday, 12 September 2021 at 14:31:04 UTC, NonNull wrote: If several plugins are built by different third parties, each dynamic library will have its own GC and copy of druntime right now. How can I organize that there is one separate dynamic library to share these among all plugins? Shouldn't the runtime not already be shared on Linux? The `Runtime.loadLibrary` specs say `If the library contains a D runtime it will be integrated with the current runtime.` This should be true for the GC too. At least the memory is shared because as I remember I could access __gshared variables, whereas on Windows nothing like this works and any DLL will spawn a new thread (for each thread you use too).
