Re: Problem with coupling shared object symbol visibility with protection
On Monday, 16 February 2015 at 20:19:27 UTC, Jacob Carlborg wrote: On 2015-02-16 10:40, Benjamin Thaut wrote: This is in fact not a breaking change because export is broken anyway. Due to bug 922 export can't be used in cross platform libraries. I haven't seen a single library on dub that uses export (most likely for exactly that reason). Most likely due to that the support for dynamic libraries is fairly new only works on Linux (?). And the way it currently works on Linux `export` is completely ignored at all - hard to break something that has never worked.
Re: Problem with coupling shared object symbol visibility with protection
Hi, I also want to say a big thank you to all of you involved in this topic and especially to Benjamin. Proper DLL handling in D I would really appreciate. I think this topic is the break through. Kind regards André On Tuesday, 17 February 2015 at 18:03:06 UTC, Benjamin Thaut wrote: So i looked at the Dll Test within the dmd test framework and when I make export an attribute, like I suggested, this perticular test will compile & run without any code changes. This is another reason why I suspect that making export an attribute will only break very little if any code at all. Am 16.02.2015 um 09:08 schrieb Walter Bright: At this point I suggest simply making those private helper functions public and export them. It gets your project moving without waiting for language changes (and this is a breaking change). I have to big fears with doing this 1) I finish everything up any finally do the pull request. Then the reviewers will realize that using export in all required places will completely undermine D's module level protection system and reject the PR. Then all my work is in vain. 2) Even if 1 does not happen, that means from now on the broken export will be used in _actual_ code. Because it kind of works. Going forward this would mean that changing export into an attribute will break actual production code. So in my opinion we have to implement Dll Support on windows and fix export simulatiously, otherwise fixing export will lead to big breaking changes instead of a few small ones. Kind Regards Benjamin Thaut
Re: Problem with coupling shared object symbol visibility with protection
So i looked at the Dll Test within the dmd test framework and when I make export an attribute, like I suggested, this perticular test will compile & run without any code changes. This is another reason why I suspect that making export an attribute will only break very little if any code at all. Am 16.02.2015 um 09:08 schrieb Walter Bright: At this point I suggest simply making those private helper functions public and export them. It gets your project moving without waiting for language changes (and this is a breaking change). I have to big fears with doing this 1) I finish everything up any finally do the pull request. Then the reviewers will realize that using export in all required places will completely undermine D's module level protection system and reject the PR. Then all my work is in vain. 2) Even if 1 does not happen, that means from now on the broken export will be used in _actual_ code. Because it kind of works. Going forward this would mean that changing export into an attribute will break actual production code. So in my opinion we have to implement Dll Support on windows and fix export simulatiously, otherwise fixing export will lead to big breaking changes instead of a few small ones. Kind Regards Benjamin Thaut
Re: Problem with coupling shared object symbol visibility with protection
On Monday, 16 February 2015 at 21:41:00 UTC, Walter Bright wrote: On 2/16/2015 12:19 PM, Jacob Carlborg wrote: Most likely due to that the support for dynamic libraries is fairly new only works on Linux (?). Part of the test suite creates a DLL using D for Windows. Martin recently added shared library support for FreeBSD also: https://github.com/D-Programming-Language/druntime/pull/1068
Re: Problem with coupling shared object symbol visibility with protection
On 2/16/2015 12:19 PM, Jacob Carlborg wrote: Most likely due to that the support for dynamic libraries is fairly new only works on Linux (?). Part of the test suite creates a DLL using D for Windows.
Re: Problem with coupling shared object symbol visibility with protection
On 2015-02-16 10:40, Benjamin Thaut wrote: This is in fact not a breaking change because export is broken anyway. Due to bug 922 export can't be used in cross platform libraries. I haven't seen a single library on dub that uses export (most likely for exactly that reason). Most likely due to that the support for dynamic libraries is fairly new only works on Linux (?). -- /Jacob Carlborg
Re: Problem with coupling shared object symbol visibility with protection
On Monday, 16 February 2015 at 09:59:07 UTC, Walter Bright wrote: -- Here is a list of all things wrong with export: 32 & 64 bit issues: 1) Exporting a global variable leads to a linker error 2) Exporting thread local variables should be an error (at least it is in c++) 3) The module info should be exported as soon the module has any exported symbols 4) __gshared members of a class are not exported 5) The TypeInfo Object of the TestClass is not exported 6) The TypeInfo Object of TestStruct is not exported - None of these are addressed by making export an attribute. I never said that. I said that making export an attribute is _not_ an breaking change because it is already broken up to a point where it currently can't be used anyway (especially because of 922). My current implementation fixes both 922 and 9816 but would greatly benefit from making export an attribute because otherwise D's entire protection system would be undermined like described multiple times throughout this thread.
Re: Problem with coupling shared object symbol visibility with protection
On 2/16/2015 1:40 AM, Benjamin Thaut wrote: On Monday, 16 February 2015 at 08:08:17 UTC, Walter Bright wrote: At this point I suggest simply making those private helper functions public and export them. It gets your project moving without waiting for language changes (and this is a breaking change). This is in fact not a breaking change because export is broken anyway. Due to bug 922 export can't be used in cross platform libraries. I haven't seen a single library on dub that uses export (most likely for exactly that reason). Also export on windows is broken as well, see Bug 9816. So making export an attribute would most likely not break anything because it already is broken and wouldn't work if you tried to use it. From https://issues.dlang.org/show_bug.cgi?id=9816: -- Here is a list of all things wrong with export: 32 & 64 bit issues: 1) Exporting a global variable leads to a linker error 2) Exporting thread local variables should be an error (at least it is in c++) 3) The module info should be exported as soon the module has any exported symbols 4) __gshared members of a class are not exported 5) The TypeInfo Object of the TestClass is not exported 6) The TypeInfo Object of TestStruct is not exported - None of these are addressed by making export an attribute.
Re: Problem with coupling shared object symbol visibility with protection
On Monday, 16 February 2015 at 08:08:17 UTC, Walter Bright wrote: At this point I suggest simply making those private helper functions public and export them. It gets your project moving without waiting for language changes (and this is a breaking change). This is in fact not a breaking change because export is broken anyway. Due to bug 922 export can't be used in cross platform libraries. I haven't seen a single library on dub that uses export (most likely for exactly that reason). Also export on windows is broken as well, see Bug 9816. So making export an attribute would most likely not break anything because it already is broken and wouldn't work if you tried to use it.
Re: Problem with coupling shared object symbol visibility with protection
On Monday, 16 February 2015 at 08:08:17 UTC, Walter Bright wrote: On 1/31/2015 11:52 AM, Benjamin Thaut wrote: @Walter: Making export a attribute seems to be the preferred choice in this discussion. Additionaly this was the result of the last discussion around export, a year ago, although for different reasons. The last Discussion resulted in DIP 45 which also proposes making export an attribute. Before I start with the implementation, would you be ok with making export an attribute or would you veto it? At this point I suggest simply making those private helper functions public and export them. It gets your project moving without waiting for language changes (and this is a breaking change). This is not about some private project but about fixing language itself.
Re: Problem with coupling shared object symbol visibility with protection
On 1/31/2015 11:52 AM, Benjamin Thaut wrote: @Walter: Making export a attribute seems to be the preferred choice in this discussion. Additionaly this was the result of the last discussion around export, a year ago, although for different reasons. The last Discussion resulted in DIP 45 which also proposes making export an attribute. Before I start with the implementation, would you be ok with making export an attribute or would you veto it? At this point I suggest simply making those private helper functions public and export them. It gets your project moving without waiting for language changes (and this is a breaking change).
Re: Problem with coupling shared object symbol visibility with protection
On Sunday, 1 February 2015 at 09:24:46 UTC, Benjamin Thaut wrote: So you'd want bar to be duplicated on both sides ? This is gonna cause problems with di files. No. He wants the compiler to automatically detect that the template foo might call bar. As a result the compiler should export bar when compiling the shared library so that any user of foo does not run into a "unresolved symbol reference" linker error. Bar would still only exist once: within the shared library. Correct. Sadly it won't work that way as I was already pointed at. Otherwise it is perfect solution :P
Re: Problem with coupling shared object symbol visibility with protection
Am 31.01.2015 um 23:42 schrieb deadalnix:
On Wednesday, 28 January 2015 at 13:48:45 UTC, Dicebot wrote:
Isn't that what your first proposed solution is about? That was my
understanding and I liked that understanding :) To be 100% clear :
export void foo(T : int)(T x)
{
bar(x);
}
private void bar(int x) { }
I'd expect `bar` to be exported into resulting binary so that it can
be linked against by any users of `foo` but for any attempt to call
`bar` directly from D code to fail because of protection violation. If
someone wants to circumvent protection by forging mangling - shooting
own feet is allowed.
So you'd want bar to be duplicated on both sides ? This is gonna cause
problems with di files.
No. He wants the compiler to automatically detect that the template foo
might call bar. As a result the compiler should export bar when
compiling the shared library so that any user of foo does not run into a
"unresolved symbol reference" linker error. Bar would still only exist
once: within the shared library.
Re: Problem with coupling shared object symbol visibility with protection
On Wednesday, 28 January 2015 at 13:48:45 UTC, Dicebot wrote:
Isn't that what your first proposed solution is about? That was
my understanding and I liked that understanding :) To be 100%
clear :
export void foo(T : int)(T x)
{
bar(x);
}
private void bar(int x) { }
I'd expect `bar` to be exported into resulting binary so that
it can be linked against by any users of `foo` but for any
attempt to call `bar` directly from D code to fail because of
protection violation. If someone wants to circumvent protection
by forging mangling - shooting own feet is allowed.
So you'd want bar to be duplicated on both sides ? This is gonna
cause problems with di files.
Re: Problem with coupling shared object symbol visibility with protection
Am 31.01.2015 um 13:07 schrieb Martin Nowak: That's probably how it should behave, though an attribute applying only to public members unless explicitly added is unprecedented. Still seems like the right choice here, but might require some additional compiler logic. Well you don't have to implement it that way with in the compiler. The only thing that matters is, that the users sees this transitive behavior of export. It is not neccessary that the attribute actually gets applied recursivly within the compiler implementation. Only the export behavior needs to be implemented recursivly and I already did a implementation for that. It currently works for the export protection level and would be trivial to adapt for a export attribute. It would also not require any additional logic for applying attributes recursivly under certain conditions. @Walter: Making export a attribute seems to be the preferred choice in this discussion. Additionaly this was the result of the last discussion around export, a year ago, although for different reasons. The last Discussion resulted in DIP 45 which also proposes making export an attribute. Before I start with the implementation, would you be ok with making export an attribute or would you veto it? Kind Regards Benjamin Thaut
Re: Problem with coupling shared object symbol visibility with protection
On Saturday, 31 January 2015 at 09:25:10 UTC, Benjamin Thaut
wrote:
Well, export is going to remain transitive. So the first
approach is still going to work. The only difference is going
to be that you can "force export" private declarations. So for
most modules it is hopefully going to be enough to put "export
{ }" around the public part of the module and force export
some of the needed private declarations. For a module without
templates a single "export { }" should be enough.
That's probably how it should behave, though an attribute
applying only to public members unless explicitly added is
unprecedented. Still seems like the right choice here, but might
require some additional compiler logic.
Re: Problem with coupling shared object symbol visibility with protection
Am 31.01.2015 um 06:11 schrieb Dicebot:
On Friday, 30 January 2015 at 19:10:06 UTC, Martin Nowak wrote:
It has a serious drawback of increasing attribute noise even more
though. First approach allows for more automatic inference.
But with D restrictions it seems the least bad option.
Well, export is going to remain transitive. So the first approach is
still going to work. The only difference is going to be that you can
"force export" private declarations. So for most modules it is hopefully
going to be enough to put "export { }" around the public part of the
module and force export some of the needed private declarations. For a
module without templates a single "export { }" should be enough.
Re: Problem with coupling shared object symbol visibility with protection
On Friday, 30 January 2015 at 19:10:06 UTC, Martin Nowak wrote:
On Tuesday, 20 January 2015 at 12:23:32 UTC, Benjamin Thaut
wrote:
2) Make export an attribute. If export is no longer an
protection level but instead an attribute this issue can
easily be solved by doing.
export public void templateFunc(T)()
{
someHelperFunc();
}
export private void someHelperFunc()
{
}
Clearly the better solution.
Export and protection have something in common but are not
identical.
It has a serious drawback of increasing attribute noise even more
though. First approach allows for more automatic inference.
But with D restrictions it seems the least bad option.
Re: Problem with coupling shared object symbol visibility with protection
On Tuesday, 20 January 2015 at 12:23:32 UTC, Benjamin Thaut wrote:
2) Make export an attribute. If export is no longer an
protection level but instead an attribute this issue can easily
be solved by doing.
export public void templateFunc(T)()
{
someHelperFunc();
}
export private void someHelperFunc()
{
}
Clearly the better solution.
Export and protection have something in common but are not
identical.
Re: Problem with coupling shared object symbol visibility with protection
Am 30.01.2015 um 11:39 schrieb Martin Nowak:
If you mean float, then it
will instatiate the template when compiled individually and use b's
instantiation when b and c are compiled together.
If this is true, then please explain this behavior:
module a; // --> a.dll
struct Storage(T)
{
T var;
__gshared T s_var;
}
module b; // --> b.dll
import a;
export __gshared Storage!int g_var1;
module c; // --> c.exe
import a;
import b;
__gshared Storage!int g_var2;
void main(string[] args)
{
g_var1.var = 2;
g_var2.var = 3;
g_var1.s_var = 2;
g_var2.s_var = 3;
}
c.obj : error LNK2019: Verweis auf nicht aufgelöstes externes Symbol
"_D1a14__T7StorageTiZ7Storage5s_vari" in Funktion "_Dmain".
If your statement would be true module c should not use the instance
from module b because they are not compiled together. But as you can
clearly see from the linker error message module c does not instanciate
the template, because if module c would instanciate the template there
would not be a unresolved symbol.
Compiling module c with "-allinst" solves the problem. Putting "export"
in front of struct Storage(T) also solves the problem. And thats exactly
the reason why it is neccessary to be able to export templates. (Where
export means, export all template instanciations)
Re: Problem with coupling shared object symbol visibility with protection
On Thursday, 29 January 2015 at 13:24:36 UTC, Benjamin Thaut wrote: module c: SomeTemplate!uint var3; // will this use instaction from b? Or instanciate itself? That's the first instantiation with uint. If you mean float, then it will instatiate the template when compiled individually and use b's instantiation when b and c are compiled together.
Re: Problem with coupling shared object symbol visibility with protection
On Thursday, 29 January 2015 at 10:21:25 UTC, Walter Bright wrote:
On 1/28/2015 5:19 AM, Benjamin Thaut wrote:
On Wednesday, 28 January 2015 at 11:01:09 UTC, Walter Bright
wrote:
The example had marked the template itself as 'export'. This
raises the
specter of which binary the template instantiation exists in.
The export in this context actually means "export all
instanciations of this
template". And this is needed to avoid using -allinst
everywhere.
The problem is what happens when the client side instantiates
the template, which it must in order to use it.
Well if there already is a statically known instanciation it will
not instanciate it. (I didn't change that behvaior). The question
is what happens when you have something like this:
module a:
struct SomeTemplate(T){}
alias knownInstance = SomeTemplate!int;
module b:
SomeTemplate!int var1; // will use instanciation from a (unless
-allinst)
SomeTemplate!float var2; // will instanciate
alias knownInstance2 = SomeTemplate!uint;
module c:
SomeTemplate!uint var3; // will this use instaction from b? Or
instanciate itself?
I don't know enough about D's template implementation to answer
the question regarding c.var3. Depending on the answer to this
question I can answer what should happen if a export marked
template is instanciated outside of its module. (e.g. by the user)
Please also correct me if any of the above assumptions are
incorrect.
Re: Problem with coupling shared object symbol visibility with protection
On 1/28/2015 5:27 AM, Benjamin Thaut wrote: Also sorry for the harsh answer, this was a classical double misunderstanding. No problemo.
Re: Problem with coupling shared object symbol visibility with protection
On 1/28/2015 5:19 AM, Benjamin Thaut wrote: On Wednesday, 28 January 2015 at 11:01:09 UTC, Walter Bright wrote: The example had marked the template itself as 'export'. This raises the specter of which binary the template instantiation exists in. The export in this context actually means "export all instanciations of this template". And this is needed to avoid using -allinst everywhere. The problem is what happens when the client side instantiates the template, which it must in order to use it.
Re: Problem with coupling shared object symbol visibility with protection
On Wednesday, 28 January 2015 at 15:01:30 UTC, Benjamin Thaut wrote: With that in mind second option is starting to look more attractive despite the grammar change - forcing good chunk template API into structs does not sound very convenient :( Well and then there is the third option Walter proposed: - Make everything export even if it means that it gets callable by the user directly. It is hardly a good option - I consider LTO/WPO a very important future goal for any native language.
Re: Problem with coupling shared object symbol visibility with protection
On Wednesday, 28 January 2015 at 14:31:54 UTC, Dicebot wrote: Yes, I see the problem now. "static if" isn't even the worst offender, any kind of string mixins that generate the call to `bar` are impossible to detect without having actual `foo` instance. Sorry for misinterpretation. With that in mind second option is starting to look more attractive despite the grammar change - forcing good chunk template API into structs does not sound very convenient :( Well and then there is the third option Walter proposed: - Make everything export even if it means that it gets callable by the user directly.
Re: Problem with coupling shared object symbol visibility with protection
On Wednesday, 28 January 2015 at 14:16:03 UTC, Benjamin Thaut wrote: Well this would be ultimate goal. Although it is not possible to automatically detect that bar needs to be exported because that would mean you would have to analyze all possible instantiations of the template foo. (static if...) So Automatically detecting that bar needs to be exported is not possible. We either have to make export into an attribute or use the solution where bar is actually nested into a struct which is exported instead. This is also described in my initial post with examples. Yes, I see the problem now. "static if" isn't even the worst offender, any kind of string mixins that generate the call to `bar` are impossible to detect without having actual `foo` instance. Sorry for misinterpretation. With that in mind second option is starting to look more attractive despite the grammar change - forcing good chunk template API into structs does not sound very convenient :(
Re: Problem with coupling shared object symbol visibility with protection
On Wednesday, 28 January 2015 at 13:48:45 UTC, Dicebot wrote:
Isn't that what your first proposed solution is about? That was
my understanding and I liked that understanding :) To be 100%
clear :
export void foo(T : int)(T x)
{
bar(x);
}
private void bar(int x) { }
I'd expect `bar` to be exported into resulting binary so that
it can be linked against by any users of `foo` but for any
attempt to call `bar` directly from D code to fail because of
protection violation. If someone wants to circumvent protection
by forging mangling - shooting own feet is allowed.
Well this would be ultimate goal. Although it is not possible to
automatically detect that bar needs to be exported because that
would mean you would have to analyze all possible instantiations
of the template foo. (static if...)
So Automatically detecting that bar needs to be exported is not
possible. We either have to make export into an attribute or use
the solution where bar is actually nested into a struct which is
exported instead. This is also described in my initial post with
examples.
Re: Problem with coupling shared object symbol visibility with protection
On Wednesday, 28 January 2015 at 13:30:17 UTC, Benjamin Thaut
wrote:
On Wednesday, 28 January 2015 at 11:42:19 UTC, Dicebot wrote:
2) first proposed solution does not allow to mark private
functions as "export". It implicitly exports those if they are
needed for actual public/export template function to work.
This is not the same - those functions still can't be called
via provide .di binding, it simply keeps them available for
linking.
So you would prefer to keep symbols private and just make them
available for linking? Or what exactly is the message here?
Isn't that what your first proposed solution is about? That was
my understanding and I liked that understanding :) To be 100%
clear :
export void foo(T : int)(T x)
{
bar(x);
}
private void bar(int x) { }
I'd expect `bar` to be exported into resulting binary so that it
can be linked against by any users of `foo` but for any attempt
to call `bar` directly from D code to fail because of protection
violation. If someone wants to circumvent protection by forging
mangling - shooting own feet is allowed.
Re: Problem with coupling shared object symbol visibility with protection
On Wednesday, 28 January 2015 at 11:42:19 UTC, Dicebot wrote: 2) first proposed solution does not allow to mark private functions as "export". It implicitly exports those if they are needed for actual public/export template function to work. This is not the same - those functions still can't be called via provide .di binding, it simply keeps them available for linking. So you would prefer to keep symbols private and just make them available for linking? Or what exactly is the message here? 3) there is a big maintenance benefit from encouraging people to explicitly mark their API, template or not. It is a clear sign "this is a part of my libraries you are expected to use directly" and that was what my idea for attribute inference abused. Agree here. Especially if you want to keep your shared library backwards compatible, so that people can simply drop in the new binary. To do that you need very fine control over what is exported and what is not.
Re: Problem with coupling shared object symbol visibility with protection
On Wednesday, 28 January 2015 at 11:01:09 UTC, Walter Bright wrote: The example had marked the template itself as 'export'. This raises the specter of which binary the template instantiation exists in. Also sorry for the harsh answer, this was a classical double misunderstanding.
Re: Problem with coupling shared object symbol visibility with protection
On Wednesday, 28 January 2015 at 11:01:09 UTC, Walter Bright wrote: The example had marked the template itself as 'export'. This raises the specter of which binary the template instantiation exists in. The export in this context actually means "export all instanciations of this template". And this is needed to avoid using -allinst everywhere.
Re: Problem with coupling shared object symbol visibility with protection
My 5 cents: 1) "export all" approach is extremely limiting and I'd like to see it go on Linux too. One of major problems with it is that many forms of link-time optimizations (such as --gc-sections) become simply impossible when compiler can't know what symbols are actually supposed to be available externally. 2) first proposed solution does not allow to mark private functions as "export". It implicitly exports those if they are needed for actual public/export template function to work. This is not the same - those functions still can't be called via provide .di binding, it simply keeps them available for linking. 3) there is a big maintenance benefit from encouraging people to explicitly mark their API, template or not. It is a clear sign "this is a part of my libraries you are expected to use directly" and that was what my idea for attribute inference abused.
Re: Problem with coupling shared object symbol visibility with protection
On 1/27/2015 2:05 PM, Rainer Schuetze wrote:
On 26.01.2015 23:24, Walter Bright wrote:
The problem here is that you don't want to make someHelperFunc()
export because that would mean users could call it directly, but
you want it to be available for cross shared library calls. The
cross shared library call happens if a template is instanced from a
different shared library / executable than the module it was
originally located in.
exporting a template and then having the user instantiate outside of
the library doesn't make a whole lot of sense, because the
instantiation won't be there in the library. The library will have to
instantiate every use case. If the compiler knows the library
instantiated it, it won't re-instantiate it locally.
The problem is not about into which binary the template is generated to (this
must be the binary where it is used),
The example had marked the template itself as 'export'. This raises the specter
of which binary the template instantiation exists in.
but how to access private non-templated methods called by the template.
From the core.time.FracSec example:
export struct FracSec
{
///...
static FracSec from(string units)(long value)
if(units == "msecs" ||
units == "usecs" ||
units == "hnsecs" ||
units == "nsecs")
{
immutable hnsecs = cast(int)convert!(units, "hnsecs")(value);
_enforceValid(hnsecs);
return FracSec(hnsecs);
}
private static void _enforceValid(int hnsecs)
{
if(!_valid(hnsecs))
throw new TimeException("FracSec must ...");
}
///...
}
_enforceValid() could also be a free function. It is likely to be compiled into
druntime.dll, but needs to be exported from the DLL to be callable by the
instantiation of the template function in another DLL. The "private" forbids
exporting, though.
I tend to view a DLL's exports as being inherently not private, hence D's export
design being a "super" public. At the risk of sounding flip, I suggest simply
removing the 'private' from free functions one wishes to export. If the user is
calling undocumented functions that start with '_', we can presume they know
what they're doing.
If you still want to hide the free function, it can be done like this:
struct Bar(T) {
void callit() { Impl.freefunc(); }
}
private struct Impl {
export static void freefunc() { }
}
Re: Problem with coupling shared object symbol visibility with protection
On Tuesday, 27 January 2015 at 22:29:41 UTC, Rainer Schuetze wrote: I would not mind if we export all symbols on Windows aswell. It doesn't seem to bother a lot of people for the linux version, even though it's unsafer and slower than on Windows (at least that was my experience more than 10 years ago). It might get us a first working version without adding "export" throughout the druntime/phobos source code. There are multiple reasons why I don't want to simply export every symbol: 1) Before I started this implementation I synchronized with Martin Nowak regrading hish plans for D shared libraries. It turns out that he wants to annotate all of druntime and phobos with export asap and use it to control symbol visibility on linux. He wants to get away from the "export everything" on linux because it hurts performance and prevents some optimizations. 2) Every data symbol that is considered for exporting adds a slight performance overhead through a additional indirection, even for static builds. That is because the compiler can't know if the symbol is imported or not, as export means both import and export at the same time. So if the compiler simply assumes that all symbols are exported this would add a lot of unnecessary overhead even in static builds. (but also in dynamic ones) 3) If we start with export everything on Windows now, it will be hared to go back to export only whats annotated. I don't have a clear favorite, but the second version makes it clearer that visibility and protection are separate issues. A note on: > export public void templateFunc(T)() I don't think it is well defined what exporting a template is supposed to mean. My guess: whenever an instance of the template is created, its symbols are exported. This could make for a lot of duplicate symbols across multiple DLLs, though. Obviously its not yet well defined. But we can define it. And you are right, it means that all instances are exported. And we need that behavior because otherwise you have to spray in -allinst everywhere. Believe me I tried. Maybe there should be a method of explicitly exporting/importing a template instance from another DLL, e.g. export alias symbol = templateFunc!int; I would rather not do that. You don't have to explicitly import template instances from static libraries either. We should try to keep the behavior of static and dynamic libraries as similar as possible. The ideal situation would be that you can simply compile something that was a static library into a dynamic one without doing any code changes (other then writing export: at the beginning of every file)
Re: Problem with coupling shared object symbol visibility with protection
On 20.01.2015 13:23, Benjamin Thaut wrote:
I'm currently working on Windows DLL support which has stronger rules
than linux shared objects for which symbols actually get exported from a
shared library. But as we want to replicate the same behaviour on linux
using symbol visibility (e.g. gcc 4 -fVisibility=hidden) this issue also
applies to linux once implemented.
Currently export means two things:
- the symbol is publicy accessible (same as public)
- the symbol will be accisble across shared library boundaries
This has the following issue:
export void templateFunc(T)()
{
someHelperFunc();
}
private void someHelperFunc()
{
}
And you don't even have to go into phobos to hit this problem. It is
already in druntime see core.time.FracSec._enforceValid
This works with the current linux shared objects because they simply
export all symbols. But once only symbols with export get exported this
breaks.
I would not mind if we export all symbols on Windows aswell. It doesn't
seem to bother a lot of people for the linux version, even though it's
unsafer and slower than on Windows (at least that was my experience more
than 10 years ago). It might get us a first working version without
adding "export" throughout the druntime/phobos source code.
The problem here is that you don't want to make someHelperFunc() export
because that would mean users could call it directly, but you want it to
be available for cross shared library calls. The cross shared library
call happens if a template is instanced from a different shared library
/ executable than the module it was originally located in.
There are two solutions for this.
1) Given that export is transitive (that means if a struct or class is
declared export every member that is _not_ private will be accessible
across shared library boundaries. This behaviour is required to make the
export protection level work on windows)
You can now do the following:
export struct SomeStruct
{
static public void templateFunc(T)()
{
someHelperFunc();
}
static package void someHelperFunc()
{
}
}
Because of the transitivity someHelperFunc will be exported but still
not be callable by the user directly. This can be used to fix the issue
in core.time but may not be so well suited if you want the template to
be on module level instead of nesting it into a struct.
2) Make export an attribute. If export is no longer an protection level
but instead an attribute this issue can easily be solved by doing.
export public void templateFunc(T)()
{
someHelperFunc();
}
export private void someHelperFunc()
{
}
But this would require grammar changes. Which are generally avoided if
possible.
I don't have a clear favorite, but the second version makes it clearer
that visibility and protection are separate issues.
A note on:
> export public void templateFunc(T)()
I don't think it is well defined what exporting a template is supposed
to mean. My guess: whenever an instance of the template is created, its
symbols are exported. This could make for a lot of duplicate symbols
across multiple DLLs, though. Maybe there should be a method of
explicitly exporting/importing a template instance from another DLL, e.g.
export alias symbol = templateFunc!int;
Please note, that the problem raised above by Benjamin applies just as
well to non-exported templates.
Re: Problem with coupling shared object symbol visibility with protection
On 26.01.2015 23:24, Walter Bright wrote:
The problem here is that you don't want to make someHelperFunc()
export because that would mean users could call it directly, but
you want it to be available for cross shared library calls. The
cross shared library call happens if a template is instanced from a
different shared library / executable than the module it was
originally located in.
exporting a template and then having the user instantiate outside of
the library doesn't make a whole lot of sense, because the
instantiation won't be there in the library. The library will have to
instantiate every use case. If the compiler knows the library
instantiated it, it won't re-instantiate it locally.
The problem is not about into which binary the template is generated to
(this must be the binary where it is used), but how to access private
non-templated methods called by the template.
From the core.time.FracSec example:
export struct FracSec
{
///...
static FracSec from(string units)(long value)
if(units == "msecs" ||
units == "usecs" ||
units == "hnsecs" ||
units == "nsecs")
{
immutable hnsecs = cast(int)convert!(units, "hnsecs")(value);
_enforceValid(hnsecs);
return FracSec(hnsecs);
}
private static void _enforceValid(int hnsecs)
{
if(!_valid(hnsecs))
throw new TimeException("FracSec must ...");
}
///...
}
_enforceValid() could also be a free function. It is likely to be
compiled into druntime.dll, but needs to be exported from the DLL to be
callable by the instantiation of the template function in another DLL.
The "private" forbids exporting, though.
Re: Problem with coupling shared object symbol visibility with protection
Am 26.01.2015 um 23:24 schrieb Walter Bright: exporting a template and then having the user instantiate outside of the library doesn't make a whole lot of sense, because the instantiation won't be there in the library. The library will have to instantiate every use case. If the compiler knows the library instantiated it, it won't re-instantiate it locally. Sorry, but wtf? So we just throw all of phobos away? Given this argument you can _never_ make phobos into a shared lirary becuse you can't possibliy pre-instaniate all possible template permutations in phobos. Your suggestion is completely un-pratical. Even C++ allows you to instanciate templates provided by a shared library. I'd be thinking that what a shared library exports is fixed, and expecting the user to make more instantiations makes for a fairly unresolvable issue. The solution is design the templates to be expanded only on one side of the dll/exe boundary, not straddle it. Again this just makes it impossible to make phobos ever work as a shared library. A Language as template heavy as D should clearly allow users to instanciate templates across dll/exe boundaries. Why am I getting the impression, everytime I read one of your comments regarding dlls on windows, that you don't want them to be useable? DO you have a special hate against Windows or their shared library solution? Kind Regards Benjamin Thaut
Re: Problem with coupling shared object symbol visibility with protection
On 1/20/2015 4:23 AM, Benjamin Thaut wrote:
I'm currently working on Windows DLL support which has stronger rules than linux
shared objects for which symbols actually get exported from a shared library.
But as we want to replicate the same behaviour on linux using symbol visibility
(e.g. gcc 4 -fVisibility=hidden) this issue also applies to linux once
implemented.
Currently export means two things:
- the symbol is publicy accessible (same as public)
- the symbol will be accisble across shared library boundaries
This has the following issue:
export void templateFunc(T)()
{
someHelperFunc();
}
private void someHelperFunc()
{
}
And you don't even have to go into phobos to hit this problem. It is already in
druntime see core.time.FracSec._enforceValid
This works with the current linux shared objects because they simply export all
symbols. But once only symbols with export get exported this breaks.
The problem here is that you don't want to make someHelperFunc() export because
that would mean users could call it directly, but you want it to be available
for cross shared library calls. The cross shared library call happens if a
template is instanced from a different shared library / executable than the
module it was originally located in.
exporting a template and then having the user instantiate outside of the library
doesn't make a whole lot of sense, because the instantiation won't be there in
the library. The library will have to instantiate every use case. If the
compiler knows the library instantiated it, it won't re-instantiate it locally.
There are two solutions for this.
1) Given that export is transitive (that means if a struct or class is declared
export every member that is _not_ private will be accessible across shared
library boundaries. This behaviour is required to make the export protection
level work on windows)
You can now do the following:
export struct SomeStruct
{
static public void templateFunc(T)()
{
someHelperFunc();
}
static package void someHelperFunc()
{
}
}
Because of the transitivity someHelperFunc will be exported but still not be
callable by the user directly. This can be used to fix the issue in core.time
but may not be so well suited if you want the template to be on module level
instead of nesting it into a struct.
2) Make export an attribute. If export is no longer an protection level but
instead an attribute this issue can easily be solved by doing.
export public void templateFunc(T)()
{
someHelperFunc();
}
export private void someHelperFunc()
{
}
But this would require grammar changes. Which are generally avoided if possible.
There would be a third option, which I rather avoid. Doing a
"pramga(forceExport)" or something like that.
My implementation, which I ran into this issue with, currently usses approach 1.
What do you think how this sould be solved?
I'd be thinking that what a shared library exports is fixed, and expecting the
user to make more instantiations makes for a fairly unresolvable issue. The
solution is design the templates to be expanded only on one side of the dll/exe
boundary, not straddle it.
Re: Problem with coupling shared object symbol visibility with protection
There are uses in Phobos where workaround 1) would require some code
changes:
private @property File trustedStdout() @trusted { return stdout; }
void write(T...)(T args) if (!is(T[0] : File))
{
trustedStdout.write(args);
}
My workaround so far:
export struct _impl1
{
package @property static File trustedStdout() @trusted { return stdout; }
}
alias trustedStdout = _impl1.trustedStdout;
Re: Problem with coupling shared object symbol visibility with protection
Thanks for keeping to poke this issue - symbol visibility is
currently a big undefined minefield in D ABI.
Your welcome. At this point I'm so desperate for D Dll support
that I stopped poking and started implementing it myself. I'm 3
unresolved symbol references away from actually building phobos
into a dll (druntime already is).
By the way you could also export non accessible symbols like this:
static public void templateFunc(T)()
{
Impl.someHelperFunc();
}
export struct Impl
{
package:
static void someHelperFunc()
{
}
}
Re: Problem with coupling shared object symbol visibility with protection
On Tuesday, 20 January 2015 at 12:23:32 UTC, Benjamin Thaut wrote:
I'm currently working on Windows DLL support which has stronger
rules than linux shared objects for which symbols actually get
exported from a shared library. But as we want to replicate the
same behaviour on linux using symbol visibility (e.g. gcc 4
-fVisibility=hidden) this issue also applies to linux once
implemented.
Currently export means two things:
- the symbol is publicy accessible (same as public)
- the symbol will be accisble across shared library boundaries
This has the following issue:
export void templateFunc(T)()
{
someHelperFunc();
}
private void someHelperFunc()
{
}
And you don't even have to go into phobos to hit this problem.
It is already in druntime see core.time.FracSec._enforceValid
This works with the current linux shared objects because they
simply export all symbols. But once only symbols with export
get exported this breaks.
The problem here is that you don't want to make
someHelperFunc() export because that would mean users could
call it directly, but you want it to be available for cross
shared library calls. The cross shared library call happens if
a template is instanced from a different shared library /
executable than the module it was originally located in.
There are two solutions for this.
1) Given that export is transitive (that means if a struct or
class is declared export every member that is _not_ private
will be accessible across shared library boundaries. This
behaviour is required to make the export protection level work
on windows)
You can now do the following:
export struct SomeStruct
{
static public void templateFunc(T)()
{
someHelperFunc();
}
static package void someHelperFunc()
{
}
}
Because of the transitivity someHelperFunc will be exported but
still not be callable by the user directly. This can be used to
fix the issue in core.time but may not be so well suited if you
want the template to be on module level instead of nesting it
into a struct.
2) Make export an attribute. If export is no longer an
protection level but instead an attribute this issue can easily
be solved by doing.
export public void templateFunc(T)()
{
someHelperFunc();
}
export private void someHelperFunc()
{
}
But this would require grammar changes. Which are generally
avoided if possible.
There would be a third option, which I rather avoid. Doing a
"pramga(forceExport)" or something like that.
My implementation, which I ran into this issue with, currently
usses approach 1. What do you think how this sould be solved?
Walter: What was the general idea behind export when you
designed it, and how can it be used to solve this problem?
Kind Regards
Benjamin Thaut
Just as heads up in case D ever comes to Aix, I don't know how it
looks like nowadays, but Aix back in 2000 used to be have similar
behavior to Windows.
The .def files in Windows were .exp (I think) files on Aix.
--
Paulo
Re: Problem with coupling shared object symbol visibility with protection
I like the first version more as it fits better the natural way people design their interfaces and reduces attributed noise. Can't say if it will trigger some hidden issues. First version also fits better some of ideas I had about automated API generation/verification (http://forum.dlang.org/post/otejdbgnhmyvbyaxa...@forum.dlang.org) Thanks for keeping to poke this issue - symbol visibility is currently a big undefined minefield in D ABI.
Problem with coupling shared object symbol visibility with protection
I'm currently working on Windows DLL support which has stronger rules
than linux shared objects for which symbols actually get exported from a
shared library. But as we want to replicate the same behaviour on linux
using symbol visibility (e.g. gcc 4 -fVisibility=hidden) this issue also
applies to linux once implemented.
Currently export means two things:
- the symbol is publicy accessible (same as public)
- the symbol will be accisble across shared library boundaries
This has the following issue:
export void templateFunc(T)()
{
someHelperFunc();
}
private void someHelperFunc()
{
}
And you don't even have to go into phobos to hit this problem. It is
already in druntime see core.time.FracSec._enforceValid
This works with the current linux shared objects because they simply
export all symbols. But once only symbols with export get exported this
breaks.
The problem here is that you don't want to make someHelperFunc() export
because that would mean users could call it directly, but you want it to
be available for cross shared library calls. The cross shared library
call happens if a template is instanced from a different shared library
/ executable than the module it was originally located in.
There are two solutions for this.
1) Given that export is transitive (that means if a struct or class is
declared export every member that is _not_ private will be accessible
across shared library boundaries. This behaviour is required to make the
export protection level work on windows)
You can now do the following:
export struct SomeStruct
{
static public void templateFunc(T)()
{
someHelperFunc();
}
static package void someHelperFunc()
{
}
}
Because of the transitivity someHelperFunc will be exported but still
not be callable by the user directly. This can be used to fix the issue
in core.time but may not be so well suited if you want the template to
be on module level instead of nesting it into a struct.
2) Make export an attribute. If export is no longer an protection level
but instead an attribute this issue can easily be solved by doing.
export public void templateFunc(T)()
{
someHelperFunc();
}
export private void someHelperFunc()
{
}
But this would require grammar changes. Which are generally avoided if
possible.
There would be a third option, which I rather avoid. Doing a
"pramga(forceExport)" or something like that.
My implementation, which I ran into this issue with, currently usses
approach 1. What do you think how this sould be solved?
Walter: What was the general idea behind export when you designed it,
and how can it be used to solve this problem?
Kind Regards
Benjamin Thaut
