Re: std.reflection prototype
On Thursday, 28 September 2017 at 01:33:35 UTC, bitwise wrote:
On Wednesday, 27 September 2017 at 20:38:51 UTC, Gheorghe
Gabriel wrote:
On Monday, 30 March 2015 at 01:11:55 UTC, bitwise wrote:
[...]
Hi, your link is not working any more and I really need your
implementation.
Gabriel
I took the code down because there were design flaws I had to
work out. I've reworked the code significantly at this point. I
can throw a copy up on github in a day or two if you need it.
There are still some issues to overcome though, which may or
may not be a problem depending on what you're doing.
Thank you, I really need it for my project.
I want to load scripts in a 3D scene without compiling an
restarting the whole engine.
example:
module game1.player;
class Player : Actor {
this(string name) {
super(name)
}
~this() {}
void jump() { ... }
@Editable("Slider")("min = 0; max = 5;")
ushort lives = 5;
}
In this moment I want to load this script and create a scene
object.
I should load Player class, then all its public methods, then its
public members and check theire attributes.
Re: std.reflection prototype
On Wednesday, 27 September 2017 at 20:38:51 UTC, Gheorghe Gabriel wrote: On Monday, 30 March 2015 at 01:11:55 UTC, bitwise wrote: [...] Hi, your link is not working any more and I really need your implementation. Gabriel I took the code down because there were design flaws I had to work out. I've reworked the code significantly at this point. I can throw a copy up on github in a day or two if you need it. There are still some issues to overcome though, which may or may not be a problem depending on what you're doing.
Re: std.reflection prototype
On Monday, 30 March 2015 at 01:11:55 UTC, bitwise wrote:
I came across this post a while back and decided to implement
it:
http://forum.dlang.org/thread/juf7sk$16rl$1...@digitalmars.com
My implementation:
https://github.com/bitwise-github/D-Reflection
The above conversation seemed to stop abruptly, so I went on to
assume that no one chose to champion the task.
At the time, I looked around for other conversations or
attempts at runtime reflection for D, but couldn't really find
anything. I did find the ModuleInfo/reflection stuff in
object.d, but it seemed like an effort that may have been
abandoned. Also, the above conversation seemed to suggest it
should be opt-in, which also made me wonder if the stuff in
object.d was abandoned or for a different purpose.
Looking again today, someone seems to have been working on it a
bit.. For example, MemberInfo_field and MemberInfo_function
were empty last time I checked.
So what's the current state of things?
Is anybody working on it?
Although MemberInfo_field exists, it doesn't seem to be
available from TypeInfo_Class... Is that the eventual goal?
Is there anything I can do to help get things moving?
Any comments on my implementation would be welcome as well.
(https://github.com/bitwise-github/D-Reflection)
main.d shows some general use cases, but basically, if the
reflect(T) template is used on a class,
that class, and any child types will be reflected. Recursive
reflection only propagates downward, or else it could leak
sideways and unnecessarily reflect several modules.
Most of the reflection information is available at compile
time. For example:
enum name =
reflectModule!(test).findClass("Test").findField("variable").name;
pragma(msg, name); // "variable" will be outputted.
To make a module available for runtime reflection, the
following can be used:
mixin(runtimeReflection!test);
At this point, the above example can be rewritten as:
string name =
getModuleReflection("tests.test").findClass("Test3").findField("static_variable").name;
writeln(name);
Hi, your link is not working any more and I really need your
implementation.
Gabriel
Re: std.reflection prototype
On Fri, 17 Apr 2015 05:30:04 -0400, Jacob Carlborg wrote:
On 2015-04-17 01:25, bitwise wrote:
Ok, that sounds right. D has no multiple or virtual inheritance, so I
guess things should be fine. C# is the same way(single inheritance,
interfaces) which is likely designed to avoid these kinds of issues.
I would be modifying the offTi() property though, not getMembers().
getMembers() actually works right now, although it's kinda useless.
Basically, if you put any function named "getMembers" at module scope,
the address of that function can be retrieved using
ModuleInfo.xgetMembers().
We need a "getMembers" on ClassInfo. Because "typeid" will resolve the
dynamic type. The problem is this code:
class Bar {}
class Foo : Bar {}
void main ()
{
Bar a = new Foo;
writeln(typeid(a));
}
Compile time reflection cannot be used to get the members in this case.
Sorry, that's what I meant(using this on classes, not modules).
xgetMembers/getMembers seems to be designed so that the caller can write
their own getMembers() function, whereas offTi() is designed to return an
array that was automatically generated.
offTi() is actually defined in TypeInfo:
https://github.com/D-Programming-Language/druntime/blob/cfcf7480b2faea0af9ab6ddba8e3b0d9f05c4415/src/object_.d#L301
and overridden for TypeInfo_Class:
https://github.com/D-Programming-Language/druntime/blob/cfcf7480b2faea0af9ab6ddba8e3b0d9f05c4415/src/object_.d#L831
but TypeInfo_Class.m_offTi is never populated:
https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L436
So while this code technically works, there will be no output:
class Test {
int a = 4;
int b = 5;
int c = 6;
}
void main()
{
foreach(const(OffsetTypeInfo) off; typeid(Test).offTi())
writeln(off.offset);
}
Re: std.reflection prototype
On 2015-04-17 01:25, bitwise wrote:
Ok, that sounds right. D has no multiple or virtual inheritance, so I
guess things should be fine. C# is the same way(single inheritance,
interfaces) which is likely designed to avoid these kinds of issues.
I would be modifying the offTi() property though, not getMembers().
getMembers() actually works right now, although it's kinda useless.
Basically, if you put any function named "getMembers" at module scope,
the address of that function can be retrieved using
ModuleInfo.xgetMembers().
We need a "getMembers" on ClassInfo. Because "typeid" will resolve the
dynamic type. The problem is this code:
class Bar {}
class Foo : Bar {}
void main ()
{
Bar a = new Foo;
writeln(typeid(a));
}
Compile time reflection cannot be used to get the members in this case.
--
/Jacob Carlborg
Re: std.reflection prototype
On Thu, 16 Apr 2015 02:22:25 -0400, Jacob Carlborg wrote:
In D I would assume this would eventually be possible:
class Foo { int a; }
class Bar : Foo { int b; }
Foo f = new Bar;
auto offset = typeid(f).getMemeber("b").offset;
auto ptr = cast(int*)(cast(void*)f + offset);
*ptr = 4;
Ok, that sounds right. D has no multiple or virtual inheritance, so I
guess things should be fine. C# is the same way(single inheritance,
interfaces) which is likely designed to avoid these kinds of issues.
I would be modifying the offTi() property though, not getMembers().
getMembers() actually works right now, although it's kinda useless.
Basically, if you put any function named "getMembers" at module scope, the
address of that function can be retrieved using ModuleInfo.xgetMembers().
///
module mainMod;
template isField(alias T) {
enum hasInit = is(typeof(typeof(T).init));
enum isManifestConst = __traits(compiles, { enum e = T; });
enum isField = hasInit && !isManifestConst;
}
MemberInfo[] getMembers()
{
mixin("alias " ~ .stringof[7..$] ~ " mod;");
MemberInfo[] members;
foreach(memberName; __traits(allMembers, mod))
{
alias Identity!(__traits(getMember, mod, memberName)) member;
static if(isField!member)
members ~= new MemberInfo_field(memberName, typeid(member), 0);
}
return members;
}
int a = 1;
int b = 2;
int c = 3;
void main()
{
foreach(mi; ModuleInfo)
{
if(mi.name == "mainModule")
{
alias MemberInfo[] function() getMembers_type;
getMembers_type getMembers =
cast(getMembers_type)mi.xgetMembers();
foreach(m; getMembers())
writeln(m.name);
}
}
}
Output:
a
b
c
offTi() however, even though it's documented, returns an empty array. I'm
not sure what the motivation originally was for having an array of
TypeInfo and offsets without the field names, but the dead code predates
D2. I suppose you could do binary serialization...
Anyways, I've put the reflection library to rest for now. Hopefully I'll
have some time in the next little while to try and get offTi() working.
Re: std.reflection prototype
On 2015-04-16 01:32, bitwise wrote:
One reason is that casting with multiple inheritance offsets the
pointer, and I forget exactly virtual inheritance works, but I'm sure it
breaks things too..
In D I would assume this would eventually be possible:
class Foo { int a; }
class Bar : Foo { int b; }
Foo f = new Bar;
auto offset = typeid(f).getMemeber("b").offset;
auto ptr = cast(int*)(cast(void*)f + offset);
*ptr = 4;
--
/Jacob Carlborg
Re: std.reflection prototype
One more thing.
I had to add a virtual destructor to A to get this to work:
class A {
public: int a; virtual ~A(){}
};
//
A *d2 = (A*)d;
cout << boolalpha << (typeid(*d2) == typeid(D)) << endl;
C *d3 = (C*)d;
cout << boolalpha << (typeid(*d3) == typeid(D)) << endl;
Output is now:
true
false
Without virtual destructor:
false
false
So you can't rely on serialization working properly using only
TypeInfo/offsets
Re: std.reflection prototype
On Wed, 15 Apr 2015 15:48:28 -0400, Jacob Carlborg wrote:
On 2015-04-15 17:26, bitwise wrote:
Right now, this is the def:
/**
* Array of pairs giving the offset and type information for each
* member in an aggregate.
*/
struct OffsetTypeInfo
{
size_t offset;/// Offset of member from start of object
TypeInfo ti;/// TypeInfo for this member
}
If "string name" esd added, and then offTi[] esd actually populated,
then I suppose you could do this:
class Test {
int a = 4;
private int b = 5;
void print(){ writeln(b); }
}
void main()
{
Test test = new Test;
// offsetof would instead come from the TypeInfo/OffsetTypeInfo
int* b = cast(int*)(cast(void*)test + Test.b.offsetof);
*b = 1234;
test.print();
}
But AFAIK, this is NOT ok in C++ because of the way inheritance works..
is this safe in D?
I'm not sure, I would assume so. Why is this not safe in C++?
One reason is that casting with multiple inheritance offsets the pointer,
and I forget exactly virtual inheritance works, but I'm sure it breaks
things too..
#include
using namespace std;
class A {
public: int a;
};
class B {
public: int b;
};
class C {
public: int c;
};
class D : public A, public B, public C {
public: int d;
};
int main(int argc, const char * argv[])
{
D *d = new D;
cout << (intptr_t)d << endl;
cout << (intptr_t)(C*)d << endl;
cout << (intptr_t)(B*)d << endl;
cout << (intptr_t)(A*)d << endl;
return 0;
}
possible output:
1098416
1098424
1098420
1098416
Re: std.reflection prototype
On 2015-04-15 17:26, bitwise wrote:
Right now, this is the def:
/**
* Array of pairs giving the offset and type information for each
* member in an aggregate.
*/
struct OffsetTypeInfo
{
size_t offset;/// Offset of member from start of object
TypeInfo ti;/// TypeInfo for this member
}
If "string name" esd added, and then offTi[] esd actually populated,
then I suppose you could do this:
class Test {
int a = 4;
private int b = 5;
void print(){ writeln(b); }
}
void main()
{
Test test = new Test;
// offsetof would instead come from the TypeInfo/OffsetTypeInfo
int* b = cast(int*)(cast(void*)test + Test.b.offsetof);
*b = 1234;
test.print();
}
But AFAIK, this is NOT ok in C++ because of the way inheritance works..
is this safe in D?
I'm not sure, I would assume so. Why is this not safe in C++?
--
/Jacob Carlborg
Re: std.reflection prototype
On Wed, 15 Apr 2015 11:26:48 -0400, bitwise wrote:
But AFAIK, this is NOT ok in C++ because of the way inheritance works..
is this safe in D?
To clarify, I'm talking about doing doing things by Object pointer or void
pointer, which seems to work fine:
class Test {
int a = 4;
private int b = 5;
void print(){ writeln(b); }
}
struct Test2 {
int a = 4;
private int b = 5;
void print(){ writeln(b); }
}
void main()
{
Test test = new Test;
Object obj = test;
int* b = cast(int*)(cast(void*)obj + Test.b.offsetof);
*b = 1234;
test.print();
Test2 test2 = Test2();
void *obj = &test2;
int* b = cast(int*)(obj + Test2.b.offsetof);
*b = 1234;
test2.print();
}
Re: std.reflection prototype
On Wed, 15 Apr 2015 05:31:24 -0400, Jacob Carlborg wrote:
It needs to be possible to set and get a value of an instance variable
based on it's name, through runtime reflection. It also needs to bypass
protection, i.e. "private".
Right now, this is the def:
/**
* Array of pairs giving the offset and type information for each
* member in an aggregate.
*/
struct OffsetTypeInfo
{
size_t offset;/// Offset of member from start of object
TypeInfo ti;/// TypeInfo for this member
}
If "string name" esd added, and then offTi[] esd actually populated, then
I suppose you could do this:
class Test {
int a = 4;
private int b = 5;
void print(){ writeln(b); }
}
void main()
{
Test test = new Test;
// offsetof would instead come from the TypeInfo/OffsetTypeInfo
int* b = cast(int*)(cast(void*)test + Test.b.offsetof);
*b = 1234;
test.print();
}
But AFAIK, this is NOT ok in C++ because of the way inheritance works.. is
this safe in D?
Re: std.reflection prototype
On 2015-04-14 18:33, bitwise wrote: This is one of the main reasons I like the OffsetTypeInfo/offTi() option. I think a "name" field would have to be added to OffsetTypeInfo for it to be useful, and I have a feeling that would increase the binary size by quite a bit.. So I'll have to test it and see. It needs to be possible to set and get a value of an instance variable based on it's name, through runtime reflection. It also needs to bypass protection, i.e. "private". -- /Jacob Carlborg
Re: std.reflection prototype
On Tue, 14 Apr 2015 04:16:16 -0400, Jacob Carlborg wrote: On 2015-04-14 07:03, bitwise wrote: You may want to look at "Orange". https://github.com/jacob-carlborg/orange The API is clean, and non-invasive. The only downside for me, is that in order to serialize a class by a base pointer, you have to register it at compile time. Yeah, this will require runtime reflection to avoid the registering of the type. This is one of the main reasons I like the OffsetTypeInfo/offTi() option. I think a "name" field would have to be added to OffsetTypeInfo for it to be useful, and I have a feeling that would increase the binary size by quite a bit.. So I'll have to test it and see.
Re: std.reflection prototype
On 2015-04-14 07:03, bitwise wrote: You may want to look at "Orange". https://github.com/jacob-carlborg/orange The API is clean, and non-invasive. The only downside for me, is that in order to serialize a class by a base pointer, you have to register it at compile time. Yeah, this will require runtime reflection to avoid the registering of the type. -- /Jacob Carlborg
Re: std.reflection prototype
On Mon, 13 Apr 2015 08:57:26 -0400, rcorre wrote:
For me, having a solid reflection library like this is one of the most
important improvements D can make right now.
At this point, I've kinda hit a wall.
Generating a hierarchical/object-oriented representation of the type
system at compile-time using templates seems to be an all-or-nothing
procedure.
Below is an example of the base() method of ClassRefl:
const(ClassRefl) base() const {
static if(!is(T == Object)) {
alias BaseClassesTuple!T base_type;
return reflect!(base_type[0]);
}
}
The method returns "reflect!(base_type[0])", which, when instantiated,
will create another ClassRefl for the base class. That ClassRefl will also
have a base() method, which will instantiate more reflections/templates,
and so on... I also had a parent() method at one point, but since a module
can be the parent of a class, it was possible that reflecting a single
class using reflect!T could build entire reflection hierarchies for
several modules. As soon as you reflect anything right now, all dependent
reflections must also be instantiated. The amount of bloat(compiled binary
size), and the increase in compile time would most likely be a deal
breaker for a lot of people. My approach could probably be optimized, but
I'm not sure it would be enough to make it usable.
I'm out of ideas right now, but open to suggestions if you have a solution
to the above problem.
How can you get the static type of a reflected field? For example:
alias T = reflect!Foo.fields[0].field_type
doSomethingWithType!T
I don't believe an answer exists with my current design.
How can you get the user-defined attributes of a given ScopeRefl?
I don't think this is possible either with the current design. The current
design must also work at runtime, and since UDAs can be any symbol, I'm
not sure there is a good way to generalize this. I could wrap all
attributes in an AttributeRefl class, but there wouldn't be much to tell
about the wrapped attribute beside it's type, and maybe a string
representation of it(more bloat).
I'm sure that more could be done to complement __traits and std.traits,
but I don't think that includes using my reflection library in it's
current state.
I can say that the techniques I used in jsonizer got pretty hacky --
there's a whole lot of __traits and one monstrous
mixin template.
You may want to look at "Orange".
https://github.com/jacob-carlborg/orange
The API is clean, and non-invasive. The only downside for me, is that in
order to serialize a class by a base pointer, you have to register it at
compile time.
If I understand correctly, more work needs to be done in druntime to
better support reflection, but I've read this thread about 3 times and
I'm still having a hard time figuring out whats going on :)
The RTInfo(T) template would, in theory, automatically generate some
custom metadata for every type. The compiler automatically instantiates
the RTInfo(T) template for each type right now. The idea was suggested
that a programmer could create their own RTInfo(T) that the compiler would
use that instead. This option was thrown out because it doesn't work with
separate compilation. Right now, RTInfo(T) is defined in object.d, which
is implicitly imported into every module that you compile, but if you had
your own RTInfo defined in your own module, it wouldn't be available to
any other modules at compile time, nor would it be available to any third
party libraries you may be using.
The OffsetTypeInfo thing would be compiler generated. However, the oldest
version of DMD I can find on github (2009) says this:
// offTi[]
dtdword(&dt, 0);
dtdword(&dt, 0);// null for now, fix later
So I'm not sure why it was never implemented. I may eventually attempt a
pull request for the above fix.
For the most part, this looks really nice. Thanks for putting it
together, bitwise!
Thanks ;)
Re: std.reflection prototype
For me, having a solid reflection library like this is one of
the most important improvements D can make right now.
After releasing jsonizer (a json serializer) I started talking to
the authors of PainlessJson (another json serializer that was
released around the same time) about how there was a substantial
amount of overlap between our implementations, a large portion of
which was related to reflection rather than actually handling
json. While I won't speak for painlessjson, I can say that the
techniques I used in jsonizer got pretty hacky -- there's a whole
lot of __traits and one monstrous mixin template.
For the most part, this looks really nice. Thanks for putting it
together, bitwise!
That being said, there are a few limitiations I see to this
implementation:
How can you get the user-defined attributes of a given ScopeRefl?
It would be awesome to say:
foreach(field ; reflect!Foo.fields) {
static if (field.hasAttribute(...)) { ... }
}
I guess the difficulty here is that a user-defined attribute in
D can be pretty much anything from a primitive type to a struct
instance to some arbitrary symbol.
How can you get the static type of a reflected field? For example:
alias T = reflect!Foo.fields[0].field_type
doSomethingWithType!T
This is available in FieldRefImpl, but disappears on FieldRefl so
it can be stuck into an array with a bunch of other FieldRefl,
and I'm not sure if there is a way of getting that information
back (other than resorting back to __traits on the user-end,
which kind of defeats the purpose).
If I understand correctly, more work needs to be done in druntime
to better support reflection, but I've read this thread about 3
times and I'm still having a hard time figuring out whats going
on :)
If these changes were made to druntime, would they obviate the
need for a reflection library, or simply make it easier to
implement a reflection library? Is more help needed? If so, where
should I start looking? I'm not familiar with druntime at all,
but wouldn't mind trying to make sense of it.
Re: std.reflection prototype
It most certainly is. I've continued to dig and try to reason about the history behind this problem. This code has actually been around since D1, but was never completed: https://github.com/D-Programming-Language/dmd/blob/master/src/dsymbol.c#L1237 I think finishing up getMembers() rather than trying to output OffsetTypeInfo would be a better idea. Thoughts?
Re: std.reflection prototype
On 6/04/2015 6:08 a.m., bitwise wrote: I'm pretty sure this function here is converting the typeinfo to a linked list of data that gets outputted to the object file: .. Scratch that, I think this is what I'm looking for: https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L308 It most certainly is.
Re: std.reflection prototype
I'm pretty sure this function here is converting the typeinfo to a linked list of data that gets outputted to the object file: .. Scratch that, I think this is what I'm looking for: https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L308
Re: std.reflection prototype
I don't think so. I looked into it, and it looks to be pretty
low hanging and easy to do.
I'm just not setup *grumbles*.
I'm trying to get it figured out right now.
It looks like this will load all the static data for me:
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_win32.d#L53
I'm guessing the TypeInfo will be part of the regular data
segment. I'm a little confused why the ModuleInfo has it's own
special segment.. Would that have to be done for all the
MemberInfo too? Anyways, as far as I can tell, druntime would not
have to be altered to implement offTi.
Looking at the compiler now, I'm trying to figure out where the
TypeInfo would be outputted.
It seems like this is it:
https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L1013
I'm pretty sure this function here is converting the typeinfo to
a linked list of data that gets outputted to the object file:
[code]
TypeInfo_toDt(&s->Sdt, tid);
[/code]
https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L1027
But, inside that call, I find this:
[code]
TypeInfoDtVisitor v(pdt);
d->accept(&v);
[/code]
https://github.com/D-Programming-Language/dmd/blob/master/src/typinf.c#L602
which finally leads me to...
void TypeInfoDtVisitor::visit(TypeInfoClassDeclaration *d)
{
//printf("TypeInfoClassDeclaration::toDt() %s\n",
tinfo->toChars());
assert(0);
}
which is not helpful.
Anyways, I'll keep looking, but help is welcome :)
Re: std.reflection prototype
On 2015-04-04 21:49, bitwise wrote: One more question: Does anyone know why TypeInfo_class.getMembers() was removed? [1] I found an old post saying that it never worked and returned an empty array, so that is most likely the answer Yes. but although getMembers was removed, Walter seems to have left behind the classes MemberInfo, MemberInfo_field, and MemberInfo_function, so I'm guessing there is still hope of having them implemented one day. [2] I don't know why they were not removed. -- /Jacob Carlborg
Re: std.reflection prototype
On 2015-04-04 18:16, bitwise wrote: Ok, I think I understand what you're suggesting now, which is that you want a library to be able to override RTInfo in order to add it's own metadata to all types, which raises the question, what if more than one library wants to add metadata? And I think this question was addressed by the suggestion of the AA for RTInfo where the module's fully qualified name was the key, which won't work because of separate compilation... right? Yes, exactly. I don't know if it works with separate complication or not. But it will make it more complicated. So in my case I could just update my RTInfo to generate a reflection for each type, and make it accessible using a UFC or something. If I understand correctly though, the idea has been dismissed as being impossible due to separate compilation, right? Is there any leads on this at this point? I don't think it's impossible, but more it's complicated. I don't think I got a really good elaborated answer form Martin why he didn't like the AA approach, except for being complicated. -- /Jacob Carlborg
Re: std.reflection prototype
On 5/04/2015 4:41 a.m., bitwise wrote: I love the idea of it. But first we need to finish off what support we damn well have in druntime. m_offTi for TypeInfo_Class currently is not being generated. We also need some form of RTInfo for modules. Not just for symbols. This alone will open up quite a few doors! Ok, I'm starting to understand this idea a little better. Is anyone currently working on generating m_offTi? I don't think so. I looked into it, and it looks to be pretty low hanging and easy to do. I'm just not setup *grumbles*.
Re: std.reflection prototype
One more question: Does anyone know why TypeInfo_class.getMembers() was removed? [1] I found an old post saying that it never worked and returned an empty array, so that is most likely the answer, but although getMembers was removed, Walter seems to have left behind the classes MemberInfo, MemberInfo_field, and MemberInfo_function, so I'm guessing there is still hope of having them implemented one day. [2] [1] https://github.com/D-Programming-Language/druntime/commit/f957b4ef7222bae5da7f3b4104ab42061dd41319#diff-7eac7eb46e31907f148813e793155274 [2] https://github.com/D-Programming-Language/druntime/blob/bd3f4cb2122edd1a7c107f86936c20bba15191b6/src/object.di#L276
Re: std.reflection prototype
I love the idea of it. But first we need to finish off what support we damn well have in druntime. m_offTi for TypeInfo_Class currently is not being generated. We also need some form of RTInfo for modules. Not just for symbols. This alone will open up quite a few doors! Ok, I'm starting to understand this idea a little better. Is anyone currently working on generating m_offTi?
Re: std.reflection prototype
On Thursday, 2 April 2015 at 06:57:25 UTC, Jacob Carlborg wrote: On 2015-04-02 02:28, bitwise wrote: If I'm understanding correctly, doing it this way is to avoid making changes to the compiler, right? I don't understand this decision because it seems that most of the needed infrastructure is already built into ModuleInfo, and that it just needs to be completed. It would eliminate the problem of template/code bloat from a library like mine, and at the same time, would not require the user to register any types. As I said, the reason for implementing RTInfo for modules was to _not_ have to register anything. There are other good use cases for both RTInfo and RMInfo (runtime module info), they are a more generic solution. Two other threads about unit testing [1], [2] is a good use case. RMInfo can be used to collect all unit test functions and create a custom runner. Here's [3] one example where a unit test runner makes it possible to have CTFE unit tests. This proof of concept only scans the current module, here RMInfo would be really handy to scan all modules. Here's [4] one example where RTInfo is used to check virtual methods. All virtual methods are required to be marked with @virtual. There's a lot of missing info in ModuleInfo and TypeInfo. For example MemberInfo_function contains no information about parameters, return types, attributes and so on. [1] http://forum.dlang.org/thread/mfcgj3$12a0$1...@digitalmars.com [2] http://forum.dlang.org/thread/mfci6o$13oa$1...@digitalmars.com [3] http://forum.dlang.org/thread/ks1brj$1l6c$1...@digitalmars.com [4] http://forum.dlang.org/thread/kok86c$126l$1...@digitalmars.com Ok, I think I understand what you're suggesting now, which is that you want a library to be able to override RTInfo in order to add it's own metadata to all types, which raises the question, what if more than one library wants to add metadata? And I think this question was addressed by the suggestion of the AA for RTInfo where the module's fully qualified name was the key, which won't work because of separate compilation... right? So in my case I could just update my RTInfo to generate a reflection for each type, and make it accessible using a UFC or something. If I understand correctly though, the idea has been dismissed as being impossible due to separate compilation, right? Is there any leads on this at this point?
Re: std.reflection prototype
On 2015-04-02 02:28, bitwise wrote: If I'm understanding correctly, doing it this way is to avoid making changes to the compiler, right? I don't understand this decision because it seems that most of the needed infrastructure is already built into ModuleInfo, and that it just needs to be completed. It would eliminate the problem of template/code bloat from a library like mine, and at the same time, would not require the user to register any types. As I said, the reason for implementing RTInfo for modules was to _not_ have to register anything. There are other good use cases for both RTInfo and RMInfo (runtime module info), they are a more generic solution. Two other threads about unit testing [1], [2] is a good use case. RMInfo can be used to collect all unit test functions and create a custom runner. Here's [3] one example where a unit test runner makes it possible to have CTFE unit tests. This proof of concept only scans the current module, here RMInfo would be really handy to scan all modules. Here's [4] one example where RTInfo is used to check virtual methods. All virtual methods are required to be marked with @virtual. There's a lot of missing info in ModuleInfo and TypeInfo. For example MemberInfo_function contains no information about parameters, return types, attributes and so on. [1] http://forum.dlang.org/thread/mfcgj3$12a0$1...@digitalmars.com [2] http://forum.dlang.org/thread/mfci6o$13oa$1...@digitalmars.com [3] http://forum.dlang.org/thread/ks1brj$1l6c$1...@digitalmars.com [4] http://forum.dlang.org/thread/kok86c$126l$1...@digitalmars.com -- /Jacob Carlborg
Re: std.reflection prototype
Currently in object.d in druntime there's
a template, RTInfo, declared something
like this:
[snip]
Thanks for the breakdown! Those pull requests make a lot more
sense when read in the proper context ;)
The main reason for implementing RTInfo
for modules is to implement reflection.
With the template instantiated for each
module you could scan the module for class and methods and
build up reflection
data completely transparently without
the user needing to register any types
or similar.
If I'm understanding correctly, doing it this way is to avoid
making changes to the compiler, right?
I don't understand this decision because it seems that most of
the needed infrastructure is already built into ModuleInfo, and
that it just needs to be completed. It would eliminate the
problem of template/code bloat from a library like mine, and at
the same time, would not require the user to register any types.
The downside would be susceptibility to things like .NET
Reflector, making binaries very easy to crack/decompile, and of
course, the extra space it took for the metadata. But, the
generation of the metadata could be made optional via command
line switch. Metadata could either be disabled completely, or
partially with something like --no-verbose-rtti, which could omit
all the human readable portions of the metadata, while leaving
behind whatever was needed for precise GC or whatever else may
need it. Of course, the inverse would also be an option, to omit
all type info and enable it manually via --verbose-rtti.
Failing all of the above, I suppose my reflection library does
the job well enough for my needs. Some work could be done to
optimize the binary size, but feature-wise, I find it more than
complete enough. However, I am thinking about making it more
versatile in terms of what is generated, because that seems to be
a common request.
Something like this might make sense:
module test;
class Test1{}
@Reflected class Test2{}
@NotReflected class Test3{}
///
module reflection;
enum ReflectionMode {
Inclusive, // include all classes unless specified
Exclusive // exclude all classes unless specified
}
template moduleReflection(alias mod, ReflectionMode m) { ... }
///
module test;
mixin(moduleReflection!(test, ReflectionMode.Inclusive));
///
ReflectionMode.Inclusive would generate reflection for Test1 and
Test2,
ReflectionMode.Exclusive would generate reflection for only Test2.
I'm not sure it's possible to make recursiveness of reflection
optional without losing compile-time availability of major
features though.
This is needed for sure. For example do you
count how many subjects on .learn are about
__traits(allMembers,... or __traits(getMember,...
Yeah, I agree these are confusing... along with type-tuples, and
CTFE. I think all three of these are hard to grasp without
knowledge of how a compiler works.
About the design i would rather see this as
a user choice, eg the reflexion as a mixin
template that you can mix in a class or a struct you want to be
filled with info.
This remark is related to the size problem
pointed in the previous posts.
I would like to make things as flexible/optional as possible
without being intrusive to user types. UDAs are about as far as
I'm willing to go with per-type boilerplate.
Re: std.reflection prototype
On Monday, 30 March 2015 at 01:11:55 UTC, bitwise wrote:
I came across this post a while back and decided to implement
it:
http://forum.dlang.org/thread/juf7sk$16rl$1...@digitalmars.com
My implementation:
https://github.com/bitwise-github/D-Reflection
The above conversation seemed to stop abruptly, so I went on to
assume that no one chose to champion the task.
At the time, I looked around for other conversations or
attempts at runtime reflection for D, but couldn't really find
anything. I did find the ModuleInfo/reflection stuff in
object.d, but it seemed like an effort that may have been
abandoned. Also, the above conversation seemed to suggest it
should be opt-in, which also made me wonder if the stuff in
object.d was abandoned or for a different purpose.
Looking again today, someone seems to have been working on it a
bit.. For example, MemberInfo_field and MemberInfo_function
were empty last time I checked.
So what's the current state of things?
Is anybody working on it?
Although MemberInfo_field exists, it doesn't seem to be
available from TypeInfo_Class... Is that the eventual goal?
Is there anything I can do to help get things moving?
Any comments on my implementation would be welcome as well.
(https://github.com/bitwise-github/D-Reflection)
main.d shows some general use cases, but basically, if the
reflect(T) template is used on a class,
that class, and any child types will be reflected. Recursive
reflection only propagates downward, or else it could leak
sideways and unnecessarily reflect several modules.
Most of the reflection information is available at compile
time. For example:
enum name =
reflectModule!(test).findClass("Test").findField("variable").name;
pragma(msg, name); // "variable" will be outputted.
To make a module available for runtime reflection, the
following can be used:
mixin(runtimeReflection!test);
At this point, the above example can be rewritten as:
string name =
getModuleReflection("tests.test").findClass("Test3").findField("static_variable").name;
writeln(name);
This is needed for sure. For example do you count how many
subjects on .learn are about __traits(allMembers,... or
__traits(getMember,...
Run-time serialization is an example.
About the design i would rather see this as a user choice, eg the
reflexion as a mixin template that you can mix in a class or a
struct you want to be filled with info. This remark is related to
the size problem pointed in the previous posts.
Re: std.reflection prototype
On 2015-03-30 23:31, bitwise wrote:
https://github.com/D-Programming-Language/druntime/pull/775
https://github.com/D-Programming-Language/dmd/pull/2271
I've read over the comments of the above two pull-requests, and I'm
confused about what is trying to be achieved.
Currently in object.d in druntime there's a template, RTInfo, declared
something like this:
template RTInfo (T)
{
enum RTInfo = null;
}
The compiler will automatically instantiate this template for all user
defined types. What the template evaluates to will be placed in the
TypeInfo for that given type in the "rtInfo" property.
This allows two things:
* Add custom type checking on user defined types:
template RTInfo (T)
{
static if (T.stringof == "Foo")
static assert(false, "A type named 'Foo' is not allowed");
enum RTInfo = null;
}
* Add custom type info for a given type which is accessible at runtime:
template RTInfo (T)
{
enum RTInfo = T.stringof;
}
The original reason why this was added to the language was to extract
type info necessary to build a precise garbage collector.
The problem with this is to use this feature you need to modify object.d
in druntime. The first pull request tries to come up with a solution
that doesn't require modifying druntime.
The second pull request implemented the same thing as RTInfo but for
modules instead of types. So the compiler would automatically
instantiate this template with for each module it encounters during
compilation.
The main reason for implementing RTInfo for modules is to implement
reflection. With the template instantiated for each module you could
scan the module for class and methods and build up reflection data
completely transparently without the user needing to register any types
or similar.
--
/Jacob Carlborg
Re: std.reflection prototype
On Mon, 30 Mar 2015 21:13:48 +, bitwise wrote: >> On Linux and OSX, the build for just the unittest library would >> generate an over 40MB executable. On Windows with Optlink though, it >> was only 4MB, so perhaps there are optimizations that could be made to >> reduce the size on OSX / Linux. > > This sounds like a bug.. if it CAN be done with an executable size of > 4MB, why would it not be so on OSX/Linux? 'cause DMD on posix doesn't strip binaries by default. it's funny to scare windows people. signature.asc Description: PGP signature
Re: std.reflection prototype
https://github.com/D-Programming-Language/druntime/pull/775
https://github.com/D-Programming-Language/dmd/pull/2271
I've read over the comments of the above two pull-requests, and
I'm confused about what is trying to be achieved.
A ModuleInfo for each module already exists and is available at
runtime. Each ModuleInfo has the method localClasses() to get all
it's classes. Why not just add an array of MethodInfo and
FieldInfo to TypeInfo_Class?
It seems like the above two pulls are trying to add some kind of
custom per-module info. For my purposes, some very basic info for
suffice.
At the very minimum, if TypeInfo_Class could be completed to
contain these, I would be happy:
// one per field
class FieldInfo
{
string name();
TypeInfo type();
void *getAddress(void *instance);
}
// one per overload
class MethodInfo
{
string name();
TypeInfo returnType();
TypeInfo[] parameterTypes();
bool isProperty();
bool isStatic();
void *getAddress(void *instance);
}
The above two would be more than enough to do serialization and
runtime modification of unknown objects.
What am I missing here?
Re: std.reflection prototype
On Linux and OSX, the build for just the unittest library would generate an over 40MB executable. On Windows with Optlink though, it was only 4MB, so perhaps there are optimizations that could be made to reduce the size on OSX / Linux. This sounds like a bug.. if it CAN be done with an executable size of 4MB, why would it not be so on OSX/Linux? Also, i was reading through this pull request: https://github.com/D-Programming-Language/druntime/pull/775 and yglukhov mentioned something about needing 5 minutes and 12GB or ram to build phobos with whatever they're talking about. To be honest though, after reading the above pull-comments and this one below, I'm still confused about what they're trying to do. https://github.com/D-Programming-Language/dmd/pull/2271 Also, from what I can tell you generate a class for every symbol. This could generate a significant amount of bloat as well because it would generate further TypeInfo for each of these classes. I could be wrong here, but if that's the case, perhaps changing to structs would be a better choice if possible? You're right that I am creating a class per symbol, along with a bunch of other templates. It probably would be a good idea to try and eliminate the templates where possible. I'm not sure why you are suggesting structs instead of classes though.
Re: std.reflection prototype
On 2015-03-30 09:06, Rikki Cattermole wrote:
You, me and Walter should have a chat then. I could pretty easily come
up with a way to add data into RTInfo.
I've already come up with a way, any template with the @rtInfo UDA is
treated the same way as RTInfo is now. The problem is then how to store
the data in TypeInfo. Since it would be possible to have multiple data
generated for a given type I was thinking it could be stored in an
associative array. The keys would be the name of the module which
generated the data and the the values would be the data.
Something like this:
module foo.bar;
@rtInfo template Foo (T)
{
enum Foo = "bar";
}
assert(typeid(T).rtInfo["foo.bar"] == "bar");
If I recall correctly Martin Nowak didn't like this approach. The
associate array would need to be built at load time of the application
due to separate compilation.
BTW, here [1] is the pull request and the reason why it was closed.
[1]
https://github.com/D-Programming-Language/dmd/pull/2271#issuecomment-59621060
--
/Jacob Carlborg
Re: std.reflection prototype
On 30/03/2015 7:59 p.m., Jacob Carlborg wrote: On 2015-03-30 04:09, Rikki Cattermole wrote: We also need some form of RTInfo for modules. I made a pull request for that but unfortunately it hasn't been accepted yet. The reason seems to be that we need to come up with a way to do custom RTInfo without modifying druntime, that can also be used for this template. You, me and Walter should have a chat then. I could pretty easily come up with a way to add data into RTInfo.
Re: std.reflection prototype
On 2015-03-30 04:09, Rikki Cattermole wrote: We also need some form of RTInfo for modules. I made a pull request for that but unfortunately it hasn't been accepted yet. The reason seems to be that we need to come up with a way to do custom RTInfo without modifying druntime, that can also be used for this template. -- /Jacob Carlborg
Re: std.reflection prototype
On Monday, 30 March 2015 at 05:06:14 UTC, bitwise wrote: This definitely should be a library solution. In terms of what Andre was suggesting, I think my implementation is 90% of the way there, so if the will was there to add something like std.reflection to phobos, it wouldn't be much of a leap. But it seems silly, to not use code in druntime. I kind of agree. If I had things my way, all reflection info for all classes would be compiler generated so that it was guaranteed to be there for tools, interop and such. If people were really worried about code bloat, the reflection info could simply be turned off with a -no-reflection flag or something. In terms of playing around with the compiler and the type system, I'm not sure how helpful I could be right now. I'm pretty sure comprehension will not be a problem, but I've got quite a bit of reading to do before I'm up to speed with whats going on with dmd and the type system. When I tried mine (https://shardsoft.com/stash/projects/SHARD/repos/shardtools/browse/source/ShardTools/Reflection.d), which was admittedly a while back, the template bloat was actually a significant problem. On Linux and OSX, the build for just the unittest library would generate an over 40MB executable. On Windows with Optlink though, it was only 4MB, so perhaps there are optimizations that could be made to reduce the size on OSX / Linux. Either way though, that's way too much bloat for something in druntime, and I don't think it's necessary. While it might be nice to use RTInfo to generate reflection data for your whole program, most people would be perfectly okay with at most a few modules, possibly recursively. Also, from what I can tell you generate a class for every symbol. This could generate a significant amount of bloat as well because it would generate further TypeInfo for each of these classes. I could be wrong here, but if that's the case, perhaps changing to structs would be a better choice if possible?
Re: std.reflection prototype
On 30/03/2015 6:06 p.m., bitwise wrote:
This definitely should be a library solution.
In terms of what Andre was suggesting, I think my implementation is 90%
of the way there, so if the will was there to add something like
std.reflection to phobos, it wouldn't be much of a leap.
But it seems silly, to not use code in druntime.
I kind of agree. If I had things my way, all reflection info for all
classes would be compiler generated so that it was guaranteed to be
there for tools, interop and such. If people were really worried about
code bloat, the reflection info could simply be turned off with a
-no-reflection flag or something.
In terms of playing around with the compiler and the type system, I'm
not sure how helpful I could be right now. I'm pretty sure comprehension
will not be a problem, but I've got quite a bit of reading to do before
I'm up to speed with whats going on with dmd and the type system.
I have so many views upon all this, thanks to my new (not released as
not ready) web service framework. But druntime right now is not ready
for proper reflection. There needs to be some modifications to it and
dmd. But mostly druntime for RTInfo.
For example what shouldn't needed to be done. Here is my install
information for my autoconfig system. (Allows for publically importing
any module I want AND handling any symbol that the system has used.
#Install
## Configure per type
RTInfo requires a single modification.
[Based upon a forum
thread](http://forum.dlang.org/post/majnjuhxdefjuqjlp...@forum.dlang.org).
```D
template RTInfo(T, string moduleName = __MODULE__)
{
enum RTInfo = cast(void*)0x12345678;
static if (__traits(compiles, {import core.config;})) {
import core.config;
alias checkResult = TypeCheck!(T, moduleName);
}
}
```
## Auto imports
COPY/core/configimports.d needs to be copied to import/core/configimports.d
Also needed is object.di under imports to be modified.
Add ``D public import core.configimports;`` anywhere you want.
Re: std.reflection prototype
This definitely should be a library solution. In terms of what Andre was suggesting, I think my implementation is 90% of the way there, so if the will was there to add something like std.reflection to phobos, it wouldn't be much of a leap. But it seems silly, to not use code in druntime. I kind of agree. If I had things my way, all reflection info for all classes would be compiler generated so that it was guaranteed to be there for tools, interop and such. If people were really worried about code bloat, the reflection info could simply be turned off with a -no-reflection flag or something. In terms of playing around with the compiler and the type system, I'm not sure how helpful I could be right now. I'm pretty sure comprehension will not be a problem, but I've got quite a bit of reading to do before I'm up to speed with whats going on with dmd and the type system.
Re: std.reflection prototype
On 30/03/2015 5:39 p.m., bitwise wrote: But first we need to finish off what support we damn well have in druntime. m_offTi for TypeInfo_Class currently is not being generated. We also need some form of RTInfo for modules. Not just for symbols. This alone will open up quite a few doors! In the "time for std.reflection" convo, it seemed like Andre was talking about a library solution, and not something compiler generated.. has that idea since been thrown out? This definitely should be a library solution. But it seems silly, to not use code in druntime. Either that, or if it is like m_offTi, remove it. After all, it was never implemented. Has there been other conversations on this that I missed? No.
Re: std.reflection prototype
But first we need to finish off what support we damn well have in druntime. m_offTi for TypeInfo_Class currently is not being generated. We also need some form of RTInfo for modules. Not just for symbols. This alone will open up quite a few doors! In the "time for std.reflection" convo, it seemed like Andre was talking about a library solution, and not something compiler generated.. has that idea since been thrown out? Has there been other conversations on this that I missed? Thanks
Re: std.reflection prototype
On 30/03/2015 2:11 p.m., bitwise wrote:
I came across this post a while back and decided to implement it:
http://forum.dlang.org/thread/juf7sk$16rl$1...@digitalmars.com
My implementation:
https://github.com/bitwise-github/D-Reflection
The above conversation seemed to stop abruptly, so I went on to assume
that no one chose to champion the task.
At the time, I looked around for other conversations or attempts at
runtime reflection for D, but couldn't really find anything. I did find
the ModuleInfo/reflection stuff in object.d, but it seemed like an
effort that may have been abandoned. Also, the above conversation seemed
to suggest it should be opt-in, which also made me wonder if the stuff
in object.d was abandoned or for a different purpose.
Looking again today, someone seems to have been working on it a bit..
For example, MemberInfo_field and MemberInfo_function were empty last
time I checked.
So what's the current state of things?
Is anybody working on it?
Although MemberInfo_field exists, it doesn't seem to be available from
TypeInfo_Class... Is that the eventual goal?
Is there anything I can do to help get things moving?
Any comments on my implementation would be welcome as well.
(https://github.com/bitwise-github/D-Reflection)
main.d shows some general use cases, but basically, if the reflect(T)
template is used on a class,
that class, and any child types will be reflected. Recursive reflection
only propagates downward, or else it could leak sideways and
unnecessarily reflect several modules.
Most of the reflection information is available at compile time. For
example:
enum name =
reflectModule!(test).findClass("Test").findField("variable").name;
pragma(msg, name); // "variable" will be outputted.
To make a module available for runtime reflection, the following can be
used:
mixin(runtimeReflection!test);
At this point, the above example can be rewritten as:
string name =
getModuleReflection("tests.test").findClass("Test3").findField("static_variable").name;
writeln(name);
I love the idea of it.
But first we need to finish off what support we damn well have in druntime.
m_offTi for TypeInfo_Class currently is not being generated.
We also need some form of RTInfo for modules. Not just for symbols. This
alone will open up quite a few doors!
std.reflection prototype
I came across this post a while back and decided to implement it:
http://forum.dlang.org/thread/juf7sk$16rl$1...@digitalmars.com
My implementation:
https://github.com/bitwise-github/D-Reflection
The above conversation seemed to stop abruptly, so I went on to
assume that no one chose to champion the task.
At the time, I looked around for other conversations or attempts
at runtime reflection for D, but couldn't really find anything. I
did find the ModuleInfo/reflection stuff in object.d, but it
seemed like an effort that may have been abandoned. Also, the
above conversation seemed to suggest it should be opt-in, which
also made me wonder if the stuff in object.d was abandoned or for
a different purpose.
Looking again today, someone seems to have been working on it a
bit.. For example, MemberInfo_field and MemberInfo_function were
empty last time I checked.
So what's the current state of things?
Is anybody working on it?
Although MemberInfo_field exists, it doesn't seem to be available
from TypeInfo_Class... Is that the eventual goal?
Is there anything I can do to help get things moving?
Any comments on my implementation would be welcome as well.
(https://github.com/bitwise-github/D-Reflection)
main.d shows some general use cases, but basically, if the
reflect(T) template is used on a class,
that class, and any child types will be reflected. Recursive
reflection only propagates downward, or else it could leak
sideways and unnecessarily reflect several modules.
Most of the reflection information is available at compile time.
For example:
enum name =
reflectModule!(test).findClass("Test").findField("variable").name;
pragma(msg, name); // "variable" will be outputted.
To make a module available for runtime reflection, the following
can be used:
mixin(runtimeReflection!test);
At this point, the above example can be rewritten as:
string name =
getModuleReflection("tests.test").findClass("Test3").findField("static_variable").name;
writeln(name);
