Re: Dynamic binding to the Mono runtime API
On 6/4/17 04:15, Jakub Szewczyk wrote: On Sunday, 4 June 2017 at 09:43:23 UTC, Adam Wilson wrote: On 6/4/17 01:18, Jakub Szewczyk wrote: This is an interface to the Mono libraries, D/CLI would [...] My interest is less in code ports than bindings to the actual code. My experience with code ports or translations is that often subtle bugs creep in during translation due to the fact that each language has different idioms. What I am thinking about is a tool that loads an assembly, examines it's types and methods via this API and emits D code that directly interfaces into the .NET types via this API. The tricky part here is mapping the .NET dependencies into D. The moment the library exposes a type from a dependency, that dependency ALSO needs to be included somehow. All libraries reference "mscorlib", AKA the BCL, so we'd have to provide a "mono-bcl" package on DUB. That's what I actually meant, "porting" was a misused term on my part, "binding" would be a better word, sorry for that. As for the dependency problem - I think that a linking layer generator would accept a list of input assemblies (and optionally, specific classes) to which it should generate bindings, the core Mono types could be automatically translated to D equivalents, and the rest could be left as an opaque reference, like MonoObject* in C, also providing support for very basic reflection through the Mono methods if it turned out to be useful for anyone. Mono actually supports some kind of GC bridging as far as I understand, [...] On the GC side I was mostly thinking about GC Handles so that the objects don't get collected out from underneath us. That is something is trivial to code-gen. As for exceptions, I like the catch->translate->rethrow mechanism. And if the exception is unknown we could simply throw a generic exception. The important thing is to get close to the D experience, not try to map it perfectly. Yes, GCHandles to keep Mono objects in D and a wrapper based on that GC bridge to keep D references from being collected by Mono. I have previously implemented a very similar mechanism for Lua in a small wrapper layer, and it worked perfectly. I can make a static library version, [...] Thank you for this! I find static libraries easier to deal with. I'm sure other people have differing opinions, so having both would make everyone happy. It's now public as v1.1.0, I've tested that it works with the tiny sample, the only important part is that the library to link must be specified by the project using this binding, because those paths may vary across systems, and they cannot be specified in code like the dynamic link ones. However, a simple "libs":["mono-2.0"] entry in dub.json should be enough for most use cases. Thank you for this, I've tried it and it works! I did some in depth research and prototyping in D, and it looks like the only way to enumerate the types in an assembly is to use the Metadata Table API's map everything that way. That's a little beyond the scope of my free time so I'll have to shelve the idea from now. :( -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Re: Dynamic binding to the Mono runtime API
On 2017-06-04 20:19, Jakub Szewczyk wrote: The problem is, it emits completely wrong code whereever a function is necessary, like in function pointers. I can try to isolate the change to global-level function declarations only, to make it generate correct code that doesn't require running two versions of DStep in parallel to extract all the information, when I do that I'll submit a PR. Cool, that would be great. -- /Jacob Carlborg
Re: Dynamic binding to the Mono runtime API
On Sunday, 4 June 2017 at 15:43:09 UTC, Jacob Carlborg wrote: On 2017-06-04 10:18, Jakub Szewczyk wrote: Btw, I've manually ported the basic and configuration headers, so that no mistakes are made, and then used DStep and a modified DStep to generate the rest of the headers - my modification was only to change the way function declarations are generated, to make them in derelict form of alias da_function = void function(...);, and the rest was done with quite a lot of editor(VSCode) shortcuts. Would you like to contribute your changes under a flag to DStep? The problem is, it emits completely wrong code whereever a function is necessary, like in function pointers. I can try to isolate the change to global-level function declarations only, to make it generate correct code that doesn't require running two versions of DStep in parallel to extract all the information, when I do that I'll submit a PR.
Re: Dynamic binding to the Mono runtime API
On 2017-06-04 10:18, Jakub Szewczyk wrote: Btw, I've manually ported the basic and configuration headers, so that no mistakes are made, and then used DStep and a modified DStep to generate the rest of the headers - my modification was only to change the way function declarations are generated, to make them in derelict form of alias da_function = void function(...);, and the rest was done with quite a lot of editor(VSCode) shortcuts. Would you like to contribute your changes under a flag to DStep? -- /Jacob Carlborg
Re: Dynamic binding to the Mono runtime API
On 2017-06-03 23:44, extrawurst wrote: On Saturday, 3 June 2017 at 17:30:05 UTC, Jakub Szewczyk wrote: Mono runtime is a cross-platform, open-source alternative to Microsoft's .NET framework [1], and it can be embedded in other applications as a "scripting" VM, but with JIT-compilation enhanced performance and support of many languages such as C#, F# or IronPython [2]. It provides a C API, so I've bound it to D as a Derelict-based project, available at https://github.com/kubasz/derelict-mono, and as a DUB package (http://code.dlang.org/packages/derelict-mono). It currently wraps the Mono 5.0 API. There's also a simple example of calling a C# main from D code, and C# code calling a native function implemented in D. [...] Cool work! Thanks! Did u use dstep for the translation or manual labor? I'm guessing no, since these bindings are using function pointers while DStep will translate function declarations to function declarations in D as well, not function pointers. -- /Jacob Carlborg
Re: Dynamic binding to the Mono runtime API
On Sunday, 4 June 2017 at 09:43:23 UTC, Adam Wilson wrote: On 6/4/17 01:18, Jakub Szewczyk wrote: This is an interface to the Mono libraries, D/CLI would [...] My interest is less in code ports than bindings to the actual code. My experience with code ports or translations is that often subtle bugs creep in during translation due to the fact that each language has different idioms. What I am thinking about is a tool that loads an assembly, examines it's types and methods via this API and emits D code that directly interfaces into the .NET types via this API. The tricky part here is mapping the .NET dependencies into D. The moment the library exposes a type from a dependency, that dependency ALSO needs to be included somehow. All libraries reference "mscorlib", AKA the BCL, so we'd have to provide a "mono-bcl" package on DUB. That's what I actually meant, "porting" was a misused term on my part, "binding" would be a better word, sorry for that. As for the dependency problem - I think that a linking layer generator would accept a list of input assemblies (and optionally, specific classes) to which it should generate bindings, the core Mono types could be automatically translated to D equivalents, and the rest could be left as an opaque reference, like MonoObject* in C, also providing support for very basic reflection through the Mono methods if it turned out to be useful for anyone. Mono actually supports some kind of GC bridging as far as I understand, [...] On the GC side I was mostly thinking about GC Handles so that the objects don't get collected out from underneath us. That is something is trivial to code-gen. As for exceptions, I like the catch->translate->rethrow mechanism. And if the exception is unknown we could simply throw a generic exception. The important thing is to get close to the D experience, not try to map it perfectly. Yes, GCHandles to keep Mono objects in D and a wrapper based on that GC bridge to keep D references from being collected by Mono. I have previously implemented a very similar mechanism for Lua in a small wrapper layer, and it worked perfectly. I can make a static library version, [...] Thank you for this! I find static libraries easier to deal with. I'm sure other people have differing opinions, so having both would make everyone happy. It's now public as v1.1.0, I've tested that it works with the tiny sample, the only important part is that the library to link must be specified by the project using this binding, because those paths may vary across systems, and they cannot be specified in code like the dynamic link ones. However, a simple "libs":["mono-2.0"] entry in dub.json should be enough for most use cases.
Re: Dynamic binding to the Mono runtime API
On 6/4/17 01:18, Jakub Szewczyk wrote: This is an interface to the Mono libraries, D/CLI would require quite a lot of compiler changes, both on the front-end and back-end side, but thanks to metaprogramming a wrapper library can get very close to such an interface. I plan on making an automated D to Mono bridge, akin to LuaD, so that it can be used as a scripting platform for e.g. games. I haven't thought about doing it the other way around, but now it seems like a very interesting idea! Unfortunately no XAML (http://www.mono-project.com/docs/gui/wpf/), but many other libraries, such as XWT (https://github.com/mono/xwt) could be ported this way, I'll certainly look into it. My interest is less in code ports than bindings to the actual code. My experience with code ports or translations is that often subtle bugs creep in during translation due to the fact that each language has different idioms. What I am thinking about is a tool that loads an assembly, examines it's types and methods via this API and emits D code that directly interfaces into the .NET types via this API. The tricky part here is mapping the .NET dependencies into D. The moment the library exposes a type from a dependency, that dependency ALSO needs to be included somehow. All libraries reference "mscorlib", AKA the BCL, so we'd have to provide a "mono-bcl" package on DUB. One solution is to simply include the exposed dependency in the generated code. This would work because while the D code would have distinct types for the same class, the underlying .NET types is the same. The drawback with this approach is that you can't share the instances across D interfaces as the types are different. For example, Library1.B and Library2.C both rely on Dependency.A. In D you would have Library1.B.A and Library2.C.A and these two types, while the same in practice, are different types to the compiler. Maybe a clever use of alias can solve this at code-gen time... more research is required there. The other solution is too have the code-gen throw an error when it encounters a type from a dependency. This ensures that the types are the same across libraries in D, but at the cost of increased complexity for the developer when running the tool (specifying extra deps) and when building their code (ensuring all relevant packages are built/referenced). I'm not sold on either one. And it'd be best if it's possible to support both somehow. As for the WPF remark, my brain immediately jump to trying to interface to .NET/Core using a similar mechanism, but alas some research time indicates that this is not possible outside of COM. *le sigh* My apologize for the random side-track. Mono actually supports some kind of GC bridging as far as I understand, as there is a sgen-bridge.h header just for that, and it has apparently been used in the C#-Java Xamarin interace on Android. As for exceptions - D functions have to be wrapped in a nothrow wrapper, but that can be completely automated with templates. The other way around is also quite simple - when invoking a Mono function a pointer can be given that will set an exception reference if one is thrown from the .NET side, and a wrapper can easily rethrow it back to D. On the GC side I was mostly thinking about GC Handles so that the objects don't get collected out from underneath us. That is something is trivial to code-gen. As for exceptions, I like the catch->translate->rethrow mechanism. And if the exception is unknown we could simply throw a generic exception. The important thing is to get close to the D experience, not try to map it perfectly. I can make a static library version, it's just some regex substitutions done on the functions file actually, most probably I'll publish it today. It will be a dub subconfiguration like it is the case for GLFW3. Btw, I've manually ported the basic and configuration headers, so that no mistakes are made, and then used DStep and a modified DStep to generate the rest of the headers - my modification was only to change the way function declarations are generated, to make them in derelict form of alias da_function = void function(...);, and the rest was done with quite a lot of editor(VSCode) shortcuts. Thank you for this! I find static libraries easier to deal with. I'm sure other people have differing opinions, so having both would make everyone happy. I am very excited about this! -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Re: Dynamic binding to the Mono runtime API
On Sunday, 4 June 2017 at 05:47:32 UTC, Adam Wilson wrote: On 6/3/17 10:30, Jakub Szewczyk wrote: ... I work with C# professionally and this is some SERIOUSLY cool work. Thank you for this! Thank you all! I've looked over the code a bit and I have a couple of questions. This appears to be an interface to the runtime itself, not a BCL interface correct? It looks like this could be used to could this be used to read into a Mono Class Libraries, and if so would so some sort of automated code generation tool be required? It looks to me like the binding will be non-trivial, with GC, exceptions, etc. that all need to be handled at the call-site. Can we get a static library version of this, or is there a dependency on dynamic libraries? I have to admit I am very impressed. I have spent a lot of time building code generators before and I have to admit that the idea of binding to arbitrary .NET libraries via code generation is extremely appealing to me. I am seriously tempted to take this and start building a binding generator... I seriously need more free time! Way too many cool and useful things happening in D for my limited free time. A D binding for XAML ... THAT would sight to behold! This is an interface to the Mono libraries, D/CLI would require quite a lot of compiler changes, both on the front-end and back-end side, but thanks to metaprogramming a wrapper library can get very close to such an interface. I plan on making an automated D to Mono bridge, akin to LuaD, so that it can be used as a scripting platform for e.g. games. I haven't thought about doing it the other way around, but now it seems like a very interesting idea! Unfortunately no XAML (http://www.mono-project.com/docs/gui/wpf/), but many other libraries, such as XWT (https://github.com/mono/xwt) could be ported this way, I'll certainly look into it. Mono actually supports some kind of GC bridging as far as I understand, as there is a sgen-bridge.h header just for that, and it has apparently been used in the C#-Java Xamarin interace on Android. As for exceptions - D functions have to be wrapped in a nothrow wrapper, but that can be completely automated with templates. The other way around is also quite simple - when invoking a Mono function a pointer can be given that will set an exception reference if one is thrown from the .NET side, and a wrapper can easily rethrow it back to D. I can make a static library version, it's just some regex substitutions done on the functions file actually, most probably I'll publish it today. It will be a dub subconfiguration like it is the case for GLFW3. Btw, I've manually ported the basic and configuration headers, so that no mistakes are made, and then used DStep and a modified DStep to generate the rest of the headers - my modification was only to change the way function declarations are generated, to make them in derelict form of alias da_function = void function(...);, and the rest was done with quite a lot of editor(VSCode) shortcuts.
Re: Dynamic binding to the Mono runtime API
On 6/3/17 10:30, Jakub Szewczyk wrote: Mono runtime is a cross-platform, open-source alternative to Microsoft's .NET framework [1], and it can be embedded in other applications as a "scripting" VM, but with JIT-compilation enhanced performance and support of many languages such as C#, F# or IronPython [2]. It provides a C API, so I've bound it to D as a Derelict-based project, available at https://github.com/kubasz/derelict-mono, and as a DUB package (http://code.dlang.org/packages/derelict-mono). It currently wraps the Mono 5.0 API. There's also a simple example of calling a C# main from D code, and C# code calling a native function implemented in D. PS: Because I don't own a Mac I have no idea what the correct paths to the Mono shared library are, so it'd be great if someone could post/create a PR of them. [1] http://www.mono-project.com/ [2] http://www.mono-project.com/docs/advanced/embedding/scripting/ I work with C# professionally and this is some SERIOUSLY cool work. Thank you for this! I've looked over the code a bit and I have a couple of questions. This appears to be an interface to the runtime itself, not a BCL interface correct? It looks like this could be used to could this be used to read into a Mono Class Libraries, and if so would so some sort of automated code generation tool be required? It looks to me like the binding will be non-trivial, with GC, exceptions, etc. that all need to be handled at the call-site. Can we get a static library version of this, or is there a dependency on dynamic libraries? I have to admit I am very impressed. I have spent a lot of time building code generators before and I have to admit that the idea of binding to arbitrary .NET libraries via code generation is extremely appealing to me. I am seriously tempted to take this and start building a binding generator... I seriously need more free time! Way too many cool and useful things happening in D for my limited free time. A D binding for XAML ... THAT would sight to behold! -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Re: Dynamic binding to the Mono runtime API
On Saturday, 3 June 2017 at 17:30:05 UTC, Jakub Szewczyk wrote: Mono runtime is a cross-platform, open-source alternative to Microsoft's .NET framework [1], and it can be embedded in other applications as a "scripting" VM, but with JIT-compilation enhanced performance and support of many languages such as C#, F# or IronPython [2]. It provides a C API, so I've bound it to D as a Derelict-based project, available at https://github.com/kubasz/derelict-mono, and as a DUB package (http://code.dlang.org/packages/derelict-mono). It currently wraps the Mono 5.0 API. There's also a simple example of calling a C# main from D code, and C# code calling a native function implemented in D. PS: Because I don't own a Mac I have no idea what the correct paths to the Mono shared library are, so it'd be great if someone could post/create a PR of them. [1] http://www.mono-project.com/ [2] http://www.mono-project.com/docs/advanced/embedding/scripting/ This is very cool - thank you for doing this. It could prove very helpful. Have you thought of/any interest in looking at automatically generating C# wrapper and bindings for D code? (I'm interested myself mostly in nested templated structs/arrays rather than classes). It may not be relevant for you, but if it is please drop me an email on laeeth ... at kaleidic.io Thanks. Laeeth.
Re: Dynamic binding to the Mono runtime API
On Saturday, 3 June 2017 at 17:30:05 UTC, Jakub Szewczyk wrote: Mono runtime is a cross-platform, open-source alternative to Microsoft's .NET framework [1], and it can be embedded in other applications as a "scripting" VM, but with JIT-compilation enhanced performance and support of many languages such as C#, F# or IronPython [2]. It provides a C API, so I've bound it to D as a Derelict-based project, available at https://github.com/kubasz/derelict-mono, and as a DUB package (http://code.dlang.org/packages/derelict-mono). It currently wraps the Mono 5.0 API. There's also a simple example of calling a C# main from D code, and C# code calling a native function implemented in D. [...] Cool work! Thanks! Did u use dstep for the translation or manual labor? I just checked on my mac and it works from the get go! Cheers, Stephan
Dynamic binding to the Mono runtime API
Mono runtime is a cross-platform, open-source alternative to Microsoft's .NET framework [1], and it can be embedded in other applications as a "scripting" VM, but with JIT-compilation enhanced performance and support of many languages such as C#, F# or IronPython [2]. It provides a C API, so I've bound it to D as a Derelict-based project, available at https://github.com/kubasz/derelict-mono, and as a DUB package (http://code.dlang.org/packages/derelict-mono). It currently wraps the Mono 5.0 API. There's also a simple example of calling a C# main from D code, and C# code calling a native function implemented in D. PS: Because I don't own a Mac I have no idea what the correct paths to the Mono shared library are, so it'd be great if someone could post/create a PR of them. [1] http://www.mono-project.com/ [2] http://www.mono-project.com/docs/advanced/embedding/scripting/
