Re: DMD Frontend working in WebAssembly
On Saturday, 16 October 2021 at 01:08:11 UTC, max haughton wrote: issues with references being held and confusing the GC, also. A reminder (if one were needed) that one can still have memory leaks with GC :)
Re: DMD Frontend working in WebAssembly
On Friday, 15 October 2021 at 21:58:29 UTC, Paul Backus wrote: On Friday, 15 October 2021 at 15:19:26 UTC, Imperatorn wrote: Btw, why is dmd a memory hog? Because it doesn't free memory. It just allocates, and relies on the OS to clean up at process exit. See https://www.digitalmars.com/articles/b87.html lowmem actually isn't much better sometimes. There are issues with references being held and confusing the GC, also. You can see this, if you use either massif or enable the GC. The peak heap collapses but at not much expense in runtime. Sometimes however, the GC makes no difference - I noticed this when benchmarking AliasAssign, you'd expect the old assigned-to memory to be freed but this doesn't seem to be the case empirically.
Re: DMD Frontend working in WebAssembly
On Friday, 15 October 2021 at 15:19:26 UTC, Imperatorn wrote: Btw, why is dmd a memory hog? Because it doesn't free memory. It just allocates, and relies on the OS to clean up at process exit. See https://www.digitalmars.com/articles/b87.html
Re: DMD Frontend working in WebAssembly
On Friday, 15 October 2021 at 11:52:24 UTC, hatf0 wrote: On Friday, 15 October 2021 at 07:28:55 UTC, Sebastiaan Koppe wrote: [...] Very cool! This all would not be possible with your wasm forks -- they are the saving grace here. Regarding upstreaming the patch... that will be rather challenging, but I would not mind contributing. I definitely think the WASI bindings (if not all of the files in `core.sys.wasi.*`) should be upstreamed first, as they're the simplest to implement, and then the various druntime patches should follow. GC also needs some investigation (or malloc), as I keep get spurious OOM errors. Could be because dmd is one hell of a memory hog, but who knows. Btw, why is dmd a memory hog?
Re: DMD Frontend working in WebAssembly
On Friday, 15 October 2021 at 11:52:24 UTC, hatf0 wrote: Very cool! This all would not be possible with your wasm forks -- they are the saving grace here. Nice that you got it running. GC also needs some investigation (or malloc), as I keep get spurious OOM errors. Could be because dmd is one hell of a memory hog, but who knows. So far I haven't hit those when running druntime/phobos tests, but they could be caused by pointers being hidden from GC because they sit on the wasm stack/registers. The only way to solve that would be to (somehow) have LDC put them on the shadow stack whenever they aren't. I know of some experiments where they did this with a boehm style GC on emscripten. Maybe I need to look more into how other languages (like go) handle this.
Re: DMD Frontend working in WebAssembly
On Friday, 15 October 2021 at 07:28:55 UTC, Sebastiaan Koppe wrote: On Thursday, 14 October 2021 at 22:58:37 UTC, hatf0 wrote: On Thursday, 14 October 2021 at 22:56:07 UTC, hatf0 wrote: Hi all, I've just managed to get the full DMD front-end to work in WebAssembly (with skoppe's druntime fork). Cool stuff. I recently picked up my wasm forks again and updated to wasi 12, ldc 1.25.0 and the accompanying druntime/phobos versions and got all tests to pass (those that I didn't version out). I am currently in the process of updating to latest ldc and latest druntime/phobos. Phobos has a bunch of merge conflicts, but it should be doable if I find a few hours. Then there is the process of upstreaming a 5000+ line patch and I probably need some help there. I reckon the WASI bindings are pretty standalone so those should be easy to add, but there is a sprawl of additions across almost all files due to not supporting threads, fibers and catching exceptions. Very cool! This all would not be possible with your wasm forks -- they are the saving grace here. Regarding upstreaming the patch... that will be rather challenging, but I would not mind contributing. I definitely think the WASI bindings (if not all of the files in `core.sys.wasi.*`) should be upstreamed first, as they're the simplest to implement, and then the various druntime patches should follow. GC also needs some investigation (or malloc), as I keep get spurious OOM errors. Could be because dmd is one hell of a memory hog, but who knows.
Re: DMD Frontend working in WebAssembly
On Friday, 15 October 2021 at 09:54:06 UTC, WebFreak001 wrote: On Thursday, 14 October 2021 at 22:56:07 UTC, hatf0 wrote: Hi all, I've just managed to get the full DMD front-end to work in WebAssembly (with skoppe's druntime fork). This doesn't do code-gen or anything (but it potentially could?), and has some OS-specific functionality stubbed out. No clue about GC -- haven't run into that issue, haven't thought about it yet! You can find my work here if you're interested: https://github.com/hatf0/dmd-fe-wasm-wrapper This repo also serves as a semi-decent guide on how to get started building files that target WebAssembly (specifically WASI), if you're into that. neat! Given the low footprint of WASM apps maybe this could maybe be used to have small isolated, cross-platform DMD tools that run one-shot? Could for example run the WASM binaries from the IDE and because it's WASM running in the own process, with memory that we can free, it avoids low startup times, especially on windows. Definitely -- the sky is the limit at this point. Once I figure out the funkiness that's going on with the JS WASI runtimes, I've thought of doing things like having ddoc generation occurring at the edge (i.e. CloudFlare Workers). This would allow for dynamically generation of the documentation pages for Dub packages (or even, generating them on the client-side), and would mean less infrastructure complexity overall. Or hell, doing linting or other code analysis at package submission at the edge. There's *so* much that's possible since the full front-end is operable. Do we have some WASM runner in D? We have https://code.dlang.org/packages/wasmer, which wraps the wasmer C runtime. This package /may/ need some tidying up (as wasmer is on v1.0.0 and this is still on a RC).
Re: DMD Frontend working in WebAssembly
On Thursday, 14 October 2021 at 22:56:07 UTC, hatf0 wrote: Hi all, I've just managed to get the full DMD front-end to work in WebAssembly (with skoppe's druntime fork). This doesn't do code-gen or anything (but it potentially could?), and has some OS-specific functionality stubbed out. No clue about GC -- haven't run into that issue, haven't thought about it yet! You can find my work here if you're interested: https://github.com/hatf0/dmd-fe-wasm-wrapper This repo also serves as a semi-decent guide on how to get started building files that target WebAssembly (specifically WASI), if you're into that. neat! Given the low footprint of WASM apps maybe this could maybe be used to have small isolated, cross-platform DMD tools that run one-shot? Could for example run the WASM binaries from the IDE and because it's WASM running in the own process, with memory that we can free, it avoids low startup times, especially on windows. Do we have some WASM runner in D?
Re: DMD Frontend working in WebAssembly
On Thursday, 14 October 2021 at 22:56:07 UTC, hatf0 wrote: Hi all, I've just managed to get the full DMD front-end to work in WebAssembly (with skoppe's druntime fork). This doesn't do code-gen or anything (but it potentially could?), and has some OS-specific functionality stubbed out. No clue about GC -- haven't run into that issue, haven't thought about it yet! You can find my work here if you're interested: https://github.com/hatf0/dmd-fe-wasm-wrapper This repo also serves as a semi-decent guide on how to get started building files that target WebAssembly (specifically WASI), if you're into that. Nice work guys ☀️
Re: DMD Frontend working in WebAssembly
On Thursday, 14 October 2021 at 22:58:37 UTC, hatf0 wrote: On Thursday, 14 October 2021 at 22:56:07 UTC, hatf0 wrote: Hi all, I've just managed to get the full DMD front-end to work in WebAssembly (with skoppe's druntime fork). Cool stuff. I recently picked up my wasm forks again and updated to wasi 12, ldc 1.25.0 and the accompanying druntime/phobos versions and got all tests to pass (those that I didn't version out). I am currently in the process of updating to latest ldc and latest druntime/phobos. Phobos has a bunch of merge conflicts, but it should be doable if I find a few hours. Then there is the process of upstreaming a 5000+ line patch and I probably need some help there. I reckon the WASI bindings are pretty standalone so those should be easy to add, but there is a sprawl of additions across almost all files due to not supporting threads, fibers and catching exceptions.
Re: DMD Frontend working in WebAssembly
On Thursday, 14 October 2021 at 22:56:07 UTC, hatf0 wrote: Hi all, I've just managed to get the full DMD front-end to work in WebAssembly (with skoppe's druntime fork). This doesn't do code-gen or anything (but it potentially could?), and has some OS-specific functionality stubbed out. No clue about GC -- haven't run into that issue, haven't thought about it yet! You can find my work here if you're interested: https://github.com/hatf0/dmd-fe-wasm-wrapper This repo also serves as a semi-decent guide on how to get started building files that target WebAssembly (specifically WASI), if you're into that. Oooh -- I totally forgot: for some reason, this doesn't work with the wasmer JS runtime. I haven't done any investigation into it, since it works just fine with the wasmer Rust runtime. More work needs to be done before we can fully bring DMD to the browser, but this could totally open up the idea of having something like Swift Playgrounds within the browser (or perhaps some toys like AST inspection, or whatnot).
