Re: SAOC Experience Report: Porting a fork-based GC
On Tuesday, 23 July 2019 at 11:28:35 UTC, Francesco Mecca wrote: On Monday, 22 July 2019 at 15:58:08 UTC, a11e99z wrote: On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote: Francesco Mecca ha written an experience report for the D thanks for answers
Re: SAOC Experience Report: Porting a fork-based GC
On Monday, 22 July 2019 at 15:58:08 UTC, a11e99z wrote: On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote: Francesco Mecca ha written an experience report for the D Blog about his SAOC 2018 project, porting Leandro Lucarella's old GC from D1 to D2. The blog: https://dlang.org/blog/wp-admin/post.php?post=2148=edit Reddit: https://www.reddit.com/r/programming/comments/cgdk1r/symmetry_autumn_of_code_experience_report_porting/ interesting idea do Mark phase through fork process. I want more technical details: - about Windows: will such idea work on Windows? does Windows support similar technique? (VirtualQueryEx/ReadProcessMemory/mmap/or_something) I have no idea about Windows. You can read about it here: https://rainers.github.io/visuald/druntime/concurrentgc.html - can GC-process run full time and just exchange with parent process about changes in memory configuration? or GC-process can not to see changes in parent process after fork? I mean GC-process can see snapshots of parent-process memory in realtime when it has just readonly-access. It could given that the bitarrays that are needed for the algorithm could be mmapped. In order to have a children running full time we would need a way to signal that a new allocation is taking place and to start a mark phase. We could experiment in this direction (it shouldn't be much work at the current state) but to me it seems overkill given that it is related to the mark routines only and not the whole garbage collection cycle. - does GC-process uses some AA for building graph of objects or set Mark-bits in situ? in last case COW makes copies so memory differs and GC-process should be forked every GC-cycle. Last case, but the needed bits could be mmapped in order to be shared. Impact is minimal. - digits in PR benchmark scary me: max pause 969ms or 260ms. why not to use Golang-GC? Go-GC is non-compacting/non-moving GC that has STW(stop the world) ~10ms and Google can spend money to build best GC that can help to D-users too. First of all I am not aware of the go-gc being provided as a library, it should be tightly coupled to the go runtime. Also, I think it uses write barriers and D doesn't have them. Moreover, the go-gc runs in background all the time, trashing the cache and stealing cpu cycles. I don't know if the situation has changed but that is a major compromise. Regarding dlang I would suggest to stop worrying about the GC and profile. The GC is highly deterministic (a property that few GC out there share) so that the user knows when it runs. Currently there are many facilities for profiling (and more could be added) so that bottlenecks caused by the GC can be solved on a case by case basis. Post such as this: https://blog.cloudflare.com/go-dont-collect-my-garbage/ shows that in real world scenarios people try to tweak the many knobs that GC implementers provide in order to adjust overall performance. The main difference is that in D you can isolate the few cases that are impacted by the GC and avoid garbage collection in that path.
Re: SAOC Experience Report: Porting a fork-based GC
On 23/07/2019 9:50 PM, Francesco Mecca wrote: On Monday, 22 July 2019 at 16:52:08 UTC, rikki cattermole wrote: I may have another try at it, if somebody can give me pointers on where the actual reading occurs in the GC. Because it has left me a bit stumped. Come talk in the IRC channel. I am there and people are super helpful. I'm one of those helpful people :)
Re: SAOC Experience Report: Porting a fork-based GC
On Monday, 22 July 2019 at 16:52:08 UTC, rikki cattermole wrote: I may have another try at it, if somebody can give me pointers on where the actual reading occurs in the GC. Because it has left me a bit stumped. Come talk in the IRC channel. I am there and people are super helpful.
Re: SAOC Experience Report: Porting a fork-based GC
On Monday, 22 July 2019 at 20:57:19 UTC, Meta wrote: On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote: > [...] A pull request to the D runtime was my final milestone. I was ready at the beginning of February, but I started to procrastinate. I’d had no previous communication with any of the reviewers and I was timorous about engaging with them. I spent a lot of time refactoring my code back and forth and delaying my pull request. At a certain point, I even considered abandoning the final milestone and providing the GC as a library. In the meantime, Rainer Scheutze published a threaded implementation of the mark phase that reduced the mark time in the GC and I lost faith in my project. [...] at the dconf I had the possibility to talk about this with many people and I think that so far we are witnessing major improvements regarding the GSOC students.
Re: SAOC Experience Report: Porting a fork-based GC
On Monday, 22 July 2019 at 20:57:19 UTC, Meta wrote: [snip] This seems like a major failure in the process that this was allowed to happen - good work almost went abandoned. How can we prevent this in future SAoC/GSoC? Without knowing what the mentor did/didn't do, an obvious answer seems like there should be a follow-up to ensure that the work done is actually getting in to the compiler/runtime/etc. To go so far and trip right at the finish line is unfortunate (glad to see that a PR is now open). I've been really liking the progress updates that the GSoC students are doing this year.
Re: SAOC Experience Report: Porting a fork-based GC
On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote: Francesco Mecca ha written an experience report for the D Blog about his SAOC 2018 project, porting Leandro Lucarella's old GC from D1 to D2. The blog: https://dlang.org/blog/wp-admin/post.php?post=2148=edit Reddit: > https://www.reddit.com/r/programming/comments/cgdk1r/symmetry_autumn_of_code_experience_report_porting/ A pull request to the D runtime was my final milestone. I was ready at the beginning of February, but I started to procrastinate. I’d had no previous communication with any of the reviewers and I was timorous about engaging with them. I spent a lot of time refactoring my code back and forth and delaying my pull request. At a certain point, I even considered abandoning the final milestone and providing the GC as a library. In the meantime, Rainer Scheutze published a threaded implementation of the mark phase that reduced the mark time in the GC and I lost faith in my project. This seems like a major failure in the process that this was allowed to happen - good work almost went abandoned. How can we prevent this in future SAoC/GSoC? Without knowing what the mentor did/didn't do, an obvious answer seems like there should be a follow-up to ensure that the work done is actually getting in to the compiler/runtime/etc. To go so far and trip right at the finish line is unfortunate (glad to see that a PR is now open).
Re: SAOC Experience Report: Porting a fork-based GC
On 23/07/2019 3:58 AM, a11e99z wrote: On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote: Francesco Mecca ha written an experience report for the D Blog about his SAOC 2018 project, porting Leandro Lucarella's old GC from D1 to D2. The blog: https://dlang.org/blog/wp-admin/post.php?post=2148=edit Reddit: https://www.reddit.com/r/programming/comments/cgdk1r/symmetry_autumn_of_code_experience_report_porting/ interesting idea do Mark phase through fork process. I want more technical details: - about Windows: will such idea work on Windows? does Windows support similar technique? (VirtualQueryEx/ReadProcessMemory/mmap/or_something) The problem with Windows is not so much reading another process memory, is the fact that you cannot create another process which is copied from your own memory space AND be able to have live threads running in it. What we can do (which will be good for things like audio and game engines) is make the GC only suspend the current thread and read from a snapshot of the heap space of the process. This we can do. Unfortunately I remembered that marking is a bit more complex than just locating memory a bit too late tonight. No wonder I never tried before now to implement this (I have evaluated it). I may have another try at it, if somebody can give me pointers on where the actual reading occurs in the GC. Because it has left me a bit stumped.
Re: SAOC Experience Report: Porting a fork-based GC
On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote: Francesco Mecca ha written an experience report for the D Blog about his SAOC 2018 project, porting Leandro Lucarella's old GC from D1 to D2. The blog: https://dlang.org/blog/wp-admin/post.php?post=2148=edit OT: this link try to login me in WordPress and to redirect to D-blog probably cuz "wp-admin" and "=edit" in URL
Re: SAOC Experience Report: Porting a fork-based GC
On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote: Francesco Mecca ha written an experience report for the D Blog about his SAOC 2018 project, porting Leandro Lucarella's old GC from D1 to D2. The blog: https://dlang.org/blog/wp-admin/post.php?post=2148=edit Reddit: https://www.reddit.com/r/programming/comments/cgdk1r/symmetry_autumn_of_code_experience_report_porting/ Should have included mention of D in the title
Re: SAOC Experience Report: Porting a fork-based GC
On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote: Francesco Mecca ha written an experience report for the D Blog about his SAOC 2018 project, porting Leandro Lucarella's old GC from D1 to D2. The blog: https://dlang.org/blog/wp-admin/post.php?post=2148=edit Reddit: https://www.reddit.com/r/programming/comments/cgdk1r/symmetry_autumn_of_code_experience_report_porting/ interesting idea do Mark phase through fork process. I want more technical details: - about Windows: will such idea work on Windows? does Windows support similar technique? (VirtualQueryEx/ReadProcessMemory/mmap/or_something) - can GC-process run full time and just exchange with parent process about changes in memory configuration? or GC-process can not to see changes in parent process after fork? I mean GC-process can see snapshots of parent-process memory in realtime when it has just readonly-access. - does GC-process uses some AA for building graph of objects or set Mark-bits in situ? in last case COW makes copies so memory differs and GC-process should be forked every GC-cycle. - digits in PR benchmark scary me: max pause 969ms or 260ms. why not to use Golang-GC? Go-GC is non-compacting/non-moving GC that has STW(stop the world) ~10ms and Google can spend money to build best GC that can help to D-users too.
Re: SAOC Experience Report: Porting a fork-based GC
On Monday, 22 July 2019 at 14:04:32 UTC, rikki cattermole wrote: https://dlang.org/blog/2019/07/22/symmetry-autumn-of-code-experience-report-porting-a-fork-based-gc/ Thanks!
Re: SAOC Experience Report: Porting a fork-based GC
https://dlang.org/blog/2019/07/22/symmetry-autumn-of-code-experience-report-porting-a-fork-based-gc/
SAOC Experience Report: Porting a fork-based GC
Francesco Mecca ha written an experience report for the D Blog about his SAOC 2018 project, porting Leandro Lucarella's old GC from D1 to D2. The blog: https://dlang.org/blog/wp-admin/post.php?post=2148=edit Reddit: https://www.reddit.com/r/programming/comments/cgdk1r/symmetry_autumn_of_code_experience_report_porting/
