Re: DConf 2013 Day 2 Talk 5: A Precise Garbage Collector for D by Rainer Schütze
Interesting talk. It seems there are a few features D could provide which would make for some more powerful GCs: - Hook for when generating code which reads/writes a pointer allowing GC to put in read/write barriers where necessary. - Hook for union assignment so that the GC could emplace the correct type bits - Hook for passing pointers to extern(XXX) functions so that the GC can pin the arguments allowing a compacting GC - Hook for function call/return so that stack type-info can be stored. There's actually another way to do this which would have less overhead - at compile time the compiler could write the address range of each function along with a bitmap for possible pointers in that function to some known location. When the GC runs it could unwind the stack and look each frame pointer up in the list of functions to find the bit pattern to use. When calling a function that does some low-level stuff and might prevent the unwinding from working correctly it would be possible to switch back to conservative mode until the function returns.
Re: DConf 2013 Day 2 Talk 4: Web Development in D by Vladimir Panteleev
On Monday, 3 June 2013 at 12:14:48 UTC, Andrei Alexandrescu wrote: On reddit: http://www.reddit.com/r/programming/comments/1fkr5s/dconf_2013_day_2_talk_4_web_development_in_d_by/ On hackernews: https://news.ycombinator.com/item?id=5812723 On facebook: https://www.facebook.com/dlang.org/posts/650767074936977 On twitter: https://twitter.com/D_Programming/status/341527862815367168 Andrei Great talk! Would love to see the improvements to phobos suggested. An idea for the virtual address space problem on 32-bit: - Assuming each stack has a marker page at the end to prevent overflow, by simply exchanging the stack memory with a separate block of memory you can reduce the number of marker pages saving up to one page per fiber - Could use some fast compression method to save a not recently used stack in memory - As a last resort can save stack to a file and read it in again when it is required. Since events are queued the event loop can easily peek ahead in the queue and start loading in a stack early so that it is ready by the time that event gets to the front.
Re: DConf 2013 Day 2 Talk 4: Web Development in D by Vladimir Panteleev
On Monday, 3 June 2013 at 18:51:45 UTC, Vladimir Panteleev wrote: On Monday, 3 June 2013 at 18:07:57 UTC, Diggory wrote: Great talk! Would love to see the improvements to phobos suggested. An idea for the virtual address space problem on 32-bit: - Assuming each stack has a marker page at the end to prevent overflow, by simply exchanging the stack memory with a separate block of memory you can reduce the number of marker pages saving up to one page per fiber - Could use some fast compression method to save a not recently used stack in memory - As a last resort can save stack to a file and read it in again when it is required. Since events are queued the event loop can easily peek ahead in the queue and start loading in a stack early so that it is ready by the time that event gets to the front. Thanks! One of the main advantages of fibers over threads is the low overhead for switching - basically you save registers and change ESP. By comparison, switching to another thread requires an OS kernel system call. Copying entire stacks might negate this performance benefit. It's worth noting that it looks like the average stack size will grow for D programs in the future, as the push is made to minimize heap allocations throughout Phobos and use the stack more. Yes, although it would theoretically be possible to swap the stack by swapping the mappings rather than the memory itself, although I doubt many OSes would support that kind of functionality... I guess it's not too much to ask to use a 64-bit OS for when 1000s of connections need to be handled!
Re: dmd 2.063 released with 260 bugfixes and enhancements
On Thursday, 30 May 2013 at 17:28:49 UTC, Dmitry Olshansky wrote: 30-May-2013 21:16, Simen Kjaeraas пишет: On 2013-05-30, 17:16, Andrei Alexandrescu wrote: For the full story, mosey to the redesigned changelog: http://dlang.org/changelog.html Kudos to Andrej for this. *This* is how a great changelog looks. Joins the cheering crowd :) Nice job, Andrej! Agreed, also very satisfying to see ones own bug fixes in there :P
Re: DConf 2013 Day 2 Talk 2: Shared Libraries in D by Martin Nowak
With regard to the last point in the talk where Walter was suggesting not calling finalizers on objects whose code has been unloaded - would it not make more sense to simply call all the finalizers before unloading the library? If the finalizer is not called you will potentially get resource leaks - although there's no guarantee currently that it will be called, isn't there the assumption that if it's not called the object still exists, even if it's not referenced anywhere? ie. you still have some guarantee that if you keep allocating objects and then unreferencing them that the number of unfinalized objects will never exceed some large but fixed value.
Re: DConf 2013 Day 2 Talk 2: Shared Libraries in D by Martin Nowak
On Thursday, 30 May 2013 at 04:00:18 UTC, Steven Schveighoffer wrote: On Wed, 29 May 2013 22:12:54 -0400, Diggory digg...@googlemail.com wrote: With regard to the last point in the talk where Walter was suggesting not calling finalizers on objects whose code has been unloaded - would it not make more sense to simply call all the finalizers before unloading the library? In fact, I think in discussions after the talk (not recorded on video), we came to the same conclusion. If you are unloading the library and have any pointers to classes from that library, that is a programming error. So it should be safe to destroy all known objects from that library. -Steve Ah, good to know :)
Re: DConf 2013 Day 1 Talk 6: Concurrent Garbage Collection for D by Leandro Lucarella
On Monday, 27 May 2013 at 17:56:10 UTC, Brian Rogoff wrote: On Friday, 24 May 2013 at 19:44:19 UTC, Jonathan M Davis wrote: On Friday, May 24, 2013 20:30:54 Juan Manuel Cabo wrote: I'd like to know if there is interest in a precise garbage collector. There is interest in it, and Rainer Schütze did a talk on it at DConf. At the current pace (assuming that Andrei actually posts one on Monday even though it's a federal holiday in the US), it'll be posted on June 3rd (and if he skips Monday, then it'll probably be June 5th). And actually, the precise GC changes stand a much better chance of making it into druntime in the short term than any concurrency changes do. - Jonathan M Davis That's very promising. The lack of precise garbage collection and the unclear story with regards to programming sans-GC (maybe it's clear to someone, but not to me) is far more of a deal breaker for me than the lack of non-nullable pointers. I hope that you're right and that this gets sorted out soon. -- Brian It's actually possible to improve the precision of the GC without any additional type info. As long as you can give some unique ID to each type when you allocate it then the GC can learn the layout of that type on the fly. For example a simple algorithm would be: - When a new ID is first see create new type-info that is all pointers. - While scanning an instance of that type, if a pointer points to a value in the higher half, or a sufficiently low value which is not equal to zero, then remove this pointer from the type-info. You would have to disable this for unions, but for the rest it should work fine. Plus with more intelligent algorithms you can handle more cases. You could even save the type-info to a file and reuse it later to improve performance.
Re: DConf 2013 Day 1 Talk 6: Concurrent Garbage Collection for D by Leandro Lucarella
On Tuesday, 21 May 2013 at 20:08:16 UTC, Leandro Lucarella wrote: I'm interested in what you're describing, but I don't know how can you achieve this without fork()ing (or clone()ing in Linux). What does remap shared memory using COW in a context where fork() doesn't happen? Why do you even need shared memory if fork() doesn't happen? If remap shared memory using COW means get a different address for the same block of memory until a write happens in that block, then you can't scan the roots anymore. I'm *very* interested in your suggestion. After doing some further investigation I think I've found a fairly awesome way of doing garbage collection on windows, hopefully linux has a similar mechanism. It doesn't require memory mapped files or anything special, it can be done retroactively, and it allows a kind of reverse copy on write which is what's actually needed. When the GC is run: - Use VirtualProtect to mark all mutable memory pages as read-only - Add a vectored exception handler to handle the access violation exception - Resume the GC thread In the exception handler: - Copy the page being modified to a new page - Mark the original page as writable again - Tell the GC to use the new page instead - Continue The main problem I see is the need to synchronise between the exception handler and the GC, when it would be nice if the exception handler was as light-weight as possible. However this can be largely solved by doing finer grained synchronisation/some clever stuff. For example, the handler can add the new mapping first, then do the page copy, and only then wait for the GC to acknowledge the new mapping, so the GC has the whole time that the copying is taking place to see the change. The GC would also of course have to wait for the copying to complete before continuing but pausing the GC for a short time isn't really an issue at all.
Re: DConf 2013 Day 1 Talk 6: Concurrent Garbage Collection for D by Leandro Lucarella
On Friday, 24 May 2013 at 07:48:49 UTC, Vladimir Panteleev wrote: I don't know why you say that. The satellite process would be the same executable file as the main process, but started with a special switch (or environment variable), which will be handled by the D runtime. Since the two processes use the same image, the executable code will share the same pages in memory, resulting in a small overhead. I'm fairly sure windows only shares memory between instances of the same DLL, not executables. When implementing WinMain, you have to call the runtime initialization function, which will initialize the GC. GC initialization for the satellite process need not exit. You're still executing part of the user's code and it's going to be exceedingly hard to satisfactorily explain why the main function will be called twice except the second time it will mysteriously stop half-way through because the runtime init function doesn't return... I don't think this is true. Although the image subsystem is auto-detected by the entry point you're using (CONSOLE for main, WINDOWS for WinMain), you can specify it explicitly using the /SUBSYSTEM linker switch (/SUBSYSTEM:WINDOWS). The only time I've been able to get it to not show a console window is when using WinMain... Sorry? Could you provide an example (with a specific security software package)? Any security software with the active defense type stuff, Commodo does it for example. The things they ask you about vary and it's completely pointless IMO, but they do ask for permission for practically everything and with two processes that's twice as much to ask you about. Anyway, my main point is that it's not the kind of thing you want to impose as standard on all D applications, there may be other problems it causes depending on what the program is for. If it exists at all it should be opt-in and the default should still be as performant as possible. I've tried writing a generational GC for D that used page protection for write barriers a while ago. IIRC, I ran into performance issues (the page faults were rather expensive). Hmm, it shouldn't really be much slower than the technique Leandro was using - both rely on the exact same number of page faults - and his results looked very promising.
Re: DConf 2013 Day 1 Talk 6: Concurrent Garbage Collection for D by Leandro Lucarella
On Friday, 24 May 2013 at 09:56:25 UTC, Diggory wrote: The only time I've been able to get it to not show a console window is when using WinMain... OK, nvm, it seems it's just that the SUBSYSTEM setting in VisualD has no effect, you have to set it through a .def file.
Re: DConf 2013 Day 1 Talk 6: Concurrent Garbage Collection for D by Leandro Lucarella
On Friday, 24 May 2013 at 10:20:12 UTC, Vladimir Panteleev wrote: Leandro wasn't using Windows ;) True but assuming windows isn't completely degenerate the overhead of a page fault is going to be pretty close to identical on the same cpu architecture. On Friday, 24 May 2013 at 11:15:21 UTC, Artur Skawina wrote: A page fault per every page written to between every GC run + a user space callback for every such fault + a page sized copy every time is not going to perform very well... (There are ways to avoid these costs, but no idea if it's easily doable on windows...) artur It's only between the time a GC run is started and the time it finishes, exactly the same as in the talk. A generational GC should be able to do complete mark cycles fairly infrequently.
Re: DConf 2013 Day 1 Talk 6: Concurrent Garbage Collection for D by Leandro Lucarella
On 64-bit windows there is also the GetWriteWatch function which lets you access the dirty flag in the page table = no page faults = super efficient concurrent generational GC. Just a shame it doesn't exist on 32-bit systems for some reason.
Re: DConf 2013 Day 1 Talk 6: Concurrent Garbage Collection for D by Leandro Lucarella
On Thursday, 23 May 2013 at 12:39:04 UTC, Vladimir Panteleev wrote: On Tuesday, 21 May 2013 at 04:52:25 UTC, Diggory wrote: Either way, at least on windows the separate process would have to be persistent as creating a new process has a lot more overhead attached to it than on posix systems. Implicitly creating a child process is also something a D programmer might not want, again this is more windows specific where processes do not have the same strict hierarchy as on posix. What would be some of the potential downsides of a long-running satellite process? A concurrent GC would probably be opt-in or opt-out, either way. You'd either have to distribute a separate .exe for the GC process or embed the second .exe inside the first and extract it at runtime to a temporary location, or call the same .exe with some flag which tells it to turn into a GC. None of those options are particularly nice, all of them require significant interference from the GC, it's not just a case of calling some function to start the GC. This is especially a problem in cases where the runtime can't insert the init handler such as when using WinMain, which you currently have to do unless you want a console window to pop up. Next there's the fact that you have a separate top level process for every D process. This is going to cause problems with security software since you now have to give both processes permissions in order to run. In addition you have to consider the case where one or other of the two processes is paused or killed unexpectedly - you don't want orphaned GC processes hanging around wasting resources.
Re: HibernateD and DDBC - ORM and DB abstraction layer for D
On Tuesday, 21 May 2013 at 22:24:06 UTC, Nick Sabalausky wrote: On Mon, 06 May 2013 11:14:56 +0200 Kagamin s...@here.lot wrote: On Monday, 29 April 2013 at 09:38:10 UTC, David wrote: Null blows up your code, doesn't. There's no difference between null and empty string in D. That's not true: assert( !is null); // Passes Or did I misunderstand what you meant? Strings are slices which are a pointer and a length. I think a slice compares equal to null only if the pointer part is null. However, a slice with a null pointer and a length of zero is still a valid empty slice, which is slightly odd behaviour compared to other languages... An empty string literal initialises the pointer to non-null because string literals are null terminated, so the memory block actually has a length of one, even though the slice has length zero.
Re: DConf 2013 Day 1 Talk 6: Concurrent Garbage Collection for D by Leandro Lucarella
On Tuesday, 21 May 2013 at 20:08:16 UTC, Leandro Lucarella wrote: I'm interested in what you're describing, but I don't know how can you achieve this without fork()ing (or clone()ing in Linux). What does remap shared memory using COW in a context where fork() doesn't happen? Why do you even need shared memory if fork() doesn't happen? If remap shared memory using COW means get a different address for the same block of memory until a write happens in that block, then you can't scan the roots anymore. I'm *very* interested in your suggestion. I do mean mapping the same physical block of memory into two virtual memory blocks, such that a write will cause the OS to make a copy and sever the link. Given that we're dealing with significantly large blocks of memory all in multiples of the page size, it should be a simple matter to map from one copy of the memory to the other and back again using basic pointer arithmetic, so scanning the roots should not be a problem. You need shared memory because a file handle is needed when calling mmap to enable the COW functionality, and shared memory lets you get a file handle to a block of memory.
Re: DConf 2013 Day 1 Talk 6: Concurrent Garbage Collection for D by Leandro Lucarella
On Monday, 20 May 2013 at 13:55:05 UTC, Regan Heath wrote: On Mon, 20 May 2013 13:50:25 +0100, Andrei Alexandrescu seewebsiteforem...@erdani.org wrote: On reddit: http://www.reddit.com/r/programming/comments/1eovfu/dconf_2013_day_1_talk_6_concurrent_garbage/ This may be the Windows Copy On Write feature mentioned in the QA at the end: http://support.microsoft.com/kb/103858 .. but it's not clear to me how useful this is for fork emulation or similar. R Fork isn't needed at all really in the technique described, this is all that's needed: - Map a copy of the memory using copy-on-write - Run some code concurrently It just happens that fork does both of these things, but you can equally well do the two things using separate calls. In fact you should be able to avoid the deadlock issue by not using fork but just remapping some shared memory using copy on write. The GC can exist in a separate thread which pauses itself after every run. To run the GC it's then just a case of: - stop the world - copy registers to stack - remap shared memory using COW - resume the world - resume the GC thread And that would work on all modern OSes, plus you don't have the overhead of creating a new process or even a new thread. Also immutable memory doesn't need to be mapped, the GC thread can access it directly.
Re: DConf 2013 Day 1 Talk 6: Concurrent Garbage Collection for D by Leandro Lucarella
On Tuesday, 21 May 2013 at 00:00:13 UTC, deadalnix wrote: On Monday, 20 May 2013 at 22:52:33 UTC, Diggory wrote: And that would work on all modern OSes, plus you don't have the overhead of creating a new process or even a new thread. Also immutable memory doesn't need to be mapped, the GC thread can access it directly. Copy on WRITE usually don't happen on immutable memory. I never said it did... I said that when using fork the immutable memory must be mapped into the new process, when using a thread it does not since threads share the same address space.
Re: DConf 2013 Day 1 Talk 6: Concurrent Garbage Collection for D by Leandro Lucarella
On Tuesday, 21 May 2013 at 04:26:18 UTC, Vladimir Panteleev wrote: On Monday, 20 May 2013 at 13:55:05 UTC, Regan Heath wrote: On Mon, 20 May 2013 13:50:25 +0100, Andrei Alexandrescu seewebsiteforem...@erdani.org wrote: On reddit: http://www.reddit.com/r/programming/comments/1eovfu/dconf_2013_day_1_talk_6_concurrent_garbage/ This may be the Windows Copy On Write feature mentioned in the QA at the end: http://support.microsoft.com/kb/103858 Yes, basically. I can't find where I've read that mapping COW memory from within the same process is only supported on NT versions. However, doing it from within the same process is not practical anyway, since that would imply at least halving the address space. Either way, at least on windows the separate process would have to be persistent as creating a new process has a lot more overhead attached to it than on posix systems. Implicitly creating a child process is also something a D programmer might not want, again this is more windows specific where processes do not have the same strict hierarchy as on posix. On 64-bit systems there shouldn't be a problem with address space, and even on 32-bit systems, the objects which take up the vast majority of the address space are usually arrays of primitive types which don't need to be scanned by the GC. Certainly for 64-bit systems I think it's worth at least doing some performance comparisons, and I think using a thread will turn out to be more efficient. If it turns out that there are merits to both options then having a GC option to use a process or a thread might make sense.
Re: DConf 2013 Day 1 Talk 2: Copy and Move Semantics in D by Ali Cehreli
On Sunday, 12 May 2013 at 14:16:26 UTC, Andrei Alexandrescu wrote: I was also thinking mon/wed/fri would be a great schedule! Andrei +1
Re: DConf 2013 Day 1 Talk 2: Copy and Move Semantics in D by Ali Cehreli
He's serious. If you post them all at once, then each video gets minimal impact. A lot of people will look at one, maybe two, and then not bother with the rest, because all of them showed up at once, whereas if they're posted over a longer period of time, then each video will have larger a impact, because it'll be showing up by itself, and it increases the chances of more casual people viewing it. Yes. This sucks for those who didn't get the chance to go to the conference and who will definitely view all of them regardless of how quickly they're released, but it produces better PR for the language this way. We'll keep getting new posts on reddit or wherever over a period of several weeks as opposed to it being more of a blip on people's radar and then gone. - Jonathan M Davis If there is such a long delay all that will happen is that people will forget/lose interest and stop watching the talks. It makes much more sense to release one each day or couple of days both to keep people interested and to avoid irritating everyone...
Re: Standalone opengl bindings
On Sunday, 5 May 2013 at 08:52:27 UTC, Jacob Carlborg wrote: On 2013-05-05 05:58, Diggory wrote: I wrote this mainly for my own benefit, but I figured it might be useful to someone else. It's a small program which generates opengl bindings for D directly from the specification. https://github.com/Diggsey/OpenGL-D-Bindings It generates a single file gl.d which by default will do no more or less than what you'd get by including gl/gl.h. This will give you access to GL 1.1 functions. Sorry, but I can't understand why you would want to have this instead of Derelict. A few reasons: - It's one file that does JUST the opengl stuff. With derelict, the functions it provides overlap with the platform SDK. Sure you can prevent conflicts by aliasing them but it's messy and a waste. - Derelict requires countless files over two different sub-projects just to get simple opengl support. You would certainly want to build it separately and then link it in, with this it's just a case of adding one file to your project. - I'm building on top of this, this way it's one fewer dependency to worry about. - Because it's generated automatically from the spec it will always be up to date (currently 4.3, whereas derelict is only on 2.1!).
Re: Standalone opengl bindings
On Sunday, 5 May 2013 at 10:47:39 UTC, David wrote: I don't understand it, which function e.g. conflicts? Derelict defines a bunch of platform specific functions (wgl***, some pixel format functions, etc.) which belong in the platform sdk, not in the opengl bindings. Well, for Derelict I have to add ~5 files (all of the opengl subfolder) Whereas with this I have to add 1 file which declares precisely those functions which I need and no more. - I'm building on top of this, this way it's one fewer dependency to worry about. I use Derelict with a git submodule and only compile the folders in I need, works great. I'm sure it does, that doesn't stop it from being a dependency. As I said I did this for my own benefit, I'm not forcing people to use it. I think you're misinformed, Derelict always supported more than that, current version supports 4.3 as well Ah I was looking at Derelict rather than Derelict2 apparently - slightly confusing having the latest version in a branch...
Standalone opengl bindings
I wrote this mainly for my own benefit, but I figured it might be useful to someone else. It's a small program which generates opengl bindings for D directly from the specification. https://github.com/Diggsey/OpenGL-D-Bindings It generates a single file gl.d which by default will do no more or less than what you'd get by including gl/gl.h. This will give you access to GL 1.1 functions. You can then call loadGL passing in the opengl version and the names of any extensions you want to use and it will automatically load in the relevant functions. It does this at compile time so at runtime it just loads precisely those functions which you asked for. It provides only the platform independent part of opengl - it expects wgl***/glX*** functions to be provided by the platform sdk. Unfortunately the built in windows bindings for D are fairly useless, but I've forked druntime and phobos to use the windows bindings from the dsource bindings project instead, which are far better: https://github.com/Diggsey/druntime https://github.com/Diggsey/phobos/tree/win32-bindings
Re: DConf 2013 on twitter
Just curious if there were any plans to put videos/slideshows from the presentations online after the conference?
