Re: GC deadlocks on linux
On Mon, 02 Mar 2015 08:15:12 +, Kagamin wrote: On Saturday, 28 February 2015 at 05:32:51 UTC, ketmar wrote: On Sat, 28 Feb 2015 06:09:16 +0100, Martin Nowak wrote: Meanwhile the author of daemonized came up with another idea, using exec instead of fork. https://github.com/NCrashed/daemonize/issues/2 ahem. http://forum.dlang.org/post/mc35ap$2dvo$5...@digitalmars.com Bug 6846 is supposedly caused by usage of GC. Can't recent spawnProcess from phobos be used to start a process? It works around GC after fork sufficiently enough. it doesn't workaround in any way, it does exactly what it is supposed to do: it running *new* process. no workarounds needed for that, and it does no allocations in child process after forking point. signature.asc Description: PGP signature
Re: GC deadlocks on linux
On Saturday, 28 February 2015 at 05:32:51 UTC, ketmar wrote: On Sat, 28 Feb 2015 06:09:16 +0100, Martin Nowak wrote: Meanwhile the author of daemonized came up with another idea, using exec instead of fork. https://github.com/NCrashed/daemonize/issues/2 ahem. http://forum.dlang.org/post/mc35ap$2dvo$5...@digitalmars.com Bug 6846 is supposedly caused by usage of GC. Can't recent spawnProcess from phobos be used to start a process? It works around GC after fork sufficiently enough.
Re: GC deadlocks on linux
I mean, if it used GC after fork, like in bug 6846, that would be untidy (it's complicated because GC is always a temptation).
Re: GC deadlocks on linux
On Sat, 28 Feb 2015 06:09:16 +0100, Martin Nowak wrote: Meanwhile the author of daemonized came up with another idea, using exec instead of fork. https://github.com/NCrashed/daemonize/issues/2 ahem. http://forum.dlang.org/post/mc35ap$2dvo$5...@digitalmars.com signature.asc Description: PGP signature
Re: GC deadlocks on linux
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/18/2015 09:27 PM, Byron Heads wrote: I have a medium size daemon application that uses several threads, libasync, and daemonize. On windows it runs correctly with GC enabled, but on linux the GC causes a deadlock while allocating memory. Have you been able to resolve the issue? There were a number of suggestions in the thread, but we never heard back from you. Meanwhile the author of daemonized came up with another idea, using exec instead of fork. https://github.com/NCrashed/daemonize/issues/2 -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBAgAGBQJU8U14AAoJELJzgRYSuxk5wnEP/R6nFZECdFsNSFK/Bmsys5WI RmYt8FyS5WJt/2tM/jKgN6k5WJLP+ZniSHPTMdWO4pGxJTaK6zGvMXVYEHPFgIDp RPcJhZ/J1KXyfOyR3by/23opvGaWqAJXIx3yBfYBhvjNNpjQrGxqQEwdT4sbDvz1 AtUZ+Tc1CoBo+3hRNESyk9FNa3c58adu5SWkpErJezrg4TFzYg7sKytbZEtb5T6U OWLVlMqY8Q6AyskbEUqxfQt8lba9fM5Eeg8gbzf7ShCZwrzviBkrhdTWrodv8kHo HeMZmRIBKmz/L0Ce/7NSV+U0htEB+DAB2LSYRKhy/qYwoLHi8UNruqv3pf4PU7Ly 4atq5XTcFNS/ywL8qkP8OMmcdBN0pBQNsDfmG2w1DsWANtK/cLqwS50O8TXCxpv1 hlQ/CgRD6jJWujleaDOhuOZWYzJ0Xwk1a5BGjoO5MkQaHeRZgalSN4rkmoPZQz1t H2kA03JMTVkEx2AllPKHdQCcNEV0wpI7sJNRWh9kewtdzW0SQtlV2NYM9U2gXPJe u+zYuWRLGpWOrqItzZGt+Z+NSrNziV0cO/IpogQRjMPtLXsRgaFofzaOO81l7OWk SWnE3bT2PO2sdZ0Z3uf/c+KAosDJ+5AhD9FXmuEemIc4S4/1yKJSVj7BtsomBxE+ hGfLvRuNjUDQil+WjC0t =MNgH -END PGP SIGNATURE-
Re: GC deadlocks on linux
I think, it's better to diagnose, what problem the program encounters, than to explain, what happens in kernel and glibc. The first step is to see, where it hangs and get a stacktrace.
Re: GC deadlocks on linux
On Thursday, 19 February 2015 at 22:12:03 UTC, Steven Schveighoffer wrote: I'm not sure what the issue here is, but I don't think forking is as unstable as you seem to think, or maybe I'm reading this wrong. I can agree there are many gotchas with forking. You need to very carefully configure what happens to resources before/after forking. So if you use third party libraries with threading all bets are off... If you fork early in the process' lifespan, before acquiring resources, then it is much easier... http://linux.die.net/man/2/fork
Re: GC deadlocks on linux
On Fri, 20 Feb 2015 09:04:29 +, Ola Fosheim Grøstad wrote: If you fork early in the process' lifespan, before acquiring resources, then it is much easier... and even if `fork()` is the first line of code in `main()`, there cannot be any guarantees. there can be module constructor doing something, or 3rd party library that already initialized something, or... you got the idea. ;-) signature.asc Description: PGP signature
Re: GC deadlocks on linux
On Fri, 20 Feb 2015 08:03:00 +, Kagamin wrote: I think, it's better to diagnose, what problem the program encounters, than to explain, what happens in kernel and glibc. The first step is to see, where it hangs and get a stacktrace. this way of thinking is exactly why i recommend to avoid `fork()` unless one can explain *everything* it does. signature.asc Description: PGP signature
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 20:27:08 UTC, Byron Heads wrote: Adding core.memory.GC.disable; to main causes the application to work correctly (and quickly till it runs out of memory :D ) GC.disable shouldn't run OOM, BTW. http://dlang.org/phobos/core_memory.html#.GC.disable
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 20:41:12 UTC, Dicebot wrote: Any chance you are using gdm-3.12.x? I was so mad when I have encountered this: https://issues.dlang.org/show_bug.cgi?id=4890 Indeed, maybe http://dlang.org/phobos-prerelease/core_thread.html#.thread_setGCSignals might help. We should probably try to improve druntime to only use a single signal for suspending and resuming.
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 20:27:08 UTC, Byron Heads wrote: I have a medium size daemon application that uses several threads, libasync, and daemonize. On windows it runs correctly with GC enabled, but on linux the GC causes a deadlock while allocating memory. Can you reliably reproduce the deadlock? If so please attach a gdb to the deadlocked process and provide us back traces of all threads? If you can share the code (maybe privately), that would help as well.
Re: GC deadlocks on linux
On Friday, 20 February 2015 at 12:47:58 UTC, ketmar wrote: 3rd party library that already initialized something, or... you got the idea. ;-) Yeah, either use plain C or avoid 3rd party libraries... I guess that includes phobos ;)
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 20:27:08 UTC, Byron Heads wrote: System: DMD 2.066.1 Can you also try the latest beta please, maybe we already fixed something. http://forum.dlang.org/post/54e49e2d.2010...@dawg.eu
Re: GC deadlocks on linux
On Fri, 20 Feb 2015 14:33:29 +, Ola Fosheim Grøstad wrote: On Friday, 20 February 2015 at 12:47:58 UTC, ketmar wrote: 3rd party library that already initialized something, or... you got the idea. ;-) Yeah, either use plain C or avoid 3rd party libraries... I guess that includes phobos ;) maybe even druntime to some extent. and, of course, no other D libraries, as who knows what they can do in their module initialisers... signature.asc Description: PGP signature
Re: GC deadlocks on linux
On Friday, 20 February 2015 at 14:33:31 UTC, Ola Fosheim Grøstad wrote: Yeah, either use plain C or avoid 3rd party libraries... I guess that includes phobos ;) AFAIK, in early days of unix there were no threads, processes were single-threaded, fork was the way to concurrency and exec was the most efficient way to run a program in memory-constrained conditions of 70s (kbytes of RAM!). Plain unix-like single-threaded processes, plain C heap, which didn't serialize access, because no threads, and worked fine with just virtual memory, which fork got right. That's the model, which should work the best with fork+exec.
Re: GC deadlocks on linux
On Friday, 20 February 2015 at 22:07:56 UTC, Kagamin wrote: AFAIK, in early days of unix there were no threads, processes were single-threaded, fork was the way to concurrency and exec was the most efficient way to run a program in memory-constrained conditions of 70s (kbytes of RAM!). Plain unix-like single-threaded processes, plain C heap, which didn't serialize access, because no threads, and worked fine with just virtual memory, which fork got right. That's the model, which should work the best with fork+exec. Indeed, actually, not only for the early days, but for the first 20 years or so! :-D Single thread, C, fork and a pipe + limited use of shared memory is a quite clean model. The underlying principle in Unix is to build complex software from many isolated simple units. This robust philosophy somehow got lost in the quest for bleeding edge.
Re: GC deadlocks on linux
On Friday, 20 February 2015 at 15:17:22 UTC, Martin Nowak wrote: Might want to try using libasync without multiple threads. http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them Or you temporarily disable the GC before forking and enable it again afterwards.
Re: GC deadlocks on linux
On 02/18/2015 09:35 PM, Byron Heads wrote: I am in the daemonize library https://github.com/NCrashed/daemonize Might want to try using libasync without multiple threads. http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them
Re: GC deadlocks on linux
On Fri, 20 Feb 2015 22:29:04 +, Ola Fosheim Grøstad wrote: This robust philosophy somehow got lost in the quest for bleeding edge. i still missing it. sure, we can write our code in this style today, but with all that libraries that can create threads without you knowing about it (heh, have you ever used `getaddrinfo_a(GAI_NOWAIT)`? a hideous solution!)... signature.asc Description: PGP signature
Re: GC deadlocks on linux
On 2/19/15 12:01 PM, ketmar wrote: On Thu, 19 Feb 2015 16:33:58 +, Byron Heads wrote: Now I am not sure. This code runs correctly: as i told you before, `fork()` is hard. you can experiment for monthes seeing strange bugs here and there, and seeing no bugs, and strange bugs, and... there are alot of things going on under the hood, and alot of things aren't. strictly speaking you *CAN'T* get cloned process in the same state as it's parent. it may work, though, if stars are in a good shape. Just as an example, Sociomantic uses a concurrent GC based on forking. I'm not sure what the issue here is, but I don't think forking is as unstable as you seem to think, or maybe I'm reading this wrong. I can agree there are many gotchas with forking. -Steve
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 21:21:11 UTC, Byron Heads wrote:
On Wednesday, 18 February 2015 at 21:05:10 UTC, Byron Heads
wrote:
On Wednesday, 18 February 2015 at 20:55:56 UTC, ketmar wrote:
On Wed, 18 Feb 2015 20:35:44 +, Byron Heads wrote:
On Wednesday, 18 February 2015 at 20:33:40 UTC, ketmar wrote:
On Wed, 18 Feb 2015 20:27:07 +, Byron Heads wrote:
are you forking? ;-)
I am in the daemonize library
https://github.com/NCrashed/daemonize
can you drop that and just let the program run on foreground?
i suspect
that it may solve your problem. you can detach your program
with setsid
and to avoid forking.
`fork()` is a very fragile thing, and it may be the source of
heisenbugs.
stop `fork()`ing may be the easiest solution (if it solves
something, of
course ;-).
By passing daemonize with the GC enabled is working.. going to
dig into the signals and see if thats it.
My guess is this is going to be related to forking, going to
see if I can make a test case.
Now I am not sure. This code runs correctly:
import std.stdio;
import std.concurrency;
import core.sys.posix.unistd;
import core.sys.posix.sys.stat;
import core.memory;
import std.c.linux.linux;
extern(C) nothrow {
int __libc_current_sigrtmin();
int close(int rd);
}
void foo(Tid tid) {
writeln(1);
foreach(i; 0..1024) {
ubyte[] buffer = new ubyte[](8_192);
auto f = new float[1024];
GC.collect;
}
writeln(2);
send(tid, true);
}
extern(C) void sigsig(int sig) nothrow pure @system @nogc {
}
void main() {
auto pid = fork();
if(pid 0) {
writeln(fork failed);
} if (pid 0) {
writeln(Spawned , pid);
return;
}
umask(0);
auto sid = setsid();
if(sid 0) {
writeln(failed to set sid);
}
assert(signal(SIGABRT, sigsig) != SIG_ERR);
assert(signal(SIGTERM, sigsig) != SIG_ERR);
assert(signal(SIGQUIT, sigsig) != SIG_ERR);
assert(signal(SIGINT, sigsig) != SIG_ERR);
assert(signal(SIGQUIT, sigsig) != SIG_ERR);
assert(signal(SIGHUP, sigsig) != SIG_ERR);
assert(signal(__libc_current_sigrtmin+1, sigsig) != SIG_ERR);
writeln(spawning);
spawn(foo, thisTid);
spawn(foo, thisTid);
spawn(foo, thisTid);
foreach(i; 0..1024) {
auto x = new long[1024];
GC.collect;
}
receiveOnly!bool;
receiveOnly!bool;
receiveOnly!bool;
GC.collect;
writeln(done);
}
Re: GC deadlocks on linux
On Thu, 19 Feb 2015 16:33:58 +, Byron Heads wrote: Now I am not sure. This code runs correctly: as i told you before, `fork()` is hard. you can experiment for monthes seeing strange bugs here and there, and seeing no bugs, and strange bugs, and... there are alot of things going on under the hood, and alot of things aren't. strictly speaking you *CAN'T* get cloned process in the same state as it's parent. it may work, though, if stars are in a good shape. i can give you a simple test that will tell you if you should avoid `fork ()` in your code: unless you can explain what is going on in kernel, glibc, pthreads and druntime exactly in the moment of forking, without looking to mans, documentation and source code of all components -- avoid using `fork()`. i can explain ~2/3, so i'm not using `fork()` in my code. ;-) signature.asc Description: PGP signature
Re: GC deadlocks on linux
On Thu, 19 Feb 2015 17:12:02 -0500, Steven Schveighoffer wrote: On 2/19/15 12:01 PM, ketmar wrote: On Thu, 19 Feb 2015 16:33:58 +, Byron Heads wrote: Now I am not sure. This code runs correctly: as i told you before, `fork()` is hard. you can experiment for monthes seeing strange bugs here and there, and seeing no bugs, and strange bugs, and... there are alot of things going on under the hood, and alot of things aren't. strictly speaking you *CAN'T* get cloned process in the same state as it's parent. it may work, though, if stars are in a good shape. Just as an example, Sociomantic uses a concurrent GC based on forking. it's completely different thing. *this* use of fork -- to make snapshot of running application -- is perfectly ok. I'm not sure what the issue here is, but I don't think forking is as unstable as you seem to think, or maybe I'm reading this wrong. I can agree there are many gotchas with forking. forking is stable -- in the sense that it's doing exactly what it was asked to do. but that can be something different from what someone *think* it should do. `fork()` is described as making a copy of the running process, and it's right... for some extent. but what copy is exactly? all threads except the one are lost. pid and tid was changed. what about signal masks? and thousands of other things which are taken as granted -- thread locks, for example, which can store tid of the thread that was aquired the lock? what about other libraries, which can do things you don't know about? and so on... sure, `fork()` is perfectly usable... when you know what is going on and can foresee all consequences. but it's hard, and i'd better recommend to avoid `fork()` at all if programmer is not closely familiar with the thing. in most cases avoiding `fork()` is much easier than trying to find out what (can/is) going wrong with the program that was working ok without forking. what i trying to tell OP is that if his code working ok without forking, it's much easier to just go this way and forget about `fork()`, not wasting time to find out what is wrong with it. signature.asc Description: PGP signature
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 21:05:10 UTC, Byron Heads wrote: On Wednesday, 18 February 2015 at 20:55:56 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:35:44 +, Byron Heads wrote: On Wednesday, 18 February 2015 at 20:33:40 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:27:07 +, Byron Heads wrote: are you forking? ;-) I am in the daemonize library https://github.com/NCrashed/daemonize can you drop that and just let the program run on foreground? i suspect that it may solve your problem. you can detach your program with setsid and to avoid forking. `fork()` is a very fragile thing, and it may be the source of heisenbugs. stop `fork()`ing may be the easiest solution (if it solves something, of course ;-). By passing daemonize with the GC enabled is working.. going to dig into the signals and see if thats it. My guess is this is going to be related to forking, going to see if I can make a test case.
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 21:21:11 UTC, Byron Heads wrote: On Wednesday, 18 February 2015 at 21:05:10 UTC, Byron Heads wrote: On Wednesday, 18 February 2015 at 20:55:56 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:35:44 +, Byron Heads wrote: On Wednesday, 18 February 2015 at 20:33:40 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:27:07 +, Byron Heads wrote: are you forking? ;-) I am in the daemonize library https://github.com/NCrashed/daemonize can you drop that and just let the program run on foreground? i suspect that it may solve your problem. you can detach your program with setsid and to avoid forking. `fork()` is a very fragile thing, and it may be the source of heisenbugs. stop `fork()`ing may be the easiest solution (if it solves something, of course ;-). By passing daemonize with the GC enabled is working.. going to dig into the signals and see if thats it. My guess is this is going to be related to forking, going to see if I can make a test case. Might be related: https://issues.dlang.org/show_bug.cgi?id=6846
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 20:33:40 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:27:07 +, Byron Heads wrote: are you forking? ;-) I am in the daemonize library https://github.com/NCrashed/daemonize
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 20:41:12 UTC, Dicebot wrote: Any chance you are using gdm-3.12.x? I was so mad when I have encountered this: https://issues.dlang.org/show_bug.cgi?id=4890 Dont think so $dpkg --get-selections | grep gdm doesn't return anything also running via ssh
Re: GC deadlocks on linux
On Wed, 18 Feb 2015 20:27:07 +, Byron Heads wrote: are you forking? ;-) signature.asc Description: PGP signature
Re: GC deadlocks on linux
Any chance you are using gdm-3.12.x? I was so mad when I have encountered this: https://issues.dlang.org/show_bug.cgi?id=4890
Re: GC deadlocks on linux
On Wed, 18 Feb 2015 20:35:44 +, Byron Heads wrote: On Wednesday, 18 February 2015 at 20:33:40 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:27:07 +, Byron Heads wrote: are you forking? ;-) I am in the daemonize library https://github.com/NCrashed/daemonize can you drop that and just let the program run on foreground? i suspect that it may solve your problem. you can detach your program with setsid and to avoid forking. `fork()` is a very fragile thing, and it may be the source of heisenbugs. stop `fork()`ing may be the easiest solution (if it solves something, of course ;-). signature.asc Description: PGP signature
Re: GC deadlocks on linux
On Wed, 18 Feb 2015 20:35:44 +, Byron Heads wrote: On Wednesday, 18 February 2015 at 20:33:40 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:27:07 +, Byron Heads wrote: are you forking? ;-) I am in the daemonize library https://github.com/NCrashed/daemonize p.s. and check if you don't disable signals. druntime uses SIGUSR1 to communicate with threads (even if you aren't using more that one thread in your code). it doesn't on windows though (there are no signals there ;- ), but your posix code can accidentally block/intercept the signals druntime uses for internal bookkeeping. signature.asc Description: PGP signature
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 20:55:56 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:35:44 +, Byron Heads wrote: On Wednesday, 18 February 2015 at 20:33:40 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:27:07 +, Byron Heads wrote: are you forking? ;-) I am in the daemonize library https://github.com/NCrashed/daemonize can you drop that and just let the program run on foreground? i suspect that it may solve your problem. you can detach your program with setsid and to avoid forking. `fork()` is a very fragile thing, and it may be the source of heisenbugs. stop `fork()`ing may be the easiest solution (if it solves something, of course ;-). By passing daemonize with the GC enabled is working.. going to dig into the signals and see if thats it.
Re: GC deadlocks on linux
On Wed, 18 Feb 2015 21:21:10 +, Byron Heads wrote: My guess is this is going to be related to forking, going to see if I can make a test case. it's better to avoid forking at all. it's *very* hard to do forking right, it's almost impossible to do it right even in single-threaded app, and it's impossible to do it right in multithreaded app. `fork()` is a can full of worms, it may surprise you even on seemingly simple code. i suggest you to just drop forking and use shell script instead. you can lost alot of time trying to force your forked code to work right, and then it will break again on any move (or even when moon will change it's phase). signature.asc Description: PGP signature
Re: GC deadlocks on linux
On Wed, 18 Feb 2015 21:21:10 +, Byron Heads wrote: p.s. if you really-really-really need to avoid shell scripts, you can `exec` your program with STDIN, STDOUT and STDERR redirected to /dev/ null, passing some special argument to it. i.e. it shouldn't fork, just exec itself again with added command-line option. just make sure that you did `setsid()` and so on, so the second instance is completely detached. signature.asc Description: PGP signature
