Re: Forks, slices and threads: Can you make GSlice deadlock?
On Thu, Dec 6, 2012 at 3:25 PM, David Nečas y...@physics.muni.cz wrote: So, they are all trying to lock allocator-slab_mutex in GSlice. But nothing seems to hold it. Could that be messed up by forking somehow? Could you please advise how to debug it further, to rule out the possibility of GLib bug, if nothing else? See https://bugzilla.gnome.org/show_bug.cgi?id=679683 g_test_trap_fork is just a bad idea, doubly so with threads (and every glib program is using threads nowadays) ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Forks, slices and threads: Can you make GSlice deadlock?
On Fri, Dec 07, 2012 at 05:53:57AM -0500, Matthias Clasen wrote: See https://bugzilla.gnome.org/show_bug.cgi?id=679683 g_test_trap_fork is just a bad idea, doubly so with threads (and every glib program is using threads nowadays)a Thanks, somehow, I missed that bug. Regards, Yeti ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Forks, slices and threads: Can you make GSlice deadlock?
On Thu, 6 Dec 2012 21:25:47 +0100 David Nečas y...@physics.muni.cz wrote: can g_slice_alloc() be made to deadlock simply by some bad sequence of GLib function calls, considering the calling program does not, of course, hold any GLib lock explicitly? (Without a GLib bug, that is.) I am starting to suspect a problem in GSlice interaction with threads. But I cannot report anything to bugzilla because I am unable to get to the core of the problem. My program (test program for a library) does g_test_trap_fork() and the child creates worker threads with g_thread_new(), sends them tasks with GAsyncQueue and cancels the tasks using GCancellables. Occasionally, a seemingly innocent g_thread_new() call deadlocks in g_slice_alloc(), see the backtrace below for how and where exactly. If it happens, it happens in the child soon after forking. I canNOT reproduce any deadlock if: - G_SLICE=always-malloc is set, - g_test_trap_fork() is not used and the test is run directly in the main program, - under valgrind (also, it reports no errors), - I print anything to stderr in g_slice_alloc() – infuriating, but so it works. I am sure you already know this, but assuming you are using the POSIX backend, POSIX does not allow you to fork() from a multi-threaded process and then call any non-async-signal-safe functions in the child (all you are really supposed to do is set up your pipes and then exec()). So for your scheme to work, the parent before forking must be single threaded. The kind of symptom you report is what you would expect if it is multi-threaded. Chris ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Forks, slices and threads: Can you make GSlice deadlock?
On Fri, Dec 07, 2012 at 08:44:01PM +, Chris Vine wrote: ... So for your scheme to work, the parent before forking must be single threaded. As I have learnt the hard way, this is not something I can ensure even the program does not use any multi-threading itself because libraries may. Even apparently very innocent programs can exhibit weird threading-related bugs. And while it is a sort of gray area, starting to use threads internally is rarely considered an ABI breakage. So, unfortunately, the conditions you listed can be reduced to ‘it will never work’ in practice. Anyway, I do not attempt to do this kind of fork not-exec in real Gtk+ applications. So I'm looking forward to a better solution for unit tests which is discussed in the bugzilla bug mentioned by Matthias. Regards, Yeti ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Forks, slices and threads: Can you make GSlice deadlock?
On Fri, 7 Dec 2012 22:37:27 +0100 David Nečas y...@physics.muni.cz wrote: On Fri, Dec 07, 2012 at 08:44:01PM +, Chris Vine wrote: ... So for your scheme to work, the parent before forking must be single threaded. As I have learnt the hard way, this is not something I can ensure even the program does not use any multi-threading itself because libraries may. Even apparently very innocent programs can exhibit weird threading-related bugs. And while it is a sort of gray area, starting to use threads internally is rarely considered an ABI breakage. So, unfortunately, the conditions you listed can be reduced to ‘it will never work’ in practice. Anyway, I do not attempt to do this kind of fork not-exec in real Gtk+ applications. So I'm looking forward to a better solution for unit tests which is discussed in the bugzilla bug mentioned by Matthias. Ah, OK. I am not a glib developer. I just (i) follow this mailing list because I maintain a library which happens to use glib, and (ii) know what POSIX says. It would be hard rations if introducing threads were treated as ABI or API breakage, and anyway a threading option (now compulsory) has been in glib from the outset. Having now read the bug reports another poster has referred to, the better conclusion is that the test function in question was wrong at the outset. Chris ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Forks, slices and threads: Can you make GSlice deadlock?
Hello, can g_slice_alloc() be made to deadlock simply by some bad sequence of GLib function calls, considering the calling program does not, of course, hold any GLib lock explicitly? (Without a GLib bug, that is.) I am starting to suspect a problem in GSlice interaction with threads. But I cannot report anything to bugzilla because I am unable to get to the core of the problem. My program (test program for a library) does g_test_trap_fork() and the child creates worker threads with g_thread_new(), sends them tasks with GAsyncQueue and cancels the tasks using GCancellables. Occasionally, a seemingly innocent g_thread_new() call deadlocks in g_slice_alloc(), see the backtrace below for how and where exactly. If it happens, it happens in the child soon after forking. I canNOT reproduce any deadlock if: - G_SLICE=always-malloc is set, - g_test_trap_fork() is not used and the test is run directly in the main program, - under valgrind (also, it reports no errors), - I print anything to stderr in g_slice_alloc() – infuriating, but so it works. When it deadlocks the main thread looks: #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136 #1 0x003451009c8c in _L_lock_1024 () from /lib64/libpthread.so.0 #2 0x003451009c35 in __pthread_mutex_lock (mutex=0x25442e0) at pthread_mutex_lock.c:105 #3 0x7fe496b01fb6 in g_mutex_lock (mutex=0x7fe496d9f8e0) at gthread-posix.c:208 #4 0x7fe496adef29 in magazine_cache_pop_magazine (ix=4, countp=0x2544368) at gslice.c:718 #5 0x7fe496adf235 in thread_memory_magazine1_reload (tmem=0x2544310, ix=4) at gslice.c:794 #6 0x7fe496adf4df in g_slice_alloc (mem_size=72) at gslice.c:992 #7 0x7fe496adf572 in g_slice_alloc0 (mem_size=72) at gslice.c:1032 #8 0x7fe496b02981 in g_system_thread_new (thread_func=0x7fe496ae8fc9 g_thread_proxy, stack_size=0, error=0x7fffb1ded5a0) at gthread-posix.c:1101 #9 0x7fe496ae9207 in g_thread_new_internal (name=0x491da9 canceller, proxy=0x7fe496ae8fc9 g_thread_proxy, func=0x461790 cancel_cancel, data=0x25822f0, stack_size=0, error=0x7fffb1ded5a0) at gthread.c:884 #10 0x7fe496ae90d5 in g_thread_new (name=0x491da9 canceller, func=0x461790 cancel_cancel, data=0x25822f0) at gthread.c:835 #11 0x00461667 in master_cancel_one (nproc=1) at master.c:232 #12 master_cancel_one (nproc=1) at master.c:208 #13 0x7fe496ae792d in test_case_run (tc=0x2555c90) at gtestutils.c:1679 #14 g_test_run_suite_internal (suite=suite@entry=0x2557c80, path=optimized out, path@entry=0x7fe496b5c2be ) at gtestutils.c:1732 #15 0x7fe496ae7aa6 in g_test_run_suite_internal (suite=suite@entry=0x2557c60, path=optimized out, path@entry=0x7fe496b5c2be ) at gtestutils.c:1743 #16 0x7fe496ae7aa6 in g_test_run_suite_internal (suite=suite@entry=0x2557c40, path=optimized out, path@entry=0x7fffb1deef4d /master) at gtestutils.c:1743 #17 0x7fe496ae7aa6 in g_test_run_suite_internal (suite=suite@entry=0x2545c20, path=optimized out, path@entry=0x7fffb1deef43 testlibgwy/master) at gtestutils.c:1743 #18 0x7fe496ae7e0b in g_test_run_suite (suite=0x2545c20) at gtestutils.c:1788 #19 0x7fe496ae7e55 in g_test_run () at gtestutils.c:1308 #20 0x00412e11 in main (argc=1, argv=0x7fffb1dedae8) at testlibgwy.c:88 There is also a worker thread waiting on my own GConf with my own lock at the moment: #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:166 #1 0x7fe496b02575 in g_cond_wait (cond=0x258b5a8, mutex=0x258b5a0) at gthread-posix.c:746 #2 0x7fe496a9b09b in g_async_queue_pop_intern_unlocked (queue=queue@entry=0x258b5a0, wait=wait@entry=1, end_time=end_time@entry=-1) at gasyncqueue.c:421 #3 0x7fe496a9b546 in g_async_queue_pop (queue=queue@entry=0x258b5a0) at gasyncqueue.c:455 #4 0x7fe4973b60e3 in worker_thread_main (thread_data=optimized out) at master.c:190 #5 0x7fe496ae9082 in g_thread_proxy (data=0x258e850) at gthread.c:797 #6 0x003451007d14 in start_thread (arg=0x7fe49534c700) at pthread_create.c:309 #7 0x0034508f168d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 and several threads are in the following state: #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136 #1 0x003451009c8c in _L_lock_1024 () from /lib64/libpthread.so.0 #2 0x003451009c35 in __pthread_mutex_lock (mutex=0x25442e0) at pthread_mutex_lock.c:105 #3 0x7fe496b01fb6 in g_mutex_lock (mutex=0x7fe496d9f8e0) at gthread-posix.c:208 #4 0x7fe496adf163 in private_thread_memory_cleanup (data=0x7fe4740008c0) at gslice.c:774 #5 0x003451007b12 in __nptl_deallocate_tsd () at pthread_create.c:157 #6 0x003451007d22 in start_thread (arg=0x7fe495b4d700) at pthread_create.c:316 #7 0x0034508f168d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 That's all the threads. So, they are all trying to lock allocator-slab_mutex in
