There's been numerous ProcControlAPI issues in this category--the Linux debug interface can be very inconsistent when multithreaded processes exit. The lastest set of ProcControlAPI fixes for this category went into the dyninst GIT master branch in February. It may be worth testing this with the latest Dyninst from GIT.
Marc, you can also turn on the environment variable DYNINST_DEBUG_PROCCONTROL, run until it hangs, and send Emily/Bill/myself the resulting output. That's usually sufficient for diagnosing these.
-Matt On Thu, 2 Apr 2015, Emily Gember-Jacobson wrote:
Hi Marc, I haven't been able to reproduce this (8.2 on RHEL 6). What OS are you on? Nothing stands out to me as problematic in your mutator. Matt or Josh, do either of you have ideas here? Best, Emily On Thursday, April 2, 2015, Marc Brünink <[email protected]> wrote: Hi Emily, actually I have a hanging mutator every now and then with this very simple mutator/mutatee pair during destruction of the mutator. Using 8.2. If you enable the thread in the mutatee, you can get a hanging mutator at a different location. All hangs happen on exit. Marc On 31/03/2015 21:59, Emily Gember-Jacobson wrote: Hi Marc, Bill is out of the office this week, so I'll try to look into this for you. Would it be possible for you to share the code to reproduce this hang? Thanks, Emily On Tue, Mar 31, 2015 at 5:55 AM, Marc Brünink <[email protected] <mailto:[email protected]>> wrote: I am struggling with a hung mutator on exit of the mutatee. It looks like it is related to event ordering. The process hangs if the post-UserThreadDestroy event is queued before the pre-Exit. -------------- [pcEventMuxer.C/83]: PCEventMuxer waiting for events, non-blocking pcEventMuxer.C[407]: Begin callbackMux, process pointer = 0x33ae2dc0, event post-UserThreadDestroy pcEventMuxer.C[439]: Added event post-UserThreadDestroy to mailbox, size now 1 pcEventMuxer.C[411]: after muxing event, mailbox size is 1 pcEventMuxer.C[407]: Begin callbackMux, process pointer = 0x33ae2dc0, event pre-LWPDestroy pcEventMuxer.C[439]: Added event pre-LWPDestroy to mailbox, size now 2 pcEventMuxer.C[411]: after muxing event, mailbox size is 2 [pcEventMuxer.C:86] after PC event handling, 2 events in mailbox pcEventMuxer.C[464]: Returning event post-UserThreadDestroy from mailbox pcEventHandler.C[65]: attempting to handle event post-UserThreadDestroy on thread 4976/4977 pcEventHandler.C[109]: stopping process for event handling dynProcess.C[1048]: Stopping process 4976 pcEventMuxer.C[231]: Begin callbackMux, process pointer = 0x33ae2dc0, event pre-Exit [pcEventMuxer.C:232] Exit callback pcEventMuxer.C[439]: Added event pre-Exit to mailbox, size now 2 pcEventMuxer.C[236]: after muxing event, mailbox size is 2 ... hang ... -------------- In contrast if the pre-Exit is queued before, everything work fine because the stopping of the process is skipped. -------------- [pcEventMuxer.C/83]: PCEventMuxer waiting for events, non-blocking [46912496468160]pcEventMuxer.__C[231]: Begin callbackMux, process pointer = 0x20cde620, event pre-Exit [pcEventMuxer.C:232] Exit callback pcEventMuxer.C[439]: Added event pre-Exit to mailbox, size now 1 pcEventMuxer.C[236]: after muxing event, mailbox size is 1 pcEventMuxer.C[407]: Begin callbackMux, process pointer = 0x20cde620, event post-UserThreadDestroy pcEventMuxer.C[439]: Added event post-UserThreadDestroy to mailbox, size now 2 pcEventMuxer.C[411]: after muxing event, mailbox size is 2 pcEventMuxer.C[407]: Begin callbackMux, process pointer = 0x20cde620, event pre-LWPDestroy pcEventMuxer.C[439]: Added event pre-LWPDestroy to mailbox, size now 3 pcEventMuxer.C[411]: after muxing event, mailbox size is 3 pcEventMuxer.C[407]: Begin callbackMux, process pointer = 0x20cde620, event post-LWPDestroy pcEventMuxer.C[439]: Added event post-LWPDestroy to mailbox, size now 4 pcEventMuxer.C[411]: after muxing event, mailbox size is 4 [pcEventMuxer.C:86] after PC event handling, 4 events in mailbox pcEventMuxer.C[464]: Returning event pre-Exit from mailbox pcEventHandler.C[65]: attempting to handle event pre-Exit on thread 4952/4952 -------------- A truncated backtrace: -------------- #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/__linux/x86_64/pthread_cond___wait.S:185 #1 0x00002aaaaee25c38 in wait<Mutex<false> > (m=..., this=0x2aaab40009b8) at /usr/include/boost/thread/__pthread/condition_variable.__hpp:184 #2 wait (this=0x2aaab40009b8) at /home/marc/Downloads/__DyninstAPI-8.2/common/src/__dthread.h:121 #3 MailboxMT::dequeue (this=0x2aaab40008c0, block=true) at /home/marc/Downloads/__DyninstAPI-8.2/proccontrol/__src/mailbox.C:156 #4 0x00002aaaaee4aa01 in int_process::__waitAndHandleEvents (block=block@entry=false) at /home/marc/Downloads/__DyninstAPI-8.2/proccontrol/__src/process.C:1045 #5 0x00002aaaaee733af in Dyninst::ProcControlAPI::__ProcessSet::stopProcs (this=0xca39e0) at /home/marc/Downloads/__DyninstAPI-8.2/proccontrol/__src/procset.C:1495 #6 0x00002aaaaee39953 in Dyninst::ProcControlAPI::__Process::stopProc (this=<optimized out>) at /home/marc/Downloads/__DyninstAPI-8.2/proccontrol/__src/process.C:6406 -------------- Questions: 1. Is this a known issue in Dyninst 8.2? 2. Any idea what's wrong? It might very well be a problem in my mutator. Marc _________________________________________________ Dyninst-api mailing list [email protected] <mailto:[email protected]> https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>
_______________________________________________ Dyninst-api mailing list [email protected] https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
