Ah ok, I didn't know that. About how reproducible is the error, I run it three times (without the change you suggested) and every time stopped at around 32000 threads. Now I added appProc->continueExecution() and it happened again after creating 32322 threads, so it seems this is not the problem.
Gerard 2015-02-18 20:29 GMT+01:00 Bill Williams <[email protected]>: > On 02/18/2015 01:00 PM, Gerard wrote: > >> I'm not sure at which point it hangs, but I have set up an example that >> I think mimics what happens with my application. In this example the >> mutatee hangs after creating 32294 threads. I'm using the latest master >> version. >> >> Okay, so there's at least one obvious problem here (and this is at least > in part the fault of our manuals): if the mutatee ever becomes stopped, we > won't continue it. The standard logic should be something more like: > > do > { > appProc->continueExecution(); > bpatch.waitForStatusChange(); > } while(!appProc->isTerminated()); > > assuming that you're starting from a stopped state (e.g. create or attach > and then some instrumentation). While it's about 99% safe to assume that if > your mutator isn't stopping a process, it won't be stopped during its > execution, it's not actually guaranteed, and waitForStatusChange will > return whenever there exists a process that's become stopped or terminated. > > If there's no third-party source of SIGSTOP (and there shouldn't be), > there's still something weird (if rare) going on. I'm running with debug > logs (DYNINST_DEBUG_PROCCONTROL), which will take some time (and some disk) > but should tell me whether the above change is a sufficient fix for you. > > --bw > > Thanks, >> >> Gerard >> >> 2015-02-16 18:51 GMT+01:00 Bill Williams <[email protected] >> <mailto:[email protected]>>: >> >> On 02/16/2015 10:35 AM, Gerard wrote: >> >> Hi, >> >> I'm having another problem. I increased the MAX_THREADS variable >> to >> 10240 because I need to instrument a mutatee that creates around >> 10000 >> threads. After increasing the variable I could instrument the >> mutatee >> without problems, but I added more snippets and the process hung >> again, >> this time after creating 3999 threads. Now, it doesn't matter if I >> increase even more the constant, it still hangs. >> >> Is there any other reason why the mutatee could hang or is there >> any >> other limit somewhere? >> >> I'm not aware of other hardcoded limits, and if you already had >> tramp guards disabled, it seems likely that you're running into some >> form of bad behavior at scale that's independent of MAX_THREADS. >> >> Is it possible for you to put together a simple reproducer >> (mutator/mutatee) and send that to me? Do you know whether the >> process is hanging at a particular point (thread >> creation/destruction, process exit)? >> >> Thanks, >> >> Gerard >> >> 2015-02-04 11:22 GMT+01:00 Gerard <[email protected] >> <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>>: >> >> Thanks! That was exactly the problem. I have increased the >> constant >> MAX_THREADS and now I don't have this problem anymore. Is >> there any >> reason why the constant is set to 32 instead of a higher >> value? >> >> I also have tried to find how to dynamically change >> the DYNINST_max_num_threads value but I haven't found where >> is it >> implemented (version 8.2.1). And about disabling tramp >> guards, I >> already had them disabled so it seems that this workaround >> does not >> solve the problem. >> >> Gerard >> >> 2015-02-03 20:13 GMT+01:00 Barton Miller <[email protected] >> <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>>: >> >> Disabling tramp guards certainly works if you really >> know that >> you're not recursing. That can be subtle and error >> prone, which >> is why tramp guards were invented. Proceed cautiously >> here. >> >> --bart >> >> >> On 2/3/2015 12:02 PM, Matthew LeGendre wrote: >> >> Another possible fix may be to disable tramp >> guards. Tramp >> guards are used to prevent recursive >> instrumentation. For >> example, if you instrument malloc() with >> instrumentation >> that calls malloc(), then tramp guards will prevent >> you from >> going into infinite recursion. >> >> If you already know that your instrumentation can't >> infinitely recurse, >> then disabling tramp guards will give a big >> performance win >> and may work around this hang. To disable tramp >> guards, put >> a call to BPatch::setTrampRecursive(____true) >> before you >> insert instrumentation. >> >> -Matt >> ___________________________________________________ >> Dyninst-api mailing list >> [email protected] <mailto:[email protected]> >> <mailto:[email protected].__edu <mailto:[email protected] >> >> >> https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api> >> >> <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>> >> >> >> ___________________________________________________ >> Dyninst-api mailing list >> [email protected] <mailto:[email protected]> >> <mailto:[email protected].__edu <mailto:[email protected] >> >> >> https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api> >> >> <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>> >> >> >> >> >> >> _________________________________________________ >> Dyninst-api mailing list >> [email protected] <mailto:[email protected]> >> https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api> >> >> >> >> -- >> --bw >> >> Bill Williams >> Paradyn Project >> [email protected] <mailto:[email protected]> >> >> >> > > -- > --bw > > Bill Williams > Paradyn Project > [email protected] >
_______________________________________________ Dyninst-api mailing list [email protected] https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
