Then let's hope that debug logs will tell us where is the problem! I'm using Gentoo Linux with kernel 3.18.3-gentoo.
Gerard 2015-02-18 20:42 GMT+01:00 Bill Williams <[email protected]>: > On 02/18/2015 01:37 PM, Gerard wrote: > >> Ah ok, I didn't know that. >> >> About how reproducible is the error, I run it three times (without the >> change you suggested) and every time stopped at around 32000 threads. >> Now I added appProc->continueExecution() and it happened again after >> creating 32322 threads, so it seems this is not the problem. >> >> Then it's got to be that somewhere in here, we're messing up internal > stop/continue state without that propagating out to the user level. Debug > logs will tell me something eventually...sadly, they're verbose and > time-consuming. > > Which kernel version/distribution are you using, by the way? > > Gerard >> >> 2015-02-18 20:29 GMT+01:00 Bill Williams <[email protected] >> <mailto:[email protected]>>: >> >> On 02/18/2015 01:00 PM, Gerard wrote: >> >> I'm not sure at which point it hangs, but I have set up an >> example that >> I think mimics what happens with my application. In this example >> the >> mutatee hangs after creating 32294 threads. I'm using the latest >> master >> version. >> >> Okay, so there's at least one obvious problem here (and this is at >> least in part the fault of our manuals): if the mutatee ever becomes >> stopped, we won't continue it. The standard logic should be >> something more like: >> >> do >> { >> appProc->continueExecution(); >> bpatch.waitForStatusChange(); >> } while(!appProc->isTerminated()__); >> >> assuming that you're starting from a stopped state (e.g. create or >> attach and then some instrumentation). While it's about 99% safe to >> assume that if your mutator isn't stopping a process, it won't be >> stopped during its execution, it's not actually guaranteed, and >> waitForStatusChange will return whenever there exists a process >> that's become stopped or terminated. >> >> If there's no third-party source of SIGSTOP (and there shouldn't >> be), there's still something weird (if rare) going on. I'm running >> with debug logs (DYNINST_DEBUG_PROCCONTROL), which will take some >> time (and some disk) but should tell me whether the above change is >> a sufficient fix for you. >> >> --bw >> >> Thanks, >> >> Gerard >> >> 2015-02-16 18:51 GMT+01:00 Bill Williams <[email protected] >> <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>>: >> >> >> On 02/16/2015 10:35 AM, Gerard wrote: >> >> Hi, >> >> I'm having another problem. I increased the MAX_THREADS >> variable to >> 10240 because I need to instrument a mutatee that >> creates around >> 10000 >> threads. After increasing the variable I could >> instrument the >> mutatee >> without problems, but I added more snippets and the >> process hung >> again, >> this time after creating 3999 threads. Now, it doesn't >> matter if I >> increase even more the constant, it still hangs. >> >> Is there any other reason why the mutatee could hang or >> is there any >> other limit somewhere? >> >> I'm not aware of other hardcoded limits, and if you already >> had >> tramp guards disabled, it seems likely that you're running >> into some >> form of bad behavior at scale that's independent of >> MAX_THREADS. >> >> Is it possible for you to put together a simple reproducer >> (mutator/mutatee) and send that to me? Do you know whether >> the >> process is hanging at a particular point (thread >> creation/destruction, process exit)? >> >> Thanks, >> >> Gerard >> >> 2015-02-04 11:22 GMT+01:00 Gerard <[email protected] >> <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> <mailto:[email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>>>: >> >> Thanks! That was exactly the problem. I have >> increased the >> constant >> MAX_THREADS and now I don't have this problem >> anymore. Is >> there any >> reason why the constant is set to 32 instead of a >> higher value? >> >> I also have tried to find how to dynamically change >> the DYNINST_max_num_threads value but I haven't >> found where >> is it >> implemented (version 8.2.1). And about disabling >> tramp >> guards, I >> already had them disabled so it seems that this >> workaround >> does not >> solve the problem. >> >> Gerard >> >> 2015-02-03 20:13 GMT+01:00 Barton Miller >> <[email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> <mailto:[email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>>>: >> >> Disabling tramp guards certainly works if you >> really >> know that >> you're not recursing. That can be subtle and >> error >> prone, which >> is why tramp guards were invented. Proceed >> cautiously >> here. >> >> --bart >> >> >> On 2/3/2015 12:02 PM, Matthew LeGendre wrote: >> >> Another possible fix may be to disable tramp >> guards. Tramp >> guards are used to prevent recursive >> instrumentation. For >> example, if you instrument malloc() with >> instrumentation >> that calls malloc(), then tramp guards >> will prevent >> you from >> going into infinite recursion. >> >> If you already know that your >> instrumentation can't >> infinitely recurse, >> then disabling tramp guards will give a big >> performance win >> and may work around this hang. To disable >> tramp >> guards, put >> a call to >> BPatch::setTrampRecursive(______true) >> before you >> insert instrumentation. >> >> -Matt >> >> _____________________________________________________ >> Dyninst-api mailing list >> [email protected] <mailto:[email protected]> >> <mailto:[email protected].__edu <mailto:[email protected] >> >> >> <mailto:[email protected]. >> <mailto:[email protected].>____edu >> <mailto:[email protected].__edu <mailto:[email protected] >> >>> >> https://lists.cs.wisc.edu/______mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api> >> >> <https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>> >> >> >> <https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api> >> >> <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>>> >> >> >> >> _____________________________________________________ >> Dyninst-api mailing list >> [email protected] <mailto:[email protected]> >> <mailto:[email protected].__edu <mailto:[email protected] >> >> >> <mailto:[email protected]. >> <mailto:[email protected].>____edu >> <mailto:[email protected].__edu <mailto:[email protected] >> >>> >> https://lists.cs.wisc.edu/______mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api> >> >> <https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>> >> >> >> <https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api> >> >> <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>>> >> >> >> >> >> >> ___________________________________________________ >> Dyninst-api mailing list >> [email protected] <mailto:[email protected]> >> <mailto:[email protected].__edu <mailto:[email protected] >> >> >> https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api> >> >> <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api >> <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>> >> >> >> >> -- >> --bw >> >> Bill Williams >> Paradyn Project >> [email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> >> >> >> >> -- >> --bw >> >> Bill Williams >> Paradyn Project >> [email protected] <mailto:[email protected]> >> >> >> > > -- > --bw > > Bill Williams > Paradyn Project > [email protected] >
_______________________________________________ Dyninst-api mailing list [email protected] https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
