Commit merged into master, thanks for your feedback! Mathieu
----- On Mar 15, 2016, at 2:48 PM, Jeffrey Chen <cp...@hotmail.com> wrote: > Hi Mathieu: > Will this fix be checked-in? Thanks. > From: Mathieu Desnoyers <mathieu.desnoy...@efficios.com> > Sent: Thursday, March 10, 2016 3:27 PM > To: Jeffrey Chen > Cc: Jonathan Rajotte Julien; lttng-dev > Subject: Re: segfault when exiting process with background thread > ----- On Mar 10, 2016, at 5:59 PM, Jeffrey Chen <cp...@hotmail.com> wrote: >> Thanks, Mathieu. >> Yes, the fix works as a workaround. This is also the same workaround we have >> now. >> It seems the problem is due to a race condition, that it checked whether " >> tracepoint_dlopen.rcu_read_lock_sym_bp " is not NULL. If not NULL, it calls >> to >> that. But, at this moment, memset cleared the memory. >> This fix is okay with us, but I am not certain if it would break other >> component, since I do not understand LTTng enough to tell. Thanks. > In addition to memset clearing that memory, dlclose() of the > lttng-ust-tracepoint > shared object will clear the memory containing the called functions. > Therefore, > we need to skip both steps of the destructor if we want threads to survive > after > this destructor execution. > Thanks, > Mathieu >> From: Mathieu Desnoyers <mathieu.desnoy...@efficios.com> >> Sent: Thursday, March 10, 2016 10:52 AM >> To: Jeffrey Chen >> Cc: Jonathan Rajotte Julien; lttng-dev >> Subject: Re: segfault when exiting process with background thread >> Hi Jeffrey, >> I CC'd your @live.com address on the RFC patch. Please let me know if the >> approach >> is OK with you (and try it out). >> See: >> https://lists.lttng.org/pipermail/lttng-dev/2016-March/025608.html >> Thanks, >> Mathieu >> ----- On Mar 10, 2016, at 1:35 PM, Jeffrey Chen <cp...@hotmail.com> wrote: >>> Thanks Mathieu. >>> Is there a plan at LTTng side to fix this issue? If so, we could wait for >>> the >>> fix. If not, we will have to workaround the problem for now (probably by >>> applying your fix ourselves). Thanks. >>> From: Mathieu Desnoyers <mathieu.desnoy...@efficios.com> >>> Sent: Tuesday, March 8, 2016 6:09 PM >>> To: Jonathan Rajotte Julien >>> Cc: lttng-dev >>> Subject: Re: [lttng-dev] segfault when exiting process with background >>> thread >>> The attached patch works around the issue. The problem is >>> the tracepoint destructors that run in parallel with the thread >>> that still exists. >>> If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared >>> object on dlclose of the instrumented code. Not sure if we should >>> care though... >>> Thoughts ? >>> Thanks, >>> Mathieu >>> ----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers >>> <mathieu.desnoy...@efficios.com> wrote: >>>> Reproduced it on master (ust and tools). >>>> Repro with: >>>> lttng create >>>> lttng enable-event -u -a >>>> lttng start >>>> then run hello a few times, it eventually segfaults in the spawned thread. >>>> Looking into it. >>>> Thanks, >>>> Mathieu >>>> ----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien >>>> <jonathan.rajotte-jul...@efficios.com> wrote: >>>>> Hi Jeffrey, >>>>> Could you provide the version for lttng-ust, lttng-tools, lttng-modules >>>>> (if >>>>> installed)? >>>>> Were you able to reproduce the issue on master (git) ? >>>>> What are the exact order of command to reproduce the problem ? >>>>> Thanks >>>>> On 2016-03-08 06:43 PM, Jeffrey Chen wrote: >>>>>> Hi LTTng community: >>>>>> We are using LTTng for our production environment. We have been noticing >>>>>> segfault problem when our process exit. We think the problem is due to >>>>>> the >>>>>> background thread continue to write traces, while LTTng cleanup its >>>>>> memory. We >>>>>> do not know how to fix this issue without changing LTTng code base. I >>>>>> have >>>>>> written a very simple app that could repro the problem. Most of my codes >>>>>> are >>>>>> copy pasted from the LTTng doc sample. Is there any fix that the LTTng >>>>>> side >>>>>> could do? >>>>>> hello.c >>>>>> #include <stdio.h> >>>>>> #include "hello-tp.h" >>>>>> #include <pthread.h> >>>>>> void* doSomeThing(void *arg) >>>>>> { >>>>>> int x; >>>>>> for (x = 0; x < 100000; ++x) { >>>>>> tracepoint(hello_world, my_first_tracepoint, x, "test"); >>>>>> } >>>>>> } >>>>>> int main(int argc, char *argv[]) >>>>>> { >>>>>> int x; >>>>>> getchar(); >>>>>> pthread_t inc_x_thread; >>>>>> if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) { >>>>>> fprintf(stderr, "Error creating thread\n"); >>>>>> return 1; >>>>>> } >>>>>> tracepoint(hello_world, my_first_tracepoint, 23, "hi there!"); >>>>>> return 0; >>>>>> } >>>>>> hello-tp.c >>>>>> #define TRACEPOINT_CREATE_PROBES >>>>>> #define TRACEPOINT_DEFINE >>>>>> #include "hello-tp.h" >>>>>> hello-tp.h >>>>>> #undef TRACEPOINT_PROVIDER >>>>>> #define TRACEPOINT_PROVIDER hello_world >>>>>> #undef TRACEPOINT_INCLUDE >>>>>> #define TRACEPOINT_INCLUDE "./hello-tp.h" >>>>>> #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ) >>>>>> #define _HELLO_TP_H >>>>>> #include <lttng/tracepoint.h> >>>>>> TRACEPOINT_EVENT( >>>>>> hello_world, >>>>>> my_first_tracepoint, >>>>>> TP_ARGS( >>>>>> int, my_integer_arg, >>>>>> char*, my_string_arg >>>>>> ), >>>>>> TP_FIELDS( >>>>>> ctf_string(my_string_field, my_string_arg) >>>>>> ctf_integer(int, my_integer_field, my_integer_arg) >>>>>> ) >>>>>> ) >>>>>> #endif /* _HELLO_TP_H */ >>>>>> #include <lttng/tracepoint-event.h> >>>>>> Compile >>>>>> gcc -c -I. hello-tp.c >>>>>> gcc -c hello.c >>>>>> gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread >>>>>> _______________________________________________ >>>>>> lttng-dev mailing list lttng-dev@lists.lttng.org >>>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >>>>> -- >>>>> Jonathan R. Julien >>>>> Efficios >>>>> _______________________________________________ >>>>> lttng-dev mailing list >>>>> lttng-dev@lists.lttng.org >>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >>>> -- >>>> Mathieu Desnoyers >>>> EfficiOS Inc. >>>> http://www.efficios.com >>>> _______________________________________________ >>>> lttng-dev mailing list >>>> lttng-dev@lists.lttng.org >>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >>> -- >>> Mathieu Desnoyers >>> EfficiOS Inc. >>> http://www.efficios.com >> -- >> Mathieu Desnoyers >> EfficiOS Inc. >> http://www.efficios.com > -- > Mathieu Desnoyers > EfficiOS Inc. > http://www.efficios.com -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
_______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev