Re: [lttng-dev] lttng-dev Digest, Vol 156, Issue 3
Will try the dummy app to check if the basic setup is good. Wondering if being able to trace shared libraries is something not supported. Regards, Ramesh On Tue, Apr 6, 2021 at 6:46 PM Jonathan Rajotte-Julien < jonathan.rajotte-jul...@efficios.com> wrote: > > I followed your command sequence and noticed a bunch of files being > > created. However when I tried to run babeltrace on these outputs I don't > > see any of the functions that should have been called. Including below > the > > search for symbols. > > At least userspace tracing seems to work. > > I would recommended, again, that you first try tracing on a dummy > application, a > really simple one. Then move to the usage of cyg_profile on that dummy app > then > to your application. > > The cyg profile events are the following: > > lttng_ust_cyg_profile_fast:func_entry > lttng_ust_cyg_profile_fast:func_exit > > and, when using liblttng-ust-cyg-profile.so: > > lttng_ust_cyg_profile:func_entry > lttng_ust_cyg_profile:func_exit > > I would recommend that you first start grep-ing (lttng_ust_cyg) for this > in the > trace to see if any is getting hit and recorded. If it is not the case, > take a > step back try with a dummy app and if nothing works with the dummy app we > can at > least try to help you from there and remove all other variable since you > will be > able to share the dummy app with us. > > Cheers > ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] lttng-dev Digest, Vol 156, Issue 3
> I followed your command sequence and noticed a bunch of files being > created. However when I tried to run babeltrace on these outputs I don't > see any of the functions that should have been called. Including below the > search for symbols. At least userspace tracing seems to work. I would recommended, again, that you first try tracing on a dummy application, a really simple one. Then move to the usage of cyg_profile on that dummy app then to your application. The cyg profile events are the following: lttng_ust_cyg_profile_fast:func_entry lttng_ust_cyg_profile_fast:func_exit and, when using liblttng-ust-cyg-profile.so: lttng_ust_cyg_profile:func_entry lttng_ust_cyg_profile:func_exit I would recommend that you first start grep-ing (lttng_ust_cyg) for this in the trace to see if any is getting hit and recorded. If it is not the case, take a step back try with a dummy app and if nothing works with the dummy app we can at least try to help you from there and remove all other variable since you will be able to share the dummy app with us. Cheers ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] lttng-dev Digest, Vol 156, Issue 3
Hi, I think you need to take a step back and figure out how lttng is deployed and more importantly the overall architecture of lttng and its principal moving pieces. First, you need a functioning and running lttng-sessiond process. >From lttng-console-log.txt: root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# lttng list -u Error: Unable to list UST events: No session daemon is available The error is clear here, no lttng-sessiond is running or at least, the lttng CLI cannot communicate with it. Here the sessiond was killed before performing this operation based on CmdSequence.txt. You seem to have forgot to start a lttng-sessiond before performing the call to your application and `lttng` commands. My previous email stated the following: With the app running and having the LD_PRELOAD correctly set, and a sessiond (sessiond should have read lttng-sessiond) running. Then, if we look at kernel modules loading. >From lttng-sessiond.txt: DEBUG1 - 19:50:34.293777619 [27093/27093]: libkmod: could not find module by name='lttng_ring_buffer_client_discard' (in log_kmod() at modprobe.c:108) Error: Unable to load required module lttng-ring-buffer-client-discard Warning: No kernel tracer available libkmod expect the presence of lttng-modules kernel modules in: /lib/modules/$(uname -r)/ >From lttng-system-env.txt: root@RocrLnx23:~# find /lib/modules/$(uname -r)/ | grep lttng Yield nothing. I suspect that your modules are installed under: ~/git/compute/out/ubuntu-18.04/18.04/lib/modules/$(uname -r)/ Now, currently, lttng-sessiond when built with kmod support does not support defining the modules locations (as far as I know) since we pass (NULL, NULL) to the creation context (kmod_new). The default behaviour is to look into /lib/modules/ . Albeit it would be nice to support such scenarios, this will not help you in the short term. I would recommend the usage of symbolic link here to provide "/lib/modules/$(uname -r)/". Note that this is overly complicated by the deployment scenario you are currently using, which does not represent a typical deployment for most of our users. In any cases, again, this is only useful if you plan on doing kernel tracing. I would suggest that you might want to put effort in making sure Userspace Tracing works. Here I think the only thing missing is to have a running lttng-sessiond process while you launch ./rocminfo Concretely: $ sudo -s # systemctl stop lttng-sessiond.service # pkill lttng-sessiond # cd ~/git/compute/out/ubuntu-18.04/18.04/lib # export LD_LIBRARY_PATH=`pwd`:$LD_LIBRARY_PATH # lttng-sessiond -vvv --verbose-consumer -b > /tmp/lttng-sessiond.log 2>&1 # lttng create my_userspace_tracing_session # lttng enable-event -u -a # lttng start # LD_PRELOAD=liblttng-ust-cyg-profile-fast.so LTTNG_UST_DEBUG=1 ./rocminfo # lttng destroy # babeltrace # pkill lttng-sessiond Note: here the manual starting and termination of lttng-sessiond is only for the purpose of testing. In a normal deployment lttng-sessiond is a daemon. Note that the sequence of `lttng` commands is important here since I expect ./rocminfo to not have a long lifetime. In other word, the `rocminfo` will have finished and unregistered from lttng-sessiond perspective before you are able to manually either list its tracepoints or setup a tracing session after you started the application. Please read the following and try to perform it under your deployment scenario: https://lttng.org/docs/v2.12/#doc-tracing-your-own-user-application Hope this helps. Cheers On Mon, Apr 05, 2021 at 08:32:00PM -0500, Ramesh Errabolu via lttng-dev wrote: > Jonathan, et al > > Attaching here with logs of the run. > > 1. Command sequence I ran > 2. Console log of commands as I ran them > 3. Log of lttng-service daemon *lttng-sessiond.txt* > 4. Info about the system, things such as kernel, lttng kernel modules, etc > 5. Lastly I tried to build lttng-modules for my kernel 5.9.x. This failed > as lacking support for it. > > Any help info is appreciated. > > Regards, > Ramesh > > > On Fri, Apr 2, 2021 at 11:00 AM wrote: > > > Send lttng-dev mailing list submissions to > > lttng-dev@lists.lttng.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > or, via email, send a message with subject or body 'help' to > > lttng-dev-requ...@lists.lttng.org > > > > You can reach the person managing the list at > > lttng-dev-ow...@lists.lttng.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of lttng-dev digest..." > > > > > > Today's Topics: > > > >1. Re: Can't trace function calls (Jonathan Rajotte-Julien) > > > > > > -- > > > > Message: 1 > > Date: Fri, 2 Apr 2021 11:07:19 -0400 > > From: Jonathan
Re: [lttng-dev] lttng-dev Digest, Vol 156, Issue 3
Jonathan, et al Attaching here with logs of the run. 1. Command sequence I ran 2. Console log of commands as I ran them 3. Log of lttng-service daemon *lttng-sessiond.txt* 4. Info about the system, things such as kernel, lttng kernel modules, etc 5. Lastly I tried to build lttng-modules for my kernel 5.9.x. This failed as lacking support for it. Any help info is appreciated. Regards, Ramesh On Fri, Apr 2, 2021 at 11:00 AM wrote: > Send lttng-dev mailing list submissions to > lttng-dev@lists.lttng.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > or, via email, send a message with subject or body 'help' to > lttng-dev-requ...@lists.lttng.org > > You can reach the person managing the list at > lttng-dev-ow...@lists.lttng.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of lttng-dev digest..." > > > Today's Topics: > >1. Re: Can't trace function calls (Jonathan Rajotte-Julien) > > > -- > > Message: 1 > Date: Fri, 2 Apr 2021 11:07:19 -0400 > From: Jonathan Rajotte-Julien > To: Ramesh Errabolu > Cc: lttng-dev@lists.lttng.org > Subject: Re: [lttng-dev] Can't trace function calls > Message-ID: <20210402150719.GB79283@joraj-alpa> > Content-Type: text/plain; charset=us-ascii > > On Wed, Mar 31, 2021 at 12:55:53PM -0500, Ramesh Errabolu wrote: > > root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# ls ~/ | grep > -i ltt > > root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# lttng create > > my-kernel-session --output=~/my-kernel-trace > > Session my-kernel-session created. > > Traces will be output to > > /home/user1/git/compute/out/ubuntu-18.04/18.04/bin/~/my-kernel-trace > > root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# *lttng list > > --kerne*l > > *Error: Unable to list kernel events: Kernel tracer not available* > > Well this would be the first thing to look at. > > First let's deactivate the lttng-sessiond.service installed by the > packages. > >systemctl stop lttng-sessiond.service > > You might want to reanable it later. > > > > > root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# ps -ef | grep > ltt > > root 1002 1 0 12:16 ?00:00:00 /usr/bin/lttng-sessiond > > root 1054 1002 0 12:16 ?00:00:00 */usr/bin/lttng-sessiond* > > root 3145 2861 0 12:51 pts/000:00:00 grep --color=auto ltt > > root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# > > Make sure after the systemctl call that no lttng-sessiond process is > running. > > Now let's launch a lttng-sessiond by hand with a bit more verbosity. For > now > let's stick to the root user. > > # lttng-sessiond -vvv --verbose-consumer -b > /tmp/lttng-sessiond.log > 2>&1 > # pkill lttng-sessiond > > Please share the content of /tmp/lttng-sessiond.log using a pasting service > (paste.ubuntu.com). > > Also please provide the output of: > > # uname -a > # find /lib/modules/$(uname -r)/ | grep lttng > # dmesg | grep lttng > > > But again, the cyg-profile helper library is meant for Userspace tracing. > > With the app running and having the LD_PRELOAD correctly set, and a > sessiond > running. > > # lttng list -u > > If there is nothing, well you can start the application with the following > and > share the output of it (make sure to remove any output from your > application if > sensitive data is present) > > # LD_PRELOAD= LTTNG_UST_DEBUG=1 your_application_here > > Note that debug log will be outputted on stderr. > > Cheers > > -- > Jonathan Rajotte-Julien > EfficiOS > > > -- > > Subject: Digest Footer > > ___ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > > -- > > End of lttng-dev Digest, Vol 156, Issue 3 > * > $ sudo -s # # systemctl stop lttng-sessiond.service # ps -ef | grep -i lttng # lttng-sessiond -vvv --verbose-consumer -b > /tmp/lttng-sessiond.log 2>&1 # pkill lttng-sessiond # uname -a # find /lib/modules/$(uname -r)/ | grep lttng # dmesg | grep lttng # # cd ~/git/compute/out/ubuntu-18.04/18.04/lib # export LD_LIBRARY_PATH=`pwd`:$LD_LIBRARY_PATH # echo $LD_LIBRARY_PATH # # cd ../bin # LD_PRELOAD=liblttng-ust-cyg-profile-fast.so ./rocminfo # lttng list -u # # LD_PRELOAD=liblttng-ust-cyg-profile-fast.so LTTNG_UST_DEBUG=1 ./rocminfo # # # # root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/lib# root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/lib# export LD_LIBRARY_PATH=`pwd`:$LD_LIBRARY_PATH root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/lib# root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/lib# echo $LD_LIBRARY_PATH /home/user1/git/compute/out/ubuntu-18.04/18.04/lib:
