Re: [lttng-dev] [lttng-relayd] is there existing cases for relayd to stream over Android usb based adb?

[Kienan Stewart via lttng-dev]

Hi Amanda,

I'd like to confirm my understanding the situation.

Android device
  - Running lttng-sessiond with one or more configured sessions

Development device
  - Connected to the android device over usb using adb

You want want the data captured on the android device to be streamed via 
the usb connection rather than the other networks on the android device.


Could you expand on the commands you used to set up the tracing sessions 
and relay, and where each of those commands were run?


It sounds to me like you might want to be doing something like the 
following:


(Development device) Start lttng-relayd:
  - tcp://0.0.0.0:5342 and :5343 will be bound on the development device
  - tcp://127.0.0.1:5344 will be available for the live reader

(Development device) Create the reverses for the following ports: 5342 
and 5343
  - At this point :5342 and :5343 should be available on the android 
device and reach the relayd running on the development device


(Android device) Start lttng-sessiond
(Android device) Create session(s): `lttng create -U tcp://localhost/
  - Using `-U/--set-url`, no relayd will be spawned on the android device
(Android device) Start session(s)

This setup should have the relayd running on the development and writing 
the traces there and/or viewing them with a live viewer. On the android 
device, the UST applications (if any) will connect to the local sessiond 
and consumers, which will shuttle the information over :5342 and :5343 
to the developer device via the reverse sockets.


Please note that I didn't have time to test this, so there might be some 
mistakes. As I requested above, clear details of the exact commands you 
use for the tracing setup would be very helpful to have the clearest 
understanding of what you're doing.


hope this helps,
kienan

On 5/30/24 1:53 AM, Wu, Yannan via lttng-dev wrote:

Hihi, there,

I am currently working on enabling lttng live mode over android usb adb. 
Here is the situation, during debugging some network related issues, we 
dont want the trace data to be streamed via network to cause extra load 
to the system being profiled. Then we select to connect lttng-relayd 
with adb via port forwarding so that the data is "forward" to the host.


*Here is the set up and the problem:*

for the device:  adb reverse tcp:5342 tcp:5342; adb reverse tcp:5343 
tcp:5343; adb reverse tcp:5344 tcp:5344

Then starting up lttng with --live enabled.

*What is expected:*
lttng start streaming to the localhost.
*What is seen: *
the lttng-relayd failed to start. For unable binding to the socket.

*The cause of this issue: *

both adb reverse and lttng relayd need binding to the socket which is 
conflict with each other.



So what I wanna ask is, for embedded system use cases, do we have 
successful use cases among team that could stream the trace data in live 
mode to the host with usb based adb? If not, any idea or suggestion to 
me on how to process forward?


Amanda





___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] [PATCH] Fix mm_vmscan_lru_isolate tracepoint for RHEL 9.4 kernel

[Kienan Stewart via lttng-dev]

Hi Martin,

thanks for the patch.

I changed the version range slightly. The RHEL kernel 5.14.0-427.13.1 
still has the `isolate_mode` parameter in the `mm_vmscan_lru_isolate` 
tracepoint; it was only removed in 5.14.0-427.16.1.


I also forward ported the patch to the master branch.

The updated patches will be reviewed at: 
https://review.lttng.org/q/topic:%22buildfix-el9.4%22


thanks,
kienan

On 5/17/24 10:30 AM, Martin Hicks via lttng-dev wrote:



Redhat has moved to using the format first found in the 6.7 kernel
for the mm_vmscan_lru_isolate tracepoint.

Signed-off-by: Martin Hicks 
---
  include/instrumentation/events/mm_vmscan.h | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/include/instrumentation/events/mm_vmscan.h 
b/include/instrumentation/events/mm_vmscan.h
index ea6f4b7..49a9eae 100644
--- a/include/instrumentation/events/mm_vmscan.h
+++ b/include/instrumentation/events/mm_vmscan.h
@@ -369,7 +369,9 @@ LTTNG_TRACEPOINT_EVENT_MAP(mm_shrink_slab_end,
  )
  #endif
  
-#if (LTTNG_LINUX_VERSION_CODE >= LTTNG_KERNEL_VERSION(6,7,0))

+#if (LTTNG_LINUX_VERSION_CODE >= LTTNG_KERNEL_VERSION(6,7,0) || \
+ LTTNG_RHEL_KERNEL_RANGE(5,14,0,427,0,0, 5,15,0,0,0,0))
+
  LTTNG_TRACEPOINT_EVENT(mm_vmscan_lru_isolate,
  
  	TP_PROTO(int classzone_idx,

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Capturing snapshot on kernel panic

[Kienan Stewart via lttng-dev]

Hi Damien,

I want to expand on one of the options that could work for your case.

On 5/16/24 9:37 AM, Kienan Stewart via lttng-dev wrote:

Hi Damien,


On 5/15/24 6:24 PM, Damien Berget via lttng-dev wrote:

Good day,
we have been using LTTng successfully to capture snapshots on user 
defined tracepoints and it did provide invaluable to debug our issues. 
Thanks to all the contributors of this project!


We'd like to know if it would be possible to trigger on a kernel 
panic? I might be dubiously possible as you would still need to have 
the file-system working to write the results but I should ask.




For userspace tracing, I think the recommendation is usually to use a 
dax/pmem device and have the buffers for the session mapped there. After 
a panic, the contents of the buffers can be restored using lttng-crash[1].


Note that dax/pem isn't supported by the kernel space tracer at this time.

If I recall, there are other ways to things in the panic sequence (that 
aren't lttng specific), but I'm personally not as familiar with the 
details of that stage of linux.




It's possible to kexec-tools to load a new kernel post-panic[1]. If your 
system uses kexec, the contents of RAM aren't necessarily flushed, and 
if both the initial kernel and post-panic kernel started by kexec have 
the same configuration for an emulated PMEM device using the memmap 
paramenter [2,3] that region of memory can have a daxfs created in it 
post-clean boot.


Note: some systems may not flush the memory during a warm reboot, but 
this is dependent on the BIOS.


When your system boots you could do something like the following:

 * If it's a clean boot, create the daxfs
 * If it's an "unclean" boot (e.g. the daxfs already exists, or a 
kernel parameter informs you that it started post-panic) then you can 
copy/move/use lttng-crash to persistent storage for analysis
 * Start tracing using a snapshot session and the userspace buffers on 
the daxfs.


In this type of situation the "snapshot" command is never invoked 
directly, but the recovery of the buffers to create a snapshot is possible.


[1]: https://www.kernel.org/doc/html/latest/admin-guide/kdump/kdump.html
[2]: 
https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
[3]: 
https://docs.pmem.io/persistent-memory/getting-started-guide/creating-development-environments/linux-environments/linux-memmap


thanks,
kienan

Looking at available kernel syscall, the "reboot" one seems like a 
good candidate, however I was not able to capture a snapshot on it. I 
have tested the setup below with "--name=chdir" syscall and it 
works, "cd" to a directory will create a trace. But no dice with reboot.




The details of how this work will depend on your system. For example, my 
installations tend to use systemd as PID 1. The broad strokes seem to 
be: `/usr/sbin/reboot` is actually a link to `systemctl`, which I 
believe then kicks off the reboot.service, the PID 1 is swapped to 
/usr/lib/systemd/systemd-shutdown, sigterm then sigkill are sent to all 
processes, unmounts, syncs, calls the reboot system call [2,3].


As both the sigterm and the unmounts are done before the syscall, 
lttng-sessiond and the consumers will have already shutdown by the time 
it enters.


While this doesn't necessarily help your original question of panics, if 
you want to snapshot before shutdown or reboot and are using systemd, 
it's possible to leave a script or binary in a known directory so that 
it's invoked prior to the rest of the shutdown sequence[4].


[1]: https://lttng.org/docs/v2.13/#doc-persistent-memory-file-systems
[2]: 
https://github.com/systemd/systemd/blob/6533c14997700f74e9ea42121303fc1f5c63e62b/src/shutdown/shutdown.c
[3]: 
https://github.com/systemd/systemd/blob/main/src/shared/reboot-util.c#L77

[4]: https://www.systutorials.com/docs/linux/man/8-systemd-reboot/

hope this helps,
kienan


Would you have any suggestions?
Thanks for your help,
Cheers
Damien



# Prep output dir
mkdir /application/trace/
rm -rf /application/trace/*

# Create session
sudo lttng destroy snapshot-trace-session
sudo lttng create snapshot-trace-session --snapshot 
--output="/application/trace/"

sudo lttng enable-channel --kernel --num-subbuf=8 channelk
sudo lttng enable-channel --userspace --num-subbuf=8 channelu

# Configure session
sudo lttng enable-event --kernel --syscall --all --channel channelk
sudo lttng enable-event --kernel --tracepoint "sched*" --channel channelk
sudo lttng enable-event --userspace --all --channel channelu
sudo lttng add-context -u -t vtid -t procname
sudo lttng remove-trigger trig_reboot
sudo lttng add-trigger --name=trig_reboot \
         --condition=event-rule-matches --type=kernel:syscall:entry \
         --name=reboot\
         --action=snapshot-session snapshot-trace-session \
         --rate-policy=once-after:1

# start & list info
sudo lttng start
sudo 

Re: [lttng-dev] Capturing snapshot on kernel panic

[Kienan Stewart via lttng-dev]

Hi Damien,


On 5/15/24 6:24 PM, Damien Berget via lttng-dev wrote:

Good day,
we have been using LTTng successfully to capture snapshots on user 
defined tracepoints and it did provide invaluable to debug our issues. 
Thanks to all the contributors of this project!


We'd like to know if it would be possible to trigger on a kernel panic? 
I might be dubiously possible as you would still need to have the 
file-system working to write the results but I should ask.




For userspace tracing, I think the recommendation is usually to use a 
dax/pmem device and have the buffers for the session mapped there. After 
a panic, the contents of the buffers can be restored using lttng-crash[1].


Note that dax/pem isn't supported by the kernel space tracer at this time.

If I recall, there are other ways to things in the panic sequence (that 
aren't lttng specific), but I'm personally not as familiar with the 
details of that stage of linux.


Looking at available kernel syscall, the "reboot" one seems like a good 
candidate, however I was not able to capture a snapshot on it. I have 
tested the setup below with "--name=chdir" syscall and it works, "cd" to 
a directory will create a trace. But no dice with reboot.




The details of how this work will depend on your system. For example, my 
installations tend to use systemd as PID 1. The broad strokes seem to 
be: `/usr/sbin/reboot` is actually a link to `systemctl`, which I 
believe then kicks off the reboot.service, the PID 1 is swapped to 
/usr/lib/systemd/systemd-shutdown, sigterm then sigkill are sent to all 
processes, unmounts, syncs, calls the reboot system call [2,3].


As both the sigterm and the unmounts are done before the syscall, 
lttng-sessiond and the consumers will have already shutdown by the time 
it enters.


While this doesn't necessarily help your original question of panics, if 
you want to snapshot before shutdown or reboot and are using systemd, 
it's possible to leave a script or binary in a known directory so that 
it's invoked prior to the rest of the shutdown sequence[4].


[1]: https://lttng.org/docs/v2.13/#doc-persistent-memory-file-systems
[2]: 
https://github.com/systemd/systemd/blob/6533c14997700f74e9ea42121303fc1f5c63e62b/src/shutdown/shutdown.c
[3]: 
https://github.com/systemd/systemd/blob/main/src/shared/reboot-util.c#L77

[4]: https://www.systutorials.com/docs/linux/man/8-systemd-reboot/

hope this helps,
kienan


Would you have any suggestions?
Thanks for your help,
Cheers
Damien



# Prep output dir
mkdir /application/trace/
rm -rf /application/trace/*

# Create session
sudo lttng destroy snapshot-trace-session
sudo lttng create snapshot-trace-session --snapshot 
--output="/application/trace/"

sudo lttng enable-channel --kernel --num-subbuf=8 channelk
sudo lttng enable-channel --userspace --num-subbuf=8 channelu

# Configure session
sudo lttng enable-event --kernel --syscall --all --channel channelk
sudo lttng enable-event --kernel --tracepoint "sched*" --channel channelk
sudo lttng enable-event --userspace --all --channel channelu
sudo lttng add-context -u -t vtid -t procname
sudo lttng remove-trigger trig_reboot
sudo lttng add-trigger --name=trig_reboot \
         --condition=event-rule-matches --type=kernel:syscall:entry \
         --name=reboot\
         --action=snapshot-session snapshot-trace-session \
         --rate-policy=once-after:1

# start & list info
sudo lttng start
sudo lttng list snapshot-trace-session
sudo lttng list-triggers

# test it...
sudo reboot

#=== reconnect and Nothing :(
$ ls -alu /application/trace/
drwxr-xr-x    2 u  u       4096 May 15  2024 .
drwxr-xr-x   10 u  u       4096 May 15  2024 ..


___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng UST Benchmarks

[Kienan Stewart via lttng-dev]

Hi Aditya,

It has been suggested to me that the following publication[1] would also 
be of interest. It gives a good comparison of micro-benchmarking tracers.


[1]: https://dl.acm.org/doi/10.1145/3158644

thanks,
kienan

On 4/25/24 1:53 PM, Kienan Stewart via lttng-dev wrote:

Hi Aditya,

On 4/24/24 11:25 AM, Aditya Kurdunkar via lttng-dev wrote:
Hello everyone, I was working on a use case where I am working on 
enabling LTTng on an embedded ARM device running the OpenBMC linux 
distribution. I have enabled the lttng yocto recipe and I am able to 
trace my code. The one thing I am concerned about is the performance 
overhead. Although the documentation mentions that LTTng has the 
lowest overhead amongst all the available solutions, I am concerned 
about the overhead of the LTTng UST in comparison to 
other available tracers/profilers. I have used the benchmarking setup 
from lttng-ust/tests/benchmark at master · lttng/lttng-ust 
(github.com) 
<https://github.com/lttng/lttng-ust/tree/master/tests/benchmark> to 
benchmark the overhead of the tracepoints (on the device). The 
benchmark, please correct me if I am wrong, gives the overhead of a 
single tracepoint in your code.


This seems to be what it does.

Although this might be fine for now, I
was just wondering if there are any published benchmarks comparing 
LTTng with the available tracing/profiling solutions. 


I don't know of any published ones that do an exhaustive comparison.

There is this one[1] which references a comparison with some parts of 
eBPF. The source for the benchmarking is also available[2].


If not, how can I go

about benchmarking the overhead of the applications?



I'm not really sure how to answer you here.

I guess the most pertinent to your use case is to test your application 
with and without tracing to see the complete effect?


It would be good to have a dedicated system, disable CPU frequency 
scaling, and to perform the tests repeatedly and measure the mean, 
median, and standard deviation.


You could pull methodological inspiration from prior publications[3], 
which while outdated in terms of software version and hardware 
demonstrate the process of creating and comparing benchmarks.


It would also be useful to identify how your application and tracing 
setup works, and to understand which parts of the system you are 
interested in measuring.


For example, the startup time of tracing rapidly spawning processes will 
depend on the type of buffering scheme in use, if the tracing 
infrastructure is loaded before or after forking, etc.


Your case might be a long running application and you aren't interested 
in startup time performance but more concretely the impact of the static 
instrumentation on one of your hot paths.


If you're not sure what kind of tracing setups work best in your case, 
or would like us to characterize at certain aspect of the tool-set's 
performance, EfficiOS[4] offers consultation and support for 
instrumentation and performance in applications.


I have come across the lttng/lttng-ust-benchmarks (github.com) 
<https://github.com/lttng/lttng-ust-benchmarks> repository which has 
no documentation on how to run it, apart from one commit message on 
how to run the benchmark script.




To run those benchmarks when you have babeltrace2, lttng-tools, urcu, 
lttng-ust, and optional lttng-modules installed:


```
$ make
$ python3 ./benchmark.py
```

This should produce a file, `benchmarks.json`

You can also inspect how the CI job runs it: 
https://ci.lttng.org/view/LTTng-ust/job/lttng-ust-benchmarks_master_linuxbuild/



Any help is really appreciated. Thank you.

Regards,
Aditya

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


[1]: 
https://tracingsummit.org/ts/2022/files/Tracing_Summit_2022-LTTng_Beyond_Ring-Buffer_Based_Tracing_Jeremie_Galarneau_.pdf

[2]: https://github.com/jgalar/LinuxCon2022-Benchmarks
[3]: https://www.dorsal.polymtl.ca/files/publications/desnoyers.pdf
[4]: https://www.efficios.com/contact/

thanks,
kienan
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng UST Benchmarks

[Kienan Stewart via lttng-dev]

Hi Aditya,

On 4/24/24 11:25 AM, Aditya Kurdunkar via lttng-dev wrote:
Hello everyone, I was working on a use case where I am working on 
enabling LTTng on an embedded ARM device running the OpenBMC linux 
distribution. I have enabled the lttng yocto recipe and I am able to 
trace my code. The one thing I am concerned about is the performance 
overhead. Although the documentation mentions that LTTng has the lowest 
overhead amongst all the available solutions, I am concerned about the 
overhead of the LTTng UST in comparison to 
other available tracers/profilers. I have used the benchmarking setup 
from lttng-ust/tests/benchmark at master · lttng/lttng-ust (github.com) 
 to 
benchmark the overhead of the tracepoints (on the device). The 
benchmark, please correct me if I am wrong, gives the overhead of a 
single tracepoint in your code.


This seems to be what it does.

Although this might be fine for now, I
was just wondering if there are any published benchmarks comparing LTTng 
with the available tracing/profiling solutions. 


I don't know of any published ones that do an exhaustive comparison.

There is this one[1] which references a comparison with some parts of 
eBPF. The source for the benchmarking is also available[2].


If not, how can I go

about benchmarking the overhead of the applications?



I'm not really sure how to answer you here.

I guess the most pertinent to your use case is to test your application 
with and without tracing to see the complete effect?


It would be good to have a dedicated system, disable CPU frequency 
scaling, and to perform the tests repeatedly and measure the mean, 
median, and standard deviation.


You could pull methodological inspiration from prior publications[3], 
which while outdated in terms of software version and hardware 
demonstrate the process of creating and comparing benchmarks.


It would also be useful to identify how your application and tracing 
setup works, and to understand which parts of the system you are 
interested in measuring.


For example, the startup time of tracing rapidly spawning processes will 
depend on the type of buffering scheme in use, if the tracing 
infrastructure is loaded before or after forking, etc.


Your case might be a long running application and you aren't interested 
in startup time performance but more concretely the impact of the static 
instrumentation on one of your hot paths.


If you're not sure what kind of tracing setups work best in your case, 
or would like us to characterize at certain aspect of the tool-set's 
performance, EfficiOS[4] offers consultation and support for 
instrumentation and performance in applications.


I have come across the lttng/lttng-ust-benchmarks (github.com) 
 repository which has no 
documentation on how to run it, apart from one commit message on how to 
run the benchmark script.




To run those benchmarks when you have babeltrace2, lttng-tools, urcu, 
lttng-ust, and optional lttng-modules installed:


```
$ make
$ python3 ./benchmark.py
```

This should produce a file, `benchmarks.json`

You can also inspect how the CI job runs it: 
https://ci.lttng.org/view/LTTng-ust/job/lttng-ust-benchmarks_master_linuxbuild/



Any help is really appreciated. Thank you.

Regards,
Aditya

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


[1]: 
https://tracingsummit.org/ts/2022/files/Tracing_Summit_2022-LTTng_Beyond_Ring-Buffer_Based_Tracing_Jeremie_Galarneau_.pdf

[2]: https://github.com/jgalar/LinuxCon2022-Benchmarks
[3]: https://www.dorsal.polymtl.ca/files/publications/desnoyers.pdf
[4]: https://www.efficios.com/contact/

thanks,
kienan
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] [babeltrace2]about python self-defined plugin loading

[Kienan Stewart via lttng-dev]

My apologies,

there is a typo in my previous e-mail, the library directory should be:

`/babeltrace2/plugin-providers/'

thanks,
kienan

On 4/25/24 11:41 AM, Kienan Stewart via lttng-dev wrote:

Hi Amanda,

could you double-check to ensure that babeltrace2 was built with 
`--enable-python-plugins`, and that `import bt2` works?


There should be a babeltrace2-python-plugin-provider.so in 
`/babeltrace2/plugin-provides`


thanks,
kienan


On 4/24/24 11:28 PM, Wu, Yannan via lttng-dev wrote:

Hi, There,

I am trying to construct a customized filter and sink based on 
babeltrace2 python binding. However, nether mine plugin nor the 
plugins sample I could find on the internet all dont work.


For example, 
https://github.com/simark/babeltrace-fun-plugins/tree/master/my-first-components <https://github.com/simark/babeltrace-fun-plugins/tree/master/my-first-components>


I just downloaded the py file and run the exact command, it failed. 
The log is as following:



babeltrace2 --plugin-path . -c source.demo.MyFirstSource -c 
sink.demo.MyFirstSink
04-24 16:52:04.349 919805 919805 E CLI 
add_descriptor_to_component_descriptor_set@babeltrace2.c:1720 Cannot 
find component class: plugin-name="demo", 
comp-cls-name="MyFirstSource", comp-cls-type=1
04-24 16:52:04.349 919805 919805 E CLI 
cmd_run_ctx_init@babeltrace2.c:1882 Cannot find an operative message 
interchange protocol version to use to create the `run` command's 
graph: status=ERROR
04-24 16:52:04.349 919805 919805 E CLI cmd_run@babeltrace2.c:2465 
Cannot initialize the command's context.


ERROR:    [Babeltrace CLI] (babeltrace2.c:2465)
   Cannot initialize the command's context.
CAUSED BY [Babeltrace CLI] (babeltrace2.c:1882)
   Cannot find an operative message interchange protocol version to 
use to create the `run` command's graph: status=ERROR

CAUSED BY [Babeltrace CLI] (babeltrace2.c:1720)
   Cannot find component class: plugin-name="demo", 
comp-cls-name="MyFirstSource", comp-cls-type=1


babeltrace2 --version
Babeltrace 2.0.7 "Amqui" [v2.0.6-1-g825a0ed6d]

Amqui (/ɒmkwiː/) is a town in eastern Québec, Canada, at the base of 
the Gaspé peninsula in Bas-Saint-Laurent. Located at the
confluence of the Humqui and Matapédia Rivers, its proximity to 
woodlands makes it a great destination for outdoor activities such as

camping, hiking, and mountain biking.
yannanwu@ue91e96f2951b5c:~/trees/lttng_test_run$

Is the cli changed or something? How can I make it right?

Besides, is it possible we create a pipeline in python and make use 
the the python drafted plugin? Can you advise me how?


Amanda



___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] [babeltrace2]about python self-defined plugin loading

[Kienan Stewart via lttng-dev]

Hi Amanda,

could you double-check to ensure that babeltrace2 was built with 
`--enable-python-plugins`, and that `import bt2` works?


There should be a babeltrace2-python-plugin-provider.so in 
`/babeltrace2/plugin-provides`


thanks,
kienan


On 4/24/24 11:28 PM, Wu, Yannan via lttng-dev wrote:

Hi, There,

I am trying to construct a customized filter and sink based on 
babeltrace2 python binding. However, nether mine plugin nor the plugins 
sample I could find on the internet all dont work.


For example, 
https://github.com/simark/babeltrace-fun-plugins/tree/master/my-first-components 


I just downloaded the py file and run the exact command, it failed. The 
log is as following:



babeltrace2 --plugin-path . -c source.demo.MyFirstSource -c 
sink.demo.MyFirstSink
04-24 16:52:04.349 919805 919805 E CLI 
add_descriptor_to_component_descriptor_set@babeltrace2.c:1720 Cannot 
find component class: plugin-name="demo", comp-cls-name="MyFirstSource", 
comp-cls-type=1
04-24 16:52:04.349 919805 919805 E CLI 
cmd_run_ctx_init@babeltrace2.c:1882 Cannot find an operative message 
interchange protocol version to use to create the `run` command's graph: 
status=ERROR
04-24 16:52:04.349 919805 919805 E CLI cmd_run@babeltrace2.c:2465 Cannot 
initialize the command's context.


ERROR:    [Babeltrace CLI] (babeltrace2.c:2465)
   Cannot initialize the command's context.
CAUSED BY [Babeltrace CLI] (babeltrace2.c:1882)
   Cannot find an operative message interchange protocol version to use 
to create the `run` command's graph: status=ERROR

CAUSED BY [Babeltrace CLI] (babeltrace2.c:1720)
   Cannot find component class: plugin-name="demo", 
comp-cls-name="MyFirstSource", comp-cls-type=1


babeltrace2 --version
Babeltrace 2.0.7 "Amqui" [v2.0.6-1-g825a0ed6d]

Amqui (/ɒmkwiː/) is a town in eastern Québec, Canada, at the base of the 
Gaspé peninsula in Bas-Saint-Laurent. Located at the
confluence of the Humqui and Matapédia Rivers, its proximity to 
woodlands makes it a great destination for outdoor activities such as

camping, hiking, and mountain biking.
yannanwu@ue91e96f2951b5c:~/trees/lttng_test_run$

Is the cli changed or something? How can I make it right?

Besides, is it possible we create a pipeline in python and make use the 
the python drafted plugin? Can you advise me how?


Amanda



___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Monitor user space threads without user events

[Kienan Stewart via lttng-dev]

Hi Zvika,

with a session that has kernel events enabled, I think you could use the 
sched_switch event.


thanks,
kienan

On 4/6/24 12:20 AM, Zvi Vered via lttng-dev wrote:

Hello,

My user space process contains 2 threads:

First thread is blocked upon a kernel event created by a hardware 
interrupt handled by the kernel.

The  thread is sending ioctl which is blocked till this interrupt occurs.

Second thread has just a 5msec sleep. It does something and then sleeps 
for 5msec. Forever.


Is it possible to see when each thread is running (and not blocked or 
sleeps) ?
I know how to do it with user events. I wonder if it's possible without 
any extra code.


Thank you,
Zvika

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] babeltrace2 on windows

[Kienan Stewart via lttng-dev]

Hi Zvika,

babeltrace2 with the python bindings can be built on windows. Our CI 
does this in two environments: cygwin64 and msys2-ucrt64.


thanks,
kienan


On 4/4/24 4:08 PM, Zvi Vered via lttng-dev wrote:

Hello,

I'm required to analyze an lttng trace with Babeltrace2 python API.
The host is running windows 10.
But according to : https://babeltrace.org/#bt2-get 


Windows is not mentioned at all.

I prefer python but I can also consider using the C library for windows.

Thank you,
Zvika


___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] lttng-modules(13.9) : insmod order

[Kienan Stewart via lttng-dev]

Hi Zvika,

a sessiond launched as root will automatically try to load the 
appropriate kernel modules as mentioned here[1], using either modprobe 
or libkmod depending on how the build was configured.


thanks,
kienan

[1]: https://lttng.org/docs/v2.13/#doc-lttng-modules


On 3/30/24 10:52 PM, Zvi Vered via lttng-dev wrote:

Hello,

For some reason, I failed to integrate v13.9 into buildroot 2023.02.2
But I compiled all modules with the my kernel source: 5.4.249

Can you please tell what is the right order of *insmod *for the kernel 
modules ?


All I need is to record IRQ events and user space events.

Thank you,
Zvika

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTNG Continuously Crashing on Debian-12

[Kienan Stewart via lttng-dev]

Hi Lakshmi,

I'd like to encourage once again to review the bug reporting guidelines 
at https://lttng.org/community/ and include all the necessary 
information including steps to reproduce the issue. Thank you.


On 3/12/24 1:51 PM, Lakshmi Deverkonda wrote:

Hi,

We see that python3 based lttng is continuously crashing on debian-12.
Kernel Version is 6.1.0. Is there some special handling that has to be 
taken care for debian-12?




To the best of my knowledge there are no special cases for Debian 12 in 
upstream lttng-ust. The Debian source package carries two minor patches 
from what I can see, neither of which seem directly related to the 
python agent.



Following is the core decode.


Could you install the relevant "-dbgsym" packages for libc and 
liblttng-ust or fetch the debug symbols from debuginfod so that the 
addresses are resolved into meaningful symbols?



Program terminated with signal SIGABRT, Aborted.
#0  0x7fb95dac9e2c in ?? () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7fb95d96c040 (LWP 19426))]
(gdb) bt
#0  0x7fb95dac9e2c in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x7fb95da7afb2 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x7fb95da65472 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x7fb95da65395 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x7fb95da73eb2 in __assert_fail () from 
/lib/x86_64-linux-gnu/libc.so.6

#5  0x7fb95d9efbbf in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1
#6  0x7fb95d9f0f23 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1
#7  0x7fb95d9ece2f in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1
#8  0x7fb95d9dc537 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1
#9  0x7fb95d9c73c2 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1
#10 0x7fb95d9c8003 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1
#11 0x7fb95d9c8003 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1
#12 0x7fb95d9c8003 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1
#13 0x7fb95d9c8fce in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1
#14 0x7fb95d9c2408 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1
#15 0x7fb95dd6812a in ?? () from /lib64/ld-linux-x86-64.so.2
#16 0x7fb95dd6b764 in ?? () from /lib64/ld-linux-x86-64.so.2
#17 0x7fb95da7d55d in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#18 0x7fb95da7d69a in exit () from /lib/x86_64-linux-gnu/libc.so.6
#19 0x7fb95da66251 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#20 0x7fb95da66305 in __libc_start_main () from 
/lib/x86_64-linux-gnu/libc.so.6

#21 0x00627461 in _start ()

Regards,
Lakshmi


thanks,
kienan
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] tracing user event from multiple dockers

[Kienan Stewart via lttng-dev]

Hi Zvika,

To share a sessiond the LTTng home and run directories, and the shared 
memory should be shared between the containers.


Eg. https://github.com/mjeanson/gcloud-lttng/blob/master/pods/demo1.yaml.in

In a scenario with multiple sessionds (eg. one per container), the 
traces could be combined in post using babeltrace.


thanks,
kienan


On 3/2/24 12:31 AM, Zvi Vered via lttng-dev wrote:

Hello,

I have an application working with multiple dockers.
Each docker is a fedora O.S running an application.
The host is centos 8.5. This host will be upgraded to RHEL 9.2 in the 
near future.
Is it possible that all traces from multiple dockers will all be written 
to one metadata ?


versions:
lttng-tools-latest-2.13.tar.bz2
lttng-ust-latest-2.13.tar.bz2
userspace-rcu-latest-0.14.tar.bz2

Thank you,
Zvika

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Crash in application due to watchdog timeout with python3 lttng

[Kienan Stewart via lttng-dev]

Hi Lakshmi,

On 2/20/24 10:24 AM, Lakshmi Deverkonda wrote:

Hi,

Sorry, I could not reply on this thread. We tried to handle these issues 
in our application code.
One query, what is the write way to delete the lttng logger to stop 
logging the events(from the code).



I think that the design of LTTng-UST agents is such that tracing is not 
meant to be tightly controlled by the traced applications themselves.


To stop emitting events:

 * disable the python events for the session(s): `lttng disable-event 
--python --all-events --session= --channel=`

 * stop the tracing session(s)`lttng stop `
 * quit the `sessiond`

Note: the agent thread(s) in the python application will continue to 
run, but not emit events.


If you want to do things from within the application, there are some 
options:


 * Implement your own logic to stop calling `self.lttng_logger.log()`
 * Add a filter to the `lttngust.loghandler._Handler` objects on the 
root logger
 * Forcibly remove the `lttngust.loghandler._Handler` handler(s) from 
root logger (this really isn't meant to be done)
 * Use liblttngctl to perform one of the above actions (disable the 
python event(s), stopping the tracing sessions(s), ...)


thanks,
kienan




Regards,
Lakshmi

*From:* Kienan Stewart 
*Sent:* 16 February 2024 22:11
*To:* Lakshmi Deverkonda ; lttng-dev@lists.lttng.org 

*Subject:* Re: [lttng-dev] Crash in application due to watchdog timeout 
with python3 lttng

External email: Use caution opening links or attachments


Hi Lakshmi,

On 2/16/24 09:33, Lakshmi Deverkonda wrote:

This is how, we have created the logger. So the first logger is for file
logging where is as the second one is for lttng.

  self.logger  = logging.getLogger('cd')
self.lttng_logger = logging.getLogger('cd-lttng')

It seems like at the instant exactly when lttng is logging some data on
a particular thread and the same instant we receive SIGTERM for the
application,
we are unable to join that particular thread. Can you please help.



This doesn't constitute a usable reproducer for us. You are also
omitting information on the setup and usage of lttng on your system.

I get the impression you are not in a position to share your code.
EfficiOS offers support contracts with NDAs that could allow us to work
with you to analyze and improve your use of LTTng. For more info, please
feel free to contact sa...@efficios.com.


Also we see that performance of lttng is not that good for python3. My
application has around 24 threads and when logging is enabled for each
of the threads,
there is a delay upto 24s for processing the external events.
Please suggest how to proceed further on these issues.


Could you describe what you mean by 'processing external events'?

Which system(s) are involved in processing the events?

Are the 'external events' the events emitted by invoking
`self.lttng_logger.info('...')`, for example?

What versions of lttng-tools, lttng-ust, urcu, babeltrace, and python3
are you using? Are you using a relay-daemon at any point?

How are your lttng sessions configured? Eg. memory allocation, blocking
settings, behaviour on full buffers, etc. The commands you use to create
the session, enable the channels, and activate the events would be great
information to have.

While performing the logging is the system under heavy load from other
sources? What resources on the system face the most contention (CPU, IO,
memory, ...)?

We'd be more than happy to analyze the performance of python-lttngust
and work to make improvements so it can meet your needs under a
development contract. For more information, please reach out to
sa...@efficios.com.



Regards,
Lakshmi


I have taken the time to invent a fictitious example based on the few
details you have given:
https://gist.github.com/kienanstewart/879bd3bf19d852653b70a3c42caef361 

which spawns a number of python threads using the threading module.

I am using lttng-tools master, lttng-ust master at
47bc09f338f3c1199a878f77b7b18be8d2a224f6, urcu master at
81270292c23ff28aba1abd9a65f0624b657de82b, and babeltrace2 master at
b93af5a2d22e36cf547da1739d60e19791daccbd. My system is running Debian
sid with python 3.11.8.

To set up a recording session I do the following:

```
lttng create
lttng enable-event --python 'tp'
lttng start
```

To run the application, I do the following:

```
time python3 ./main.py
```

To quit the application, I send sigterm using the following command

```
killall $(pgrep -f 'python3 ./main.py')
```

After the application terminates, I stop the session and view the events

```
lttng stop
lttng view
lttng view | wc -l
```

In a 25s run of the application on my 4-thread laptop, I recorded
1010748 events.

thanks,
kienan




*From:* Lakshmi Deverkonda 
*Sent:* 13 February 2024 21:05
*To:* Kienan 

Re: [lttng-dev] Crash in application due to watchdog timeout with python3 lttng

[Kienan Stewart via lttng-dev]

Hi Lakshmi,

On 2/16/24 09:33, Lakshmi Deverkonda wrote:
This is how, we have created the logger. So the first logger is for file 
logging where is as the second one is for lttng.


  self.logger  = logging.getLogger('cd')
self.lttng_logger = logging.getLogger('cd-lttng')

It seems like at the instant exactly when lttng is logging some data on 
a particular thread and the same instant we receive SIGTERM for the 
application,

we are unable to join that particular thread. Can you please help.



This doesn't constitute a usable reproducer for us. You are also 
omitting information on the setup and usage of lttng on your system.


I get the impression you are not in a position to share your code. 
EfficiOS offers support contracts with NDAs that could allow us to work 
with you to analyze and improve your use of LTTng. For more info, please 
feel free to contact sa...@efficios.com.


Also we see that performance of lttng is not that good for python3. My 
application has around 24 threads and when logging is enabled for each 
of the threads,

there is a delay upto 24s for processing the external events.
Please suggest how to proceed further on these issues.


Could you describe what you mean by 'processing external events'?

Which system(s) are involved in processing the events?

Are the 'external events' the events emitted by invoking 
`self.lttng_logger.info('...')`, for example?


What versions of lttng-tools, lttng-ust, urcu, babeltrace, and python3 
are you using? Are you using a relay-daemon at any point?


How are your lttng sessions configured? Eg. memory allocation, blocking 
settings, behaviour on full buffers, etc. The commands you use to create 
the session, enable the channels, and activate the events would be great 
information to have.


While performing the logging is the system under heavy load from other 
sources? What resources on the system face the most contention (CPU, IO, 
memory, ...)?


We'd be more than happy to analyze the performance of python-lttngust 
and work to make improvements so it can meet your needs under a 
development contract. For more information, please reach out to 
sa...@efficios.com.




Regards,
Lakshmi


I have taken the time to invent a fictitious example based on the few 
details you have given: 
https://gist.github.com/kienanstewart/879bd3bf19d852653b70a3c42caef361 
which spawns a number of python threads using the threading module.


I am using lttng-tools master, lttng-ust master at 
47bc09f338f3c1199a878f77b7b18be8d2a224f6, urcu master at 
81270292c23ff28aba1abd9a65f0624b657de82b, and babeltrace2 master at 
b93af5a2d22e36cf547da1739d60e19791daccbd. My system is running Debian 
sid with python 3.11.8.


To set up a recording session I do the following:

```
lttng create
lttng enable-event --python 'tp'
lttng start
```

To run the application, I do the following:

```
time python3 ./main.py
```

To quit the application, I send sigterm using the following command

```
killall $(pgrep -f 'python3 ./main.py')
```

After the application terminates, I stop the session and view the events

```
lttng stop
lttng view
lttng view | wc -l
```

In a 25s run of the application on my 4-thread laptop, I recorded 
1010748 events.


thanks,
kienan




*From:* Lakshmi Deverkonda 
*Sent:* 13 February 2024 21:05
*To:* Kienan Stewart ; lttng-dev@lists.lttng.org 

*Subject:* Re: [lttng-dev] Crash in application due to watchdog timeout 
with python3 lttng
Yes. We are trying to join only the threads related to the application. 
The timeout is happening while trying to join the threads started by the 
application.


Regards,
Lakshmi

*From:* Kienan Stewart 
*Sent:* 13 February 2024 20:50
*To:* Lakshmi Deverkonda ; lttng-dev@lists.lttng.org 

*Subject:* Re: [lttng-dev] Crash in application due to watchdog timeout 
with python3 lttng

External email: Use caution opening links or attachments


Hi Lakshmi,

when the lttngust python agent starts, it attempts to connect to one or
more session daemons[1].

Each connection starts a thread that loops forever, retrying the
registration in case an exception occurs[2].

I don't think the it's designed to have `join()` called on those
threads, which I assume is happening in some of the code you or your
team have written.

My initial thought is that you should `join()` only the threads that
pertinent to your application, ignoring the lttngust agent threads and
then exit the application as normal.

[1]:
https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L334
 

[2]:
https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L83
 

Re: [lttng-dev] Crash in application due to watchdog timeout with python3 lttng

[Kienan Stewart via lttng-dev]

Hi Lakshmi,

On 2/13/24 10:35, Lakshmi Deverkonda wrote:
Yes. We are trying to join only the threads related to the application. 
The timeout is happening while trying to join the threads started by the 
application.


In that case, I suspect that the issue is not related to lttngust. I 
can't help with your internal application code.


If you're able to produce a minimal example that reproduces an issue 
wherein you have deadlock when lttngust is imported, but not when it's 
omitted I think that would be very interesting.


I would also recommend reviewing the bug reporting guidelines at 
https://lttng.org/community/ to ensure that all the necessary 
information is present.


thanks,
kienan



Regards,
Lakshmi

*From:* Kienan Stewart 
*Sent:* 13 February 2024 20:50
*To:* Lakshmi Deverkonda ; lttng-dev@lists.lttng.org 

*Subject:* Re: [lttng-dev] Crash in application due to watchdog timeout 
with python3 lttng

External email: Use caution opening links or attachments


Hi Lakshmi,

when the lttngust python agent starts, it attempts to connect to one or
more session daemons[1].

Each connection starts a thread that loops forever, retrying the
registration in case an exception occurs[2].

I don't think the it's designed to have `join()` called on those
threads, which I assume is happening in some of the code you or your
team have written.

My initial thought is that you should `join()` only the threads that
pertinent to your application, ignoring the lttngust agent threads and
then exit the application as normal.

[1]:
https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L334
 

[2]:
https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L83
 


thanks,
kienan

On 2/13/24 09:23, Lakshmi Deverkonda via lttng-dev wrote:

Hi,

We are able to integrate python3 lttng module in our application(python3
based). However, we are seeing that whenever the application terminates,
there is watchdog timeout due to timeout in joining the threads. What
could be the reason for this ? Does lttng module hold any thread event
locks ?
We are completely blocked on this issue. Could you please help ?

Here is the snippet of the core dump

(gdb) py-bt
Traceback (most recent call first):
    File "/usr/lib/python3.7/threading.py", line 1048, in
_wait_for_tstate_lock
  elif lock.acquire(block, timeout):
    File "/usr/lib/python3.7/threading.py", line 1032, in join
  self._wait_for_tstate_lock()
    File "/usr/lib/python3/dist-packages/h.py", line 231, in JoinThreads
  self.TT.join()
    File "/usr/sbin/c", line 1466, in do_exit
  H.JoinThreads()
    File "/usr/sbin/c", line 7201, in main
  do_exit(nlm, status)
    File "/usr/sbin/c", line 7233, in 
  main()
(gdb)

On a parallel note, thanks to Kienan who has been trying to provide
pointers on various issues reported so far.

Need help on this issue as well.
Thanks in advance,

Regards,
Lakshmi



___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev 


___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Crash in application due to watchdog timeout with python3 lttng

[Kienan Stewart via lttng-dev]

Hi Lakshmi,

when the lttngust python agent starts, it attempts to connect to one or 
more session daemons[1].


Each connection starts a thread that loops forever, retrying the 
registration in case an exception occurs[2].


I don't think the it's designed to have `join()` called on those 
threads, which I assume is happening in some of the code you or your 
team have written.


My initial thought is that you should `join()` only the threads that 
pertinent to your application, ignoring the lttngust agent threads and 
then exit the application as normal.


[1]: 
https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L334
[2]: 
https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L83


thanks,
kienan

On 2/13/24 09:23, Lakshmi Deverkonda via lttng-dev wrote:

Hi,

We are able to integrate python3 lttng module in our application(python3 
based). However, we are seeing that whenever the application terminates, 
there is watchdog timeout due to timeout in joining the threads. What 
could be the reason for this ? Does lttng module hold any thread event 
locks ?

We are completely blocked on this issue. Could you please help ?

Here is the snippet of the core dump

(gdb) py-bt
Traceback (most recent call first):
   File "/usr/lib/python3.7/threading.py", line 1048, in 
_wait_for_tstate_lock

     elif lock.acquire(block, timeout):
   File "/usr/lib/python3.7/threading.py", line 1032, in join
     self._wait_for_tstate_lock()
   File "/usr/lib/python3/dist-packages/h.py", line 231, in JoinThreads
     self.TT.join()
   File "/usr/sbin/c", line 1466, in do_exit
     H.JoinThreads()
   File "/usr/sbin/c", line 7201, in main
     do_exit(nlm, status)
   File "/usr/sbin/c", line 7233, in 
     main()
(gdb)

On a parallel note, thanks to Kienan who has been trying to provide 
pointers on various issues reported so far.


Need help on this issue as well.
Thanks in advance,

Regards,
Lakshmi



___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Hooking up callbacks to LTTng USDT Events

[Kienan Stewart via lttng-dev]

Hi Adel,

It sounds to me like the feature you might want to try and use is the 
combining triggers with the user space application notification action.


C.f. https://lttng.org/docs/v2.13/#doc-trigger
C.f. https://lttng.org/docs/v2.13/#doc-trigger-event-notif

thanks,
kienan

On 2/6/24 14:32, Adel Belkhiri via lttng-dev wrote:

Hi everyone,

Hope you're all good. I'm working on something where I need to set up 
routines or callbacks for specific LTTng USDT events (sticking to user 
space, not kernel). The plan is to catch these events as they happen to 
spot performance issues.


If anyone's done something like this or has pointers on how to pull it 
off, I'd really appreciate your advice.


Thanks!


___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng sessiond daemon Assertion `buf' failed and killed

[Kienan Stewart via lttng-dev]

Hi Yonghong,

thanks for the extra info. As you say without the symbol resolution the 
backtrace isn't super useful.


Based on the build-id in the file info for lttng-sessiond, it looks like 
you might be using the lttng/stable-2.13 ppa.


In that case, you could try using debuginfod with gdb to automatically 
download the symbols. See 
https://ubuntu.com/server/docs/service-debuginfod for the instructions 
enabling the symbol download from Ubuntu's debuginfod server.


thanks,
kienan

On 1/12/24 15:38, Yonghong Yan wrote:
Coredump is about 124MB and I am copying the backtrace of the coredump, 
see below. Unfortunately, the sessiond is stripped and no debugging 
info, so not able to see the the call stack that lead to the dump. guess 
I need to build the lttng with debugging info and to get the symbol and 
informed backtrace.


 From the log itself, it is asserted aat >      > DBG1 - 
00:24:25.241138386 [Client management]: Setting relayd for

      > session auto-20240112-002417 (in cmd_setup_relayd() at cmd.c:1004)
      >
      > lttng-sessiond: unix.c:185: lttcomm_recv_unix_sock: Assertion
     `buf' failed.


*yyan7@CCI13SZWP3LWS*:*/var/log*$ tail apport.log


ERROR: apport (pid 3787) Fri Jan 12 10:52:37 2024: debug: session gdbus 
call:


ERROR: apport (pid 3787) Fri Jan 12 10:52:37 2024: this executable 
already crashed 2 times, ignoring


ERROR: apport (pid 10543) Fri Jan 12 15:18:07 2024: called for pid 
10470, signal 6, core limit 18446744073709551615, dump mode 1


ERROR: apport (pid 10543) Fri Jan 12 15:18:07 2024: ignoring implausibly 
big core limit, treating as unlimited


ERROR: apport (pid 10543) Fri Jan 12 15:18:07 2024: executable: 
/usr/bin/lttng-sessiond (command line "lttng-sessiond -vvv 
--verbose-consumer -b")


ERROR: apport (pid 10543) Fri Jan 12 15:18:08 2024: debug: session gdbus 
call: (true,)



ERROR: apport (pid 10543) Fri Jan 12 15:18:08 2024: writing core dump to 
core._usr_bin_lttng-sessiond.1000.8f22faa6-512c-49ca-9089-8b3f0801da8d.10470.2023109 (limit: -1)


ERROR: apport (pid 10543) Fri Jan 12 15:18:08 2024: this executable 
already crashed 2 times, ignoring


*yyan7@CCI13SZWP3LWS*:*/var/log*$ cd /var/lib/apport/coredump

*yyan7@CCI13SZWP3LWS*:*/var/lib/apport/coredump*$ ls

core._usr_bin_lttng-sessiond.1000.8f22faa6-512c-49ca-9089-8b3f0801da8d.10470.2023109

*yyan7@CCI13SZWP3LWS*:*/var/lib/apport/coredump*$ file 
core._usr_bin_lttng-sessiond.1000.8f22faa6-512c-49ca-9089-8b3f0801da8d.10470.2023109


core._usr_bin_lttng-sessiond.1000.8f22faa6-512c-49ca-9089-8b3f0801da8d.10470.2023109:
 ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 
'lttng-sessiond -vvv --verbose-consumer -b', real uid: 1000, effective uid: 
1000, real gid: 1000, effective gid: 1000, execfn: '/usr/bin/lttng-sessiond', 
platform: 'x86_64'

*yyan7@CCI13SZWP3LWS*:*/var/lib/apport/coredump*$ du -h 
core._usr_bin_lttng-sessiond.1000.8f22faa6-512c-49ca-9089-8b3f0801da8d.10470.2023109


124Mcore._usr_bin_lttng-sessiond.1000.8f22faa6-512c-49ca-9089-8b3f0801da8d.10470.2023109

*yyan7@CCI13SZWP3LWS*:*/var/lib/apport/coredump*$ gdb 
/usr/bin/lttng-sessiond 
core._usr_bin_lttng-sessiond.1000.8f22faa6-512c-49ca-9089-8b3f0801da8d.10470.2023109


*GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1*

Copyright (C) 2022 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later 
>


This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

Type "show copying" and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

>.


Find the GDB manual and other documentation resources online at:

>.



For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from /usr/bin/lttng-sessiond...

(No debugging symbols found in /usr/bin/lttng-sessiond)

[New LWP 10479]

[New LWP 10470]

[New LWP 10476]

[New LWP 10475]

[New LWP 10472]

[New LWP 10478]

[New LWP 10480]

[New LWP 10473]

[New LWP 10477]

[New LWP 10482]

[New LWP 10542]

[New LWP 10533]

[New LWP 10484]

[New LWP 10474]

[New LWP 10481]

[New LWP 10483]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

--Type  for more, q to quit, c to continue without paging--

Core was generated by `lttng-sessiond -vvv --verbose-consumer -b'.

Program terminated with signal SIGABRT, Aborted.

#0__pthread_kill_implementation(no_tid=0, signo=6, 
threadid=140614729455104) at ./nptl/pthread_kill.c:44


44./nptl/pthread_kill.c: No such file or directory.

[Current thread is 1 (Thread 0x7fe36affd600 (LWP 10479))]

Re: [lttng-dev] LTTng sessiond daemon Assertion `buf' failed and killed

[Kienan Stewart via lttng-dev]

Hi Yonghong,

thanks for the additional information. Would you be willing and able to 
share a coredump and/or backtrace of the crash?


I was still unable to reproduce the issue using the commands you 
provided, but am interested in understanding what is happening here.


Are there any configuration options other than LTTNG_UST_DEBUG=1 set in 
your environment?


Are you using urcu/lttng from packages (if so, which repo), or built 
from source?


The test case I am using now:

```
# As root
$ lttng-sessiond -b

# As a non-root user which is not a member of the tracing group
$ export LTTNG_UST_DEBUG=1
$ lttng-relayd -v -b
$ lttng-sessiond -vvv --verbose-consumer -b
$ lttng create
$ lttng enable-event -u -a
```

thanks,
kienan

P.S. In the future, could you keep the lttng-dev in CC? thanks!

On 1/12/24 10:36, Yonghong Yan wrote:

Hi Kienan,

Thank you for checking.  It might be just my setting problem. I will 
just send it to you before we are sure it is an issue of lttng


Below is the output and steps I followed the guideline. Since I started 
the relayd and sessiond in the same terminal, the verbose msg are mixed 
together. I can regenerate it using two separate terminals. It is a Dell 
Precision Tower box and it worked well yesterday. But I rebuilt the 
system yesterday (because of NVIDIA GPU driver issue), and it then have 
this issue.



*yyan7@CCI13SZWP3LWS*:*~*$ uname -a

Linux CCI13SZWP3LWS 6.5.0-14-generic #14~22.04.1-Ubuntu SMP 
PREEMPT_DYNAMIC Mon Nov 20 18:15:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux


*yyan7@CCI13SZWP3LWS*:*~*$ cat /etc/lsb-release

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=22.04

DISTRIB_CODENAME=jammy

DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"

*yyan7@CCI13SZWP3LWS*:*~*$ lttng --version

lttng (LTTng Trace Control) 2.13.10 - Nordicité

*yyan7@CCI13SZWP3LWS*:*~*$ lsmod | grep lttng

*lttng*_probe_writeback614400

*lttng*_probe_workqueue204800

*lttng*_probe_vmscan 450560

*lttng*_probe_udp122880

*lttng*_probe_timer327680

*lttng*_probe_sunrpc 204800

*lttng*_probe_statedump573440

*lttng*_probe_sock 163840

*lttng*_probe_skb163840

*lttng*_probe_signal 163840

*lttng*_probe_scsi 204800

*lttng*_probe_sched450560

*lttng*_probe_regulator204800

*lttng*_probe_rcu122880

*lttng*_probe_printk 122880

*lttng*_probe_power204800

*lttng*_probe_net327680

*lttng*_probe_napi 122880

*lttng*_probe_module 204800

*lttng*_probe_kvm368640

*lttng*_probe_jbd2 327680

*lttng*_probe_irq163840

*lttng*_probe_gpio 122880

*lttng*_probe_block409600

*lttng*_probe_asoc 368640

*lttng*_counter_client_percpu_32_modular122880

*lttng*_counter_client_percpu_64_modular122881

*lttng*_counter163842 
*lttng*_counter_client_percpu_64_modular,*lttng*_counter_client_percpu_32_modular


*lttng*_ring_buffer_event_notifier_client204802

*lttng*_ring_buffer_metadata_mmap_client204800

*lttng*_ring_buffer_client_mmap_overwrite245760

*lttng*_ring_buffer_client_mmap_discard245760

*lttng*_ring_buffer_metadata_client204800

*lttng*_ring_buffer_client_overwrite245760

*lttng*_ring_buffer_client_discard245760

*lttng*_tracer 303513637 
*lttng*_probe_udp,*lttng*_probe_scsi,*lttng*_probe_sched,*lttng*_probe_net,*lttng*_probe_vmscan,*lttng*_probe_writeback,*lttng*_probe_power,*lttng*_probe_rcu,*lttng*_probe_module,*lttng*_ring_buffer_client_mmap_overwrite,*lttng*_probe_statedump,*lttng*_ring_buffer_client_discard,*lttng*_probe_printk,*lttng*_probe_sock,*lttng*_probe_asoc,*lttng*_counter_client_percpu_64_modular,*lttng*_probe_irq,*lttng*_ring_buffer_client_mmap_discard,*lttng*_probe_kvm,*lttng*_probe_timer,*lttng*_ring_buffer_event_notifier_client,*lttng*_counter_client_percpu_32_modular,*lttng*_probe_workqueue,*lttng*_probe_jbd2,*lttng*_probe_signal,*lttng*_probe_skb,*lttng*_probe_block,*lttng*_probe_napi,*lttng*_ring_buffer_metadata_client,*lttng*_ring_buffer_metadata_mmap_client,*lttng*_probe_gpio,*lttng*_ring_buffer_client_overwrite,*lttng*_probe_regulator,*lttng*_probe_sunrpc


*lttng*_statedump 7536641 *lttng*_tracer

*lttng*_wrapper163847 
*lttng*_statedump,*lttng*_probe_writeback,*lttng*_ring_buffer_client_mmap_overwrite,*lttng*_ring_buffer_client_discard,*lttng*_tracer,*lttng*_ring_buffer_client_mmap_discard,*lttng*_ring_buffer_client_overwrite


*lttng*_uprobes163841 *lttng*_tracer

*lttng*_clock122885 
*lttng*_ring_buffer_client_mmap_overwrite,*lttng*_ring_buffer_client_discard,*lttng*_tracer,*lttng*_ring_buffer_client_mmap_discard,*lttng*_ring_buffer_client_overwrite


*lttng*_kprobes163841 *lttng*_tracer

*lttng*_lib_ring_buffer942088 
*lttng*_ring_buffer_client_mmap_overwrite,*lttng*_ring_buffer_client_discard,*lttng*_tracer,*lttng*_ring_buffer_client_mmap_discard,*lttng*_ring_buffer_event_notifier_client,*lttng*_ring_buffer_metadata_client,*lttng*_ring_buffer_metadata_mmap_client,*lttng*_ring_buffer_client_overwrite


*lttng*_kretprobes 163841 *lttng*_tracer

*yyan7@CCI13SZWP3LWS*:*~*$ export LTTNG_UST_DEBUG=1

*yyan7@CCI13SZWP3LWS*:*~*$ lttng-relayd -v -b

DBG1 - 10:24:56.244818657 [3495/3495]: File 

Re: [lttng-dev] LTTng sessiond daemon Assertion `buf' failed and killed

[Kienan Stewart via lttng-dev]

Hi Yonghong,

in a brief test I'm unable to reproduce the error you see by running the 
following commands on an Ubuntu 22.04 installation with lttng-tools 
2.13.10, lttng-ust 2.13.6, urcu stable-0.12, and babeltrace stable-2.0.

```
$ uname -r -v
6.5.0-14-generic #14~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 20 
18:15:30 UTC 2


$ lttng-relayd -v -b
$ lttng-sessiond -v -b
$ lttng create
$ lttng enable-event -u --all
```

Could you please review the bug reporting guidelines at 
https://lttng.org/community/ and elaborate on the steps taken to 
reproduce the issue?


thanks,
kienan

On 1/12/24 08:12, Yonghong Yan via lttng-dev wrote:
I am not sure whether this is my setting problem or a bug with a 
more recent kernel. lttng-sessiond was killed when I tried to "enable 
event" after a session was created. See below part of the verbose output 
of the sessiond. It is observed on Ubuntu 22.04, kernel 6.5.0-14-generic 
#14~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC, lttng (LTTng Trace Control) 
2.13.10 - Nordicité.



The same version of LTTng works on another Ubuntu 22.04 machine, but 
with kernel 6.2.0-33-generic. Any suggestion on what I should try?



Thank you

Yonghong



c:1016)

DBG1 - 00:24:25.134138433 [Client management]: Getting session 
auto-20240112-002417 by name (in process_client_msg() at client.c:1133)


DBG1 - 00:24:25.134148894 [Client management]: Creating UST session (in 
create_ust_session() at client.c:510)


DBG1 - 00:24:25.134267958 [Client management]: Spawning consumerd (in 
spawn_consumerd() at client.c:204)


DBG1 - 00:24:25.135155823 [Client management]: Waiting for consumer 
management thread to be ready (in wait_until_thread_is_ready() at 
manage-consumer.c:46)


DBG1 - 00:24:25.135247552 [Consumer management]: Entering thread entry 
point (in launch_thread() at thread.c:65)


DBG1 - 00:24:25.135293542 [Consumer management]: [thread] Manage 
consumer started (in thread_consumer_management() at manage-consumer.c:65)


DBG1 - 00:24:25.135335776 [Client management]: Using 64-bit UST consumer 
at: /usr/lib/x86_64-linux-gnu/lttng/libexec/lttng-consumerd (in 
spawn_consumerd() at client.c:284)


DBG1 - 00:24:25.240725883 [Consumer management]: Consumer command socket 
ready (fd: 61) (in thread_consumer_management() at manage-consumer.c:204)


DBG1 - 00:24:25.240746802 [Consumer management]: Consumer metadata 
socket ready (fd: 62) (in thread_consumer_management() at 
manage-consumer.c:205)


DBG1 - 00:24:25.240775318 [Consumer management]: Sending consumer 
initialization command (in consumer_init() at consumer.c:1791)


DBG1 - 00:24:25.241066762 [Consumer management]: Marking consumer 
management thread as ready (in mark_thread_as_ready() at 
manage-consumer.c:31)


DBG1 - 00:24:25.241100481 [Client management]: Consumer management 
thread is ready (in wait_until_thread_is_ready() at manage-consumer.c:48)


DBG1 - 00:24:25.241138386 [Client management]: Setting relayd for 
session auto-20240112-002417 (in cmd_setup_relayd() at cmd.c:1004)


lttng-sessiond: unix.c:185: lttcomm_recv_unix_sock: Assertion `buf' failed.

Error: Events: No session daemon is available (channel channel0, session 
auto-20240112-002417)


[1]+Aborted (core dumped) lttng-sessiond --verbose

DBG1 - 00:24:26.357769106 [Run-as worker]: run_as worker exiting (ret = 
0) (in run_as_create_worker_no_lock() at runas.c:1526)




___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Python lttngust module import error

[Kienan Stewart via lttng-dev]

Hi Lakshmi,

could you elaborate on which version of lttngust you have and how you 
installed it?


One possibility that I can imagine is that your ldconfig cache is out of 
date for some reason - `ldconfig -p` will simply print the contents of 
the cache - perhaps the files are no longer in the location specified in 
that cache?


thanks,
kienan


On 12/26/23 11:19, Lakshmi Deverkonda via lttng-dev wrote:

Hi,

I'm observing an issue recently wrt import of python3 lttng module.
Following is the error and need your help to find out what is wrong.


import lttngust

Exception in thread system:
Traceback (most recent call last):
   File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
     self.run()
   File "/usr/lib/python3.7/threading.py", line 865, in run
     self._target(*self._args, **self._kwargs)
   File "/usr/local/lib/python3.7/dist-packages/lttngust/agent.py", line 
291, in _client_thread_target

     client = _TcpClient(name, _SESSIOND_HOST, port, reg_queue)
   File "/usr/local/lib/python3.7/dist-packages/lttngust/agent.py", line 
76, in __init__

     raise e
   File "/usr/local/lib/python3.7/dist-packages/lttngust/agent.py", line 
73, in __init__

     self._log_handler = lttngust.loghandler._Handler()
   File "/usr/local/lib/python3.7/dist-packages/lttngust/loghandler.py", 
line 32, in __init__

     self.agent_lib = ctypes.cdll.LoadLibrary(_Handler._LIB_NAME)
   File "/usr/lib/python3.7/ctypes/__init__.py", line 434, in LoadLibrary
     return self._dlltype(name)
   File "/usr/lib/python3.7/ctypes/__init__.py", line 356, in __init__
     self._handle = _dlopen(self._name, mode)
*OSError: liblttng-ust-python-agent.so: cannot open shared object file: 
No such file or directory*







I have installed *liblttng-ust-python-agent.so*
*
*
/# ldconfig -p | grep liblttng
  liblttng-ust.so.0 (libc6,x86-64) => 
/lib/x86_64-linux-gnu/liblttng-ust.so.0
  liblttng-ust-tracepoint.so.0 (libc6,x86-64) => 
/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0
  *liblttng-ust-python-agent.so.0 (libc6,x86-64) => 
/lib/x86_64-linux-gnu/liblttng-ust-python-agent.so.0*
  liblttng-ust-pthread-wrapper.so.0 (libc6,x86-64) => 
/lib/x86_64-linux-gnu/liblttng-ust-pthread-wrapper.so.0
  liblttng-ust-libc-wrapper.so.0 (libc6,x86-64) => 
/lib/x86_64-linux-gnu/liblttng-ust-libc-wrapper.so.0
  liblttng-ust-fork.so.0 (libc6,x86-64) => 
/lib/x86_64-linux-gnu/liblttng-ust-fork.so.0
  liblttng-ust-fd.so.0 (libc6,x86-64) => 
/lib/x86_64-linux-gnu/liblttng-ust-fd.so.0
  liblttng-ust-dl.so.0 (libc6,x86-64) => 
/lib/x86_64-linux-gnu/liblttng-ust-dl.so.0
  liblttng-ust-cyg-profile.so.0 (libc6,x86-64) => 
/lib/x86_64-linux-gnu/liblttng-ust-cyg-profile.so.0
  liblttng-ust-cyg-profile-fast.so.0 (libc6,x86-64) => 
/lib/x86_64-linux-gnu/liblttng-ust-cyg-profile-fast.so.0
  liblttng-ust-ctl.so.4 (libc6,x86-64) => 
/lib/x86_64-linux-gnu/liblttng-ust-ctl.so.4
  liblttng-ctl.so.0 (libc6,x86-64) => 
/lib/x86_64-linux-gnu/liblttng-ctl.so.0



Regards,
Lakshmi

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] [PATCH 3/3] vmscan: drop isolate_mode (v6.7+)

[Kienan Stewart via lttng-dev]

Hi Bruce,

just a quick follow-up to say that the fixes for building against Linux 
6.7 have been merged into the master and stable-2.13 branches of 
lttng-modules.


thanks,
kienan

On 2023-11-23 14:56, Bruce Ashfield wrote:

Aha,

So the mailing list isn't used for patch submission anymore ? I didn't
notice any updates in the READMEs, etc, so I just fell back to my
existing workflow.

I just want to know, so I won't waste time in the future.

Bruce

On Thu, Nov 23, 2023 at 2:49 PM Kienan Stewart  wrote:


Hi Bruce,

thanks for the patches! There's a similar series waiting for review:
https://review.lttng.org/q/topic:%22fix-linux-6.7.0-rc1%22

thanks,
kienan

On 2023-11-23 14:32, Bruce Ashfield via lttng-dev wrote:

From: Bruce Ashfield 

commit 3dfbb555c98ac55b9 [mm, vmscan: remove ISOLATE_UNMAPPED]
makes the isolate_mode parameter unused, so it is removed.

This updates the event to match.

Signed-off-by: Bruce Ashfield 
---

Repeating the comment from patch 1 & 2 of this series:

Someone else may or may not already have this in flight,
but I didn't see anything on the list or in the git
repo.

I ran into this failure when working on the v6.7-rc
kernel as part of yocto project kernel updates.

Tweak as necessary, as I'm no expert in the right way
to version these sort of changes.

Bruce

   include/instrumentation/events/mm_vmscan.h | 10 +-
   1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/include/instrumentation/events/mm_vmscan.h 
b/include/instrumentation/events/mm_vmscan.h
index e7e9b613..a3e9ddf2 100644
--- a/include/instrumentation/events/mm_vmscan.h
+++ b/include/instrumentation/events/mm_vmscan.h
@@ -343,12 +343,18 @@ LTTNG_TRACEPOINT_EVENT(mm_vmscan_lru_isolate,
   unsigned long nr_scanned,
   unsigned long nr_skipped,
   unsigned long nr_taken,
+#if (LTTNG_LINUX_VERSION_CODE < LTTNG_KERNEL_VERSION(6,7,0))
   isolate_mode_t isolate_mode,
+#endif
   int lru
   ),

   TP_ARGS(classzone_idx, order, nr_requested, nr_scanned, nr_skipped,
- nr_taken, isolate_mode, lru
+ nr_taken,
+#if (LTTNG_LINUX_VERSION_CODE < LTTNG_KERNEL_VERSION(6,7,0))
+ isolate_mode,
+#endif
+ lru
   ),

   TP_FIELDS(
@@ -358,7 +364,9 @@ LTTNG_TRACEPOINT_EVENT(mm_vmscan_lru_isolate,
   ctf_integer(unsigned long, nr_scanned, nr_scanned)
   ctf_integer(unsigned long, nr_skipped, nr_skipped)
   ctf_integer(unsigned long, nr_taken, nr_taken)
+#if (LTTNG_LINUX_VERSION_CODE < LTTNG_KERNEL_VERSION(6,7,0))
   ctf_integer(isolate_mode_t, isolate_mode, isolate_mode)
+#endif
   ctf_integer(int, lru, lru)
   )
   )





___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Any particular reason that the LTTNG_UST_TRACEPOINT_EVENT args and fields are limited to max 10?

[Kienan Stewart via lttng-dev]

Hi Yonghong,

My understanding is that the limit of 10 trac epoint arguments was 
initially chosen because it matched with the argumente limit in SDT 
probe points and there is optional integration between LTTng trace 
points and SDT. More recent versions of SDT, as of 1.7+, support up to 
12 arguments for probe points.


Beyond that, there may be standards/compiler-specific limits for the 
number of parameters to macros and/or function calls, but I don't think 
we're close to hitting any of those.


Note that the LTTNG_UST_TRACEPOINT_FIELDS macro doesn't share the same 
10 field limit.


Therefore, one work-around that is possible to to pack the values into a 
custom structure to reduce the number of arguments that need to be set 
in the lTTNG_UST_TRACEPOINT_ARGS macro.


eg.

```
LTTNG_UST_TRACEPOINT_EVENT(
   tp,
   many,
   LTTNG_UST_TP_ARGS(
  const struct Sample*, mystruct
   ),
   LTTNG_UST_TP_FIELDS(
  lttng_ust_field_integer(int, a, mystruct->a)
  lttng_ust_field_integer(int, b, mystruct->b)
  lttng_ust_field_integer(int, c, mystruct->c)
  lttng_ust_field_integer(int, e, mystruct->d)
  lttng_ust_field_integer(int, f, mystruct->e)
  lttng_ust_field_float(double, g, mystruct->g)
  lttng_ust_field_float(double, h, mystruct->h)
  lttng_ust_field_float(double, i, mystruct->i)
  lttng_ust_field_float(double, j, mystruct->j)
  lttng_ust_field_float(double, k, mystruct->k)
  lttng_ust_field_string(string_field, mystruct->thing)
  lttng_ust_field_string(string_field2, mystruct->thing2)
   )
)

```

Hope this helps!

thanks,
kienan

On 2023-11-27 22:26, Yonghong Yan via lttng-dev wrote:
I have a situation where I need to record more than 10 fields. If you 
can extend it to support more, but not introduce overhead of handling 
such rare cases, that will be great.


Thanks
Yonghong


___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] [PATCH 3/3] vmscan: drop isolate_mode (v6.7+)

[Kienan Stewart via lttng-dev]

Hi Bruce,

Others may chime in here as I'm not sure I have the full context off-hand.

On 2023-11-23 14:56, Bruce Ashfield wrote:

Aha,

So the mailing list isn't used for patch submission anymore ?


Patches submitted to the mailing list are reviewed. That being said, I 
think it's fair to say that the principal workflow has shifted away from 
the mailing list to https://review.lttng.org - many patches are reviewed 
there and not posted on the mailing list.


I think some patch series also get submitted to the mailing list in 
addition to https://review.lttng.org in order to solicit a wider 
feedback from the community.



I didn't
notice any updates in the READMEs, etc, so I just fell back to my
existing workflow.


You are correct that the README for lttng-modules doesn't make any 
mention of this. Thanks for bringing this up, it should definitely be 
addressed!


I'll go through the READMEs and CONTRIBUTING.md files to make sure they 
indicate the current workflows.


thanks,
kienan



I just want to know, so I won't waste time in the future.

Bruce

On Thu, Nov 23, 2023 at 2:49 PM Kienan Stewart  wrote:


Hi Bruce,

thanks for the patches! There's a similar series waiting for review:
https://review.lttng.org/q/topic:%22fix-linux-6.7.0-rc1%22

thanks,
kienan

On 2023-11-23 14:32, Bruce Ashfield via lttng-dev wrote:

From: Bruce Ashfield 

commit 3dfbb555c98ac55b9 [mm, vmscan: remove ISOLATE_UNMAPPED]
makes the isolate_mode parameter unused, so it is removed.

This updates the event to match.

Signed-off-by: Bruce Ashfield 
---

Repeating the comment from patch 1 & 2 of this series:

Someone else may or may not already have this in flight,
but I didn't see anything on the list or in the git
repo.

I ran into this failure when working on the v6.7-rc
kernel as part of yocto project kernel updates.

Tweak as necessary, as I'm no expert in the right way
to version these sort of changes.

Bruce

   include/instrumentation/events/mm_vmscan.h | 10 +-
   1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/include/instrumentation/events/mm_vmscan.h 
b/include/instrumentation/events/mm_vmscan.h
index e7e9b613..a3e9ddf2 100644
--- a/include/instrumentation/events/mm_vmscan.h
+++ b/include/instrumentation/events/mm_vmscan.h
@@ -343,12 +343,18 @@ LTTNG_TRACEPOINT_EVENT(mm_vmscan_lru_isolate,
   unsigned long nr_scanned,
   unsigned long nr_skipped,
   unsigned long nr_taken,
+#if (LTTNG_LINUX_VERSION_CODE < LTTNG_KERNEL_VERSION(6,7,0))
   isolate_mode_t isolate_mode,
+#endif
   int lru
   ),

   TP_ARGS(classzone_idx, order, nr_requested, nr_scanned, nr_skipped,
- nr_taken, isolate_mode, lru
+ nr_taken,
+#if (LTTNG_LINUX_VERSION_CODE < LTTNG_KERNEL_VERSION(6,7,0))
+ isolate_mode,
+#endif
+ lru
   ),

   TP_FIELDS(
@@ -358,7 +364,9 @@ LTTNG_TRACEPOINT_EVENT(mm_vmscan_lru_isolate,
   ctf_integer(unsigned long, nr_scanned, nr_scanned)
   ctf_integer(unsigned long, nr_skipped, nr_skipped)
   ctf_integer(unsigned long, nr_taken, nr_taken)
+#if (LTTNG_LINUX_VERSION_CODE < LTTNG_KERNEL_VERSION(6,7,0))
   ctf_integer(isolate_mode_t, isolate_mode, isolate_mode)
+#endif
   ctf_integer(int, lru, lru)
   )
   )





___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] [PATCH 3/3] vmscan: drop isolate_mode (v6.7+)

[Kienan Stewart via lttng-dev]

Hi Bruce,

thanks for the patches! There's a similar series waiting for review: 
https://review.lttng.org/q/topic:%22fix-linux-6.7.0-rc1%22


thanks,
kienan

On 2023-11-23 14:32, Bruce Ashfield via lttng-dev wrote:

From: Bruce Ashfield 

commit 3dfbb555c98ac55b9 [mm, vmscan: remove ISOLATE_UNMAPPED]
makes the isolate_mode parameter unused, so it is removed.

This updates the event to match.

Signed-off-by: Bruce Ashfield 
---

Repeating the comment from patch 1 & 2 of this series:

Someone else may or may not already have this in flight,
but I didn't see anything on the list or in the git
repo.

I ran into this failure when working on the v6.7-rc
kernel as part of yocto project kernel updates.

Tweak as necessary, as I'm no expert in the right way
to version these sort of changes.

Bruce

  include/instrumentation/events/mm_vmscan.h | 10 +-
  1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/include/instrumentation/events/mm_vmscan.h 
b/include/instrumentation/events/mm_vmscan.h
index e7e9b613..a3e9ddf2 100644
--- a/include/instrumentation/events/mm_vmscan.h
+++ b/include/instrumentation/events/mm_vmscan.h
@@ -343,12 +343,18 @@ LTTNG_TRACEPOINT_EVENT(mm_vmscan_lru_isolate,
unsigned long nr_scanned,
unsigned long nr_skipped,
unsigned long nr_taken,
+#if (LTTNG_LINUX_VERSION_CODE < LTTNG_KERNEL_VERSION(6,7,0))
isolate_mode_t isolate_mode,
+#endif
int lru
),
  
  	TP_ARGS(classzone_idx, order, nr_requested, nr_scanned, nr_skipped,

-   nr_taken, isolate_mode, lru
+   nr_taken,
+#if (LTTNG_LINUX_VERSION_CODE < LTTNG_KERNEL_VERSION(6,7,0))
+   isolate_mode,
+#endif
+   lru
),
  
  	TP_FIELDS(

@@ -358,7 +364,9 @@ LTTNG_TRACEPOINT_EVENT(mm_vmscan_lru_isolate,
ctf_integer(unsigned long, nr_scanned, nr_scanned)
ctf_integer(unsigned long, nr_skipped, nr_skipped)
ctf_integer(unsigned long, nr_taken, nr_taken)
+#if (LTTNG_LINUX_VERSION_CODE < LTTNG_KERNEL_VERSION(6,7,0))
ctf_integer(isolate_mode_t, isolate_mode, isolate_mode)
+#endif
ctf_integer(int, lru, lru)
)
  )

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] 答复: lttng-analysis error

[Kienan Stewart via lttng-dev]

Hi Tao,

could you keep lttng-dev@lists.lttng.org in CC: on replies in the future?

On 2023-10-31 23:39, 姜涛 Tao(AD) wrote:

Hi Kienan,

Thank you very much! It works! By the way,  which version of 
lttng-analysis could handle the ‘pid’ error?




There's no version of lttng-analyses available that includes a fix or 
work-around by default.


The project is experimental and isn't currently being maintained. If 
your employer is interested in supporting either this project or others 
please don't hesitate to reach out to EfficiOS.


thanks,
kienan


*发件人**:*Kienan Stewart 
*日期**:*星期三, 2023年11月1日03:36
*收件人**:*姜涛Tao(AD) , 
lttng-dev@lists.lttng.org 

*主题**:*Re: [lttng-dev] lttng-analysis error

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] lttng-analysis error

[Kienan Stewart via lttng-dev]

Hi Tao,

based on a quick test I performed, there may be a work-around that will 
work for you.


The issue that I see is that at least the 
`lttng_statedump_file_descriptor` events no longer have a pid field 
which the older lttng-analyses code assumes is present.


To find this out
When running the manual recording steps, could you add the following 
command before running `lttng start`:


```
lttng add-context --kernel --type=pid
```

To pinpoint the source of the error, I used the `--debug` parameter for 
`lttng-analyses` as follows:


```
# lttng-schedtop --debug 
lttng-traces/lttng-analysis-27010-20231031-182735/
Checking the trace for lost events... 

Traceback (most recent call last): 
   ] ETA:  --:--:--
  File 
"/usr/local/lib/python3.7/dist-packages/lttnganalyses/cli/command.py", 
line 73, in _run_step
fn() 

  File 
"/usr/local/lib/python3.7/dist-packages/lttnganalyses/cli/command.py", 
line 365, in _run_analysis
self._automaton.process_event(event) 

  File 
"/usr/local/lib/python3.7/dist-packages/lttnganalyses/linuxautomaton/automaton.py", 
line 81, in process_event

sp.process_event(ev)
  File 
"/usr/local/lib/python3.7/dist-packages/lttnganalyses/linuxautomaton/sp.py", 
line 33, in process_event
self._cbs[name](ev) 

  File 
"/usr/local/lib/python3.7/dist-packages/lttnganalyses/linuxautomaton/statedump.py", 
line 91, in _process_lttng_statedump_file_descriptor

pid = event['pid']
  File 
"/usr/local/lib/python3.7/dist-packages/babeltrace/babeltrace.py", line 
912, in __getitem__
raise KeyError(field_name) 

KeyError: 'pid' 


Error: Cannot run analysis: 'pid'
```

thanks,
kienan

On 2023-10-31 07:44, 姜涛 Tao(AD) via lttng-dev wrote:
Hi everyone, I use lttng-analysis tools to parse the trace files but get 
error as below:


I try two methods to catch the trace data: “automatic” using 
lttng-analysis-record and “manual” using lttng 
create/enable-event/start/stop/destroy, but always get the errors when 
using lttng-analysis tools(0.6.0 and 0.6.1 version).


I load the lttng modules(2.12.14 version) successfully:

文本 描述已自动生成

The environment is as below: Ubuntu20.04 aarch64, kernel version: 
5.10.120 rt patch.


Any help will be appreciated! Thanks!

免责声明：本邮件所包含信息发给指定个人或机构，邮件可能包含保密或专属信 
息。未经接收者许可，不得阅读、转发或传播邮件内容，或根据邮件内容采取任何 
相关行动。如果错误地收到了此邮件，请与收件人联系并自行删除邮件内容。 
Disclaimer：The information transmitted is intended only for the person 
or entity to which it is addressed and may contain confidential and/or 
privileged material. Any review, retransmission, dissemination or other 
use of, or taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is prohibited. If 
you received this in error , please contact the sender and delete the 
material from any computer.


___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Need help: Kernel tracer not available and can't find the lttng-probe-latency_tracker

[Kienan Stewart via lttng-dev]

Hi Tao,

due to lack of financing, the latency_tracker project 
(https://github.com/efficios/latency-tracker/) isn't being actively 
maintained at this time.


Another option to get information on event latencies is to have 
"_entry"- and "_exit"-style events and perform post-processing of the 
traces with the Trace Compass Incubator "inandout" plugin.


https://git.eclipse.org/r/plugins/gitiles/tracecompass.incubator/org.eclipse.tracecompass.incubator/+/refs/heads/master/doc/org.eclipse.tracecompass.incubator.inandout.doc.user/doc/User-Guide.mediawiki

On 2023-10-25 02:41, 姜涛 Tao(AD) via lttng-dev wrote:
Hi everyone, I have read this blog: 
https://lttng.org/blog/2016/01/06/monitoring-realtime-latencies/ 
, and 
I think latency_tracker project will be very useful in resovling my 
problem under linux rt patch kernel 5.10, but it doesn't support linux 
newer kernel, so I find the span-latency-tracker project later. But when 
I use lttng create session，I get the error :
Error: Event wakeup_latency: Kernel tracer not available (channel 
channel0, session auto-20231024-134242)
Error: Event block_latency: Kernel tracer not available (channel 
channel0, session auto-20231024-134242)


文本 描述已自动生成

And I find this tip: "To use the tracepoints provided by this module 
with LTTng, you first have to connect LTTng probes to them. A patch 
ready to be applied to the last lttng-modules is provided in extras/ 
. When installed, compiled, load the new probe module with modprobe lttng-probe-latency_tracker", but I can't find the "lttng-probe-latency_tracker" module in the lttng-module compiled modules or in the span-latency-tracker compiled modules.


This appears to be a fork of the latency_tracker project. We're not in a 
position to offer support for any changes that have been made there.




I also create the issue on the github: 
https://github.com/adel-belkhiri/span-latency-tracker/issues/1 
.

Any help will be appreciated! Thanks!

免责声明：本邮件所包含信息发给指定个人或机构，邮件可能包含保密或专属信 
息。未经接收者许可，不得阅读、转发或传播邮件内容，或根据邮件内容采取任何 
相关行动。如果错误地收到了此邮件，请与收件人联系并自行删除邮件内容。 
Disclaimer：The information transmitted is intended only for the person 
or entity to which it is addressed and may contain confidential and/or 
privileged material. Any review, retransmission, dissemination or other 
use of, or taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is prohibited. If 
you received this in error , please contact the sender and delete the 
material from any computer.


___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


thanks,
kienan

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Can't receive a lot of functions that i registerd as trace-points

[Kienan Stewart via lttng-dev]

Hi Lin,

On 2023-10-25 04:54, Lin ArcX wrote:

Are you seeing a situation where another tool is recording occurrences
of the events that you think aren't being seen during your tracing
session?


No, nothing, I'm running my client/server application inside wsl and no 
other tools are running.




Perhaps your test program isn't causing the WSL kernel to hit the 
`net_if_rx_ni_entry`, `net_napi_gro_receive_entry`, and 
`net_if_receive_skb_list_entry tracepoints` during your tracing session?


I recommend using `perf` to double-check that there are indeed none of 
those tracepoints being hit during a tracing session.


thanks,
kienan
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Can't receive a lot of functions that i registerd as trace-points

[Kienan Stewart via lttng-dev]

Hi Lin,

could you keep the lttng-dev list in CC on your replies in the future?


On 2023-10-24 09:25, Lin ArcX wrote:

Thanks for your response. i fixed the names. here is new commands:
# enable event for traced functions
sudo lttng enable-event --kernel irq_handler_entry,napi_poll,net_dev_queue

# different name with kernel
sudo lttng enable-event --kernel 
net_if_rx_entry,net_if_rx_ni_entry,net_napi_gro_receive_entry,net_if_receive_skb_list_entry,net_if_receive_skb


Now In new tracefile i can see a lot of new events come. but still can't 
see any trace from these:


net_if_rx_ni_entry
net_napi_gro_receive_entry
net_if_receive_skb_list_entry


This looks fine to me.

On a linux 4.19 system, I ran a test with using lttng-modules 2.12.14. I 
saw events for "net_napi_gro_receive_entry".


In parallel with the LTTng tracing session, I ran

  perf stat -e 
net:netif_rx_ni_entry,net:netif_receive_skb_list_entry,net:napi_gro_receive_entry


The above allowed me to confirm that there were 
"net:napi_gro_receive_entry_events", but no "net:netif_rx_ni_entry" or 
"net:netif_receive_skb_list_entry" occurrences in my test.


Are you seeing a situation where another tool is recording occurrences 
of the events that you think aren't being seen during your tracing session?


thanks,
kienan
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Can't receive a lot of functions that i registerd as trace-points

[Kienan Stewart via lttng-dev]

Hi Lin,

Some of the event rules you have listed are for items that aren't 
instrumented. Others have the wrong name.


I've put some more specific indications below. You can check which 
events are available in the kernel domain with


`lttng list --kernel`

And for syscalls:

`lttng list --kernel --syscall`

There is also documentation available on adding LTTng probes for 
existing ftrace tracepoints at 
https://lttng.org/docs/v2.13/#doc-linux-add-lttng-layer (for 2.13) or 
https://lttng.org/docs/v2.12/#doc-linux-add-lttng-layer




Hello,

I have a simple udp server/client and I want to trace these functions:

     Event rules:
       netif_rx_ni_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
[enabled]
       netif_rx_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint) 


In LTTng, these events start with "net_if_", not "netif_"


[enabled]
       napi_gro_receive_entry (loglevel: TRACE_EMERG (0)) (type: 
tracepoint) [enabled]


Try "net_napi_gro_receive_entry" instead.

       softirq_raise (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
[enabled]
       irq_handler_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
[enabled]

       napi_poll (loglevel: TRACE_EMERG (0)) (type: tracepoint) [enabled]
       softirq_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
[enabled]
       e1000_xmit_frame (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
[enabled]
       ndo_start_xmit (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
[enabled]


lttng-modules doesn't provide any specific instrumentation for the e1000 
and ndo drivers. Are you using additional kernel modules that add the 
instrumentation?


If you are, please ensure that the module is loaded, or it can be added 
to the list of probes loaded by the sessiond using `--extra-kmod-probes` 
(c.f. 
https://lttng.org/man/8/lttng-sessiond/v2.13/#doc-opt--extra-kmod-probes)


If not, you will need to implement the probes as mentioned earlier. If 
you need probes within a specific timeline, either patches and/or

financial support are welcome to create and maintain them. If you're
interested in funding this, please feel free to reach out to EfficiOS.


       net_dev_start_xmit (loglevel: TRACE_EMERG (0)) (type: tracepoint) > 
[enabled]


This might be "net_dev_xmit" instead of "net_dev_start_xmit"

       dev_queue_xmit (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
[enabled]
       net_dev_queue (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
[enabled]
       netif_receive_skb_list_entry (loglevel: TRACE_EMERG (0)) (type: 
tracepoint) [enabled]
       netif_receive_skb (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
[enabled]


These are also prefixed with "net_if_" instead of "netif_".

       inet_sock_set_state (loglevel: TRACE_EMERG (0)) (type: 
tracepoint) [enabled]


I don't think this is instrumented in lttng-modules at this time.



I start tracing, and when it is finished, I will open my trace directory 
into TraceCompass, But I can see just some traces from these functions:


- irq_handler_entry
- napi_poll
- net_dev_queue

I'm using  2.12.14.

Do you have any idea?

Regards,



Hope this helps.

thanks,
kienan


___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] [EXTERNAL] Re: LTTng enable-event performance issue during active session

[Kienan Stewart via lttng-dev]

Hi Zach,

the change to address the enable-event performance has been merged into 
both lttng-ust master and stable-2.13 if you would like to build from 
source before the next release.


thanks,
kienan


On 2023-10-09 06:29, Kramer, Zach wrote:

Hi Kienan,

Thank you for the thorough explanation and proposed changeset.

My use case was to synchronize with a user-configurable set of events at 
runtime, which was easiest done individually.
In the meantime I worked around the performance hit by ensuring all available 
events have already been created on the session before starting it.

Looking forward to the changeset.

Thanks,
Zach

-Original Message-
From: Kienan Stewart 
Sent: Friday, 6 October 2023 18:13
To: lttng-dev@lists.lttng.org; Kramer, Zach 
Subject: [EXTERNAL] Re: [lttng-dev] LTTng enable-event performance issue during 
active session

Hi Zach,

there's a proposed change to lttng-ust which addresses the issue you are seeing, 
but it hasn't finished clearing review & testing yet.
https://review.lttng.org/c/lttng-ust/+/11006

thanks,
kienan

On 2023-10-05 18:04, Kienan Stewart via lttng-dev wrote:

Hi Zach

apologies for the delay in replying to your question.

On 2023-09-26 08:27, Kramer, Zach via lttng-dev wrote:

Hi,

I am observing a performance issue with regards to enabling events
while a session is active and was wondering if this is expected.

LTTng versions:

   * lttng-tools: 2.13.9
   * lttng-ust: 2.13.6
Steps to reproduce:

  1. Ensure many userspace tracepoints are available in `lttng list -u`
 e.g. 100
  2. Create a new session
  3. Start session
  4. Enable new events on session

The time it takes to enable each new event has increasing cost e.g.

  1. Event 1: 1ms
  2. Event 100: 15ms
  3. Event 1000: 150ms
  4. àin total about 1.5 minutes to enable 1000 events

If either:

  1. No userspace tracepoints are available  2. Or session is not
started until /after /the events are enabled

Then the time it takes to enable new events is constant (e.g. 1ms).

Below is a bash script demonstrating this behavior:

# Pre-requisite: have many userspace tracepoints available

lttng create foo

lttng enable-channel -u -s foo bar

lttng start foo

total_t1=$(date +%s%3N);

for i in {1..100}

do

  t1=$(date +%s%3N);

  lttng enable-event -u lttng_iter_$i -s foo -c bar >
/dev/null

  t2=$(date +%s%3N);

  echo "Event #$i took $((t2-t1)) ms"

done

total_t2=$(date +%s%3N);

echo ""

echo "Enabling events on active session took $((total_t2-total_t1)) ms"

echo ""

lttng destroy foo

lttng create foo

lttng enable-channel -u -s foo bar

total_t1=$(date +%s%3N);

for i in {1..100}

do

  t1=$(date +%s%3N);

  lttng enable-event -u lttng_iter_$i -s foo -c bar >
/dev/null

  t2=$(date +%s%3N);

  echo "Event #$i took $((t2-t1)) ms"

done

total_t2=$(date +%s%3N);

echo ""

echo "Enabling events on inactive session took $((total_t2-total_t1)) ms"

echo ""

lttng destroy foo

Is this reproducible for you? Any insight is appreciated.



I'm able to reproduce what you're seeing, and it's not completely
unexpected based on my understanding of the architecture of lttng.

When a session is active and an event rule is enabled/disabled, the
sessiond will notify each of the registered applications of the event
rule state changes. This communication is going through either unix or
tcp sockets, and that portion of the protocol seems to struggle with
many small changes as the protocol to communicate the changes doesn't
support batching.

Enabling multiple events in a single call reduces some of the
overhead, but the changes will still be communicated serially. This
means a single call activating thousands events can still take a long
time to process (eg. `lttng enable-event -s foo -c bar -u
tp0,tp1,...,tpN`)

Using glob patterns or `--all` will be significantly faster as the UST
applications can digest those types of event rule changes with a
single set of messages. In cases where you want most but not all
events, flipping the logic to enable "*" but add events or patterns to
omit with "--exclude" is going to be a better strategy.

Do you have cases where you need to activate many of events but which
couldn't be covered by using glob patterns and/or exclusions?

thanks,
kienan


Many thanks,

Zach


___
lttng-dev mailing list
lttng-dev@lists.lttng.org

  ___

lttng-dev mailing list
lttng-dev@lists.lttng.org

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] lttng list -k is missing tracepoints

[Kienan Stewart via lttng-dev]

Hi Uday,

thanks for the question!

the syscalls have associated events, which should be visible when 
running the following command:


```
$ lttng list -k --syscall | grep uring
  io_uring_setup [enabled] [32/64-bit]
  io_uring_enter [enabled] [32/64-bit]
  io_uring_register [enabled] [32/64-bit]
```

The availability of those events may depend on the architecture of your 
system.


For the rest of the tracepoints exposed by io_uring, there's currently 
no probe for them in lttng-modules.


For each system that exposes tracepoints (in the linux kernel source at 
`include/trace/events/`), a small kernel module is required on the 
lttng-modules side in order to create corresponding events. These kernel 
modules require some work to create and maintain since the defined 
tracepoints aren't stable across kernel versions, and the current 
version of lttng-modules supports linux kernels >= 4.4. The probe kernel 
modules are defined in 
https://github.com/lttng/lttng-modules/tree/master/src/probes


We've noted your interest in having a probe for io_uring. If you need 
the io_uring probe within a specific timeline, either patches and/or 
financial support are welcome to create and maintain it. If you're 
interested in funding this, please feel free to reach out to EfficiOS.


Hope this helps.

thanks,
kienan




On 2023-10-09 20:29, Uday Shankar via lttng-dev wrote:

Hi,

I'm trying to use LTTNG to trace io_uring tracepoints, of which there
are many in the 6.5.4 kernel I am running:

# grep io_uring /sys/kernel/tracing/available_events
syscalls:sys_exit_io_uring_register
syscalls:sys_enter_io_uring_register
syscalls:sys_exit_io_uring_setup
syscalls:sys_enter_io_uring_setup
syscalls:sys_exit_io_uring_enter
syscalls:sys_enter_io_uring_enter
io_uring:io_uring_local_work_run
io_uring:io_uring_short_write
io_uring:io_uring_task_work_run
io_uring:io_uring_cqe_overflow
io_uring:io_uring_req_failed
io_uring:io_uring_task_add
io_uring:io_uring_poll_arm
io_uring:io_uring_submit_req
io_uring:io_uring_complete
io_uring:io_uring_fail_link
io_uring:io_uring_cqring_wait
io_uring:io_uring_link
io_uring:io_uring_defer
io_uring:io_uring_queue_async_work
io_uring:io_uring_file_get
io_uring:io_uring_register
io_uring:io_uring_create

However, lttng list -k does not show any of these tracepoints. I'm using
an lttng-modules built from the repo which I cloned last Friday:

$ modinfo lttng-wrapper | grep extra_version_git
extra_version_git:  v2.13.0-rc1-316-ga62977ca

(lttng-tools and lttng-ust are older - whatever version came with my
distro - but I don't think that should matter). Am I missing something?

Here's the full output of lttng list -k, in case it's helpful:

Kernel events:
-
   lttng_logger (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   compaction_isolate_migratepages (loglevel: TRACE_EMERG (0)) (type: 
tracepoint)
   compaction_isolate_freepages (loglevel: TRACE_EMERG (0)) (type: 
tracepoint)
   compaction_migratepages (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   asoc_snd_soc_bias_level_start (loglevel: TRACE_EMERG (0)) (type: 
tracepoint)
   asoc_snd_soc_bias_level_done (loglevel: TRACE_EMERG (0)) (type: 
tracepoint)
   asoc_snd_soc_dapm_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   asoc_snd_soc_dapm_done (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   asoc_snd_soc_dapm_widget_power (loglevel: TRACE_EMERG (0)) (type: 
tracepoint)
   asoc_snd_soc_dapm_widget_event_start (loglevel: TRACE_EMERG (0)) (type: 
tracepoint)
   asoc_snd_soc_dapm_widget_event_done (loglevel: TRACE_EMERG (0)) (type: 
tracepoint)
   asoc_snd_soc_dapm_walk_done (loglevel: TRACE_EMERG (0)) (type: 
tracepoint)
   asoc_snd_soc_dapm_path (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   asoc_snd_soc_dapm_connected (loglevel: TRACE_EMERG (0)) (type: 
tracepoint)
   asoc_snd_soc_jack_irq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   asoc_snd_soc_jack_report (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   asoc_snd_soc_jack_notify (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   gpio_direction (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   gpio_value (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   block_touch_buffer (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   block_dirty_buffer (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   block_rq_requeue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   block_rq_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   block_rq_insert (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   block_rq_issue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   block_rq_merge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   block_bio_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   block_bio_bounce (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   block_bio_backmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
   block_bio_frontmerge 

Re: [lttng-dev] LTTng enable-event performance issue during active session

[Kienan Stewart via lttng-dev]

Hi Zach,

there's a proposed change to lttng-ust which addresses the issue you are 
seeing, but it hasn't finished clearing review & testing yet. 
https://review.lttng.org/c/lttng-ust/+/11006


thanks,
kienan

On 2023-10-05 18:04, Kienan Stewart via lttng-dev wrote:

Hi Zach

apologies for the delay in replying to your question.

On 2023-09-26 08:27, Kramer, Zach via lttng-dev wrote:

Hi,

I am observing a performance issue with regards to enabling events 
while a session is active and was wondering if this is expected.


LTTng versions:

  * lttng-tools: 2.13.9
  * lttng-ust: 2.13.6
Steps to reproduce:

 1. Ensure many userspace tracepoints are available in `lttng list -u`
    e.g. 100
 2. Create a new session
 3. Start session
 4. Enable new events on session

The time it takes to enable each new event has increasing cost e.g.

 1. Event 1: 1ms
 2. Event 100: 15ms
 3. Event 1000: 150ms
 4. àin total about 1.5 minutes to enable 1000 events

If either:

 1. No userspace tracepoints are available
 2. Or session is not started until /after /the events are enabled

Then the time it takes to enable new events is constant (e.g. 1ms).

Below is a bash script demonstrating this behavior:

# Pre-requisite: have many userspace tracepoints available

lttng create foo

lttng enable-channel -u -s foo bar

lttng start foo

total_t1=$(date +%s%3N);

for i in {1..100}

do

 t1=$(date +%s%3N);

 lttng enable-event -u lttng_iter_$i -s foo -c bar > /dev/null

 t2=$(date +%s%3N);

 echo "Event #$i took $((t2-t1)) ms"

done

total_t2=$(date +%s%3N);

echo ""

echo "Enabling events on active session took $((total_t2-total_t1)) ms"

echo ""

lttng destroy foo

lttng create foo

lttng enable-channel -u -s foo bar

total_t1=$(date +%s%3N);

for i in {1..100}

do

 t1=$(date +%s%3N);

 lttng enable-event -u lttng_iter_$i -s foo -c bar > /dev/null

 t2=$(date +%s%3N);

 echo "Event #$i took $((t2-t1)) ms"

done

total_t2=$(date +%s%3N);

echo ""

echo "Enabling events on inactive session took $((total_t2-total_t1)) ms"

echo ""

lttng destroy foo

Is this reproducible for you? Any insight is appreciated.



I'm able to reproduce what you're seeing, and it's not completely 
unexpected based on my understanding of the architecture of lttng.


When a session is active and an event rule is enabled/disabled, the 
sessiond will notify each of the registered applications of the event 
rule state changes. This communication is going through either unix or 
tcp sockets, and that portion of the protocol seems to struggle with 
many small changes as the protocol to communicate the changes doesn't 
support batching.


Enabling multiple events in a single call reduces some of the overhead, 
but the changes will still be communicated serially. This means a single 
call activating thousands events can still take a long time to process 
(eg. `lttng enable-event -s foo -c bar -u tp0,tp1,...,tpN`)


Using glob patterns or `--all` will be significantly faster as the UST 
applications can digest those types of event rule changes with a single 
set of messages. In cases where you want most but not all events, 
flipping the logic to enable "*" but add events or patterns to omit with 
"--exclude" is going to be a better strategy.


Do you have cases where you need to activate many of events but which 
couldn't be covered by using glob patterns and/or exclusions?


thanks,
kienan


Many thanks,

Zach


___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng enable-event performance issue during active session

[Kienan Stewart via lttng-dev]

Hi Zach

apologies for the delay in replying to your question.

On 2023-09-26 08:27, Kramer, Zach via lttng-dev wrote:

Hi,

I am observing a performance issue with regards to enabling events while 
a session is active and was wondering if this is expected.


LTTng versions:

  * lttng-tools: 2.13.9
  * lttng-ust: 2.13.6 


Steps to reproduce:

 1. Ensure many userspace tracepoints are available in `lttng list -u`
e.g. 100
 2. Create a new session
 3. Start session
 4. Enable new events on session

The time it takes to enable each new event has increasing cost e.g.

 1. Event 1: 1ms
 2. Event 100: 15ms
 3. Event 1000: 150ms
 4. àin total about 1.5 minutes to enable 1000 events

If either:

 1. No userspace tracepoints are available
 2. Or session is not started until /after /the events are enabled

Then the time it takes to enable new events is constant (e.g. 1ms).

Below is a bash script demonstrating this behavior:

# Pre-requisite: have many userspace tracepoints available

lttng create foo

lttng enable-channel -u -s foo bar

lttng start foo

total_t1=$(date +%s%3N);

for i in {1..100}

do

     t1=$(date +%s%3N);

     lttng enable-event -u lttng_iter_$i -s foo -c bar > /dev/null

     t2=$(date +%s%3N);

     echo "Event #$i took $((t2-t1)) ms"

done

total_t2=$(date +%s%3N);

echo ""

echo "Enabling events on active session took $((total_t2-total_t1)) ms"

echo ""

lttng destroy foo

lttng create foo

lttng enable-channel -u -s foo bar

total_t1=$(date +%s%3N);

for i in {1..100}

do

     t1=$(date +%s%3N);

     lttng enable-event -u lttng_iter_$i -s foo -c bar > /dev/null

     t2=$(date +%s%3N);

     echo "Event #$i took $((t2-t1)) ms"

done

total_t2=$(date +%s%3N);

echo ""

echo "Enabling events on inactive session took $((total_t2-total_t1)) ms"

echo ""

lttng destroy foo

Is this reproducible for you? Any insight is appreciated.



I'm able to reproduce what you're seeing, and it's not completely 
unexpected based on my understanding of the architecture of lttng.


When a session is active and an event rule is enabled/disabled, the 
sessiond will notify each of the registered applications of the event 
rule state changes. This communication is going through either unix or 
tcp sockets, and that portion of the protocol seems to struggle with 
many small changes as the protocol to communicate the changes doesn't 
support batching.


Enabling multiple events in a single call reduces some of the overhead, 
but the changes will still be communicated serially. This means a single 
call activating thousands events can still take a long time to process 
(eg. `lttng enable-event -s foo -c bar -u tp0,tp1,...,tpN`)


Using glob patterns or `--all` will be significantly faster as the UST 
applications can digest those types of event rule changes with a single 
set of messages. In cases where you want most but not all events, 
flipping the logic to enable "*" but add events or patterns to omit with 
"--exclude" is going to be a better strategy.


Do you have cases where you need to activate many of events but which 
couldn't be covered by using glob patterns and/or exclusions?


thanks,
kienan


Many thanks,

Zach


___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] What is DEFINE_LTTNG_UST_SIGBUS_STATE for and other beginner questions

[Kienan Stewart via lttng-dev]

Hi Chris,

glad that you got it figured out! I created an issue based on your 
feedback. https://bugs.lttng.org/issues/1393


thanks,
kienan


On 2023-10-05 15:40, Christopher Harvey wrote:

I got this one figured out. I used strace -fp  -trace=connect on the 
target app to see why the connect was failing and it turns out there was a mismatch 
between sessiond's apps-socket paths and libust's.

time to review my build environment I guess. It's very possible my docker 
container's path environment variables are non-standard.

If I had a tiny bit of feedback to provide it would be to print out the errno 
and paths in the connect/bind commands in the DBG commands. I had to bust out 
strace to get that info. Small problem, but it could have saved me a few hours 
and some mailing list messages.

Thank you for making this nice looking software available for free. Looking 
forward to playing with it some more.

-Chris

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] What is DEFINE_LTTNG_UST_SIGBUS_STATE for and other beginner questions

[Kienan Stewart via lttng-dev]

Hi Chris,

On 2023-10-04 12:56, Christopher Harvey via lttng-dev wrote:

Hi all,

I'm just getting started with lttng and have a couple of questions. My target 
is getting userspace tracepoints to work in a c++ application I control. I 
don't need kernel or system tracing. The target is running in a docker 
container I have control over as well. Running with --privileged is possible in 
my setup. I could run lttng inside the target container, outside on the host, 
or in another container on the same host. It's all the same to me. I don't know 
which setup is best, or if it even matters given that I don't need system or 
kernel traces.

I am using:
lttng-tools-2.13.11/  and   lttng-ust-2.13.6/
with userspace-rcu-0.14.0/

There is a slight version mismatch there, I just downloaded the "-lastest" from 
the links in the guide.

Here is my tracepoint provider header:
-
#undef LTTNG_UST_TRACEPOINT_PROVIDER
#define LTTNG_UST_TRACEPOINT_PROVIDER my_provider

#undef LTTNG_UST_TRACEPOINT_INCLUDE
#define LTTNG_UST_TRACEPOINT_INCLUDE "./myLttngUstProvider.hpp"

#if !defined(_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
#define _TP_H

#include "lttng/tracepoint.h"

LTTNG_UST_TRACEPOINT_EVENT(
 /* Tracepoint provider name */
 my_provider,

 /* Tracepoint name */
 test,

 /* Input arguments */
 LTTNG_UST_TP_ARGS(
 int, arg1
 ),

 /* Output event fields */
 LTTNG_UST_TP_FIELDS(
 lttng_ust_field_integer(int, arg1, arg1)
 )
)

#endif /* _TP_H */

#include "lttng/tracepoint-event.h"


and the cpp:
---
#define LTTNG_UST_TRACEPOINT_CREATE_PROBES
#define LTTNG_UST_TRACEPOINT_DEFINE

#include "myLttngUstProvider.hpp"
#include "lttng/ust-sigbus.h"

DEFINE_LTTNG_UST_SIGBUS_STATE();


I'm suspicious of a couple of things in my setup.

First, if I omit the DEFINE_LTTNG_UST_SIGBUS_STATE(); line, my program fails to 
compile with the following error:
liblttng-ust-ctl.so: undefined reference to `lttng_ust_sigbus_state'



I wasn't able to reproduce the compilation error you are seeing. Could 
you describe in more detail how you are compiling and linking your 
tracepoint provider and application?



The getting started guide doesn't say anything about needing 
DEFINE_LTTNG_UST_SIGBUS_STATE so it makes me wonder if I have done something 
else wrong somewhere else. I ran a find *.so | xargs nm -D | grep 
lttng_ust_sigbus_state and nothing in lttng-ust provides that symbol, so it 
looks like the target application has to define it. Is that normal?



If the application is actually using liblttng-ust-ctl, then it would be 
normal to do the define. The header file explains that this is done to 
avoid unnecessarily pre-allocating space in the thread-local storage.

https://github.com/lttng/lttng-ust/blob/master/include/lttng/ust-sigbus.h#L39

Traced applications typically don't link against liblttng-ust-ctl.



If I leave DEFINE_LTTNG_UST_SIGBUS_STATE in then my program compiles. If I run 
lttng-sessiond in the target container, then run lttng list --userspace I see 
no tracepoints in the list.



Is your application actively running while you run `lttng list --userspace`?

`lttng list --userspace` will only list the userspace tracepoints of 
applications which are currently registered to the running session 
daemon. If an application disconnects, the tracepoints it registered are 
unloaded.



I tried looking at the lttng source code but there is a lot there. I'm looking 
for some next steps for debugging this setup. What are some things I could 
check to make sure the lttng daemon is running properly? What is the mechanism 
for advertising tracepoints to the lttng daemon? Is it possible that docker has 
somehow interfered with the lttng ipc mechanism? It would be a tiny bit more 
convenient for me to simply deploy lttng-tools as a separate container, can I 
consider doing that instead? I'm not running anything as root, just a normal 
user.

I appreciate any and all guidance. Looking forward to using this neat looking 
tool.

-Chris


thanks,
kienan


___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTNG LIB UST Crash

[Kienan Stewart via lttng-dev]

Hi Lakshmi,

On 2023-09-07 07:46, Lakshmi Deverkonda via lttng-dev wrote:

Thanks for replying.

Basically in our python3 application, we already have a logger which 
will redirect the logs to a log file.  By default, only info logs gets 
logged unless user explicitly turns on debug logging via cli.


For LTTNG Tracing, we would want to log all the events that is both 
info/debugs.

 >>Do you think there would be any overhead on the application?


Yes, but I'm not in position to quantify its effect on your application.

When lttngust is imported, one or two threads are spawned to register 
and communicate with the lttng-sessiond (see `lttngust.agent`). Some 
debug information about this process is available when you run the 
application with `LTTNG_UST_PYTHON_DEBUG=1` set in the environment


In terms of logging, the default behaviour is to add an instance of 
`lttngust.loghandler._Handler` to the root logger with level 
`logging.NOTSET`.


Instances of this handler load the liblttng-ust-python-agent.so dynamic 
library, and the `emit()` method encodes the record data and calls the 
library's `py_tracepoint` function. This is a thin wrapper of 
liblttng-ust, which will ultimately send the encoded event to the 
running consumerd.


One of the goals of the tracing libraries is to have a minimal footprint 
and impact on the running applications, but it is non-zero. In any case, 
if you`re interested in the overhead, you can profile your application 
with and without tracing and measure the effect.


 >> I cannot use the existing logger which does a file logging so for 
lttng only I have created just a new logger without any handler. Should 
this be fine?


I think this question is more about how you're writing your python 
application and not about LTTng / tracing userspace applications, so  I 
don't think I can answer it for you.


 >>Also, I see the default channel created for the python logging in 
lttng  "lttng_python_channel" .  From the documentation, I see that we 
cannot create another channel for python logging.
I would want to modify some of the attribute for the default channel 
such as making event-loss-mode as "overwrite" and increasing the 
trace_file_count. How can I do it ? This is one of the necessary 
requirement for our application. Can you please guide on this ?


You can create the channel explicitly with your desired configuration 
before running `lttng enable-events --python my-logger`


For example:

```
$ lttng create
Session auto-20230907-103141 created.

$ lttng enable-channel --userspace  --overwrite --tracefile-count 10 
--tracefile-size 1073741824 lttng_python_channel

ust channel lttng_python_channel enabled for session auto-20230907-103141

$ lttng enable-event --python my-logger

$ lttng list auto-20230907-103141

...
- lttng_python_channel: [enabled]

  Attributes;
Event-loss mode:  overwrite
Sub-buffer size:  524288 bytes
...
Trace file count: 10 per stream
Trace file size:  1073741824  bytes
...
```

thanks,
kienan




Regards,
Lakshmi

*From:* Kienan Stewart 
*Sent:* 06 September 2023 21:01
*To:* lttng-dev@lists.lttng.org ; Lakshmi 
Deverkonda 

*Subject:* Re: [lttng-dev] LTTNG LIB UST Crash
External email: Use caution opening links or attachments


Hi Lakshmi,

On 2023-09-06 06:02, Lakshmi Deverkonda via lttng-dev wrote:

Thanks for the reply. Issue is fixed after loading the tracing helpers.

I have one query wrt to logging wrt lttng on python3 application. Is
there any way I can avoid the file logging and only trace via lttng ?



In the example python application at
https://lttng.org/docs/v2.13/#doc-python-application 
 the log messages
are not written to disk or to stderr.

I lack the details of your application to give you a more precise answer.

Hope this helps,
kienan


Regards,
Lakshmi



*From:* Kienan Stewart 
*Sent:* 05 September 2023 21:20
*To:* Lakshmi Deverkonda 
*Subject:* Re: [lttng-dev] LTTNG LIB UST Crash
External email: Use caution opening links or attachments


Hi Lakshmi,

could you please provide us with the system details and version
information for LTTng tools and UST?

The bug reporting guidelines which cover the type of information
required to respond adequately questions can be found here:
https://lttng.org/community/#bug-reporting-guidelines 
 
>

Given that you are instrumenting a user space application, do you have a
minimal reproducer of the crash including the details of how the
application is invoked that you would be able to share?

Some types of user space applications required tracing helpers loaded
via LD_PRELOAD. More information can be found here

Re: [lttng-dev] LTTNG LIB UST Crash

[Kienan Stewart via lttng-dev]

Hi Lakshmi,

On 2023-09-06 06:02, Lakshmi Deverkonda via lttng-dev wrote:

Thanks for the reply. Issue is fixed after loading the tracing helpers.

I have one query wrt to logging wrt lttng on python3 application. Is 
there any way I can avoid the file logging and only trace via lttng ?




In the example python application at 
https://lttng.org/docs/v2.13/#doc-python-application the log messages 
are not written to disk or to stderr.


I lack the details of your application to give you a more precise answer.

Hope this helps,
kienan


Regards,
Lakshmi



*From:* Kienan Stewart 
*Sent:* 05 September 2023 21:20
*To:* Lakshmi Deverkonda 
*Subject:* Re: [lttng-dev] LTTNG LIB UST Crash
External email: Use caution opening links or attachments


Hi Lakshmi,

could you please provide us with the system details and version
information for LTTng tools and UST?

The bug reporting guidelines which cover the type of information
required to respond adequately questions can be found here:
https://lttng.org/community/#bug-reporting-guidelines 


Given that you are instrumenting a user space application, do you have a
minimal reproducer of the crash including the details of how the
application is invoked that you would be able to share?

Some types of user space applications required tracing helpers loaded
via LD_PRELOAD. More information can be found here
https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers 


If you're unable to share code or other log files due to company policy,
or require responses within a guaranteed time frame, EfficiOS offers
commercial support services: 
https://www.efficios.com/services/ 


thanks,
kienan

p.s. Sorry for forgetting to CC you in my earlier reply to the list!

On 2023-09-05 02:30, Lakshmi Deverkonda via lttng-dev wrote:

Hi All,

I am observing lttng crash while trying to interface LTTNG for one of my
python3 application.
I just tried the following things,

Added "import lttngust " in the code.

# lttng create clagd
Session clagd created.
Traces will be written in /root/lttng-traces/clagd-20230905-062210

# lttng enable-event --python clagd
#lttng start

#service clagd start
*
*
*cumulus-core: Running cl-support for core files
"clagd-ust.95410.1693895062.core"*

#0  0x7f134fb938eb in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x7f134fb7e535 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x7f134fb7e40f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x7f134fb8c1a2 in __assert_fail () from
/lib/x86_64-linux-gnu/libc.so.6
#4  0x7f134f1a9677 in lttng_ust_add_fd_to_tracker () from
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#5  0x7f134f1bdcf4 in lttng_ust_elf_create () from
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#6  0x7f134f1bf8de in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#7  0x7f134fc8f957 in dl_iterate_phdr () from
/lib/x86_64-linux-gnu/libc.so.6
#8  0x7f134f1bff6b in lttng_ust_dl_update () from
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#9  0x7f134f1c061a in do_lttng_ust_statedump () from
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#10 0x7f134f1b5ca9 in lttng_handle_pending_statedump () from
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#11 0x7f134f1ab6d1 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#12 0x7f134f1ad7eb in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#13 0x7f134ff0dfa3 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#14 0x7f134fc5506f in clone () from /lib/x86_64-linux-gnu/libc.so.6


When I stop the lttng session, I see another core

#0  0x7fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7fad9331f700 (LWP 2221103))]
(gdb) bt
#0  0x7fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x7fad95183535 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x7fad9518340f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x7fad951911a2 in __assert_fail () from
/lib/x86_64-linux-gnu/libc.so.6
#4  0x7fad947e8d9f in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#5  0x7fad947e98e3 in shm_object_table_destroy () from
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#6  0x7fad947e4d9a in channel_destroy () from
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#7  0x7fad947ba6b5 in lttng_session_destroy () from
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#8  0x7fad947b47c6 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#9  0x7fad947b4bac in lttng_ust_objd_unref () from
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#10 0x7fad947b4bac in lttng_ust_objd_unref () from
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#11 0x7fad947b4bac in lttng_ust_objd_unref () from
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#12 0x7fad947b5304 in lttng_ust_objd_table_owner_cleanup () from

Re: [lttng-dev] LTTNG LIB UST Crash

[Kienan Stewart via lttng-dev]

Hi Lakshmi,

could you please provide us with the system details and version 
information for LTTng tools and UST?


The bug reporting guidelines which cover the type of information 
required to respond adequately questions can be found here: 
https://lttng.org/community/#bug-reporting-guidelines


Given that you are instrumenting a user space application, do you have a 
minimal reproducer of the crash including the details of how the 
application is invoked that you would be able to share?


Some types of user space applications required tracing helpers loaded 
via LD_PRELOAD. More information can be found here 
https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers


If you're unable to share code or other log files due to company policy, 
or require responses within a guaranteed time frame, EfficiOS offers 
commercial support services: https://www.efficios.com/services/


thanks,
kienan

On 2023-09-05 02:30, Lakshmi Deverkonda via lttng-dev wrote:

Hi All,

I am observing lttng crash while trying to interface LTTNG for one of my 
python3 application.

I just tried the following things,

Added "import lttngust " in the code.

# lttng create clagd
Session clagd created.
Traces will be written in /root/lttng-traces/clagd-20230905-062210

# lttng enable-event --python clagd
#lttng start

#service clagd start
*
*
*cumulus-core: Running cl-support for core files 
"clagd-ust.95410.1693895062.core"*


#0  0x7f134fb938eb in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x7f134fb7e535 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x7f134fb7e40f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x7f134fb8c1a2 in __assert_fail () from 
/lib/x86_64-linux-gnu/libc.so.6
#4  0x7f134f1a9677 in lttng_ust_add_fd_to_tracker () from 
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#5  0x7f134f1bdcf4 in lttng_ust_elf_create () from 
/lib/x86_64-linux-gnu/liblttng-ust.so.0

#6  0x7f134f1bf8de in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#7  0x7f134fc8f957 in dl_iterate_phdr () from 
/lib/x86_64-linux-gnu/libc.so.6
#8  0x7f134f1bff6b in lttng_ust_dl_update () from 
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#9  0x7f134f1c061a in do_lttng_ust_statedump () from 
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#10 0x7f134f1b5ca9 in lttng_handle_pending_statedump () from 
/lib/x86_64-linux-gnu/liblttng-ust.so.0

#11 0x7f134f1ab6d1 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#12 0x7f134f1ad7eb in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#13 0x7f134ff0dfa3 in start_thread () from 
/lib/x86_64-linux-gnu/libpthread.so.0

#14 0x7f134fc5506f in clone () from /lib/x86_64-linux-gnu/libc.so.6


When I stop the lttng session, I see another core

#0  0x7fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7fad9331f700 (LWP 2221103))]
(gdb) bt
#0  0x7fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x7fad95183535 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x7fad9518340f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x7fad951911a2 in __assert_fail () from 
/lib/x86_64-linux-gnu/libc.so.6

#4  0x7fad947e8d9f in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#5  0x7fad947e98e3 in shm_object_table_destroy () from 
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#6  0x7fad947e4d9a in channel_destroy () from 
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#7  0x7fad947ba6b5 in lttng_session_destroy () from 
/lib/x86_64-linux-gnu/liblttng-ust.so.0

#8  0x7fad947b47c6 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#9  0x7fad947b4bac in lttng_ust_objd_unref () from 
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#10 0x7fad947b4bac in lttng_ust_objd_unref () from 
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#11 0x7fad947b4bac in lttng_ust_objd_unref () from 
/lib/x86_64-linux-gnu/liblttng-ust.so.0
#12 0x7fad947b5304 in lttng_ust_objd_table_owner_cleanup () from 
/lib/x86_64-linux-gnu/liblttng-ust.so.0

#13 0x7fad947b2b75 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#14 0x7fad95512fa3 in start_thread () from 
/lib/x86_64-linux-gnu/libpthread.so.0

#15 0x7fad9525a06f in clone () from /lib/x86_64-linux-gnu/libc.so.6


Can you please help here if I'm missing something. This is a critical 
task item for us but are currently stuck with multiple lttng crashes.


Regards,
Lakshmi

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] profiling areas of LTTng

[Kienan Stewart via lttng-dev]

Hi Bala,


On 2023-07-31 07:01, Bala Gundeboina via lttng-dev wrote:

Hi all,
          I are working in a Ti project, I already installed LTTng and i 
am able to trace userspace.I have a doubt regarding profiling areas of 
LTTng. In the project we are using ISP and RGB gstreamer. *can we 
profile exactly how much time taken by ISP and RGB gstreamer to process 
each frames* by using LTTng userspace tracing.


If the work being done to process each frame is being done via library 
calls, it would be possible to use a custom userspace tracing helper 
which instruments the necessary function calls, similar to those found 
[here][1] which are loaded via LD_PRELOAD.


Otherwise you will need to modify and recompile your application to [add 
tracepoints][2], or use a tool that does dynamic profiling.


i want to know is this 
possible to profile by LTTng, what are the supported profiling areas and 
peripherals of LTTng, Sorry if any grammatical mistakes made.




It is possible to add static (user-defined) instrumentation to 
applications using [liblttng-ust][3] in C/C++. [Helpers][4] are also 
available for Java (JUL/Log4j) and Python applications.


Tracing of the linux kernel and various system information is possible 
with kernel modules provided by [lttng-modules][5].


[1]: https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers
[2]: https://lttng.org/docs/v2.13/#doc-c-application
[3]: https://lttng.org/docs/v2.13/#doc-lttng-ust
[4]: https://lttng.org/docs/v2.13/#doc-lttng-ust-agents
[5]: https://lttng.org/docs/v2.13/#doc-lttng-modules

thanks,
kienan


Thanks & Regards

Bala Gundeboina
This message contains information that may be privileged or confidential 
and is the property of the KPIT Technologies Ltd. It is intended only 
for the person to whom it is addressed. If you are not the intended 
recipient, you are not authorized to read, print, retain copy, 
disseminate, distribute, or use this message or any part thereof. If you 
receive this message in error, please notify the sender immediately and 
delete all copies of this message. KPIT Technologies Ltd. does not 
accept any liability for virus infected mails.


___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev