[lttng-dev] LTTNG Continuously Crashing on Debian-12
Hi, We see that python3 based lttng is continuously crashing on debian-12. Kernel Version is 6.1.0. Is there some special handling that has to be taken care for debian-12? Following is the core decode. Program terminated with signal SIGABRT, Aborted. #0 0x7fb95dac9e2c in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [Current thread is 1 (Thread 0x7fb95d96c040 (LWP 19426))] (gdb) bt #0 0x7fb95dac9e2c in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x7fb95da7afb2 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x7fb95da65472 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x7fb95da65395 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x7fb95da73eb2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x7fb95d9efbbf in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1 #6 0x7fb95d9f0f23 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1 #7 0x7fb95d9ece2f in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1 #8 0x7fb95d9dc537 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1 #9 0x7fb95d9c73c2 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1 #10 0x7fb95d9c8003 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1 #11 0x7fb95d9c8003 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1 #12 0x7fb95d9c8003 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1 #13 0x7fb95d9c8fce in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1 #14 0x7fb95d9c2408 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.1 #15 0x7fb95dd6812a in ?? () from /lib64/ld-linux-x86-64.so.2 #16 0x7fb95dd6b764 in ?? () from /lib64/ld-linux-x86-64.so.2 #17 0x7fb95da7d55d in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #18 0x7fb95da7d69a in exit () from /lib/x86_64-linux-gnu/libc.so.6 #19 0x7fb95da66251 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #20 0x7fb95da66305 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6 #21 0x00627461 in _start () Regards, Lakshmi ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Crash in application due to watchdog timeout with python3 lttng
pplication due to watchdog timeout > with python3 lttng > External email: Use caution opening links or attachments > > > Hi Lakshmi, > > when the lttngust python agent starts, it attempts to connect to one or > more session daemons[1]. > > Each connection starts a thread that loops forever, retrying the > registration in case an exception occurs[2]. > > I don't think the it's designed to have `join()` called on those > threads, which I assume is happening in some of the code you or your > team have written. > > My initial thought is that you should `join()` only the threads that > pertinent to your application, ignoring the lttngust agent threads and > then exit the application as normal. > > [1]: > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flttng%2Flttng-ust%2Fblob%2F3287f48be61ef3491aff0a80b7185ac57b3d8a5d%2Fsrc%2Fpython-lttngust%2Flttngust%2Fagent.py%23L334=05%7C02%7Claksd%40nvidia.com%7C41cb3fd667944d89cab708dc2f0e2a78%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638436985101003000%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=mZjN5F0W%2BHNWslQAULblQHefyQIT5cpSNFylxAY2kAk%3D=0<https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L334> > > <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flttng%2Flttng-ust%2Fblob%2F3287f48be61ef3491aff0a80b7185ac57b3d8a5d%2Fsrc%2Fpython-lttngust%2Flttngust%2Fagent.py%23L334=05%7C02%7Claksd%40nvidia.com%7C41cb3fd667944d89cab708dc2f0e2a78%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638436985101009055%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=JypI4VvwxEZzTGG9621hNAvonem63ljMZiXaqvJhIQw%3D=0<https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L334>> > [2]: > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flttng%2Flttng-ust%2Fblob%2F3287f48be61ef3491aff0a80b7185ac57b3d8a5d%2Fsrc%2Fpython-lttngust%2Flttngust%2Fagent.py%23L83=05%7C02%7Claksd%40nvidia.com%7C41cb3fd667944d89cab708dc2f0e2a78%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638436985101015117%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=ZTbmTvR2uY1QoS9qKA6HqwXRKFd3pY%2F39kw2qwjaaVM%3D=0<https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L83> > > <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flttng%2Flttng-ust%2Fblob%2F3287f48be61ef3491aff0a80b7185ac57b3d8a5d%2Fsrc%2Fpython-lttngust%2Flttngust%2Fagent.py%23L83=05%7C02%7Claksd%40nvidia.com%7C41cb3fd667944d89cab708dc2f0e2a78%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638436985101021691%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=%2B6JlJyEJVLlfwCMv8eHWwnQwr298L%2Fsk3vGNUuhaAMU%3D=0<https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L83>> > > thanks, > kienan > > On 2/13/24 09:23, Lakshmi Deverkonda via lttng-dev wrote: >> Hi, >> >> We are able to integrate python3 lttng module in our application(python3 >> based). However, we are seeing that whenever the application terminates, >> there is watchdog timeout due to timeout in joining the threads. What >> could be the reason for this ? Does lttng module hold any thread event >> locks ? >> We are completely blocked on this issue. Could you please help ? >> >> Here is the snippet of the core dump >> >> (gdb) py-bt >> Traceback (most recent call first): >>File "/usr/lib/python3.7/threading.py", line 1048, in >> _wait_for_tstate_lock >> elif lock.acquire(block, timeout): >>File "/usr/lib/python3.7/threading.py", line 1032, in join >> self._wait_for_tstate_lock() >>File "/usr/lib/python3/dist-packages/h.py", line 231, in JoinThreads >> self.TT.join() >>File "/usr/sbin/c", line 1466, in do_exit >> H.JoinThreads() >>File "/usr/sbin/c", line 7201, in main >> do_exit(nlm, status) >>File "/usr/sbin/c", line 7233, in >> main() >> (gdb) >> >> On a parallel note, thanks to Kienan who has been trying to provide >> pointers on various issues reported so far. >> >> Need help on this issue as well. >> Thanks in advance, >> >> Regards, >> Lakshmi >> >> >> >> ___ >> lttng-dev mailing lis
Re: [lttng-dev] Crash in application due to watchdog timeout with python3 lttng
This is how, we have created the logger. So the first logger is for file
logging where is as the second one is for lttng.
self.logger = logging.getLogger('cd')
self.lttng_logger = logging.getLogger('cd-lttng')
It seems like at the instant exactly when lttng is logging some data on a
particular thread and the same instant we receive SIGTERM for the application,
we are unable to join that particular thread. Can you please help.
Also we see that performance of lttng is not that good for python3. My
application has around 24 threads and when logging is enabled for each of the
threads,
there is a delay upto 24s for processing the external events.
Please suggest how to proceed further on these issues.
Regards,
Lakshmi
From: Lakshmi Deverkonda
Sent: 13 February 2024 21:05
To: Kienan Stewart ; lttng-dev@lists.lttng.org
Subject: Re: [lttng-dev] Crash in application due to watchdog timeout with
python3 lttng
Yes. We are trying to join only the threads related to the application. The
timeout is happening while trying to join the threads started by the
application.
Regards,
Lakshmi
From: Kienan Stewart
Sent: 13 February 2024 20:50
To: Lakshmi Deverkonda ; lttng-dev@lists.lttng.org
Subject: Re: [lttng-dev] Crash in application due to watchdog timeout with
python3 lttng
External email: Use caution opening links or attachments
Hi Lakshmi,
when the lttngust python agent starts, it attempts to connect to one or
more session daemons[1].
Each connection starts a thread that loops forever, retrying the
registration in case an exception occurs[2].
I don't think the it's designed to have `join()` called on those
threads, which I assume is happening in some of the code you or your
team have written.
My initial thought is that you should `join()` only the threads that
pertinent to your application, ignoring the lttngust agent threads and
then exit the application as normal.
[1]:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flttng%2Flttng-ust%2Fblob%2F3287f48be61ef3491aff0a80b7185ac57b3d8a5d%2Fsrc%2Fpython-lttngust%2Flttngust%2Fagent.py%23L334=05%7C02%7Claksd%40nvidia.com%7Cbdf064d348474249f14a08dc2ca755c9%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C63843437867621%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=7tmpFtjl7RkTVgYLr2YjdlPs2oM1F%2FXOg6W51mHDCws%3D=0<https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L334>
[2]:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flttng%2Flttng-ust%2Fblob%2F3287f48be61ef3491aff0a80b7185ac57b3d8a5d%2Fsrc%2Fpython-lttngust%2Flttngust%2Fagent.py%23L83=05%7C02%7Claksd%40nvidia.com%7Cbdf064d348474249f14a08dc2ca755c9%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C63843437874777%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=PKb8rKDWFKmuuVB4YQEL8ZtAP%2B%2BYfTniUuLN9fFBctc%3D=0<https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L83>
thanks,
kienan
On 2/13/24 09:23, Lakshmi Deverkonda via lttng-dev wrote:
> Hi,
>
> We are able to integrate python3 lttng module in our application(python3
> based). However, we are seeing that whenever the application terminates,
> there is watchdog timeout due to timeout in joining the threads. What
> could be the reason for this ? Does lttng module hold any thread event
> locks ?
> We are completely blocked on this issue. Could you please help ?
>
> Here is the snippet of the core dump
>
> (gdb) py-bt
> Traceback (most recent call first):
>File "/usr/lib/python3.7/threading.py", line 1048, in
> _wait_for_tstate_lock
> elif lock.acquire(block, timeout):
>File "/usr/lib/python3.7/threading.py", line 1032, in join
> self._wait_for_tstate_lock()
>File "/usr/lib/python3/dist-packages/h.py", line 231, in JoinThreads
> self.TT.join()
>File "/usr/sbin/c", line 1466, in do_exit
> H.JoinThreads()
>File "/usr/sbin/c", line 7201, in main
> do_exit(nlm, status)
>File "/usr/sbin/c", line 7233, in
> main()
> (gdb)
>
> On a parallel note, thanks to Kienan who has been trying to provide
> pointers on various issues reported so far.
>
> Need help on this issue as well.
> Thanks in advance,
>
> Regards,
> Lakshmi
>
>
>
> ___
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev=05%7C02%7Claksd%40nvidia.com%7Cbdf064d348474249f14a08dc2ca755c9%7C43083d1572734
Re: [lttng-dev] Crash in application due to watchdog timeout with python3 lttng
Yes. We are trying to join only the threads related to the application. The timeout is happening while trying to join the threads started by the application. Regards, Lakshmi From: Kienan Stewart Sent: 13 February 2024 20:50 To: Lakshmi Deverkonda ; lttng-dev@lists.lttng.org Subject: Re: [lttng-dev] Crash in application due to watchdog timeout with python3 lttng External email: Use caution opening links or attachments Hi Lakshmi, when the lttngust python agent starts, it attempts to connect to one or more session daemons[1]. Each connection starts a thread that loops forever, retrying the registration in case an exception occurs[2]. I don't think the it's designed to have `join()` called on those threads, which I assume is happening in some of the code you or your team have written. My initial thought is that you should `join()` only the threads that pertinent to your application, ignoring the lttngust agent threads and then exit the application as normal. [1]: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flttng%2Flttng-ust%2Fblob%2F3287f48be61ef3491aff0a80b7185ac57b3d8a5d%2Fsrc%2Fpython-lttngust%2Flttngust%2Fagent.py%23L334=05%7C02%7Claksd%40nvidia.com%7Cbdf064d348474249f14a08dc2ca755c9%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C63843437867621%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=7tmpFtjl7RkTVgYLr2YjdlPs2oM1F%2FXOg6W51mHDCws%3D=0<https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L334> [2]: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flttng%2Flttng-ust%2Fblob%2F3287f48be61ef3491aff0a80b7185ac57b3d8a5d%2Fsrc%2Fpython-lttngust%2Flttngust%2Fagent.py%23L83=05%7C02%7Claksd%40nvidia.com%7Cbdf064d348474249f14a08dc2ca755c9%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C63843437874777%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=PKb8rKDWFKmuuVB4YQEL8ZtAP%2B%2BYfTniUuLN9fFBctc%3D=0<https://github.com/lttng/lttng-ust/blob/3287f48be61ef3491aff0a80b7185ac57b3d8a5d/src/python-lttngust/lttngust/agent.py#L83> thanks, kienan On 2/13/24 09:23, Lakshmi Deverkonda via lttng-dev wrote: > Hi, > > We are able to integrate python3 lttng module in our application(python3 > based). However, we are seeing that whenever the application terminates, > there is watchdog timeout due to timeout in joining the threads. What > could be the reason for this ? Does lttng module hold any thread event > locks ? > We are completely blocked on this issue. Could you please help ? > > Here is the snippet of the core dump > > (gdb) py-bt > Traceback (most recent call first): >File "/usr/lib/python3.7/threading.py", line 1048, in > _wait_for_tstate_lock > elif lock.acquire(block, timeout): >File "/usr/lib/python3.7/threading.py", line 1032, in join > self._wait_for_tstate_lock() >File "/usr/lib/python3/dist-packages/h.py", line 231, in JoinThreads > self.TT.join() >File "/usr/sbin/c", line 1466, in do_exit > H.JoinThreads() >File "/usr/sbin/c", line 7201, in main > do_exit(nlm, status) >File "/usr/sbin/c", line 7233, in > main() > (gdb) > > On a parallel note, thanks to Kienan who has been trying to provide > pointers on various issues reported so far. > > Need help on this issue as well. > Thanks in advance, > > Regards, > Lakshmi > > > > ___ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev=05%7C02%7Claksd%40nvidia.com%7Cbdf064d348474249f14a08dc2ca755c9%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C63843437880631%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=2iVi8xLrTS1Dj%2FcF3V30q0OjCvMP4kTpOUSthJvnZI0%3D=0<https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev> ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] Crash in application due to watchdog timeout with python3 lttng
Hi, We are able to integrate python3 lttng module in our application(python3 based). However, we are seeing that whenever the application terminates, there is watchdog timeout due to timeout in joining the threads. What could be the reason for this ? Does lttng module hold any thread event locks ? We are completely blocked on this issue. Could you please help ? Here is the snippet of the core dump (gdb) py-bt Traceback (most recent call first): File "/usr/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock elif lock.acquire(block, timeout): File "/usr/lib/python3.7/threading.py", line 1032, in join self._wait_for_tstate_lock() File "/usr/lib/python3/dist-packages/h.py", line 231, in JoinThreads self.TT.join() File "/usr/sbin/c", line 1466, in do_exit H.JoinThreads() File "/usr/sbin/c", line 7201, in main do_exit(nlm, status) File "/usr/sbin/c", line 7233, in main() (gdb) On a parallel note, thanks to Kienan who has been trying to provide pointers on various issues reported so far. Need help on this issue as well. Thanks in advance, Regards, Lakshmi ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] Python lttngust module import error
Hi, I'm observing an issue recently wrt import of python3 lttng module. Following is the error and need your help to find out what is wrong. >>> import lttngust Exception in thread system: Traceback (most recent call last): File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner self.run() File "/usr/lib/python3.7/threading.py", line 865, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.7/dist-packages/lttngust/agent.py", line 291, in _client_thread_target client = _TcpClient(name, _SESSIOND_HOST, port, reg_queue) File "/usr/local/lib/python3.7/dist-packages/lttngust/agent.py", line 76, in __init__ raise e File "/usr/local/lib/python3.7/dist-packages/lttngust/agent.py", line 73, in __init__ self._log_handler = lttngust.loghandler._Handler() File "/usr/local/lib/python3.7/dist-packages/lttngust/loghandler.py", line 32, in __init__ self.agent_lib = ctypes.cdll.LoadLibrary(_Handler._LIB_NAME) File "/usr/lib/python3.7/ctypes/__init__.py", line 434, in LoadLibrary return self._dlltype(name) File "/usr/lib/python3.7/ctypes/__init__.py", line 356, in __init__ self._handle = _dlopen(self._name, mode) OSError: liblttng-ust-python-agent.so: cannot open shared object file: No such file or directory >>> I have installed liblttng-ust-python-agent.so /# ldconfig -p | grep liblttng liblttng-ust.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblttng-ust.so.0 liblttng-ust-tracepoint.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0 liblttng-ust-python-agent.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblttng-ust-python-agent.so.0 liblttng-ust-pthread-wrapper.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblttng-ust-pthread-wrapper.so.0 liblttng-ust-libc-wrapper.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblttng-ust-libc-wrapper.so.0 liblttng-ust-fork.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblttng-ust-fork.so.0 liblttng-ust-fd.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblttng-ust-fd.so.0 liblttng-ust-dl.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblttng-ust-dl.so.0 liblttng-ust-cyg-profile.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblttng-ust-cyg-profile.so.0 liblttng-ust-cyg-profile-fast.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblttng-ust-cyg-profile-fast.so.0 liblttng-ust-ctl.so.4 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblttng-ust-ctl.so.4 liblttng-ctl.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblttng-ctl.so.0 Regards, Lakshmi ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] LTTNG LIB UST Crash
Thanks for replying. Basically in our python3 application, we already have a logger which will redirect the logs to a log file. By default, only info logs gets logged unless user explicitly turns on debug logging via cli. For LTTNG Tracing, we would want to log all the events that is both info/debugs. >>Do you think there would be any overhead on the application? >> I cannot use the existing logger which does a file logging so for lttng only >> I have created just a new logger without any handler. Should this be fine? >>Also, I see the default channel created for the python logging in lttng >>"lttng_python_channel" . From the documentation, I see that we cannot create >>another channel for python logging. I would want to modify some of the attribute for the default channel such as making event-loss-mode as "overwrite" and increasing the trace_file_count. How can I do it ? This is one of the necessary requirement for our application. Can you please guide on this ? Regards, Lakshmi From: Kienan Stewart Sent: 06 September 2023 21:01 To: lttng-dev@lists.lttng.org ; Lakshmi Deverkonda Subject: Re: [lttng-dev] LTTNG LIB UST Crash External email: Use caution opening links or attachments Hi Lakshmi, On 2023-09-06 06:02, Lakshmi Deverkonda via lttng-dev wrote: > Thanks for the reply. Issue is fixed after loading the tracing helpers. > > I have one query wrt to logging wrt lttng on python3 application. Is > there any way I can avoid the file logging and only trace via lttng ? > In the example python application at https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.13%2F%23doc-python-application=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C63829641743499%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=HMryNCihtLBKA49TWfsxp4%2FkzkFJhZrlyocoAsR7w78%3D=0<https://lttng.org/docs/v2.13/#doc-python-application> the log messages are not written to disk or to stderr. I lack the details of your application to give you a more precise answer. Hope this helps, kienan > Regards, > Lakshmi > > > > *From:* Kienan Stewart > *Sent:* 05 September 2023 21:20 > *To:* Lakshmi Deverkonda > *Subject:* Re: [lttng-dev] LTTNG LIB UST Crash > External email: Use caution opening links or attachments > > > Hi Lakshmi, > > could you please provide us with the system details and version > information for LTTng tools and UST? > > The bug reporting guidelines which cover the type of information > required to respond adequately questions can be found here: > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fcommunity%2F%23bug-reporting-guidelines=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C63829641743499%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=ZU4mg%2BDNKMbtxCVA3kRuyCZAaj7yUH5bLz6p4oXpROA%3D=0<https://lttng.org/community/#bug-reporting-guidelines> > > <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fcommunity%2F%23bug-reporting-guidelines=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C63829641899697%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=x0QzE9thuFtUFdNKd2qg0Fo4XSYqeTB4vbEBl88dGxE%3D=0<https://lttng.org/community/#bug-reporting-guidelines>> > > Given that you are instrumenting a user space application, do you have a > minimal reproducer of the crash including the details of how the > application is invoked that you would be able to share? > > Some types of user space applications required tracing helpers loaded > via LD_PRELOAD. More information can be found here > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.13%2F%23doc-prebuilt-ust-helpers=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C63829641899697%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=WpDm%2FpMZMb3EukOw9NF9L2GiMZpWeOAq3It8H8p%2BNSI%3D=0<https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers> > > <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.13%2F%23doc-prebuilt-ust-helpers=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C63829641899697%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
Re: [lttng-dev] LTTNG LIB UST Crash
Thanks for the reply. Issue is fixed after loading the tracing helpers. I have one query wrt to logging wrt lttng on python3 application. Is there any way I can avoid the file logging and only trace via lttng ? Regards, Lakshmi From: Kienan Stewart Sent: 05 September 2023 21:20 To: Lakshmi Deverkonda Subject: Re: [lttng-dev] LTTNG LIB UST Crash External email: Use caution opening links or attachments Hi Lakshmi, could you please provide us with the system details and version information for LTTng tools and UST? The bug reporting guidelines which cover the type of information required to respond adequately questions can be found here: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fcommunity%2F%23bug-reporting-guidelines=05%7C01%7Claksd%40nvidia.com%7C087f25ce0bc54b0b0cf008dbae27e747%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638295258729072557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=ZUKJj2JTtxWJNp9Prf8XjarnbPRPbLGXQy1Eou8AvRE%3D=0<https://lttng.org/community/#bug-reporting-guidelines> Given that you are instrumenting a user space application, do you have a minimal reproducer of the crash including the details of how the application is invoked that you would be able to share? Some types of user space applications required tracing helpers loaded via LD_PRELOAD. More information can be found here https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.13%2F%23doc-prebuilt-ust-helpers=05%7C01%7Claksd%40nvidia.com%7C087f25ce0bc54b0b0cf008dbae27e747%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638295258729072557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=g9HSdPQPqYw%2BdTXH9kKKyYjNZDX90NGArrUTW8UHp7s%3D=0<https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers> If you're unable to share code or other log files due to company policy, or require responses within a guaranteed time frame, EfficiOS offers commercial support services: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.efficios.com%2Fservices%2F=05%7C01%7Claksd%40nvidia.com%7C087f25ce0bc54b0b0cf008dbae27e747%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638295258729072557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=Yry9Y74Ydo9eHmC5uP0bhcueqYTNoXbXpyiOLIytShc%3D=0<https://www.efficios.com/services/> thanks, kienan p.s. Sorry for forgetting to CC you in my earlier reply to the list! On 2023-09-05 02:30, Lakshmi Deverkonda via lttng-dev wrote: > Hi All, > > I am observing lttng crash while trying to interface LTTNG for one of my > python3 application. > I just tried the following things, > > Added "import lttngust " in the code. > > # lttng create clagd > Session clagd created. > Traces will be written in /root/lttng-traces/clagd-20230905-062210 > > # lttng enable-event --python clagd > #lttng start > > #service clagd start > * > * > *cumulus-core: Running cl-support for core files > "clagd-ust.95410.1693895062.core"* > > #0 0x7f134fb938eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x7f134fb7e535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 > #2 0x7f134fb7e40f in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #3 0x7f134fb8c1a2 in __assert_fail () from > /lib/x86_64-linux-gnu/libc.so.6 > #4 0x7f134f1a9677 in lttng_ust_add_fd_to_tracker () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #5 0x7f134f1bdcf4 in lttng_ust_elf_create () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #6 0x7f134f1bf8de in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #7 0x7f134fc8f957 in dl_iterate_phdr () from > /lib/x86_64-linux-gnu/libc.so.6 > #8 0x7f134f1bff6b in lttng_ust_dl_update () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #9 0x7f134f1c061a in do_lttng_ust_statedump () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #10 0x7f134f1b5ca9 in lttng_handle_pending_statedump () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #11 0x7f134f1ab6d1 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #12 0x7f134f1ad7eb in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #13 0x7f134ff0dfa3 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #14 0x7f134fc5506f in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > > When I stop the lttng session, I see another core > > #0 0x7fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 > [Current thread is 1 (Thread 0x7fad9331f700 (LWP 2221103))] > (gdb) bt > #0 0x7fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x7fad95183535 in abort () from /lib/x86_64-linux-gnu/libc.s
[lttng-dev] LTTNG LIB UST Crash
Hi All, I am observing lttng crash while trying to interface LTTNG for one of my python3 application. I just tried the following things, Added "import lttngust " in the code. # lttng create clagd Session clagd created. Traces will be written in /root/lttng-traces/clagd-20230905-062210 # lttng enable-event --python clagd #lttng start #service clagd start cumulus-core: Running cl-support for core files "clagd-ust.95410.1693895062.core" #0 0x7f134fb938eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x7f134fb7e535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x7f134fb7e40f in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x7f134fb8c1a2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x7f134f1a9677 in lttng_ust_add_fd_to_tracker () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #5 0x7f134f1bdcf4 in lttng_ust_elf_create () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #6 0x7f134f1bf8de in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #7 0x7f134fc8f957 in dl_iterate_phdr () from /lib/x86_64-linux-gnu/libc.so.6 #8 0x7f134f1bff6b in lttng_ust_dl_update () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #9 0x7f134f1c061a in do_lttng_ust_statedump () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #10 0x7f134f1b5ca9 in lttng_handle_pending_statedump () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #11 0x7f134f1ab6d1 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #12 0x7f134f1ad7eb in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #13 0x7f134ff0dfa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #14 0x7f134fc5506f in clone () from /lib/x86_64-linux-gnu/libc.so.6 When I stop the lttng session, I see another core #0 0x7fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 [Current thread is 1 (Thread 0x7fad9331f700 (LWP 2221103))] (gdb) bt #0 0x7fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x7fad95183535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x7fad9518340f in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x7fad951911a2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x7fad947e8d9f in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #5 0x7fad947e98e3 in shm_object_table_destroy () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #6 0x7fad947e4d9a in channel_destroy () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #7 0x7fad947ba6b5 in lttng_session_destroy () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #8 0x7fad947b47c6 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #9 0x7fad947b4bac in lttng_ust_objd_unref () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #10 0x7fad947b4bac in lttng_ust_objd_unref () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #11 0x7fad947b4bac in lttng_ust_objd_unref () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #12 0x7fad947b5304 in lttng_ust_objd_table_owner_cleanup () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #13 0x7fad947b2b75 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 #14 0x7fad95512fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #15 0x7fad9525a06f in clone () from /lib/x86_64-linux-gnu/libc.so.6 Can you please help here if I'm missing something. This is a critical task item for us but are currently stuck with multiple lttng crashes. Regards, Lakshmi ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
