Bug#1040716: libc6: Stack Traces
On Tue, 2023-07-11 at 22:34 +0200, Aurelien Jarno wrote: > Hi, > > On 2023-07-11 15:28, Tim McConnell wrote: > > > > > > On Tue, 2023-07-11 at 21:11 +0200, Aurelien Jarno wrote: > > > Hi, > > > > > > On 2023-07-11 11:21, Tim McConnell wrote: > > > > > > > > > > > > On Mon, 2023-07-10 at 23:17 +0200, Aurelien Jarno wrote: > > > > > You might want > > > > > to upgrade to version 2.37-5 to check if it solves your issue > > > > Okay that's done and it's still doing it. The entry from > > > > Journalctl > > > > shows module libudev1 if that's of any use. > > > > > > > > Started systemd-coredump@1785-616863-0.service - Process Core > > > > Dump > > > > (PID > > > > 616863/UID 0). > > > > Jul 11 10:52:06 DebianTim systemd-coredump[616865]: Process > > > > 616847 > > > > (collectd) of user 0 dumped core. > > > > > > > > Module > > > > libudev.so.1 > > > > from deb systemd-252.11-1.amd64 > > > > Stack trace > > > > of > > > > thread 616848: > > > > #0 > > > > 0x7fce1335e9f2 __memmove_ssse3 (libc.so.6 + 0x16d9f2) > > > > #1 > > > > 0x7fce131156d9 rrd_write (librrd.so.8 + 0x346d9) > > > > #2 > > > > 0x7fce13120acd n/a (librrd.so.8 + 0x3facd) > > > > #3 > > > > 0x7fce13122962 n/a (librrd.so.8 + 0x41962) > > > > #4 > > > > 0x7fce1317c370 n/a (rrdtool.so + 0x3370) > > > > #5 > > > > 0x7fce132793ec start_thread (libc.so.6 + 0x883ec) > > > > #6 > > > > 0x7fce132f9a1c __clone3 (libc.so.6 + 0x108a1c) > > > > > > > > > > Thanks for the details. This shows that the binary crashing > > > regularly > > > is > > > collectd. This is very unlikely that the issue is linked to the > > > locales, > > > and your test is confirming that. > > > > > > It's also not clear that it's a glibc issue, it's more likely an > > > issue > > > in collectd or librrd8. It appears that systemd-coredump saved a > > > coredump when the process crashed. You should be able do use > > > "coredumpctl" to get the list of cores. You can select one > > > coredump > > > and > > > examine it with gdb using "coredumpctl debug ". Then when > > > under > > > gdb > > > you should be able to run "thread apply all bt" to get the > > > backtrace. > > > That should allows to better understand the issue. > > > > > > Regards > > > Aurelien > > > > > I'm unsure how helpful this is ( I am not a programmer) but: > > thread apply all bt > > Thanks, that's already much more useful. However I forgot to tell you > to > install the libc6-dbg package before getting the backtrace, sorry > about > that. Could you please install it and follow the same procedure > again? > > Thanks > Aurelien > It's already there, any others?
Bug#1040716: libc6: Stack Traces
Hi, On 2023-07-11 15:28, Tim McConnell wrote: > > > On Tue, 2023-07-11 at 21:11 +0200, Aurelien Jarno wrote: > > Hi, > > > > On 2023-07-11 11:21, Tim McConnell wrote: > > > > > > > > > On Mon, 2023-07-10 at 23:17 +0200, Aurelien Jarno wrote: > > > > You might want > > > > to upgrade to version 2.37-5 to check if it solves your issue > > > Okay that's done and it's still doing it. The entry from Journalctl > > > shows module libudev1 if that's of any use. > > > > > > Started systemd-coredump@1785-616863-0.service - Process Core Dump > > > (PID > > > 616863/UID 0). > > > Jul 11 10:52:06 DebianTim systemd-coredump[616865]: Process 616847 > > > (collectd) of user 0 dumped core. > > > > > > Module > > > libudev.so.1 > > > from deb systemd-252.11-1.amd64 > > > Stack trace of > > > thread 616848: > > > #0 > > > 0x7fce1335e9f2 __memmove_ssse3 (libc.so.6 + 0x16d9f2) > > > #1 > > > 0x7fce131156d9 rrd_write (librrd.so.8 + 0x346d9) > > > #2 > > > 0x7fce13120acd n/a (librrd.so.8 + 0x3facd) > > > #3 > > > 0x7fce13122962 n/a (librrd.so.8 + 0x41962) > > > #4 > > > 0x7fce1317c370 n/a (rrdtool.so + 0x3370) > > > #5 > > > 0x7fce132793ec start_thread (libc.so.6 + 0x883ec) > > > #6 > > > 0x7fce132f9a1c __clone3 (libc.so.6 + 0x108a1c) > > > > > > > Thanks for the details. This shows that the binary crashing regularly > > is > > collectd. This is very unlikely that the issue is linked to the > > locales, > > and your test is confirming that. > > > > It's also not clear that it's a glibc issue, it's more likely an > > issue > > in collectd or librrd8. It appears that systemd-coredump saved a > > coredump when the process crashed. You should be able do use > > "coredumpctl" to get the list of cores. You can select one coredump > > and > > examine it with gdb using "coredumpctl debug ". Then when under > > gdb > > you should be able to run "thread apply all bt" to get the backtrace. > > That should allows to better understand the issue. > > > > Regards > > Aurelien > > > I'm unsure how helpful this is ( I am not a programmer) but: > thread apply all bt Thanks, that's already much more useful. However I forgot to tell you to install the libc6-dbg package before getting the backtrace, sorry about that. Could you please install it and follow the same procedure again? Thanks Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://aurel32.net
Bug#1040716: libc6: Stack Traces
On Tue, 2023-07-11 at 21:11 +0200, Aurelien Jarno wrote: > Hi, > > On 2023-07-11 11:21, Tim McConnell wrote: > > > > > > On Mon, 2023-07-10 at 23:17 +0200, Aurelien Jarno wrote: > > > You might want > > > to upgrade to version 2.37-5 to check if it solves your issue > > Okay that's done and it's still doing it. The entry from Journalctl > > shows module libudev1 if that's of any use. > > > > Started systemd-coredump@1785-616863-0.service - Process Core Dump > > (PID > > 616863/UID 0). > > Jul 11 10:52:06 DebianTim systemd-coredump[616865]: Process 616847 > > (collectd) of user 0 dumped core. > > > > Module > > libudev.so.1 > > from deb systemd-252.11-1.amd64 > > Stack trace of > > thread 616848: > > #0 > > 0x7fce1335e9f2 __memmove_ssse3 (libc.so.6 + 0x16d9f2) > > #1 > > 0x7fce131156d9 rrd_write (librrd.so.8 + 0x346d9) > > #2 > > 0x7fce13120acd n/a (librrd.so.8 + 0x3facd) > > #3 > > 0x7fce13122962 n/a (librrd.so.8 + 0x41962) > > #4 > > 0x7fce1317c370 n/a (rrdtool.so + 0x3370) > > #5 > > 0x7fce132793ec start_thread (libc.so.6 + 0x883ec) > > #6 > > 0x7fce132f9a1c __clone3 (libc.so.6 + 0x108a1c) > > > > Thanks for the details. This shows that the binary crashing regularly > is > collectd. This is very unlikely that the issue is linked to the > locales, > and your test is confirming that. > > It's also not clear that it's a glibc issue, it's more likely an > issue > in collectd or librrd8. It appears that systemd-coredump saved a > coredump when the process crashed. You should be able do use > "coredumpctl" to get the list of cores. You can select one coredump > and > examine it with gdb using "coredumpctl debug ". Then when under > gdb > you should be able to run "thread apply all bt" to get the backtrace. > That should allows to better understand the issue. > > Regards > Aurelien > I'm unsure how helpful this is ( I am not a programmer) but: thread apply all bt Thread 12 (Thread 0x7fae021f96c0 (LWP 9783)): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fae021f8e60, op=393, expected=0, futex_word=0x556c7b80d488) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x556c7b80d488, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fae021f8e60, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex- internal.c:87 #2 0x7fae08a3b1bb in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x556c7b80d488, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fae021f8e60, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x7fae08a3dafc in __pthread_cond_wait_common (abstime=0x7fae021f8e60, clockid=0, mutex=0x556c7b80d4a0, cond=0x556c7b80d460) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x556c7b80d460, mutex=0x556c7b80d4a0, abstime=0x7fae021f8e60) at ./nptl/pthread_cond_wait.c:643 #5 0x556c7b7e83a2 in () #6 0x7fae08a3e3ec in start_thread (arg=) at ./nptl/pthread_create.c:444 #7 0x7fae08abea1c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 Thread 11 (Thread 0x7fae08c1a740 (LWP 9776)): #0 0x7fae08a84a25 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7fff754277f0, rem=0x7fff754277f0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48 #1 0x7fae08a893f3 in __GI___nanosleep (req=, rem=) at ../sysdeps/unix/sysv/linux/nanosleep.c:25 #2 0x556c7b7de16b in () #3 0x556c7b7de7a8 in run_loop () #4 0x556c7b7dd934 in main () Thread 10 (Thread 0x7fae041fd6c0 (LWP 9779)): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x556c7b80d3e8) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x556c7b80d3e8, expected=expected@entry=0, clockid=clockid@entry=0, abst--Type for more, q to quit, c to continue without paging-- ime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87 #2 0x7fae08a3b1bb in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x556c7b80d3e8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x7fae08a3d818 in __pthread_cond_wait_common (abstime=0x0, clockid=0,
Bug#1040716: libc6: Stack Traces
Hi, On 2023-07-11 11:21, Tim McConnell wrote: > > > On Mon, 2023-07-10 at 23:17 +0200, Aurelien Jarno wrote: > > You might want > > to upgrade to version 2.37-5 to check if it solves your issue > Okay that's done and it's still doing it. The entry from Journalctl > shows module libudev1 if that's of any use. > > Started systemd-coredump@1785-616863-0.service - Process Core Dump (PID > 616863/UID 0). > Jul 11 10:52:06 DebianTim systemd-coredump[616865]: Process 616847 > (collectd) of user 0 dumped core. > > Module libudev.so.1 > from deb systemd-252.11-1.amd64 > Stack trace of > thread 616848: > #0 > 0x7fce1335e9f2 __memmove_ssse3 (libc.so.6 + 0x16d9f2) > #1 > 0x7fce131156d9 rrd_write (librrd.so.8 + 0x346d9) > #2 > 0x7fce13120acd n/a (librrd.so.8 + 0x3facd) > #3 > 0x7fce13122962 n/a (librrd.so.8 + 0x41962) > #4 > 0x7fce1317c370 n/a (rrdtool.so + 0x3370) > #5 > 0x7fce132793ec start_thread (libc.so.6 + 0x883ec) > #6 > 0x7fce132f9a1c __clone3 (libc.so.6 + 0x108a1c) > Thanks for the details. This shows that the binary crashing regularly is collectd. This is very unlikely that the issue is linked to the locales, and your test is confirming that. It's also not clear that it's a glibc issue, it's more likely an issue in collectd or librrd8. It appears that systemd-coredump saved a coredump when the process crashed. You should be able do use "coredumpctl" to get the list of cores. You can select one coredump and examine it with gdb using "coredumpctl debug ". Then when under gdb you should be able to run "thread apply all bt" to get the backtrace. That should allows to better understand the issue. Regards Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://aurel32.net
Bug#1040716: libc6: Stack Traces
On Mon, 2023-07-10 at 23:17 +0200, Aurelien Jarno wrote: > You might want > to upgrade to version 2.37-5 to check if it solves your issue Okay that's done and it's still doing it. The entry from Journalctl shows module libudev1 if that's of any use. Started systemd-coredump@1785-616863-0.service - Process Core Dump (PID 616863/UID 0). Jul 11 10:52:06 DebianTim systemd-coredump[616865]: Process 616847 (collectd) of user 0 dumped core. Module libudev.so.1 from deb systemd-252.11-1.amd64 Stack trace of thread 616848: #0 0x7fce1335e9f2 __memmove_ssse3 (libc.so.6 + 0x16d9f2) #1 0x7fce131156d9 rrd_write (librrd.so.8 + 0x346d9) #2 0x7fce13120acd n/a (librrd.so.8 + 0x3facd) #3 0x7fce13122962 n/a (librrd.so.8 + 0x41962) #4 0x7fce1317c370 n/a (rrdtool.so + 0x3370) #5 0x7fce132793ec start_thread (libc.so.6 + 0x883ec) #6 0x7fce132f9a1c __clone3 (libc.so.6 + 0x108a1c) Stack trace of thread 616847: #0 0x7fce132bfa25 __GI___clock_nanosleep (libc.so.6 + 0xcea25) #1 0x7fce132c43f3 __GI___nanosleep (libc.so.6 + 0xd33f3) #2 0x55ade3cad16b n/a (collectd + 0x816b) #3 0x55ade3cad7a8 run_loop (collectd + 0x87a8) #4 0x55ade3cac934 main (collectd + 0x7934) #5 0x7fce132186ca __libc_start_call_main (libc.so.6 + 0x276ca) #6 0x7fce13218785 __libc_start_main_impl (libc.so.6 + 0x27785) #7 0x55ade3cace21 _start (collectd + 0x7e21) Stack trace of thread 616849: #0 0x7fce13276156 __futex_abstimed_wait_common64 (libc.so.6 + 0x85156) #1 0x7fce13278818 __pthread_cond_wait_common (libc.so.6 + 0x87818) #2 0x55ade3cb5d1b n/a (collectd + 0x10d1b) #3 0x7fce132793ec start_thread (libc.so.6 + 0x883ec) #4 0x7fce132f9a1c __clone3 (libc.so.6 + 0x108a1c) Stack trace of thread 616850: #0 0x7fce13276156 __futex_abstimed_wait_common64 (libc.so.6 + 0x85156) #1 0x7fce13278818 __pthread_cond_wait_common (libc.so.6 + 0x87818) #2 0x55ade3cb5d1b n/a (collectd + 0x10d1b) #3 0x7fce132793ec start_thread (libc.so.6 + 0x883ec) #4 0x7fce132f9a1c __clone3 (libc.so.6 + 0x108a1c) Stack trace of thread 616851: #0 0x7fce13276156 __futex_abstimed_wait_common64 (libc.so.6 + 0x85156) #1 0x7fce13278818 __pthread_cond_wait_common (libc.so.6 + 0x87818) #2 0x55ade3cb5d1b n/a (collectd + 0x10d1b) #3 0x7fce132793ec start_thread (libc.so.6 + 0x883ec) #4 0x7fce132f9a1c __clone3 (libc.so.6 + 0x108a1c) Stack trace of thread 616852: #0 0x7fce13276156 __futex_abstimed_wait_common64 (libc.so.6 + 0x85156) #1 0x7fce13278818 __pthread_cond_wait_common (libc.so.6 + 0x87818) #2
Bug#1040716: libc6: Stack Traces
On Mon, 2023-07-10 at 23:17 +0200, Aurelien Jarno wrote: > Hi, > > On 2023-07-09 17:15, Tim McConnell wrote: > > Hi Aurelien, > > The Stack Traces started showing up in my System Event logs > > yesterday > > and has totaled up to 18 times so far. As far as issues, the system > > is > > slower than usual and occasionally freezes. > > I ran apt-file search lib.so and came up with libc6 so that's > > where I > > filed the bug. I tried running gdb on it (libc6) and it told me > > file > > not found, I'd be happy to give more information just tell me how > > to > > get it. > > Thanks for the details, however I am afraid it is not enough. libc6 > is a > library, for understanding the issue, I would need to know by which > program it is used in the corresponding stack traces. The issue might > not be in the library but in another library or another program. Do > you > have some more log entries around those stack traces? > > All that said, it appears that the locales package in versions 2.37-2 > to > 2.37-4 is buggy and might cause random crashes in libc6. You might > want > to upgrade to version 2.37-5 to check if it solves your issue. The > package is currently only available in sid, but it is expected to > migrate to testing in the next days. > > Regards > Aurelien > Checking Journalctl I found this: Jul 10 19:02:11 DebianTim systemd-coredump[3869]: [] Process 992 (collectd) of user 0 dumped core. Module libudev.so.1 from deb systemd-252.11-1.amd64 Stack trace of thread 1424: #0 0x7f7cbbfce9f2 __memmove_ssse3 (libc.so.6 + 0x16d9f2) #1 0x7f7cbbd856d9 rrd_write (librrd.so.8 + 0x346d9) #2 0x7f7cbbd90acd n/a (librrd.so.8 + 0x3facd) #3 0x7f7cbbd92962 n/a (librrd.so.8 + 0x41962) #4 0x7f7cbbdec370 n/a (rrdtool.so + 0x3370) #5 0x7f7cbbee93ec start_thread (libc.so.6 + 0x883ec) #6 0x7f7cbbf69a1c __clone3 (libc.so.6 + 0x108a1c) Stack trace of thread 1431: #0 0x7f7cbbee6156 __futex_abstimed_wait_common64 (libc.so.6 + 0x85156) #1 0x7f7cbbee8afc __pthread_cond_wait_common (libc.so.6 + 0x87afc) #2 0x5586e89f93a2 n/a (collectd + 0x123a2) #3 0x7f7cbbee93ec start_thread (libc.so.6 + 0x883ec) #4 0x7f7cbbf69a1c __clone3 (libc.so.6 + 0x108a1c) Stack trace of thread 1432: #0 0x7f7cbbee6156 __futex_abstimed_wait_common64 (libc.so.6 + 0x85156) #1 0x7f7cbbee8afc __pthread_cond_wait_common (libc.so.6 + 0x87afc) #2 0x5586e89f93a2 n/a (collectd + 0x123a2) #3 0x7f7cbbee93ec start_thread (libc.so.6 + 0x883ec) #4 0x7f7cbbf69a1c __clone3 (libc.so.6 + 0x108a1c) Stack trace of thread 1427: #0 0x7f7cbbee6156 __futex_abstimed_wait_common64 (libc.so.6 + 0x85156) #1 0x7f7cbbee8818 __pthread_cond_wait_common (libc.so.6 + 0x87818) #2 0x5586e89f7d1b n/a (collectd + 0x10d1b) #3 0x7f7cbbee93ec start_thread (libc.so.6 + 0x883ec) #4 0x7f7cbbf69a1c __clone3 (libc.so.6 + 0x108a1c) Stack trace of thread 1429: #0 0x7f7cbbee6156 __futex_abstimed_wait_common64 (libc.so.6 + 0x85156) #1 0x7f7cbbee8818 __pthread_cond_wait_common (libc.so.6 + 0x87818) #2 0x5586e89f7d1b n/a (collectd + 0x10d1b) Stack trace of
Bug#1040716: libc6: Stack Traces
On Mon, 2023-07-10 at 23:17 +0200, Aurelien Jarno wrote: > Hi, > > On 2023-07-09 17:15, Tim McConnell wrote: > > Hi Aurelien, > > The Stack Traces started showing up in my System Event logs > > yesterday > > and has totaled up to 18 times so far. As far as issues, the system > > is > > slower than usual and occasionally freezes. > > I ran apt-file search lib.so and came up with libc6 so that's > > where I > > filed the bug. I tried running gdb on it (libc6) and it told me > > file > > not found, I'd be happy to give more information just tell me how > > to > > get it. > > Thanks for the details, however I am afraid it is not enough. libc6 > is a > library, for understanding the issue, I would need to know by which > program it is used in the corresponding stack traces. The issue might > not be in the library but in another library or another program. Do > you > have some more log entries around those stack traces? > > All that said, it appears that the locales package in versions 2.37-2 > to > 2.37-4 is buggy and might cause random crashes in libc6. You might > want > to upgrade to version 2.37-5 to check if it solves your issue. The > package is currently only available in sid, but it is expected to > migrate to testing in the next days. > > Regards > Aurelien > I'll check journalctl and see if that has better info. I'll also upgrade the package you were talking about to see if it fixes the issue . I will let you know.
Bug#1040716: libc6: Stack Traces
Hi, On 2023-07-09 17:15, Tim McConnell wrote: > Hi Aurelien, > The Stack Traces started showing up in my System Event logs yesterday > and has totaled up to 18 times so far. As far as issues, the system is > slower than usual and occasionally freezes. > I ran apt-file search lib.so and came up with libc6 so that's where I > filed the bug. I tried running gdb on it (libc6) and it told me file > not found, I'd be happy to give more information just tell me how to > get it. Thanks for the details, however I am afraid it is not enough. libc6 is a library, for understanding the issue, I would need to know by which program it is used in the corresponding stack traces. The issue might not be in the library but in another library or another program. Do you have some more log entries around those stack traces? All that said, it appears that the locales package in versions 2.37-2 to 2.37-4 is buggy and might cause random crashes in libc6. You might want to upgrade to version 2.37-5 to check if it solves your issue. The package is currently only available in sid, but it is expected to migrate to testing in the next days. Regards Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://aurel32.net
Bug#1040716: libc6: Stack Traces
Hi Aurelien, The Stack Traces started showing up in my System Event logs yesterday and has totaled up to 18 times so far. As far as issues, the system is slower than usual and occasionally freezes. I ran apt-file search lib.so and came up with libc6 so that's where I filed the bug. I tried running gdb on it (libc6) and it told me file not found, I'd be happy to give more information just tell me how to get it. -- Tim McConnell On Sun, 2023-07-09 at 23:34 +0200, Aurelien Jarno wrote: > Hi, > > On 2023-07-09 15:05, Tim McConnell wrote: > > Package: libc6 > > Version: 2.37-3 > > Severity: normal > > > > Thanks for your bug report. Could you please explain me what are > those > stack traces, and what is the issue you encountered? Thanks. > > Regards > Aurelien >
Bug#1040716: libc6: Stack Traces
Hi, On 2023-07-09 15:05, Tim McConnell wrote: > Package: libc6 > Version: 2.37-3 > Severity: normal > Thanks for your bug report. Could you please explain me what are those stack traces, and what is the issue you encountered? Thanks. Regards Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://aurel32.net
