Re: Runaway process

[Sander Klein]

On 2019-07-12 04:27, Willy Tarreau wrote:


If you can at least show the backtrace, this could be useful and we
can see if the core would be needed or not. Maybe this will match
another known bug.


This is the BT of yesterday:

---
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show 
copying"

and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 27066
[New LWP 27067]
[New LWP 27068]
[New LWP 27069]
[Thread debugging using libthread_db enabled]
Using host libthread_db library 
"/lib/x86_64-linux-gnu/libthread_db.so.1".
0x7f08655ef303 in epoll_wait () at 
../sysdeps/unix/syscall-template.S:84

84  ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) thread apply all bt

Thread 4 (Thread 0x7f084d058700 (LWP 27069)):
#0  0x7f08655ef303 in epoll_wait () at 
../sysdeps/unix/syscall-template.S:84

#1  0x562cea640f95 in ?? ()
#2  0x562cea6e6792 in ?? ()
#3  0x7f08c4a4 in start_thread (arg=0x7f084d058700) at 
pthread_create.c:456
#4  0x7f08655eed0f in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:97


Thread 3 (Thread 0x7f084d859700 (LWP 27068)):
#0  0x562cea6af336 in ?? ()
#1  0x562cea73cd1d in si_cs_send ()
#2  0x562cea73d90a in si_update_both ()
#3  0x562cea6a1976 in process_stream ()
#4  0x562cea770728 in process_runnable_tasks ()
#5  0x562cea6e67c1 in ?? ()
#6  0x7f08c4a4 in start_thread (arg=0x7f084d859700) at 
pthread_create.c:456
#7  0x7f08655eed0f in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:97


Thread 2 (Thread 0x7f084e05a700 (LWP 27067)):
#0  0x7f08655ef303 in epoll_wait () at 
../sysdeps/unix/syscall-template.S:84

#1  0x562cea640f95 in ?? ()
#2  0x562cea6e6792 in ?? ()
#3  0x7f08c4a4 in start_thread (arg=0x7f084e05a700) at 
pthread_create.c:456
#4  0x7f08655eed0f in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:97


Thread 1 (Thread 0x7f0866e5c180 (LWP 27066)):
#0  0x7f08655ef303 in epoll_wait () at 
../sysdeps/unix/syscall-template.S:84

#1  0x562cea640f95 in ?? ()
#2  0x562cea6e6792 in ?? ()
#3  0x562cea63e96c in main ()
---

And today I had another one:

---
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show 
copying"

and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 6982
[New LWP 6983]
[New LWP 6984]
[New LWP 6985]
[Thread debugging using libthread_db enabled]
Using host libthread_db library 
"/lib/x86_64-linux-gnu/libthread_db.so.1".
0x7fbdf0713303 in epoll_wait () at 
../sysdeps/unix/syscall-template.S:84

84  ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) thread apply all bt

Thread 4 (Thread 0x7fbdd817c700 (LWP 6985)):
#0  0x5606dd570457 in ?? ()
#1  0x5606dd5fdd1d in si_cs_send ()
#2  0x5606dd5ff45d in si_cs_io_cb ()
#3  0x5606dd6319a6 in process_runnable_tasks ()
#4  0x5606dd5a77c1 in ?? ()
#5  0x7fbdf17904a4 in start_thread (arg=0x7fbdd817c700) at 
pthread_create.c:456
#6  0x7fbdf0712d0f in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:97


Thread 3 (Thread 0x7fbdd897d700 (LWP 6984)):
#0  0x7fbdf0713303 in epoll_wait () at 
../sysdeps/unix/syscall-template.S:84

#1  0x5606dd501f95 in ?? ()
#2  0x5606dd5a7792 in ?? ()
#3  0x7fbdf17904a4 in start_thread (arg=0x7fbdd897d700) at 
pthread_create.c:456
#4  0x7fbdf0712d0f in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:97


Thread 2 (Thread 0x7fbdd917e700 (LWP 6983)):
#0  0x7fbdf0713303 in epoll_wait () at 
../sysdeps/unix/syscall-template.S:84

#1  0x5606dd501f95 in ?? ()
#2  0x5606dd5a7792 in ?? ()
#3  0x7fbdf17904a4 in start_thread (arg=0x7fbdd917e700) at 
pthread_create.c:456
#4  0x7fbdf0712d0f in clone () at 
..

Re: Runaway process

[Willy Tarreau]

Hi Sander,

On Thu, Jul 11, 2019 at 01:28:50PM +0200, Sander Klein wrote:
> On 2019-07-11 12:27, Tim Düsterhus wrote:
> > Try attaching to the process with `gdb -p 12345` with 12345 being the
> > process ID. Then:
> > 
> > 1. Get a backtrace for all threads: thread apply all bt
> > 2. Generate a core file: generate-core-file
> > 
> > If you are also able to connect to the stats socket of that process then
> > the following might be helpful as well:
> > 
> > 1. show info
> > 2. show fd
> > 3. show activity
> > 4. show sess all
> 
> I've created the backtrace and the core file. I couldn't connect to the
> stats socket anymore so no info on that.
> 
> If a dev is interested I can send it.

If you can at least show the backtrace, this could be useful and we
can see if the core would be needed or not. Maybe this will match
another known bug.

Thanks!
Willy

Re: Runaway process

[Sander Klein]

On 2019-07-11 12:27, Tim Düsterhus wrote:

Try attaching to the process with `gdb -p 12345` with 12345 being the
process ID. Then:

1. Get a backtrace for all threads: thread apply all bt
2. Generate a core file: generate-core-file

If you are also able to connect to the stats socket of that process 
then

the following might be helpful as well:

1. show info
2. show fd
3. show activity
4. show sess all


I've created the backtrace and the core file. I couldn't connect to the 
stats socket anymore so no info on that.


If a dev is interested I can send it.

Regards,

Sander

0x2E78FBE8.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: Runaway process

[Tim Düsterhus]

Sander,

Am 11.07.19 um 08:48 schrieb Sander Klein:
> I seem to have runaway HAProxy process since yesterday evening around
> 20:50. This process is eating up 100% CPU continously. (HAProxy 1.9.8)
> 
> Of course I can just kill it and go on with my life, but I was wondering
> if there was any interest to see if we can uncover a bug here. If so,
> please let me know what you need from me.
> 

Try attaching to the process with `gdb -p 12345` with 12345 being the
process ID. Then:

1. Get a backtrace for all threads: thread apply all bt
2. Generate a core file: generate-core-file

If you are also able to connect to the stats socket of that process then
the following might be helpful as well:

1. show info
2. show fd
3. show activity
4. show sess all

Afterwards you should be able to kill the process, because you extracted
all relevant information. Keep the core file somewhere safe, do not send
it to the list, it contains very private information such as TLS private
keys. It might be helpful in case a developer needs additional
information, though.

Best regards
Tim Düsterhus