Re: The master janitor goes crazy / Re: Debugging Deadlocks
Okay, just got distracted by this happening again. The way I'm reproducing it
is by ^C'ing a Cassandane run (which cleanly shuts down the cyrus instances it
had started) during the JMAP tests, and I guess I'm sometimes getting lucky on
the timing and hitting this.
On Wed, Dec 4, 2019, at 2:22 PM, ellie timoney wrote:
> I think the really useful information to collect next time this happens
> (and while the master process is still running) is:
>
> * What does lsof tell us about that ready file descriptor (in the
> example, fd 11)? I would be very interested to know if it's a client
> socket, or an internal messaging socket (that service processes use to
> tell master their status).
master 3121 cyrus 11u IPv4 49559608 0t0 TCP
localhost:9146 (LISTEN)
This is the port the http service was listening on.
> * If you can attach a debugger and step through a couple of iterations
> of master's big "for (;;) {" loop, what path is it taking? What
> decisions is it making?
So the problem pops up here:
/* connections */
if (y >= 0 && Services[i].ready_workers == 0 &&
Services[i].nactive < Services[i].max_workers &&
!service_is_fork_limited([i])) {
if (verbose > 2)
syslog(LOG_DEBUG, "listening for connections for %s/%s",
Services[i].name, Services[i].familyname);
FD_SET(y, );
if (y > maxfd) maxfd = y;
}
(gdb) p Services[i]
$28 = {name = 0x55df363956c0 "http", listen = 0x55df36395170 "127.0.0.1:9146",
proto = 0x55df36395680 "tcp", exec = 0x55df363956e0, babysit = 0,
associate = 0, family = 2, familyname = 0x55df35a8c780 "ipv4", socket = 11,
stat = {12, 13}, desired_workers = 0, max_workers = 2147483647, maxfds = 0,
maxforkrate = 0, ready_workers = 0, nforks = 1, nactive = 1,
nconnections = 1, forkrate = 0, nreadyfails = 0, lastreadyfail = 0,
last_interval_start = {tv_sec = 1578455582, tv_usec = 342312},
interval_forks = 1}
The http service has shut down already... sort of. It's hanging around as a
defunct process:
cyrus 3143 3121 0 14:53 ?00:00:00 [httpd]
We have no ready_workers, and nactive is less than max_workers, so we're adding
the service socket (fd 11) to the listen set. If verbose were large enough, I
think we would be logging "listening for connections for %s/%s" during this
loop.
So, we have data on the socket for this service, and if we weren't already
in_shutdown we'd be trying to spawn a new service process to handle it. But we
ARE in_shutdown, so we don't spawn a service, so the data on the socket remains
unhandled.
We don't simply finish shutting down, because that nactive=1 in the service
table entry means we think we still have children, so we call child_janitor and
then go round the loop again.
So we're waiting for child_janitor to clean up after the http service, I
guess.
Inside child_janitor, here's the entry:
(gdb) print **p
$36 = {pid = 3143, service_state = SERVICE_STATE_BUSY, janitor_deadline = 0,
si = 1,
desc = 0x55df36397f30 "type:SERVICE name:http
path:/dev/shm/cyrus/main/libexec/httpd", spawntime = {tv_sec = 1578455582,
tv_usec = 682977}, sighuptime = -1,
next = 0x0}
So, it thinks the http process is still busy, so we're spinning and waiting for
it... but the defunct state tells us it's already exited, and the system is
just waiting for the parent process (master) to waitpid() it before getting rid
of it completely.
It's pretty clear that there's still client data on that socket, but I'm not
sure if "shut down Cyrus while there's still a client sending data to httpd" is
enough to trigger it, or if it's more subtle. The data pending on the socket
is the reason we see pselect() called over and over again, but I don't think
it's the cause of the problem: without that pending data, we'd still be hanging
here instead of shutting down cleanly, but we'd be blocked in pselect() instead.
We _should_ have gotten a sigchld from httpd shutting down, and then reap_child
should have waitpid'd it and cleaned up the child table, so we shouldn't be
spinning like this. So I guess the failure occurred somewhere in the sigchld
handling, but by the time I notice it spinning it's too late to see what
happened. Did the sigchld go missing? Did reap_child do the wrong thing? Who
knows!
I don't have any further insight at the moment. I guess if I can find a way to
reproduce it reasonably reliably, then I can litter reap_child with debug
output and see what it reports just before it goes pear-shaped
Cheers,
ellie
Re: The master janitor goes crazy / Re: Debugging Deadlocks
So, using my strace output from the other week as an example:
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
The arguments here are:
* nfds: 13 (12+1)
* readfds: [8 9 11 12]
* writefds: NULL
* exceptfds: NULL
* timeout: NULL
* sigmask: {[], 8}
The interesting bits are:
* we don't have a timeout (so this pselect would block forever if nothing
became ready)
* we're only waiting for fds to become readable (not writeable or having
exceptions)
* we don't have a sigmask set (empty array of 8-byte objects)
The return value of 1 means that 1 of the fds was ready, and I surmise that
"(in [11])" is telling us that it was fd 11 from the readfds set that was ready
(for reading).
The fact these pselect calls are all the same tells me that either: a lot is
happening on fd 11 and we're not keeping up, or that there's data waiting on fd
11 and we keep ignoring it (so it keeps telling us it's there).
The gdb backtrace isn't really useful here I don't think, I think it's
coincidental that when we each attached a debugger we both happened to be at
that particular line in child_janitor. Once we're in shutdown, child_janitor
is the only thing doing much work, and that line is the top of its loop.
I think the really useful information to collect next time this happens (and
while the master process is still running) is:
* What does lsof tell us about that ready file descriptor (in the example, fd
11)? I would be very interested to know if it's a client socket, or an
internal messaging socket (that service processes use to tell master their
status).
* If you can attach a debugger and step through a couple of iterations of
master's big "for (;;) {" loop, what path is it taking? What decisions is it
making?
* Without the debugger, if you let it run like this for 30 seconds or more,
does a syslog line like this appear?
https://github.com/cyrusimap/cyrus-imapd/blob/96d194de82d3dbe124a359069bd21f5cba7519ba/master/master.c#L1240-L1244
Cheers,
ellie
On Mon, Dec 2, 2019, at 11:18 PM, Дилян Палаузов wrote:
> Hello Ellie,
>
> this is exactly what I see (countless pselect calls), but I have as
> second parameter of pselect a much larger array. I
> just observed that on killing master, it terminates all cyrus processes
> but two (httpd and notifyd). Then I try to
> connect to that processes (gdb). This does not work, however, since
> the processes are moved to zombie status.
>
> Greetings
> Дилян
>
> On Thu, 2019-11-28 at 10:34 +1100, ellie timoney wrote:
> > Saw something similar just now when I killed a cassandane run off
> > prematurely. One cyrus master process wound up spinning like this:
> >
> > pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> > pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> > pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> > pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> > pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> > pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> > pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> > pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> > pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> >
> > 0x555ac7124a97 in child_janitor (now=...) at master/master.c:1221
> > 1221janitor_position = janitor_position % child_table_size;
> > (gdb) bt
> > #0 0x555ac7124a97 in child_janitor (now=...) at master/master.c:1221
> > #1 0x555ac712a67a in main (argc=10, argv=0x7ffdc1fe78b8)
> > at master/master.c:2812
> >
> > Haven't dug further yet, but it looks similar to your report
> >
> > On Wed, Nov 27, 2019, at 9:17 AM, ellie timoney wrote:
> > > Can you strace the master process next time it's spinning at 100%?
> > > What is it doing at that time?
> > >
> > > On Tue, Nov 26, 2019, at 1:29 AM, Дилян Палаузов wrote:
> > > > Hello,
> > > >
> > > > > I run cyrus imap 3.0.x with some private changes.
> > > > >
> > > > > Sometimes when stop the master process, the master process utilizes
> > > > > one CPU core to 100% for 5 minutes. After the fifth
> > > > > minute, systemd enforces kill -9. When I attach to the maste process,
> > > > > I see that it some janitor does some work, but I
> > > > > have not checked the details. Has anybody experienced this?
> > > >
> > > > I run cyrus imap. At some moment I recompile and reinstall the
> > > > binaries, which in theory means that the binaries
> > > > detect this change and restart theirselves. At some moment I call
> > > > "systemctl stop cyrus-imap" which I guess sends
> > > > SIGTERM to the master process. Then the CPU utilization of the master
> > > > process goes to 100%. In the systemd service
> > > > file I have TimeoutStopSec=320 . After this time, the master process
> > > > continues running and systemd sends 9/SIGKILL. It
> > > > is not
Re: The master janitor goes crazy / Re: Debugging Deadlocks
Hello Ellie,
this is exactly what I see (countless pselect calls), but I have as second
parameter of pselect a much larger array. I
just observed that on killing master, it terminates all cyrus processes but two
(httpd and notifyd). Then I try to
connect to that processes (gdb). This does not work, however, since the
processes are moved to zombie status.
Greetings
Дилян
On Thu, 2019-11-28 at 10:34 +1100, ellie timoney wrote:
> Saw something similar just now when I killed a cassandane run off
> prematurely. One cyrus master process wound up spinning like this:
>
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
> pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
>
> 0x555ac7124a97 in child_janitor (now=...) at master/master.c:1221
> 1221 janitor_position = janitor_position % child_table_size;
> (gdb) bt
> #0 0x555ac7124a97 in child_janitor (now=...) at master/master.c:1221
> #1 0x555ac712a67a in main (argc=10, argv=0x7ffdc1fe78b8)
> at master/master.c:2812
>
> Haven't dug further yet, but it looks similar to your report
>
> On Wed, Nov 27, 2019, at 9:17 AM, ellie timoney wrote:
> > Can you strace the master process next time it's spinning at 100%?
> > What is it doing at that time?
> >
> > On Tue, Nov 26, 2019, at 1:29 AM, Дилян Палаузов wrote:
> > > Hello,
> > >
> > > > I run cyrus imap 3.0.x with some private changes.
> > > >
> > > > Sometimes when stop the master process, the master process utilizes one
> > > > CPU core to 100% for 5 minutes. After the fifth
> > > > minute, systemd enforces kill -9. When I attach to the maste process, I
> > > > see that it some janitor does some work, but I
> > > > have not checked the details. Has anybody experienced this?
> > >
> > > I run cyrus imap. At some moment I recompile and reinstall the
> > > binaries, which in theory means that the binaries
> > > detect this change and restart theirselves. At some moment I call
> > > "systemctl stop cyrus-imap" which I guess sends
> > > SIGTERM to the master process. Then the CPU utilization of the master
> > > process goes to 100%. In the systemd service
> > > file I have TimeoutStopSec=320 . After this time, the master process
> > > continues running and systemd sends 9/SIGKILL. It
> > > is not necessary that on re-installing the binaries, and then shutting
> > > down the CPU goes to 100%: it is possible that
> > > the CPU goes to 100%, without reinstalling (and thus triggering
> > > self-restarting) of the imapd/httpd binaries.
> > >
> > > It is often, but not always, that this 100% CPU loop is entered on
> > > shutdown.
> > >
> > > I have a webmail client and to speedup things it uses SquirrelMail's
> > > IMAP Proxy (http://www.imapproxy.org/ a Caching
> > > IMAP proxy). It is recommended in the installation manual of
> > > Horde/IMP. The IMAP caching proxy connects to
> > > 127.0.0.2:143 (and is therefore permitted to skip the TLS overload).
> > > In master conf I have a line
> > > “imaplocal cmd="imapd -C /usr/local/etc/cyrus/imapdlocal.conf"
> > > listen="127.0.0.2:imap" prefork=0”.
> > >
> > > When the CPU goes to 100% on shutdown I connect with gdb to the master
> > > process. Below is the full backtrace. Does
> > > somebody have an explanation why the master process enters a never
> > > ending loop?
> > >
> > > I do not say that all above information has to be involved in the
> > > anwer. Has somebody else experienced this effects?
> > > Any suggestions how to investigate this deeper?
> > >
> > > Greetings
> > > Дилян
> > >
> > > ---
> > > warning: Could not load vsyscall page because no executable was
> > > specified
> > > Reading symbols from /usr/local/libexec/master...
> > > Attaching to program: /usr/local/libexec/master, process 9247
> > > Reading symbols from /usr/local/lib/libcyrus_min.so.0...
> > > Reading symbols from /lib/libuuid.so.1...
> > > Reading symbols from /usr/local/lib/libgssapi_krb5.so.2...
> > > Reading symbols from /usr/local/lib/libkrb5.so.3...
> > > Reading symbols from /usr/local/lib/libk5crypto.so.3...
> > > Reading symbols from /usr/local/lib/libcom_err.so.3...
> > > Reading symbols from /usr/local/lib/libkrb5support.so.0...
> > > Reading symbols from /usr/local/lib/libpcreposix.so.0...
> > > (No debugging symbols found in /usr/local/lib/libpcreposix.so.0)
> > > Reading symbols from /usr/local/lib/libpcre.so.1...
> > > (No debugging symbols found in /usr/local/lib/libpcre.so.1)
> > >
Re: The master janitor goes crazy / Re: Debugging Deadlocks
Saw something similar just now when I killed a cassandane run off prematurely.
One cyrus master process wound up spinning like this:
pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
pselect6(13, [8 9 11 12], NULL, NULL, NULL, {[], 8}) = 1 (in [11])
0x555ac7124a97 in child_janitor (now=...) at master/master.c:1221
1221janitor_position = janitor_position % child_table_size;
(gdb) bt
#0 0x555ac7124a97 in child_janitor (now=...) at master/master.c:1221
#1 0x555ac712a67a in main (argc=10, argv=0x7ffdc1fe78b8)
at master/master.c:2812
Haven't dug further yet, but it looks similar to your report
On Wed, Nov 27, 2019, at 9:17 AM, ellie timoney wrote:
> Can you strace the master process next time it's spinning at 100%?
> What is it doing at that time?
>
> On Tue, Nov 26, 2019, at 1:29 AM, Дилян Палаузов wrote:
> > Hello,
> >
> > > I run cyrus imap 3.0.x with some private changes.
> > >
> > > Sometimes when stop the master process, the master process utilizes one
> > > CPU core to 100% for 5 minutes. After the fifth
> > > minute, systemd enforces kill -9. When I attach to the maste process, I
> > > see that it some janitor does some work, but I
> > > have not checked the details. Has anybody experienced this?
> >
> > I run cyrus imap. At some moment I recompile and reinstall the
> > binaries, which in theory means that the binaries
> > detect this change and restart theirselves. At some moment I call
> > "systemctl stop cyrus-imap" which I guess sends
> > SIGTERM to the master process. Then the CPU utilization of the master
> > process goes to 100%. In the systemd service
> > file I have TimeoutStopSec=320 . After this time, the master process
> > continues running and systemd sends 9/SIGKILL. It
> > is not necessary that on re-installing the binaries, and then shutting
> > down the CPU goes to 100%: it is possible that
> > the CPU goes to 100%, without reinstalling (and thus triggering
> > self-restarting) of the imapd/httpd binaries.
> >
> > It is often, but not always, that this 100% CPU loop is entered on shutdown.
> >
> > I have a webmail client and to speedup things it uses SquirrelMail's
> > IMAP Proxy (http://www.imapproxy.org/ a Caching
> > IMAP proxy). It is recommended in the installation manual of
> > Horde/IMP. The IMAP caching proxy connects to
> > 127.0.0.2:143 (and is therefore permitted to skip the TLS overload).
> > In master conf I have a line
> > “imaplocal cmd="imapd -C /usr/local/etc/cyrus/imapdlocal.conf"
> > listen="127.0.0.2:imap" prefork=0”.
> >
> > When the CPU goes to 100% on shutdown I connect with gdb to the master
> > process. Below is the full backtrace. Does
> > somebody have an explanation why the master process enters a never
> > ending loop?
> >
> > I do not say that all above information has to be involved in the
> > anwer. Has somebody else experienced this effects?
> > Any suggestions how to investigate this deeper?
> >
> > Greetings
> > Дилян
> >
> > ---
> > warning: Could not load vsyscall page because no executable was
> > specified
> > Reading symbols from /usr/local/libexec/master...
> > Attaching to program: /usr/local/libexec/master, process 9247
> > Reading symbols from /usr/local/lib/libcyrus_min.so.0...
> > Reading symbols from /lib/libuuid.so.1...
> > Reading symbols from /usr/local/lib/libgssapi_krb5.so.2...
> > Reading symbols from /usr/local/lib/libkrb5.so.3...
> > Reading symbols from /usr/local/lib/libk5crypto.so.3...
> > Reading symbols from /usr/local/lib/libcom_err.so.3...
> > Reading symbols from /usr/local/lib/libkrb5support.so.0...
> > Reading symbols from /usr/local/lib/libpcreposix.so.0...
> > (No debugging symbols found in /usr/local/lib/libpcreposix.so.0)
> > Reading symbols from /usr/local/lib/libpcre.so.1...
> > (No debugging symbols found in /usr/local/lib/libpcre.so.1)
> > Reading symbols from /usr/local/lib/libxml2.so.2...
> > Reading symbols from /usr/local/lib/liblzma.so.5...
> > (No debugging symbols found in /usr/local/lib/liblzma.so.5)
> > Reading symbols from /usr/local/lib/libical.so.3...
> > Reading symbols from /usr/local/lib/libicalss.so.3...
> > Reading symbols from /usr/local/lib/libicalvcal.so.3...
> > Reading symbols from /usr/local/lib/libicui18n.so.63...
> > Reading symbols from /usr/local/lib/libicuuc.so.63...
> > Reading symbols from /usr/local/lib/libicudata.so.63...
> > (No debugging symbols found in /usr/local/lib/libicudata.so.63)
> > Reading symbols from
Re: The master janitor goes crazy / Re: Debugging Deadlocks
Can you strace the master process next time it's spinning at 100%? What is it doing at that time? On Tue, Nov 26, 2019, at 1:29 AM, Дилян Палаузов wrote: > Hello, > > > I run cyrus imap 3.0.x with some private changes. > > > > Sometimes when stop the master process, the master process utilizes one CPU > > core to 100% for 5 minutes. After the fifth > > minute, systemd enforces kill -9. When I attach to the maste process, I see > > that it some janitor does some work, but I > > have not checked the details. Has anybody experienced this? > > I run cyrus imap. At some moment I recompile and reinstall the > binaries, which in theory means that the binaries > detect this change and restart theirselves. At some moment I call > "systemctl stop cyrus-imap" which I guess sends > SIGTERM to the master process. Then the CPU utilization of the master > process goes to 100%. In the systemd service > file I have TimeoutStopSec=320 . After this time, the master process > continues running and systemd sends 9/SIGKILL. It > is not necessary that on re-installing the binaries, and then shutting > down the CPU goes to 100%: it is possible that > the CPU goes to 100%, without reinstalling (and thus triggering > self-restarting) of the imapd/httpd binaries. > > It is often, but not always, that this 100% CPU loop is entered on shutdown. > > I have a webmail client and to speedup things it uses SquirrelMail's > IMAP Proxy (http://www.imapproxy.org/ a Caching > IMAP proxy). It is recommended in the installation manual of > Horde/IMP. The IMAP caching proxy connects to > 127.0.0.2:143 (and is therefore permitted to skip the TLS overload). > In master conf I have a line > “imaplocal cmd="imapd -C /usr/local/etc/cyrus/imapdlocal.conf" > listen="127.0.0.2:imap" prefork=0”. > > When the CPU goes to 100% on shutdown I connect with gdb to the master > process. Below is the full backtrace. Does > somebody have an explanation why the master process enters a never > ending loop? > > I do not say that all above information has to be involved in the > anwer. Has somebody else experienced this effects? > Any suggestions how to investigate this deeper? > > Greetings > Дилян > > --- > warning: Could not load vsyscall page because no executable was > specified > Reading symbols from /usr/local/libexec/master... > Attaching to program: /usr/local/libexec/master, process 9247 > Reading symbols from /usr/local/lib/libcyrus_min.so.0... > Reading symbols from /lib/libuuid.so.1... > Reading symbols from /usr/local/lib/libgssapi_krb5.so.2... > Reading symbols from /usr/local/lib/libkrb5.so.3... > Reading symbols from /usr/local/lib/libk5crypto.so.3... > Reading symbols from /usr/local/lib/libcom_err.so.3... > Reading symbols from /usr/local/lib/libkrb5support.so.0... > Reading symbols from /usr/local/lib/libpcreposix.so.0... > (No debugging symbols found in /usr/local/lib/libpcreposix.so.0) > Reading symbols from /usr/local/lib/libpcre.so.1... > (No debugging symbols found in /usr/local/lib/libpcre.so.1) > Reading symbols from /usr/local/lib/libxml2.so.2... > Reading symbols from /usr/local/lib/liblzma.so.5... > (No debugging symbols found in /usr/local/lib/liblzma.so.5) > Reading symbols from /usr/local/lib/libical.so.3... > Reading symbols from /usr/local/lib/libicalss.so.3... > Reading symbols from /usr/local/lib/libicalvcal.so.3... > Reading symbols from /usr/local/lib/libicui18n.so.63... > Reading symbols from /usr/local/lib/libicuuc.so.63... > Reading symbols from /usr/local/lib/libicudata.so.63... > (No debugging symbols found in /usr/local/lib/libicudata.so.63) > Reading symbols from /usr/local/lib/libsqlite3.so.0... > (No debugging symbols found in /usr/local/lib/libsqlite3.so.0) > Reading symbols from /usr/local/lib/libz.so.1... > (No debugging symbols found in /usr/local/lib/libz.so.1) > Reading symbols from /lib64/libm.so.6... > Reading symbols from /lib64/libdl.so.2... > Reading symbols from /lib64/libpthread.so.0... > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > Reading symbols from /lib64/libc.so.6... > Reading symbols from /lib64/ld-linux-x86-64.so.2... > Reading symbols from /lib64/libresolv.so.2... > Reading symbols from /usr/local/lib/libdb-18.1.so... > Reading symbols from /usr/local/lib64/libstdc++.so.6... > Reading symbols from /usr/local/lib64/libgcc_s.so.1... > Reading symbols from /usr/local/lib64/libssl.so.1.1... > Reading symbols from /usr/local/lib64/libcrypto.so.1.1... > Reading symbols from /lib64/libnss_db.so.2... > Reading symbols from /lib64/libnss_files.so.2... > Reading symbols from /lib64/libnss_dns.so.2... > 0x00405406 in child_janitor (now=...) at master/master.c:1192 > 1192 janitor_position = janitor_position % child_table_size; > ™(gdb) bt f > Id Target Id Frame > * 1Thread 0x7f6a08759780 (LWP 9247) "master" 0x00405406 in >
