Hi Ian,
Thanks for the reply, I get the point on zombie, I do not think the issue
here is parent not reaping child, seems like go process has not finished
execution of some
internal threads (waiting on some futex) and causing SIGCHILD not to be
sent to parent.
go process named <replicator> hit with panic and I see this went into
zombie state
$ ps -ef | grep replicator
root 87548 87507 0 Aug23 ? 00:00:00 [replicator] <defunct>
Now looking at the tasks within the process
I see the stack trace of the threads within the process still stuck on
following
bash-4.2# cat /proc/87548/task/87561/stack
[<ffffffffbb114714>] futex_wait_queue_me+0xc4/0x120
[<ffffffffbb11520a>] futex_wait+0x10a/0x250
[<ffffffffbb1182ce>] do_futex+0x35e/0x5b0
[<ffffffffbb11865b>] SyS_futex+0x13b/0x180
[<ffffffffbb003c09>] do_syscall_64+0x79/0x1b0
[<ffffffffbba00081>] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[<ffffffffffffffff>] 0xffffffffffffffff
>From the above example if we are creating some internal threads and main
thread is excited due to panic and left some detached threads, process will
be in zombie state until the threads
within the process completes.
It appears there is some run away threads hung state scenario causing this.
I am not able to reproduce it with main go routine explict panic and some
go routine still executing.
Does the above stack trace sound familiar wrt internal threads of Go
runtime ?
Thanks,
Uday
On Thursday, August 27, 2020 at 1:43:39 PM UTC-7 Ian Lance Taylor wrote:
> On Thu, Aug 27, 2020 at 10:01 AM Uday Kiran Jonnala
> <juday...@gmail.com> wrote:
> >
> > I have a situation on zombie parent scenario with golang
> >
> > A process (in the case replicator) has many goroutines internally
> >
> > We hit into panic() and I see the replicator process is in Zombie state
> >
> > <<>>>:~$ ps -ef | grep replicator
> >
> > root 87548 87507 0 Aug23 ? 00:00:00 [replicator] <defunct>
> >
> >
> >
> > Main go routine (or the supporting P) excited, but panic left the other
> P thread to be still in executing state (main P could be 87548 and
> supporting P thread 87561 is still there) in blocked state
> >
> > bash-4.2# ls -Fl /proc/87548/task/87561/fd | grep 606649l-wx------. 1
> root root 64 Aug 25 10:59 1 -> pipe:[606649]l-wx------. 1 root root 64 Aug
> 25 10:59 2 -> pipe:[606649]
> >
> > Stack trace
> >
> > bash-4.2# cat /proc/87548/task/87561/stack[<ffffffffbb114714>]
> futex_wait_queue_me+0xc4/0x120[<ffffffffbb11520a>]
> futex_wait+0x10a/0x250[<ffffffffbb1182ce>]
> do_futex+0x35e/0x5b0[<ffffffffbb11865b>]
> SyS_futex+0x13b/0x180[<ffffffffbb003c09>]
> do_syscall_64+0x79/0x1b0[<ffffffffbba00081>]
> entry_SYSCALL_64_after_hwframe+0x3d/0xa2[<ffffffffffffffff>]
> 0xffffffffffffffff
> >
> >
> >
> > We have panic internally from main go routine
> >
> > fatal error: concurrent map writes
> >
> > goroutine 666359 [running]:
> > runtime.throw(0x101d6ae, 0x15)
> >
> /home/ll/ntnx/toolchain-builds/78ae837ba07c8ef8f0ea782407d8d4626815552b.x86_64/go/src/runtime/panic.go:608
>
> +0x72 fp=0xc00374b6f0 sp=0xc00374b6c0 pc=0x42da62
> > runtime.mapassign_faststr(0xdb71c0, 0xc00023f5f0, 0xc000aca990, 0x83,
> 0xc0009d03c8)
> >
> /home/ll/ntnx/toolchain-builds/78ae837ba07c8ef8f0ea782407d8d4626815552b.x86_64/go/src/runtime/map_faststr.go:275
>
> +0x3bf fp=0xc00374b758 sp=0xc00374b6f0 pc=0x41527f
> >
> github.eng.nutanix.com/xyz/abc/metadata.UpdateRecvInProgressFlag(0xc000aca990,
>
> 0x83, 0x0)
> >
> > .......
> >
> > goroutine 665516 [chan receive, 2 minutes]:
> > zeus.(*Leadership).LeaderValue.func1(0xc003d5c120, 0x0, 0xc002e906c0,
> 0x52, 0xc00302ec60, 0x29)
> > /home/ll/ntnx/main/build/.go/src/zeus/leadership.go:244 +0x34
> > created by zeus.(*Leadership).LeaderValue
> > /home/ll/ntnx/main/build/.go/src/zeus/leadership.go:243 +0x277
> > 2020-08-03 00:35:04 rolled over log file
> > ERROR: logging before flag.Parse: I0803 00:35:04.426906 196123
> dataset.go:26] initialize zfs linking
> > ERROR: logging before flag.Parse: I0803 00:35:04.433296 196123
> dataset.go:34] completed zfs linking successfully
> > I0803 00:35:04.433447 196123 main.go:86] Gflags passed NodeUuid:
> c238e584-0eeb-48bd-b299-2a25b13602f1, External Ip: 10.15.96.163
> > I0803 00:35:04.433460 196123 main.go:99] Component name using for this
> process : abc-c238e584-0eeb-48bd-b299-2a25b13602f1
> > I0803 00:35:04.433467 196123 main.go:120] Trying to initialize DB
> >
> > If there is panic() from main P thread, as I understand we exit() and
> cleanup all P threads of the process.
> >
> > Are we hitting into the following scenario, I did not look into M-P-G
> implantation in detail.
> >
> > Example:
> >
> > #include <stdio.h>
> > #include <pthread.h>
> > #include <unistd.h>
> > #include <stdlib.h>
> >
> > void *thread_function(void *args)
> > {
> > printf("The is new thread! Sleep 20 seconds...\n");
> > sleep(100);
> > printf("Exit from thread\n");
> > pthread_exit(0);
> > }
> >
> > int main(int argc, char **argv)
> > {
> > pthread_t thrd;
> > pthread_attr_t attr;
> > int res = 0;
> > res = pthread_attr_init(&attr);
> > res = pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
> > res = pthread_create(&thrd, &attr, thread_function, NULL);
> > res = pthread_attr_destroy(&attr);
> > printf("Main thread. Sleep 5 seconds\n");
> > sleep(5);
> > printf("Exit from main process\n");
> > pthread_exit(0);
> > }
> >
> > kkk@ ~/mycode/go () $ ./a.out &
> > [1] 108418Main thread. Sleep 5 secondsThe is new thread! Sleep 20
> seconds...
> > kkk@ ~/mycode/go () $
> > Exit from main processs
> > PID TTY TIME CMD
> > 49313 pts/26 00:00:01 bash108418 pts/26 00:00:00 [a.out] <defunct>108449
> pts/26 00:00:00 ps
> >
> > See the main process is <defunct> and child is still hanging around
> >
> > kkk@ ~/mycode/go () $ sudo cat
> /proc/108418/task/108420/stack[<ffffffff810b4c1d>]
> hrtimer_nanosleep+0xbd/0x1d0[<ffffffff810b4dae>]
> SyS_nanosleep+0x7e/0x90[<ffffffff816a63c9>]
> system_call_fastpath+0x16/0x1b[<ffffffffffffffff>]
> 0xffffffffffffffffujonnala@ ~/mycode/go () $ Exit from thread
> >
> > Any help in this regard is appreciated.
>
>
> I think you are misreading something somewhere. Zombie status is a
> feature of a process, not a thread. It means that the child process
> has exited but that the parent process, the one which started the
> child process via the fork system call (or, on GNU/Linux, the clone
> system call), has not called the wait (or waitpid or wait3 or wait4)
> system call to collect its status.
>
> So don't look at threads or P's. Look at the parent process that
> started the process that became a zombie.
>
> Ian
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/f70e42f4-622d-4d91-b51d-ed00f2e11ac4n%40googlegroups.com.
