Hi -
I had bash get stuck in a busy-wait earlier this week. I was able to
see where it was spinning.
user/parlib/ucq.c L 148
0000000000494ad0 get_ucq_msg:
...
494b4b: 0f 84 6c 01 00 00 je 494cbd <get_ucq_msg+0x1ed>
494b51: 48 89 d1 mov %rdx,%rcx
494b54: 48 c1 e1 05 shl $0x5,%rcx
494b58: 4c 01 e1 add %r12,%rcx
494b5b: 80 79 28 00 cmpb $0x0,0x28(%rcx)
494b5f: 75 08 jne 494b69 <get_ucq_msg+0x99>
494b61: f3 90 pause
494b63: 80 79 28 00 cmpb $0x0,0x28(%rcx)
494b67: 74 f8 je 494b61 <get_ucq_msg+0x91>
IN THIS LOOP^^^^^
That line of code is waiting for the kernel to finish writing a message
into the UCQ slot:
/* Wait til the msg is ready (kernel sets this flag) */
while (!my_msg->ready)
cpu_relax();
I didn't have the foresight to get any other debugging info, like a
hexdump of (%rcx), let alone the value of rcx, or the value of any other
registers. (I got this spot with perf, not with trace coretf. Oh
well.)
There are a bunch of things that could be wrong, from simple
memory/register clobbers, to races in the UCQ code.
Another thing that occurred to me is that bash calls fork but doesn't
always exec (like sshd, btw). What happens if the kernel sends a
message (shared memory) to a process at the same time it forks? It
could be that the parent process gets the shared-memory write, but not
the child. Basically, the kernel would have to be doing the fork() on
one core while posting the event on another (or perhaps from IRQ
context on the same core).
This seems like a problem, in general. Fork-without-exec has a bunch
of problems like this. I dealt with a few with dropbear, such as making
sure all user-FDs (which are basically just memory structures) are
closed after a fork. There's a lot of other shared memory business
going on where fork is a problem.
I think the fix is to not use fork(), but that might be a long time
coming. (Needs changes to shells and dropbear, if it's even possible).
In the meantime, if anyone else sees this (basically bash locks up),
see if you can get any decent debug info out of it. It'll most likely
be stuck as RUNNING_S on core 0, so CTRL-B should give you a backtrace
and a set of registers. Also, since it's spinning, you can start up
another instance of the shell and use that too (which is how I got a
perf record trace).
Barret
--
You received this message because you are subscribed to the Google Groups
"Akaros" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
For more options, visit https://groups.google.com/d/optout.
