[email protected] (Stefan Kempf), 2016.01.28 (Thu) 20:57 (CET):
> Marcus MERIGHI wrote:
> > [email protected] (Stefan Kempf), 2016.01.28 (Thu) 06:48 (CET):
> > > Stuart Henderson wrote:
> > > > On 2016/01/27 20:10, Stefan Kempf wrote:
> > > > > So what I suspect to happen is that:
> > > > > - userland does a syscall
> > > > > - something goes wrong in the kernel, causing it to call
> > > > > sigexit(SIGILL), terminating the process
> > > > > - and the offending instruction you see in the core dump
> > > > > is the 'syscall' instruction.
> > > >
> > > > If this is the case, perhaps ktrace will give clues.
> > >
> > > Let's give it a try.
> > >
> > > Marcus, can you run this as root, please?
> > > ktrace /sbin/ping some.domain
> > >
> > > Or whatever way you invoked ping that made it crash.
> > >
> > > And send us the output of kdump -f ktrace.out?
> >
> > # ktrace /sbin/ping 192.168.188.189
> > PING 192.168.188.189 (192.168.188.189): 56 data bytes
> > 64 bytes from 192.168.188.189: icmp_seq=0 ttl=255 time=3.286 ms
> > Illegal instruction
>
> It's close to my guess. This is how I interpret the end of the output:
>
> > # kdump -f ./ktrace.out
> > [...]
> > 31378 ping CALL poll(0x7f7ffffd8790,1,INFTIM)
> > 31378 ping PSIG SIGALRM caught handler=0x15a413b03050 mask=0<>
>
> The process blocks in a system call, then a signal wakes it up. Before
> returning to userspace, sendsig() tries to setup a signal context.
> Since the ktrace output stops here, sendsig() must have called
> sigexit(SIGILL). This happens when the kernel is not able to copy the
> signal context onto the stack of the user process.
>
> Some reasons I can think of: the process is at the very bottom of
> the stack, the stack pointer of the user process is trashed, or:
> the stack pointer is within the stack area of the process, but
> it points to a page that was not yet mapped-in, and uvm_fault()
> fails to fault it in for some reason.
>
> Let's see what the stack pointer looks like when you get the illegal
> instruction. Can you try this please:
>
> $ top
>
> In a different shell (as root):
> # procmap <pid of top>
>
> We need to see the lines that say [ stack ]
00007F7FFDFE1000 28672K [ stack ]
00007F7FFFBE1000 4028K read/write [ stack ]
00007F7FFFFD0000 64K read/write [ stack ]
00007F7FFFFE0000 4K [ stack ]
> Now, back in top, hit ctrl+c to make it crash. Then run:
>
> $ gdb -q /usr/bin/top top.core
> (gdb) info reg
>
> And send us the output of the 'info reg' command.
rax 0x4 4
rbx 0x6773930c4a0 7109130372256
rcx 0x679c7cb2dda 7120112791002
rdx 0x1388 5000
rsi 0x1 1
rdi 0x7f7ffffdf858 140187732408408
rbp 0x1e 0x1e
rsp 0x7f7ffffdf848 0x7f7ffffdf848
r8 0x101010101010101 72340172838076673
r9 0x8080808080808080 -9187201950435737472
r10 0x679c7d11c5a 7120113179738
r11 0x246 582
r12 0x6773930bc60 7109130370144
r13 0x6773930c480 7109130372224
r14 0x7f7ffffdf8c0 140187732408512
r15 0x67738f08ae0 7109126163168
rip 0x679c7cb2dda 0x679c7cb2dda <poll+10>
eflags 0x247 583
cs 0x2b 43
ss 0x23 35
ds 0x23 35
es 0x23 35
fs 0x23 35
gs 0x23 35
Full output is in gdb.out and procmap.out, respectively.
Thanks for your instructions and for working on this!
Bye, Marcus
> !DSPAM:56aa72d7244577102954733!
gdb -q /usr/bin/top ./top.core
(no debugging symbols found)
Core was generated by `top'.
Program terminated with signal 4, Illegal instruction.
(no debugging symbols found)
Loaded symbols for /usr/bin/top
Reading symbols from /usr/lib/libcurses.so.14.0...done.
Loaded symbols for /usr/lib/libcurses.so.14.0
Reading symbols from /usr/lib/libc.so.84.2...done.
Loaded symbols for /usr/lib/libc.so.84.2
Reading symbols from /usr/libexec/ld.so...done.
Loaded symbols for /usr/libexec/ld.so
#0 0x00000679c7cb2dda in poll () at <stdin>:2
2 <stdin>: No such file or directory.
in <stdin>
(gdb) info reg
rax 0x4 4
rbx 0x6773930c4a0 7109130372256
rcx 0x679c7cb2dda 7120112791002
rdx 0x1388 5000
rsi 0x1 1
rdi 0x7f7ffffdf858 140187732408408
rbp 0x1e 0x1e
rsp 0x7f7ffffdf848 0x7f7ffffdf848
r8 0x101010101010101 72340172838076673
r9 0x8080808080808080 -9187201950435737472
r10 0x679c7d11c5a 7120113179738
r11 0x246 582
r12 0x6773930bc60 7109130370144
r13 0x6773930c480 7109130372224
r14 0x7f7ffffdf8c0 140187732408512
r15 0x67738f08ae0 7109126163168
rip 0x679c7cb2dda 0x679c7cb2dda <poll+10>
eflags 0x247 583
cs 0x2b 43
ss 0x23 35
ds 0x23 35
es 0x23 35
fs 0x23 35
gs 0x23 35
Current language: auto; currently asm
(gdb)
0000000000001000 0K [ anon ]
0000067738F00000 40K read/exec [ ?VNODE? ]
0000067739009000 8K read [ ?VNODE? ]
0000067739100000 0K [ anon ]
000006773910A000 4K read/write [ anon ]
000006773920A000 4K read [ ?VNODE? ]
000006773920B000 4K read [ anon ]
000006773930B000 4K read/write [ anon ]
000006773930C000 4K read/write [ anon ]
0000067939100000 0K [ anon ]
0000067939E72000 4K read/write [ anon ]
000006793C531000 4K read/write [ anon ]
000006793C8EF000 4K read/write [ anon ]
000006793C975000 4K read/write [ anon ]
000006794011A000 4K read/write [ anon ]
00000679433CA000 4K read/write [ anon ]
00000679435D9000 4K read/exec [ uvm_aobj ]
00000679444E3000 4K read/write [ anon ]
00000679456D4000 4K read/write [ anon ]
0000067946194000 4K read/write [ anon ]
000006794742D000 4K read/write [ anon ]
000006794A180000 4K read/write [ anon ]
000006794B05D000 4K read/write [ anon ]
000006794B69C000 4K read/write [ anon ]
000006794D844000 4K read/write [ anon ]
000006794DC9C000 4K read/write [ anon ]
000006794F26E000 4K read/write [ anon ]
000006794F51E000 4K read/write [ anon ]
000006795160F000 4K read/write [ anon ]
0000067952D29000 4K read/write [ anon ]
00000679544A4000 4K read/write [ anon ]
0000067954834000 4K read/write [ anon ]
0000067955368000 4K read/write [ anon ]
000006795D425000 4K read/write [ anon ]
000006795D8D4000 20K read/write [ anon ]
000006795EAC6000 4K read/write [ anon ]
000006796083A000 4K read/write [ anon ]
0000067961700000 4K read/write [ anon ]
000006796416D000 4K read/write [ anon ]
0000067964ADF000 12K read [ ?VNODE? ]
0000067964C2E000 4K read/write [ anon ]
00000679652DE000 4K read/write [ anon ]
0000067965B6D000 4K read/write [ anon ]
0000067967218000 4K read/write [ anon ]
0000067968929000 4K read/write [ anon ]
0000067969A1B000 4K read/write [ anon ]
000006796B981000 4K [ anon ]
000006796B982000 8K read/write [ anon ]
000006796B984000 4K [ anon ]
000006796D950000 4K read/write [ anon ]
000006796F768000 4K read/write [ anon ]
000006796F9B0000 4K read/write [ anon ]
0000067970DC6000 4K read/write [ anon ]
000006797465F000 4K read/write [ anon ]
0000067974E22000 4K read/write [ anon ]
0000067974E97000 4K read/write [ anon ]
00000679752AE000 4K read/write [ anon ]
00000679756E5000 4K read/write [ anon ]
0000067976738000 4K read/write [ anon ]
000006797805E000 4K read/write [ anon ]
0000067979A01000 4K read/write [ anon ]
000006797A138000 4K read/write [ anon ]
000006797B214000 4K read/write [ anon ]
000006797B87D000 4K read/write [ anon ]
000006797C269000 4K read/write [ anon ]
000006797EC38000 4K read/write [ anon ]
0000067982E40000 4K read/write [ anon ]
00000679872CE000 4K read/write [ anon ]
0000067987D40000 4K read/write [ anon ]
000006798A9A9000 4K read/write [ anon ]
000006798C62A000 4K read/write [ anon ]
000006798CE27000 4K read/write [ anon ]
000006798EEF9000 4K read/write [ anon ]
000006798F4FC000 4K read/write [ anon ]
0000067991DB1000 4K read/write [ anon ]
00000679956B2000 8K read/write [ anon ]
0000067995700000 44K read/exec [ ?VNODE? ]
000006799580A000 12K read [ ?VNODE? ]
000006799590D000 4K read/write [ anon ]
0000067995A0D000 4K read [ anon ]
0000067995B0D000 4K read/write [ anon ]
0000067995B0E000 8K read/write [ anon ]
0000067995B10000 4K read [ anon ]
0000067995B11000 16K read/write [ anon ]
0000067998D93000 48K read/write [ anon ]
000006799AF35000 4K read/write [ anon ]
000006799B4D8000 4K read/write [ anon ]
000006799C568000 4K read/write [ anon ]
000006799C71F000 4K read/write [ anon ]
000006799CBC2000 4K read/write [ anon ]
000006799CF39000 4K read/write [ anon ]
000006799D1C5000 4K read/write [ anon ]
000006799EC25000 4K read/write [ anon ]
000006799F590000 4K read/write [ anon ]
000006799F7AD000 4K read/write [ anon ]
00000679A1D2E000 4K read/write [ anon ]
00000679A35BA000 4K read/write [ anon ]
00000679A3FA1000 8K read/write [ anon ]
00000679A66E8000 4K read/write [ anon ]
00000679AB0BB000 4K read/write [ anon ]
00000679AD3E5000 4K read/write [ anon ]
00000679B0319000 4K read/write [ anon ]
00000679B097D000 4K read/write [ anon ]
00000679B0A64000 4K read/write [ anon ]
00000679B2164000 4K read/write [ anon ]
00000679B337F000 4K read/write [ anon ]
00000679B4252000 4K read/write [ anon ]
00000679BAFFC000 4K read/write [ anon ]
00000679BB4DC000 4K read/write [ anon ]
00000679BCAEB000 4K read/write [ anon ]
00000679BD0A1000 4K read/write [ anon ]
00000679BE1DC000 4K read/write [ anon ]
00000679BE78B000 4K read/write [ anon ]
00000679C1ABC000 4K read/write [ anon ]
00000679C3B71000 4K read/write [ anon ]
00000679C3E74000 4K read/write [ anon ]
00000679C5624000 4K read/write [ anon ]
00000679C62B7000 4K read/write [ anon ]
00000679C6F44000 4K read/write [ anon ]
00000679C7145000 4K read/write [ anon ]
00000679C7C75000 632K read/exec [ ?VNODE? ]
00000679C7D13000 1020K [ ?VNODE? ]
00000679C7E12000 100K read [ ?VNODE? ]
00000679C7E2B000 1024K [ ?VNODE? ]
00000679C7F2B000 8K read/write [ ?VNODE? ]
00000679C7F2D000 1020K [ ?VNODE? ]
00000679C802C000 4K read [ ?VNODE? ]
00000679C802D000 1020K [ ?VNODE? ]
00000679C812C000 12K read/write [ ?VNODE? ]
00000679C812F000 40K read/write [ anon ]
00000679C8139000 4K read [ anon ]
00000679C813A000 16K read/write [ anon ]
00000679CACB0000 4K read/write [ anon ]
00000679CB024000 4K read/write [ anon ]
00000679CC2E9000 4K read/write [ anon ]
00000679CE3C3000 4K read/write [ anon ]
00000679CEE9B000 4K read/write [ anon ]
00000679CF119000 4K read/write [ anon ]
00000679CF7AD000 4K read/write [ anon ]
00000679D043B000 4K read/write [ anon ]
00000679D0936000 4K read/write [ anon ]
00000679D17C4000 4K read/write [ anon ]
00000679D18F0000 4K read/write [ anon ]
00000679D25D4000 4K read/write [ anon ]
00000679D2676000 4K read/write [ anon ]
00000679D5F6D000 4K read/write [ anon ]
00000679D7B78000 268K read/exec [ ?VNODE? ]
00000679D7BBB000 1020K [ ?VNODE? ]
00000679D7CBA000 76K read [ ?VNODE? ]
00000679D7CCD000 1024K [ ?VNODE? ]
00000679D7DCD000 16K read/write [ ?VNODE? ]
00000679D7DD1000 1020K [ ?VNODE? ]
00000679D7ED0000 4K read [ ?VNODE? ]
00000679D7ED1000 1020K [ ?VNODE? ]
00000679D7FD0000 8K read/write [ ?VNODE? ]
00000679D9D72000 4K read/write [ anon ]
00000679D9ED6000 4K read/write [ anon ]
00000679DACD6000 4K read/write [ anon ]
00000679DAF6A000 4K read/write [ anon ]
00000679E0984000 4K read/write [ anon ]
00000679E4180000 4K read/write [ anon ]
00000679E45B3000 44K read/write [ anon ]
00000679E5B6C000 4K read/write [ anon ]
00000679E63C1000 4K read/write [ anon ]
00000679E7442000 4K read/write [ anon ]
00000679EA6BB000 4K read/write [ anon ]
00000679EAFE7000 4K read/write [ anon ]
00000679F1CAA000 4K read/write [ anon ]
00000679F2EB4000 4K read/write [ anon ]
00000679F68C4000 4K read/write [ anon ]
00000679F6C35000 4K read/write [ anon ]
00000679F8A62000 4K read/write [ anon ]
00000679F94D7000 4K read/write [ anon ]
00000679F958E000 4K read/write [ anon ]
00000679F9A04000 4K read/write [ anon ]
00000679FB30B000 4K read/write [ anon ]
00000679FCEAB000 4K read/write [ anon ]
00000679FDE11000 4K read/write [ anon ]
00000679FE26F000 4K read/write [ anon ]
00000679FFF09000 4K read/write [ anon ]
0000067A020D6000 4K read/write [ anon ]
0000067A023B3000 4K read/write [ anon ]
0000067A046B7000 4K read/write [ anon ]
0000067A062CB000 4K [ anon ]
0000067A062CC000 8K read/write [ anon ]
0000067A062CE000 4K [ anon ]
0000067A0736D000 4K read/write [ anon ]
0000067A09F28000 4K read/write [ anon ]
0000067A0DF1F000 4K read/write [ anon ]
0000067A0EBED000 4K read/write [ anon ]
0000067A10F8D000 4K read/write [ anon ]
0000067A11070000 4K read [ anon ]
0000067A118EB000 4K read/write [ anon ]
0000067A15377000 4K read/write [ anon ]
0000067A15683000 4K read/write [ anon ]
0000067A175DF000 4K read/write [ anon ]
0000067A17805000 4K read/write [ anon ]
0000067A1A18D000 4K read/write [ anon ]
0000067A1CCA7000 4K read/write [ anon ]
0000067A1EE6F000 4K read/write [ anon ]
0000067A1FEFC000 4K read/write [ anon ]
0000067A210C8000 4K read/write [ anon ]
0000067A23E87000 4K read/write [ anon ]
0000067A24E25000 4K read/write [ anon ]
0000067A25810000 4K read/write [ anon ]
0000067A29403000 4K read/write [ anon ]
0000067A2A7B2000 4K read/write [ anon ]
0000067A2D092000 4K read/write [ anon ]
0000067A2F57E000 4K read/write [ anon ]
0000067A30042000 4K read/write [ anon ]
0000067A301E2000 8K read/write [ anon ]
0000067A306EB000 4K read/write [ anon ]
0000067A31551000 4K read/write [ anon ]
0000067A33572000 4K read/write [ anon ]
0000067A3414E000 4K read/write [ anon ]
0000067A3572B000 4K read/write [ anon ]
0000067A366C4000 4K read/write [ anon ]
0000067A36E6F000 44K read/write [ anon ]
0000067A3878D000 4K read/write [ anon ]
0000067A38E1E000 4K read/write [ anon ]
00007F7FFDFE1000 28672K [ stack ]
00007F7FFFBE1000 4028K read/write [ stack ]
00007F7FFFFD0000 64K read/write [ stack ]
00007F7FFFFE0000 4K [ stack ]
total 6320K
