I'm trying to assist Casey Hancock with illegal instruction exceptions, reported earlier:
http://marc.info/?t=145103079400015&r=1&w=2 http://marc.info/?t=145111278100001&r=1&w=2 But I'm very weak on tracking syscalls through the userland .core files Casey has provided. I'm not sure if ktrace(1) will add any value to finding the root cause, which I assume is a branch into data, but I have no clear understanding of how to discern where it's happening, and I I could use some guidance, as otherwise it's just the blind leading the blind. At this time, I've provided Casey with a -current release(8) so I have a source tree I can ensure is in sync with executed binaries. Each failure of a userland program is an illegal instruction, and each time, there's a syscall being executed in frame 0. I've seen poll(2), kevent(2), waitpid(2), and others, and I am unsure how to -- or if I can -- get any value from the .core files produced. These appear to be valid stack traces, from the calling frame, as shown below. A cluestick would be very helpful. I'm sure there's something obvious I'm overlooking. Thanks in advance! --- stack traces and some argument variables shown below, by example --- $ gdb -q ntpd ~/i3luefire/cores/ntpd.core warning: exec file is newer than core file. Core was generated by `ntpd'. Program terminated with signal 4, Illegal instruction. Loaded symbols for /usr/obj/usr.sbin/ntpd/ntpd Reading symbols from /usr/lib/libm.so.9.0...done. Loaded symbols for /usr/lib/libm.so.9.0 Reading symbols from /usr/lib/libutil.so.12.1...done. Loaded symbols for /usr/lib/libutil.so.12.1 Reading symbols from /usr/lib/libtls.so.10.0...done. Loaded symbols for /usr/lib/libtls.so.10.0 Reading symbols from /usr/lib/libssl.so.38.0...done. Loaded symbols for /usr/lib/libssl.so.38.0 Reading symbols from /usr/lib/libcrypto.so.37.0...done. Loaded symbols for /usr/lib/libcrypto.so.37.0 Reading symbols from /usr/lib/libc.so.84.2...done. Loaded symbols for /usr/lib/libc.so.84.2 Reading symbols from /usr/libexec/ld.so...done. Loaded symbols for /usr/libexec/ld.so #0 0x00000ee8802c4dda in poll () at <stdin>:2 2 <stdin>: No such file or directory. in <stdin> (gdb) bt #0 0x00000ee8802c4dda in poll () at <stdin>:2 #1 0x00000ee64bf05e8f in main (argc=Variable "argc" is not available. ) at /usr/src/usr.sbin/ntpd/ntpd.c:262 Current language: auto; currently asm (gdb) fr 1 #1 0x00000ee64bf05e8f in main (argc=Variable "argc" is not available. ) at /usr/src/usr.sbin/ntpd/ntpd.c:262 262 if ((nfds = poll(pfd, i, timeout)) == -1) Current language: auto; currently c (gdb) p pfd $1 = (struct pollfd *) 0xee88422c180 (gdb) p i $2 = 3 (gdb) p timeout $3 = -1 (gdb) p nfds $4 = 0 (gdb) p pfd[0] $5 = {fd = 3, events = 1, revents = 0} (gdb) p pfd[1] $6 = {fd = 4, events = 1, revents = 0} (gdb) p pfd[2] $7 = {fd = 7, events = 1, revents = 0} (gdb) q $ gdb -q newsyslog ~/i3luefire/cores/newsyslog.core warning: exec file is newer than core file. Core was generated by `newsyslog'. Program terminated with signal 4, Illegal instruction. Loaded symbols for /usr/obj/usr.bin/newsyslog/newsyslog Reading symbols from /usr/lib/libc.so.84.2...done. Loaded symbols for /usr/lib/libc.so.84.2 Reading symbols from /usr/libexec/ld.so...done. Loaded symbols for /usr/libexec/ld.so #0 0x00000a250ed48108 in wait4 () at <stdin>:2 2 <stdin>: No such file or directory. in <stdin> (gdb) btr #0 0x00000a250ed48108 in wait4 () at <stdin>:2 #1 0x00000a22217053a7 in main (argc=Variable "argc" is not available. ) at /usr/src/usr.bin/newsyslog/newsyslog.c:288 Current language: auto; currently asm (gdb) fr 1 #1 0x00000a22217053a7 in main (argc=Variable "argc" is not available. ) at /usr/src/usr.bin/newsyslog/newsyslog.c:288 288 while (waitpid(-1, &status, 0) != -1) Current language: auto; currently c (gdb) p status $1 = 32639 (gdb) q $ gdb -q tmux ~/i3luefire/cores/tmux.core warning: exec file is newer than core file. Core was generated by `tmux'. Program terminated with signal 4, Illegal instruction. Loaded symbols for /usr/obj/usr.bin/tmux/tmux Reading symbols from /usr/lib/libutil.so.12.1...done. Loaded symbols for /usr/lib/libutil.so.12.1 Reading symbols from /usr/lib/libcurses.so.14.0...done. Loaded symbols for /usr/lib/libcurses.so.14.0 Reading symbols from /usr/lib/libevent.so.4.1...done. Loaded symbols for /usr/lib/libevent.so.4.1 Reading symbols from /usr/lib/libc.so.84.2...done. Loaded symbols for /usr/lib/libc.so.84.2 Reading symbols from /usr/libexec/ld.so...done. Loaded symbols for /usr/libexec/ld.so #0 0x00000de0c84e8e2a in kevent () at <stdin>:2 2 <stdin>: No such file or directory. in <stdin> (gdb) bt #0 0x00000de0c84e8e2a in kevent () at <stdin>:2 #1 0x00000de112180440 in kq_dispatch (base=Variable "base" is not available. ) at /usr/src/lib/libevent/kqueue.c:193 #2 0x00000de11217feb2 in event_base_loop (base=0xde1986bf000, flags=Variable "flags" is not available. ) at /usr/src/lib/libevent/event.c:493 #3 0x00000ddeb8a37f5a in proc_loop (tp=0xde1986bb1e0, loopcb=0xddeb8a3fa40 <server_loop>) at /usr/src/usr.bin/tmux/proc.c:219 #4 0x00000ddeb8a3fe4a in server_start (base=Variable "base" is not available. ) at /usr/src/usr.bin/tmux/server.c:184 #5 0x00000ddeb8a13af0 in client_connect (base=0xde1986bf000, path=0xde1986bb3c0 "/tmp/tmux-0/default", start_server=1) at /usr/src/usr.bin/tmux/client.c:158 #6 0x00000ddeb8a13c24 in client_main (base=0xde1986bf000, argc=0, argv=0x7f7ffffbc7c0, flags=Variable "flags" is not available. ) at /usr/src/usr.bin/tmux/client.c:269 #7 0x00000ddeb8a4653f in main (argc=0, argv=0x7f7ffffbc7b8) at /usr/src/usr.bin/tmux/tmux.c:323 Current language: auto; currently asm (gdb) fr 2 #2 0x00000de11217feb2 in event_base_loop (base=0xde1986bf000, flags=Variable "flags" is not available. ) at /usr/src/lib/libevent/event.c:493 493 res = evsel->dispatch(base, evbase, tv_p); (gdb) p evsel $1 = (const struct eventop *) 0xde1123838a0 (gdb) p evsel[0] $2 = {name = 0xde112282235 "kqueue", init = 0xde112180a80 <kq_init>, add = 0xde112180860 <kq_add>, del = 0xde112180670 <kq_del>, dispatch = 0xde1121803e0 <kq_dispatch>, dealloc = 0xde112180370 <kq_dealloc>, need_reinit = 1} (gdb) p base $3 = (struct event_base *) 0xde1986bf000 (gdb) p base[0] $4 = {evsel = 0xde1123838a0, evbase = 0xde1986bf400, event_count = 13, event_count_active = 0, event_gotterm = 0, event_break = 0, activequeues = 0xde10a142660, nactivequeues = 1, sig = {ev_signal = {ev_next = {tqe_next = 0x0, tqe_prev = 0x0}, ev_active_next = {tqe_next = 0x0, tqe_prev = 0x0}, ev_signal_next = {tqe_next = 0x0, tqe_prev = 0x0}, min_heap_idx = 0, ev_base = 0x0, ev_fd = 0, ev_events = 0, ev_ncalls = 0, ev_pncalls = 0x0, ev_timeout = {tv_sec = 0, tv_usec = 0}, ev_pri = 0, ev_callback = 0, ev_arg = 0x0, ev_res = 0, ev_flags = 0}, ev_signal_pair = {-1, -1}, ev_signal_added = 0, evsignal_caught = 0, evsigevents = {{tqh_first = 0x0, tqh_last = 0x0} <repeats 33 times>}, evsigcaught = {0 <repeats 33 times>}, sh_old = 0xde139017900, sh_old_max = 31}, eventqueue = {tqh_first = 0xddeb8e7ac80, tqh_last = 0xde0c843b208}, event_tv = {tv_sec = 22126, tv_usec = 163687}, timeheap = {p = 0xde1986c0540, n = 3, a = 8}, tv_cache = {tv_sec = 0, tv_usec = 163610}} (gdb) p tv_p $5 = (struct timeval *) 0x7f7ffffbb800 (gdb) p tv_p[0] $6 = {tv_sec = 0, tv_usec = 176235} (gdb) q $ gdb -q sftp/sftp ~/i3luefire/cores/sftp.core warning: exec file is newer than core file. Core was generated by `sftp'. Program terminated with signal 4, Illegal instruction. Loaded symbols for /usr/obj/usr.bin/ssh/sftp/sftp Reading symbols from /usr/lib/libutil.so.12.1...done. Loaded symbols for /usr/lib/libutil.so.12.1 Reading symbols from /usr/lib/libedit.so.5.2...done. Loaded symbols for /usr/lib/libedit.so.5.2 Reading symbols from /usr/lib/libtermcap.so.14.0...done. Loaded symbols for /usr/lib/libtermcap.so.14.0 Reading symbols from /usr/lib/libc.so.84.2...done. Loaded symbols for /usr/lib/libc.so.84.2 Reading symbols from /usr/libexec/ld.so...done. Loaded symbols for /usr/libexec/ld.so #0 0x000019bdbe8fe2b8 in read () at <stdin>:2 2 <stdin>: No such file or directory. in <stdin> (gdb) bt #0 0x000019bdbe8fe2b8 in read () at <stdin>:2 #1 0x000019bb5be0fbd3 in atomicio6 (f=0x19bdbe8fe2b0 <read>, fd=3, _s=0x19bd90240700, n=4, cb=0, cb_arg=0x19be4f111228) at /usr/src/usr.bin/ssh/lib/../atomicio.c:54 #2 0x000019bb5be09bbf in get_msg (conn=0x19be4f111200, m=0x19be357e2180) at /usr/src/usr.bin/ssh/sftp/../sftp-client.c:127 #3 0x000019bb5be0b65b in do_upload (conn=0x19be4f111200, local_path=0x19bdf0615a10 "ntpd.core", remote_path=0x19be2ccca400 "/i3luefire/ntpd.core", preserve_flag=0, resume=0, fsync_flag=0) at /usr/src/usr.bin/ssh/sftp/../sftp-client.c:1693 #4 0x000019bb5be04ac8 in process_put (conn=0x19be4f111200, src=Variable "src" is not available. ) at /usr/src/usr.bin/ssh/sftp/../sftp.c:726 #5 0x000019bb5be05f7f in parse_dispatch_command (conn=0x19be4f111200, cmd=0x7f7ffffdbbc0 "put ntpd.core", pwd=0x7f7ffffdbbb0, err_abort=0) at /usr/src/usr.bin/ssh/sftp/../sftp.c:1416 #6 0x000019bb5be07cb8 in interactive_loop (conn=0x19be4f111200, file1=Variable "file1" is not available. ) at /usr/src/usr.bin/ssh/sftp/../sftp.c:2111 #7 0x000019bb5be08683 in main (argc=Variable "argc" is not available. ) at /usr/src/usr.bin/ssh/sftp/../sftp.c:2389 Current language: auto; currently asm (gdb) fr 1 #1 0x000019bb5be0fbd3 in atomicio6 (f=0x19bdbe8fe2b0 <read>, fd=3, _s=0x19bd90240700, n=4, cb=0, cb_arg=0x19be4f111228) at /usr/src/usr.bin/ssh/lib/../atomicio.c:54 54 res = (f) (fd, s + pos, n - pos); Current language: auto; currently c (gdb) p fd $1 = 3 (gdb) p s No symbol "s" in current context. (gdb) p pos $2 = 0 (gdb) p n $3 = 4 (gdb) q
