Thank you for all of your help. I’ve reset the dump devices on all of my nodes to 144GB.
I’ll work on kmemdb, I have to move my monitoring VM to another host so that I’ll know when it goes down otherwise, I won’t get any alerts and the node will just stay down. We’ll at least have enough space for a dump to complete on the zvol. I’ll follow up again when everything is ready for the debugging process. On 1/8/16, 5:27 PM, "Robert Mustacchi" <[email protected]> wrote: >On 1/8/16 15:18 , John Barfield wrote: >> By any chance do you have a link to any documentation for the kernel >> debugging? > >The full documentation for the kernel debugger is going to be at >http://illumos.org/books/mdb/. However, basically what you'd want to do >is select the boot option that specifies +kmdb. At which point, you'll >end up going through and at the prompt writing the following things: > >kmem_flags/W 0xf >:c > >Which will boot things up normally. One problem here is that since kmdb >will be loaded, if it dumps, you'll be dumped into kmdb on a panic. At >which point typing ':c' to continue should proceed to get a crash dump. > >> I hate to bother the guys on IRC. > >Questions like this are completely fine, that's why we're there. Folks >sometimes have to wait a bit before someone comes around and answers the >questions, but we try to keep the IRC room friendly, such that folks can >really ask any kinds of questions. So it's not bothering us to ask >there. If you'd rather not use IRC, it's not for everyone, that's >completely fine. > >Robert > >> On 1/8/16, 5:12 PM, "Robert Mustacchi" <[email protected]> wrote: >> >>> Hi, >>> >>> I'm sorry that you're seeing these panics. The first thing that I'd do >>> is go through and increase your dump device size. The older forms of the >>> installers were a bit pessimal in how they sized them. I would recommend >>> sizing it to either half or all of memory. That is specifically the ZFS >>> volume zones/dump. You may need to remove the old dump device and then >>> create and destroy the ZFS dataset and then enable the dump device. >>> >>> The second thing that's a bit worrying is the fact that we're seeing the >>> kernel heap corruption. It may come at a large cost, but it may makes >>> sense to turn on kmem debugging to get a better sense of what's going >>> on. That can be done by setting kmem_flags to 0xf while booting via >>> kmdb. If you need help on that, feel free to reach out on our IRC >>> channel, #smartos on irc.freenode.net. >>> >>> Robert >>> >>> On 1/8/16 15:04 , John Barfield wrote: >>>> It just happened again: >>>> >>>> >>>> >>>> 2016-01-08T22:45:48.616347+00:00 00-21-28-b2-4f-ea genunix: [ID 432124 >>>> kern.notice] buffer freed to wrong cache >>>> 2016-01-08T22:45:48.616355+00:00 00-21-28-b2-4f-ea genunix: [ID 815666 >>>> kern.notice] buffer was allocated from zio_data_buf_4096, >>>> 2016-01-08T22:45:48.616360+00:00 00-21-28-b2-4f-ea genunix: [ID 530907 >>>> kern.notice] caller attempting free to zio_data_buf_8192. >>>> 2016-01-08T22:45:48.616365+00:00 00-21-28-b2-4f-ea genunix: [ID 563406 >>>> kern.notice] buffer=ffffff14b54a5000 bufctl=ffffff2cd356c020 cache: >>>> zio_data_buf_8192 >>>> 2016-01-08T22:45:48.616370+00:00 00-21-28-b2-4f-ea unix: [ID 836849 >>>> kern.notice] #012#015panic[cpu6]/thread=ffffff0116017c40: >>>> 2016-01-08T22:45:48.616375+00:00 00-21-28-b2-4f-ea genunix: [ID 812275 >>>> kern.notice] kernel heap corruption detected >>>> >>>> 2016-01-08T22:45:48.616390+00:00 00-21-28-b2-4f-ea unix: [ID 100000 >>>> kern.notice] #012 >>>> 2016-01-08T22:45:48.616402+00:00 00-21-28-b2-4f-ea genunix: [ID 802836 >>>> kern.notice] ffffff0116017a50 fffffffffba2c2f4 () >>>> 2016-01-08T22:45:48.616412+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>>> kern.notice] ffffff0116017ab0 genunix:kmem_slab_free+c1 () >>>> 2016-01-08T22:45:48.616420+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>>> kern.notice] ffffff0116017b20 genunix:kmem_magazine_destroy+6e () >>>> 2016-01-08T22:45:48.616428+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>>> kern.notice] ffffff0116017b60 genunix:kmem_depot_ws_reap+5d () >>>> 2016-01-08T22:45:48.616435+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>>> kern.notice] ffffff0116017c20 genunix:taskq_thread+2d0 () >>>> 2016-01-08T22:45:48.616441+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>>> kern.notice] ffffff0116017c30 unix:thread_start+8 () >>>> 2016-01-08T22:45:48.616449+00:00 00-21-28-b2-4f-ea unix: [ID 100000 >>>> kern.notice] >>>> 2016-01-08T22:45:48.616456+00:00 00-21-28-b2-4f-ea genunix: [ID 672855 >>>> kern.notice] syncing file systems... >>>> 2016-01-08T22:45:48.616464+00:00 00-21-28-b2-4f-ea genunix: [ID 904073 >>>> kern.notice] done >>>> 2016-01-08T22:45:48.616472+00:00 00-21-28-b2-4f-ea genunix: [ID 111219 >>>> kern.notice] dumping to /dev/zvol/dsk/zones/dump, offset 65536, content: >>>> kernel >>>> 2016-01-08T22:45:48.616480+00:00 00-21-28-b2-4f-ea genunix: [ID 100000 >>>> kern.notice] >>>> 2016-01-08T22:45:48.616488+00:00 00-21-28-b2-4f-ea genunix: [ID 665016 >>>> kern.notice] #015 36% done: 2962335 pages dumped, >>>> 2016-01-08T22:45:48.616495+00:00 00-21-28-b2-4f-ea genunix: [ID 495082 >>>> kern.notice] dump failed: error 28 >>>> >>>> >>>> >>>> >>>> >>>> John Barfield / Sr Principal Engineer >>>> >>>> +1 (214) 425-0783/ >>>> [email protected]<mailto:[email protected]> >>>> >>>> BISS, Inc. Office: +1 (214) 506-8354 >>>> >>>> 4925 Greenville Ave Suite 900 >>>> >>>> Dallas, TX 75206 >>>> >>>> support.bissinc.com<http://htmlsig.com/support.bissinc.com> >>>> >>>> This e-mail message may contain confidential or legally privileged >>>> information and is intended only for the use of the intended recipient(s). >>>> Any unauthorized disclosure, dissemination, distribution, copying or the >>>> taking of any action in reliance on the information herein is prohibited. >>>> E-mails are not secure and cannot be guaranteed to be error free as they >>>> can be intercepted, amended, or contain viruses. Anyone who communicates >>>> with us by e-mail is deemed to have accepted these risks. Company Name is >>>> not responsible for errors or omissions in this message and denies any >>>> responsibility for any damage arising from the use of e-mail. Any opinion >>>> and other statement contained in this message and any attachment are >>>> solely those of the author and do not necessarily represent those of the >>>> company. >>>> >>>> From: John Barfield >>>> <[email protected]<mailto:[email protected]>> >>>> Date: Friday, January 8, 2016 at 10:31 AM >>>> To: >>>> "[email protected]<mailto:[email protected]>" >>>> >>>> <[email protected]<mailto:[email protected]>> >>>> Subject: joyent_20150108T111855Z Kernel Panic >>>> >>>> Greetings List, >>>> >>>> Brace yourselves: >>>> >>>> I’m writing because I had 3 servers give me kernel panics yesterday and >>>> reboot themselves around the same time. (Effectively rebooting about 20 >>>> VM’s as well). >>>> >>>> /var/adm/messages available here: >>>> http://download.txtelecom.net/server-34-36-37.tar.gz for all 3 servers. >>>> >>>> If you have the time and patience, please let me know what you think :) >>>> >>>> I’m happy to provide more information if needed and potential partial >>>> dumps if someone can tell me a little bit more about how to get them and >>>> If its of any use. >>>> >>>> Also, let me know if I should post this to the illumos list as well or >>>> instead. >>>> >>>> John Barfield >>>> >>>> ________________________________ >>>> >>>> >>>> ### Rambling thoughts and info follow starting here: >>>> >>>> All 3 servers are using joyent_20150108T111855Z kernel on Sunfire X4170 >>>> M2’s with 144GB of RAM each. They have dual hex core Intel’s at 2.66ghz >>>> with hyper threading enabled. Zpool has L2Arc cache device and mirrored >>>> SLOG’s (All are SAS drives, all have 1 raidz1 vdev) >>>> >>>> Let me say that I am not a kernel engineer at all, but I started following >>>> Ben Rockwoods coredump guide trying to get a proper dump to analyze with >>>> mdb and it keeps failing with the following: >>>> >>>> 'mdb: failed to read 'platform' string from kernel: No such file or >>>> directory mdb: failed to initialize target: No such file or directory’ >>>> >>>> I’m not sure if I’m doing it wrong or not but besides the error, the logs >>>> actually had a message that said the coredump was unsuccessful. >>>> >>>> Anyway, since I don’t have a viable core dump, I went reading through the >>>> panic messages in the logs and just figured I would post them here for >>>> someone else to see what they think from /var/adm/messages. >>>> I could be way off but I think the kernel panic occurred when a VM wrote >>>> to a zvol due to Qemu/ZIO errors on servers 1 & 3, but then I see a ZFS >>>> buffer written to wrong cache on server 2. Could be a cascading event >>>> that started between VM’s on srv2 in theory. >>>> >>>> Anyway, please, if you get a chance, someone tell me what you think >>>> happened, and if an upgrade could resolve a bug (and which bug??). Unless >>>> someone thinks its naughty memory or something, blah, I’m just rambling. >>>> >>>> My thoughts on Potential bugs: >>>> https://www.illumos.org/issues/5504 >>>> http://www.listbox.com/member/archive/182191/2015/01/sort/time_rev/page/1/entry/11:84/20150126102848:0193750E-A570-11E4-93E5-83F5155B1524/ >>>> >>>> ### End Rambling. >>>> >>>> >>>> >>>> Relevant bits in the logs: >>>> >>>> Server 1: >>>> >>>> 2016-01-08T02:07:55.180609+00:00 00-21-28-b2-24-6a unix: [ID 836849 >>>> kern.notice] #012#015panic[cpu0]/thread=ffffff011765dc40: >>>> 2016-01-08T02:07:55.180621+00:00 00-21-28-b2-24-6a genunix: [ID 683410 >>>> kern.notice] BAD TRAP: type=e (#pf Page fault) rp=ffffff011765d890 >>>> addr=ffffff0b4d92d000 >>>> 2016-01-08T02:07:55.180627+00:00 00-21-28-b2-24-6a unix: [ID 100000 >>>> kern.notice] #012 >>>> 2016-01-08T02:07:55.180632+00:00 00-21-28-b2-24-6a unix: [ID 839527 >>>> kern.notice] zpool-zones: >>>> 2016-01-08T02:07:55.180637+00:00 00-21-28-b2-24-6a unix: [ID 753105 >>>> kern.notice] #pf Page fault >>>> 2016-01-08T02:07:55.180642+00:00 00-21-28-b2-24-6a unix: [ID 532287 >>>> kern.notice] Bad kernel fault at addr=0xffffff0b4d92d000 >>>> 2016-01-08T02:07:55.180658+00:00 00-21-28-b2-24-6a unix: [ID 243837 >>>> kern.notice] pid=90, pc=0xfffffffffb85414a, sp=0xffffff011765d988, >>>> eflags=0x10206 >>>> 2016-01-08T02:07:55.180669+00:00 00-21-28-b2-24-6a unix: [ID 211416 >>>> kern.notice] cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: >>>> 26f8<vmxe,xmme,fxsr,pge,mce,pae,pse,de> >>>> 2016-01-08T02:07:55.180675+00:00 00-21-28-b2-24-6a unix: [ID 624947 >>>> kern.notice] cr2: ffffff0b4d92d000 >>>> 2016-01-08T02:07:55.180680+00:00 00-21-28-b2-24-6a unix: [ID 625075 >>>> kern.notice] cr3: 1dc00000 >>>> 2016-01-08T02:07:55.180685+00:00 00-21-28-b2-24-6a unix: [ID 625715 >>>> kern.notice] cr8: c >>>> 2016-01-08T02:07:55.180690+00:00 00-21-28-b2-24-6a unix: [ID 100000 >>>> kern.notice] >>>> 2016-01-08T02:07:55.180695+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>>> kern.notice] #011rdi: ffffff0b4d92d000 rsi: ffffff2b4973c000 rdx: >>>> 1000 >>>> 2016-01-08T02:07:55.180701+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>>> kern.notice] #011rcx: 200 r8: 0 r9: >>>> ffffff26349d4000 >>>> 2016-01-08T02:07:55.180706+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>>> kern.notice] #011rax: ffffff2b4d989b30 rbx: ffffff2b3e0f9498 rbp: >>>> ffffff011765d9b0 >>>> 2016-01-08T02:07:55.180711+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>>> kern.notice] #011r10: 6 r11: 0 r12: >>>> 0 >>>> 2016-01-08T02:07:55.180716+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>>> kern.notice] #011r13: 0 r14: 0 r15: >>>> 0 >>>> 2016-01-08T02:07:55.180721+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>>> kern.notice] #011fsb: fffffd7fff071240 gsb: fffffffffbc32600 ds: >>>> 38 >>>> 2016-01-08T02:07:55.180726+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>>> kern.notice] #011 es: 38 fs: 0 gs: >>>> 0 >>>> 2016-01-08T02:07:55.180731+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>>> kern.notice] #011trp: e err: 2 rip: >>>> fffffffffb85414a >>>> 2016-01-08T02:07:55.180738+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>>> kern.notice] #011 cs: 30 rfl: 10206 rsp: >>>> ffffff011765d988 >>>> 2016-01-08T02:07:55.180743+00:00 00-21-28-b2-24-6a unix: [ID 266532 >>>> kern.notice] #011 ss: 38 >>>> 2016-01-08T02:07:55.180750+00:00 00-21-28-b2-24-6a unix: [ID 100000 >>>> kern.notice] >>>> 2016-01-08T02:07:55.180756+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>>> kern.notice] ffffff011765d770 unix:real_mode_stop_cpu_stage2_end+9fb3 () >>>> 2016-01-08T02:07:55.180761+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>>> kern.notice] ffffff011765d880 unix:trap+db3 () >>>> 2016-01-08T02:07:55.180768+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>>> kern.notice] ffffff011765d890 unix:cmntrap+e6 () >>>> 2016-01-08T02:07:55.180791+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>>> kern.notice] ffffff011765d9b0 unix:bcopy+55a () >>>> 2016-01-08T02:07:55.180800+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>>> kern.notice] ffffff011765da40 zfs:zio_done+342 () >>>> 2016-01-08T02:07:55.180807+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>>> kern.notice] ffffff011765da70 zfs:zio_execute+78 () >>>> 2016-01-08T02:07:55.180814+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>>> kern.notice] ffffff011765db30 genunix:taskq_thread+2d0 () >>>> 2016-01-08T02:07:55.180819+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>>> kern.notice] ffffff011765db40 unix:thread_start+8 () >>>> 2016-01-08T02:07:55.180826+00:00 00-21-28-b2-24-6a unix: [ID 100000 >>>> kern.notice] >>>> 2016-01-08T02:07:55.180834+00:00 00-21-28-b2-24-6a genunix: [ID 672855 >>>> kern.notice] syncing file systems... >>>> 2016-01-08T02:07:55.180840+00:00 00-21-28-b2-24-6a genunix: [ID 733762 >>>> kern.notice] 17 >>>> 2016-01-08T02:07:55.180847+00:00 00-21-28-b2-24-6a genunix: [ID 904073 >>>> kern.notice] done >>>> 2016-01-08T02:07:55.180854+00:00 00-21-28-b2-24-6a genunix: [ID 111219 >>>> kern.notice] dumping to /dev/zvol/dsk/zones/dump, offset 65536, content: >>>> kernel >>>> 2016-01-08T02:07:55.180860+00:00 00-21-28-b2-24-6a genunix: [ID 100000 >>>> kern.notice] >>>> 2016-01-08T02:07:55.180868+00:00 00-21-28-b2-24-6a genunix: [ID 665016 >>>> kern.notice] #015 51% done: 3210871 pages dumped, >>>> 2016-01-08T02:07:55.180874+00:00 00-21-28-b2-24-6a genunix: [ID 495082 >>>> kern.notice] dump failed: error 28 >>>> >>>> >>>> Server2: >>>> -01-08T03:12:39.904338+00:00 00-10-e0-0f-3f-a0 genunix: [ID 432124 >>>> kern.notice] buffer freed to wrong cache >>>> 2016-01-08T03:12:39.904347+00:00 00-10-e0-0f-3f-a0 genunix: [ID 815666 >>>> kern.notice] buffer was allocated from zio_data_buf_4096, >>>> 2016-01-08T03:12:39.904352+00:00 00-10-e0-0f-3f-a0 genunix: [ID 530907 >>>> kern.notice] caller attempting free to zio_data_buf_8192. >>>> 2016-01-08T03:12:39.904357+00:00 00-10-e0-0f-3f-a0 genunix: [ID 563406 >>>> kern.notice] buffer=ffffff1397a14000 bufctl=ffffff2a46f33238 cache: >>>> zio_data_buf_8192 >>>> 2016-01-08T03:12:39.904362+00:00 00-10-e0-0f-3f-a0 unix: [ID 836849 >>>> kern.notice] #012#015panic[cpu20]/thread=ffffff0116017c40: >>>> 2016-01-08T03:12:39.904366+00:00 00-10-e0-0f-3f-a0 genunix: [ID 812275 >>>> kern.notice] kernel heap corruption detected >>>> 2016-01-08T03:12:39.904381+00:00 00-10-e0-0f-3f-a0 unix: [ID 100000 >>>> kern.notice] #012 >>>> 2016-01-08T03:12:39.904391+00:00 00-10-e0-0f-3f-a0 genunix: [ID 802836 >>>> kern.notice] ffffff0116017a50 fffffffffba2c2f4 () >>>> 2016-01-08T03:12:39.904397+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >>>> kern.notice] ffffff0116017ab0 genunix:kmem_slab_free+c1 () >>>> 2016-01-08T03:12:39.904402+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >>>> kern.notice] ffffff0116017b20 genunix:kmem_magazine_destroy+6e () >>>> 2016-01-08T03:12:39.904406+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >>>> kern.notice] ffffff0116017b60 genunix:kmem_depot_ws_reap+5d () >>>> 2016-01-08T03:12:39.904411+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >>>> kern.notice] ffffff0116017c20 genunix:taskq_thread+2d0 () >>>> 2016-01-08T03:12:39.904416+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >>>> kern.notice] ffffff0116017c30 unix:thread_start+8 () >>>> 2016-01-08T03:12:39.904420+00:00 00-10-e0-0f-3f-a0 unix: [ID 100000 >>>> kern.notice] >>>> >>>> >>>> Server3: >>>> 2016-01-07T23:30:29.392494+00:00 00-21-28-b2-4f-ea unix: [ID 836849 >>>> kern.notice] #012#015panic[cpu6]/thread=ffffff264340e880: >>>> 2016-01-07T23:30:29.392507+00:00 00-21-28-b2-4f-ea genunix: [ID 683410 >>>> kern.notice] BAD TRAP: type=e (#pf Page fault) rp=ffffff01171e0450 >>>> addr=ffffff1529242000 2016-01-07T23:30:29.392512+00:00 00-21-28-b2-4f-ea >>>> unix: [ID 100000 kern.notice] #012 2016-01-07T23:30:29.392518+00:00 >>>> 00-21-28-b2-4f-ea unix: [ID 839527 kern.notice] qemu-system-x86_: >>>> 2016-01-07T23:30:29.392523+00:00 00-21-28-b2-4f-ea unix: [ID 753105 >>>> kern.notice] #pf Page fault 2016-01-07T23:30:29.392528+00:00 >>>> 00-21-28-b2-4f-ea unix: [ID 532287 kern.notice] Bad kernel fault at >>>> addr=0xffffff1529242000 2016-01-07T23:30:29.392545+00:00 00-21-28-b2-4f-ea >>>> unix: [ID 243837 kern.notice] pid=4465, pc=0xfffffffff7dc5577, >>>> sp=0xffffff01171e0540, eflags=0x10282 2016-01-07T23:30:29.392556+00:00 >>>> 00-21-28-b2-4f-ea unix: [ID 211416 kern.notice] cr0: >>>> 80050033<pg,wp,ne,et,mp,pe> cr4: 26f8<vmxe,xmme,fxsr,pge,mce,pae, >ps >>> e,de> 2016-01-07T23:30:29.392562+00:00 00-21-28-b2-4f-ea unix: [ID 624947 >>> kern.notice] cr2: ffffff1529242000 2016-01-07T23:30:29.392568+00:00 >>> 00-21-28-b2-4f-ea unix: [ID 625075 kern.notice] cr3: 1213265000 >>> 2016-01-07T23:30:29.392573+00:00 00-21-28-b2-4f-ea unix: [ID 625715 >>> kern.notice] cr8: c 2016-01-07T23:30:29.392578+00:00 00-21-28-b2-4f-ea >>> unix: [ID 100000 kern.notice] 2016-01-07T23:30:29.392583+00:00 >>> 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011rdi: ffffff39b2128c04 >>> rsi: ffffff39b2128c0d rdx: ffffff1529243ff7 >>> 2016-01-07T23:30:29.392589+00:00 00-21-28-b2-4f-ea unix: [ID 592667 >>> kern.notice] #011rcx: 200000098393bc0 r8: ffffff39b2128c61 r9: >>> ffffff1529243ff8 2016-01-07T23:30:29.392594+00:00 00-21-28-b2-4f-ea unix: >>> [ID 592667 kern.notice] #011rax: ffffff1529242000 rbx: ffffff1529242000 >>> rbp: ffffff01171e0610 2016-01-07T23:30:29.392600+00:00 00-21-28-b2-4f-ea >>> unix: [ID 592667 kern.notice] #011r10: ffffff1529244000 r11: >>> ffffff39b2128c59 r12: ffffff152924200d 2016-01-07T23:30:29. >392 >>> 604+00:00 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011r13: df r14: >>> ffffff39b2128c12 r15: 0 2016-01-07T23:30:29.392610+00:00 00-21-28-b2-4f-ea >>> unix: [ID 592667 kern.notice] #011fsb: fffffd7fff07e240 gsb: >>> ffffff26016beac0 ds: 4b 2016-01-07T23:30:29.392615+00:00 00-21-28-b2-4f-ea >>> unix: [ID 592667 kern.notice] #011 es: 4b fs: 0 gs: 0 >>> 2016-01-07T23:30:29.392620+00:00 00-21-28-b2-4f-ea unix: [ID 592667 >>> kern.notice] #011trp: e err: 2 rip: fffffffff7dc5577 >>> 2016-01-07T23:30:29.392625+00:00 00-21-28-b2-4f-ea unix: [ID 592667 >>> kern.notice] #011 cs: 30 rfl: 10282 rsp: ffffff01171e0540 >>> 2016-01-07T23:30:29.392630+00:00 00-21-28-b2-4f-ea unix: [ID 266532 >>> kern.notice] #011 ss: 38 2016-01-07T23:30:29.392635+00:00 00-21-28-b2-4f-ea >>> unix: [ID 100000 kern.notice] 2016-01-07T23:30:29.392640+00:00 >>> 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0330 >>> unix:real_mode_stop_cpu_stage2_end+9fb3 () 2016-01-07T23:30:29.392645+00:00 >>> 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff0 >117 >>> 1e0440 unix:trap+db3 () 2016-01-07T23:30:29.392650+00:00 00-21-28-b2-4f-ea >>> genunix: [ID 655072 kern.notice] ffffff01171e0450 unix:cmntrap+e6 () >>> 2016-01-07T23:30:29.392667+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0610 zfs:LZ4_uncompress_unknownOutputSize+147 () >>> 2016-01-07T23:30:29.392673+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0650 zfs:lz4_decompress+44 () >>> 2016-01-07T23:30:29.392678+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0690 zfs:zio_decompress_data+4b () >>> 2016-01-07T23:30:29.392682+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e06d0 zfs:zio_decompress+56 () >>> 2016-01-07T23:30:29.392687+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0700 zfs:zio_pop_transforms+7a () >>> 2016-01-07T23:30:29.392692+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0790 zfs:zio_done+21a () >>> 2016-01-07T23:30:29.392697+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 ke >rn. >>> notice] ffffff01171e07c0 zfs:zio_execute+78 () >>> 2016-01-07T23:30:29.392702+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e07f0 zfs:zio_nowait+21 () >>> 2016-01-07T23:30:29.392706+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e08e0 zfs:arc_read+4fe () >>> 2016-01-07T23:30:29.392711+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0980 zfs:dbuf_read_impl+1a5 () >>> 2016-01-07T23:30:29.392716+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e09f0 zfs:dbuf_read+de () >>> 2016-01-07T23:30:29.392721+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0a50 zfs:dmu_tx_check_ioerr+76 () >>> 2016-01-07T23:30:29.392729+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0b80 zfs:dmu_tx_count_write+3e7 () >>> 2016-01-07T23:30:29.392737+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0bd0 zfs:dmu_tx_hold_write+5a () >>> 2016-01-07T23:30:29.392744+00:00 00-21-28-b2-4f-ea genunix: [ID 6550 >72 >>> kern.notice] ffffff01171e0c80 zfs:zvol_write+1bc () >>> 2016-01-07T23:30:29.392755+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0cb0 genunix:cdev_write+2d () >>> 2016-01-07T23:30:29.392764+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0d90 specfs:spec_write+4c1 () >>> 2016-01-07T23:30:29.392770+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0e30 genunix:fop_write+8b () >>> 2016-01-07T23:30:29.392777+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0f00 genunix:pwrite+194 () >>> 2016-01-07T23:30:29.392800+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff01171e0f10 unix:brand_sys_syscall+1f5 () >>> 2016-01-07T23:30:29.392809+00:00 00-21-28-b2-4f-ea unix: [ID 100000 >>> kern.notice] 2016-01-07T23:30:29.392814+00:00 00-21-28-b2-4f-ea genunix: >>> [ID 672855 kern.notice] syncing file systems... >>> 2016-01-07T23:30:29.392822+00:00 00-21-28-b2-4f-ea genunix: [ID 904073 >>> kern.notice] done >>>> >>>> >>>> John Barfield / Sr Principal Engineer >>>> >>>> +1 (214) 425-0783/ >>>> [email protected]<mailto:[email protected]> >>>> >>>> BISS, Inc. Office: +1 (214) 506-8354 >>>> >>>> 4925 Greenville Ave Suite 900 >>>> >>>> Dallas, TX 75206 >>>> >>>> support.bissinc.com<http://htmlsig.com/support.bissinc.com> >>>> >>> >>> >> >> > > ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
