On 1/8/16 15:18 , John Barfield wrote: > By any chance do you have a link to any documentation for the kernel > debugging?
The full documentation for the kernel debugger is going to be at http://illumos.org/books/mdb/. However, basically what you'd want to do is select the boot option that specifies +kmdb. At which point, you'll end up going through and at the prompt writing the following things: kmem_flags/W 0xf :c Which will boot things up normally. One problem here is that since kmdb will be loaded, if it dumps, you'll be dumped into kmdb on a panic. At which point typing ':c' to continue should proceed to get a crash dump. > I hate to bother the guys on IRC. Questions like this are completely fine, that's why we're there. Folks sometimes have to wait a bit before someone comes around and answers the questions, but we try to keep the IRC room friendly, such that folks can really ask any kinds of questions. So it's not bothering us to ask there. If you'd rather not use IRC, it's not for everyone, that's completely fine. Robert > On 1/8/16, 5:12 PM, "Robert Mustacchi" <[email protected]> wrote: > >> Hi, >> >> I'm sorry that you're seeing these panics. The first thing that I'd do >> is go through and increase your dump device size. The older forms of the >> installers were a bit pessimal in how they sized them. I would recommend >> sizing it to either half or all of memory. That is specifically the ZFS >> volume zones/dump. You may need to remove the old dump device and then >> create and destroy the ZFS dataset and then enable the dump device. >> >> The second thing that's a bit worrying is the fact that we're seeing the >> kernel heap corruption. It may come at a large cost, but it may makes >> sense to turn on kmem debugging to get a better sense of what's going >> on. That can be done by setting kmem_flags to 0xf while booting via >> kmdb. If you need help on that, feel free to reach out on our IRC >> channel, #smartos on irc.freenode.net. >> >> Robert >> >> On 1/8/16 15:04 , John Barfield wrote: >>> It just happened again: >>> >>> >>> >>> 2016-01-08T22:45:48.616347+00:00 00-21-28-b2-4f-ea genunix: [ID 432124 >>> kern.notice] buffer freed to wrong cache >>> 2016-01-08T22:45:48.616355+00:00 00-21-28-b2-4f-ea genunix: [ID 815666 >>> kern.notice] buffer was allocated from zio_data_buf_4096, >>> 2016-01-08T22:45:48.616360+00:00 00-21-28-b2-4f-ea genunix: [ID 530907 >>> kern.notice] caller attempting free to zio_data_buf_8192. >>> 2016-01-08T22:45:48.616365+00:00 00-21-28-b2-4f-ea genunix: [ID 563406 >>> kern.notice] buffer=ffffff14b54a5000 bufctl=ffffff2cd356c020 cache: >>> zio_data_buf_8192 >>> 2016-01-08T22:45:48.616370+00:00 00-21-28-b2-4f-ea unix: [ID 836849 >>> kern.notice] #012#015panic[cpu6]/thread=ffffff0116017c40: >>> 2016-01-08T22:45:48.616375+00:00 00-21-28-b2-4f-ea genunix: [ID 812275 >>> kern.notice] kernel heap corruption detected >>> >>> 2016-01-08T22:45:48.616390+00:00 00-21-28-b2-4f-ea unix: [ID 100000 >>> kern.notice] #012 >>> 2016-01-08T22:45:48.616402+00:00 00-21-28-b2-4f-ea genunix: [ID 802836 >>> kern.notice] ffffff0116017a50 fffffffffba2c2f4 () >>> 2016-01-08T22:45:48.616412+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff0116017ab0 genunix:kmem_slab_free+c1 () >>> 2016-01-08T22:45:48.616420+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff0116017b20 genunix:kmem_magazine_destroy+6e () >>> 2016-01-08T22:45:48.616428+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff0116017b60 genunix:kmem_depot_ws_reap+5d () >>> 2016-01-08T22:45:48.616435+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff0116017c20 genunix:taskq_thread+2d0 () >>> 2016-01-08T22:45:48.616441+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >>> kern.notice] ffffff0116017c30 unix:thread_start+8 () >>> 2016-01-08T22:45:48.616449+00:00 00-21-28-b2-4f-ea unix: [ID 100000 >>> kern.notice] >>> 2016-01-08T22:45:48.616456+00:00 00-21-28-b2-4f-ea genunix: [ID 672855 >>> kern.notice] syncing file systems... >>> 2016-01-08T22:45:48.616464+00:00 00-21-28-b2-4f-ea genunix: [ID 904073 >>> kern.notice] done >>> 2016-01-08T22:45:48.616472+00:00 00-21-28-b2-4f-ea genunix: [ID 111219 >>> kern.notice] dumping to /dev/zvol/dsk/zones/dump, offset 65536, content: >>> kernel >>> 2016-01-08T22:45:48.616480+00:00 00-21-28-b2-4f-ea genunix: [ID 100000 >>> kern.notice] >>> 2016-01-08T22:45:48.616488+00:00 00-21-28-b2-4f-ea genunix: [ID 665016 >>> kern.notice] #015 36% done: 2962335 pages dumped, >>> 2016-01-08T22:45:48.616495+00:00 00-21-28-b2-4f-ea genunix: [ID 495082 >>> kern.notice] dump failed: error 28 >>> >>> >>> >>> >>> >>> John Barfield / Sr Principal Engineer >>> >>> +1 (214) 425-0783/ >>> [email protected]<mailto:[email protected]> >>> >>> BISS, Inc. Office: +1 (214) 506-8354 >>> >>> 4925 Greenville Ave Suite 900 >>> >>> Dallas, TX 75206 >>> >>> support.bissinc.com<http://htmlsig.com/support.bissinc.com> >>> >>> This e-mail message may contain confidential or legally privileged >>> information and is intended only for the use of the intended recipient(s). >>> Any unauthorized disclosure, dissemination, distribution, copying or the >>> taking of any action in reliance on the information herein is prohibited. >>> E-mails are not secure and cannot be guaranteed to be error free as they >>> can be intercepted, amended, or contain viruses. Anyone who communicates >>> with us by e-mail is deemed to have accepted these risks. Company Name is >>> not responsible for errors or omissions in this message and denies any >>> responsibility for any damage arising from the use of e-mail. Any opinion >>> and other statement contained in this message and any attachment are solely >>> those of the author and do not necessarily represent those of the company. >>> >>> From: John Barfield >>> <[email protected]<mailto:[email protected]>> >>> Date: Friday, January 8, 2016 at 10:31 AM >>> To: >>> "[email protected]<mailto:[email protected]>" >>> >>> <[email protected]<mailto:[email protected]>> >>> Subject: joyent_20150108T111855Z Kernel Panic >>> >>> Greetings List, >>> >>> Brace yourselves: >>> >>> I’m writing because I had 3 servers give me kernel panics yesterday and >>> reboot themselves around the same time. (Effectively rebooting about 20 >>> VM’s as well). >>> >>> /var/adm/messages available here: >>> http://download.txtelecom.net/server-34-36-37.tar.gz for all 3 servers. >>> >>> If you have the time and patience, please let me know what you think :) >>> >>> I’m happy to provide more information if needed and potential partial dumps >>> if someone can tell me a little bit more about how to get them and If its >>> of any use. >>> >>> Also, let me know if I should post this to the illumos list as well or >>> instead. >>> >>> John Barfield >>> >>> ________________________________ >>> >>> >>> ### Rambling thoughts and info follow starting here: >>> >>> All 3 servers are using joyent_20150108T111855Z kernel on Sunfire X4170 >>> M2’s with 144GB of RAM each. They have dual hex core Intel’s at 2.66ghz >>> with hyper threading enabled. Zpool has L2Arc cache device and mirrored >>> SLOG’s (All are SAS drives, all have 1 raidz1 vdev) >>> >>> Let me say that I am not a kernel engineer at all, but I started following >>> Ben Rockwoods coredump guide trying to get a proper dump to analyze with >>> mdb and it keeps failing with the following: >>> >>> 'mdb: failed to read 'platform' string from kernel: No such file or >>> directory mdb: failed to initialize target: No such file or directory’ >>> >>> I’m not sure if I’m doing it wrong or not but besides the error, the logs >>> actually had a message that said the coredump was unsuccessful. >>> >>> Anyway, since I don’t have a viable core dump, I went reading through the >>> panic messages in the logs and just figured I would post them here for >>> someone else to see what they think from /var/adm/messages. >>> I could be way off but I think the kernel panic occurred when a VM wrote to >>> a zvol due to Qemu/ZIO errors on servers 1 & 3, but then I see a ZFS buffer >>> written to wrong cache on server 2. Could be a cascading event that >>> started between VM’s on srv2 in theory. >>> >>> Anyway, please, if you get a chance, someone tell me what you think >>> happened, and if an upgrade could resolve a bug (and which bug??). Unless >>> someone thinks its naughty memory or something, blah, I’m just rambling. >>> >>> My thoughts on Potential bugs: >>> https://www.illumos.org/issues/5504 >>> http://www.listbox.com/member/archive/182191/2015/01/sort/time_rev/page/1/entry/11:84/20150126102848:0193750E-A570-11E4-93E5-83F5155B1524/ >>> >>> ### End Rambling. >>> >>> >>> >>> Relevant bits in the logs: >>> >>> Server 1: >>> >>> 2016-01-08T02:07:55.180609+00:00 00-21-28-b2-24-6a unix: [ID 836849 >>> kern.notice] #012#015panic[cpu0]/thread=ffffff011765dc40: >>> 2016-01-08T02:07:55.180621+00:00 00-21-28-b2-24-6a genunix: [ID 683410 >>> kern.notice] BAD TRAP: type=e (#pf Page fault) rp=ffffff011765d890 >>> addr=ffffff0b4d92d000 >>> 2016-01-08T02:07:55.180627+00:00 00-21-28-b2-24-6a unix: [ID 100000 >>> kern.notice] #012 >>> 2016-01-08T02:07:55.180632+00:00 00-21-28-b2-24-6a unix: [ID 839527 >>> kern.notice] zpool-zones: >>> 2016-01-08T02:07:55.180637+00:00 00-21-28-b2-24-6a unix: [ID 753105 >>> kern.notice] #pf Page fault >>> 2016-01-08T02:07:55.180642+00:00 00-21-28-b2-24-6a unix: [ID 532287 >>> kern.notice] Bad kernel fault at addr=0xffffff0b4d92d000 >>> 2016-01-08T02:07:55.180658+00:00 00-21-28-b2-24-6a unix: [ID 243837 >>> kern.notice] pid=90, pc=0xfffffffffb85414a, sp=0xffffff011765d988, >>> eflags=0x10206 >>> 2016-01-08T02:07:55.180669+00:00 00-21-28-b2-24-6a unix: [ID 211416 >>> kern.notice] cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: >>> 26f8<vmxe,xmme,fxsr,pge,mce,pae,pse,de> >>> 2016-01-08T02:07:55.180675+00:00 00-21-28-b2-24-6a unix: [ID 624947 >>> kern.notice] cr2: ffffff0b4d92d000 >>> 2016-01-08T02:07:55.180680+00:00 00-21-28-b2-24-6a unix: [ID 625075 >>> kern.notice] cr3: 1dc00000 >>> 2016-01-08T02:07:55.180685+00:00 00-21-28-b2-24-6a unix: [ID 625715 >>> kern.notice] cr8: c >>> 2016-01-08T02:07:55.180690+00:00 00-21-28-b2-24-6a unix: [ID 100000 >>> kern.notice] >>> 2016-01-08T02:07:55.180695+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>> kern.notice] #011rdi: ffffff0b4d92d000 rsi: ffffff2b4973c000 rdx: >>> 1000 >>> 2016-01-08T02:07:55.180701+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>> kern.notice] #011rcx: 200 r8: 0 r9: >>> ffffff26349d4000 >>> 2016-01-08T02:07:55.180706+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>> kern.notice] #011rax: ffffff2b4d989b30 rbx: ffffff2b3e0f9498 rbp: >>> ffffff011765d9b0 >>> 2016-01-08T02:07:55.180711+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>> kern.notice] #011r10: 6 r11: 0 r12: >>> 0 >>> 2016-01-08T02:07:55.180716+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>> kern.notice] #011r13: 0 r14: 0 r15: >>> 0 >>> 2016-01-08T02:07:55.180721+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>> kern.notice] #011fsb: fffffd7fff071240 gsb: fffffffffbc32600 ds: >>> 38 >>> 2016-01-08T02:07:55.180726+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>> kern.notice] #011 es: 38 fs: 0 gs: >>> 0 >>> 2016-01-08T02:07:55.180731+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>> kern.notice] #011trp: e err: 2 rip: >>> fffffffffb85414a >>> 2016-01-08T02:07:55.180738+00:00 00-21-28-b2-24-6a unix: [ID 592667 >>> kern.notice] #011 cs: 30 rfl: 10206 rsp: >>> ffffff011765d988 >>> 2016-01-08T02:07:55.180743+00:00 00-21-28-b2-24-6a unix: [ID 266532 >>> kern.notice] #011 ss: 38 >>> 2016-01-08T02:07:55.180750+00:00 00-21-28-b2-24-6a unix: [ID 100000 >>> kern.notice] >>> 2016-01-08T02:07:55.180756+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>> kern.notice] ffffff011765d770 unix:real_mode_stop_cpu_stage2_end+9fb3 () >>> 2016-01-08T02:07:55.180761+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>> kern.notice] ffffff011765d880 unix:trap+db3 () >>> 2016-01-08T02:07:55.180768+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>> kern.notice] ffffff011765d890 unix:cmntrap+e6 () >>> 2016-01-08T02:07:55.180791+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>> kern.notice] ffffff011765d9b0 unix:bcopy+55a () >>> 2016-01-08T02:07:55.180800+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>> kern.notice] ffffff011765da40 zfs:zio_done+342 () >>> 2016-01-08T02:07:55.180807+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>> kern.notice] ffffff011765da70 zfs:zio_execute+78 () >>> 2016-01-08T02:07:55.180814+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>> kern.notice] ffffff011765db30 genunix:taskq_thread+2d0 () >>> 2016-01-08T02:07:55.180819+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >>> kern.notice] ffffff011765db40 unix:thread_start+8 () >>> 2016-01-08T02:07:55.180826+00:00 00-21-28-b2-24-6a unix: [ID 100000 >>> kern.notice] >>> 2016-01-08T02:07:55.180834+00:00 00-21-28-b2-24-6a genunix: [ID 672855 >>> kern.notice] syncing file systems... >>> 2016-01-08T02:07:55.180840+00:00 00-21-28-b2-24-6a genunix: [ID 733762 >>> kern.notice] 17 >>> 2016-01-08T02:07:55.180847+00:00 00-21-28-b2-24-6a genunix: [ID 904073 >>> kern.notice] done >>> 2016-01-08T02:07:55.180854+00:00 00-21-28-b2-24-6a genunix: [ID 111219 >>> kern.notice] dumping to /dev/zvol/dsk/zones/dump, offset 65536, content: >>> kernel >>> 2016-01-08T02:07:55.180860+00:00 00-21-28-b2-24-6a genunix: [ID 100000 >>> kern.notice] >>> 2016-01-08T02:07:55.180868+00:00 00-21-28-b2-24-6a genunix: [ID 665016 >>> kern.notice] #015 51% done: 3210871 pages dumped, >>> 2016-01-08T02:07:55.180874+00:00 00-21-28-b2-24-6a genunix: [ID 495082 >>> kern.notice] dump failed: error 28 >>> >>> >>> Server2: >>> -01-08T03:12:39.904338+00:00 00-10-e0-0f-3f-a0 genunix: [ID 432124 >>> kern.notice] buffer freed to wrong cache >>> 2016-01-08T03:12:39.904347+00:00 00-10-e0-0f-3f-a0 genunix: [ID 815666 >>> kern.notice] buffer was allocated from zio_data_buf_4096, >>> 2016-01-08T03:12:39.904352+00:00 00-10-e0-0f-3f-a0 genunix: [ID 530907 >>> kern.notice] caller attempting free to zio_data_buf_8192. >>> 2016-01-08T03:12:39.904357+00:00 00-10-e0-0f-3f-a0 genunix: [ID 563406 >>> kern.notice] buffer=ffffff1397a14000 bufctl=ffffff2a46f33238 cache: >>> zio_data_buf_8192 >>> 2016-01-08T03:12:39.904362+00:00 00-10-e0-0f-3f-a0 unix: [ID 836849 >>> kern.notice] #012#015panic[cpu20]/thread=ffffff0116017c40: >>> 2016-01-08T03:12:39.904366+00:00 00-10-e0-0f-3f-a0 genunix: [ID 812275 >>> kern.notice] kernel heap corruption detected >>> 2016-01-08T03:12:39.904381+00:00 00-10-e0-0f-3f-a0 unix: [ID 100000 >>> kern.notice] #012 >>> 2016-01-08T03:12:39.904391+00:00 00-10-e0-0f-3f-a0 genunix: [ID 802836 >>> kern.notice] ffffff0116017a50 fffffffffba2c2f4 () >>> 2016-01-08T03:12:39.904397+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >>> kern.notice] ffffff0116017ab0 genunix:kmem_slab_free+c1 () >>> 2016-01-08T03:12:39.904402+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >>> kern.notice] ffffff0116017b20 genunix:kmem_magazine_destroy+6e () >>> 2016-01-08T03:12:39.904406+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >>> kern.notice] ffffff0116017b60 genunix:kmem_depot_ws_reap+5d () >>> 2016-01-08T03:12:39.904411+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >>> kern.notice] ffffff0116017c20 genunix:taskq_thread+2d0 () >>> 2016-01-08T03:12:39.904416+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >>> kern.notice] ffffff0116017c30 unix:thread_start+8 () >>> 2016-01-08T03:12:39.904420+00:00 00-10-e0-0f-3f-a0 unix: [ID 100000 >>> kern.notice] >>> >>> >>> Server3: >>> 2016-01-07T23:30:29.392494+00:00 00-21-28-b2-4f-ea unix: [ID 836849 >>> kern.notice] #012#015panic[cpu6]/thread=ffffff264340e880: >>> 2016-01-07T23:30:29.392507+00:00 00-21-28-b2-4f-ea genunix: [ID 683410 >>> kern.notice] BAD TRAP: type=e (#pf Page fault) rp=ffffff01171e0450 >>> addr=ffffff1529242000 2016-01-07T23:30:29.392512+00:00 00-21-28-b2-4f-ea >>> unix: [ID 100000 kern.notice] #012 2016-01-07T23:30:29.392518+00:00 >>> 00-21-28-b2-4f-ea unix: [ID 839527 kern.notice] qemu-system-x86_: >>> 2016-01-07T23:30:29.392523+00:00 00-21-28-b2-4f-ea unix: [ID 753105 >>> kern.notice] #pf Page fault 2016-01-07T23:30:29.392528+00:00 >>> 00-21-28-b2-4f-ea unix: [ID 532287 kern.notice] Bad kernel fault at >>> addr=0xffffff1529242000 2016-01-07T23:30:29.392545+00:00 00-21-28-b2-4f-ea >>> unix: [ID 243837 kern.notice] pid=4465, pc=0xfffffffff7dc5577, >>> sp=0xffffff01171e0540, eflags=0x10282 2016-01-07T23:30:29.392556+00:00 >>> 00-21-28-b2-4f-ea unix: [ID 211416 kern.notice] cr0: >>> 80050033<pg,wp,ne,et,mp,pe> cr4: 26f8<vmxe,xmme,fxsr,pge,mce,pae, ps >> e,de> 2016-01-07T23:30:29.392562+00:00 00-21-28-b2-4f-ea unix: [ID 624947 >> kern.notice] cr2: ffffff1529242000 2016-01-07T23:30:29.392568+00:00 >> 00-21-28-b2-4f-ea unix: [ID 625075 kern.notice] cr3: 1213265000 >> 2016-01-07T23:30:29.392573+00:00 00-21-28-b2-4f-ea unix: [ID 625715 >> kern.notice] cr8: c 2016-01-07T23:30:29.392578+00:00 00-21-28-b2-4f-ea unix: >> [ID 100000 kern.notice] 2016-01-07T23:30:29.392583+00:00 00-21-28-b2-4f-ea >> unix: [ID 592667 kern.notice] #011rdi: ffffff39b2128c04 rsi: >> ffffff39b2128c0d rdx: ffffff1529243ff7 2016-01-07T23:30:29.392589+00:00 >> 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011rcx: 200000098393bc0 r8: >> ffffff39b2128c61 r9: ffffff1529243ff8 2016-01-07T23:30:29.392594+00:00 >> 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011rax: ffffff1529242000 >> rbx: ffffff1529242000 rbp: ffffff01171e0610 2016-01-07T23:30:29.392600+00:00 >> 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011r10: ffffff1529244000 >> r11: ffffff39b2128c59 r12: ffffff152924200d 2016-01-07T23:30:29. 392 >> 604+00:00 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011r13: df r14: >> ffffff39b2128c12 r15: 0 2016-01-07T23:30:29.392610+00:00 00-21-28-b2-4f-ea >> unix: [ID 592667 kern.notice] #011fsb: fffffd7fff07e240 gsb: >> ffffff26016beac0 ds: 4b 2016-01-07T23:30:29.392615+00:00 00-21-28-b2-4f-ea >> unix: [ID 592667 kern.notice] #011 es: 4b fs: 0 gs: 0 >> 2016-01-07T23:30:29.392620+00:00 00-21-28-b2-4f-ea unix: [ID 592667 >> kern.notice] #011trp: e err: 2 rip: fffffffff7dc5577 >> 2016-01-07T23:30:29.392625+00:00 00-21-28-b2-4f-ea unix: [ID 592667 >> kern.notice] #011 cs: 30 rfl: 10282 rsp: ffffff01171e0540 >> 2016-01-07T23:30:29.392630+00:00 00-21-28-b2-4f-ea unix: [ID 266532 >> kern.notice] #011 ss: 38 2016-01-07T23:30:29.392635+00:00 00-21-28-b2-4f-ea >> unix: [ID 100000 kern.notice] 2016-01-07T23:30:29.392640+00:00 >> 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0330 >> unix:real_mode_stop_cpu_stage2_end+9fb3 () 2016-01-07T23:30:29.392645+00:00 >> 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff0 117 >> 1e0440 unix:trap+db3 () 2016-01-07T23:30:29.392650+00:00 00-21-28-b2-4f-ea >> genunix: [ID 655072 kern.notice] ffffff01171e0450 unix:cmntrap+e6 () >> 2016-01-07T23:30:29.392667+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0610 zfs:LZ4_uncompress_unknownOutputSize+147 () >> 2016-01-07T23:30:29.392673+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0650 zfs:lz4_decompress+44 () >> 2016-01-07T23:30:29.392678+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0690 zfs:zio_decompress_data+4b () >> 2016-01-07T23:30:29.392682+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e06d0 zfs:zio_decompress+56 () >> 2016-01-07T23:30:29.392687+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0700 zfs:zio_pop_transforms+7a () >> 2016-01-07T23:30:29.392692+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0790 zfs:zio_done+21a () >> 2016-01-07T23:30:29.392697+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 ke rn. >> notice] ffffff01171e07c0 zfs:zio_execute+78 () >> 2016-01-07T23:30:29.392702+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e07f0 zfs:zio_nowait+21 () >> 2016-01-07T23:30:29.392706+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e08e0 zfs:arc_read+4fe () >> 2016-01-07T23:30:29.392711+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0980 zfs:dbuf_read_impl+1a5 () >> 2016-01-07T23:30:29.392716+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e09f0 zfs:dbuf_read+de () >> 2016-01-07T23:30:29.392721+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0a50 zfs:dmu_tx_check_ioerr+76 () >> 2016-01-07T23:30:29.392729+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0b80 zfs:dmu_tx_count_write+3e7 () >> 2016-01-07T23:30:29.392737+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0bd0 zfs:dmu_tx_hold_write+5a () >> 2016-01-07T23:30:29.392744+00:00 00-21-28-b2-4f-ea genunix: [ID 6550 72 >> kern.notice] ffffff01171e0c80 zfs:zvol_write+1bc () >> 2016-01-07T23:30:29.392755+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0cb0 genunix:cdev_write+2d () >> 2016-01-07T23:30:29.392764+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0d90 specfs:spec_write+4c1 () >> 2016-01-07T23:30:29.392770+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0e30 genunix:fop_write+8b () >> 2016-01-07T23:30:29.392777+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0f00 genunix:pwrite+194 () >> 2016-01-07T23:30:29.392800+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff01171e0f10 unix:brand_sys_syscall+1f5 () >> 2016-01-07T23:30:29.392809+00:00 00-21-28-b2-4f-ea unix: [ID 100000 >> kern.notice] 2016-01-07T23:30:29.392814+00:00 00-21-28-b2-4f-ea genunix: [ID >> 672855 kern.notice] syncing file systems... 2016-01-07T23:30:29.392822+00:00 >> 00-21-28-b2-4f-ea genunix: [ID 904073 kern.notice] done >>> >>> >>> John Barfield / Sr Principal Engineer >>> >>> +1 (214) 425-0783/ >>> [email protected]<mailto:[email protected]> >>> >>> BISS, Inc. Office: +1 (214) 506-8354 >>> >>> 4925 Greenville Ave Suite 900 >>> >>> Dallas, TX 75206 >>> >>> support.bissinc.com<http://htmlsig.com/support.bissinc.com> >>> >> >> > > ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
