Ha, yeah, its set to 4GB.
Nodes have 144GB each. I’ll work on your suggestions and report back. By any chance do you have a link to any documentation for the kernel debugging? I hate to bother the guys on IRC. On 1/8/16, 5:12 PM, "Robert Mustacchi" <[email protected]> wrote: >Hi, > >I'm sorry that you're seeing these panics. The first thing that I'd do >is go through and increase your dump device size. The older forms of the >installers were a bit pessimal in how they sized them. I would recommend >sizing it to either half or all of memory. That is specifically the ZFS >volume zones/dump. You may need to remove the old dump device and then >create and destroy the ZFS dataset and then enable the dump device. > >The second thing that's a bit worrying is the fact that we're seeing the >kernel heap corruption. It may come at a large cost, but it may makes >sense to turn on kmem debugging to get a better sense of what's going >on. That can be done by setting kmem_flags to 0xf while booting via >kmdb. If you need help on that, feel free to reach out on our IRC >channel, #smartos on irc.freenode.net. > >Robert > >On 1/8/16 15:04 , John Barfield wrote: >> It just happened again: >> >> >> >> 2016-01-08T22:45:48.616347+00:00 00-21-28-b2-4f-ea genunix: [ID 432124 >> kern.notice] buffer freed to wrong cache >> 2016-01-08T22:45:48.616355+00:00 00-21-28-b2-4f-ea genunix: [ID 815666 >> kern.notice] buffer was allocated from zio_data_buf_4096, >> 2016-01-08T22:45:48.616360+00:00 00-21-28-b2-4f-ea genunix: [ID 530907 >> kern.notice] caller attempting free to zio_data_buf_8192. >> 2016-01-08T22:45:48.616365+00:00 00-21-28-b2-4f-ea genunix: [ID 563406 >> kern.notice] buffer=ffffff14b54a5000 bufctl=ffffff2cd356c020 cache: >> zio_data_buf_8192 >> 2016-01-08T22:45:48.616370+00:00 00-21-28-b2-4f-ea unix: [ID 836849 >> kern.notice] #012#015panic[cpu6]/thread=ffffff0116017c40: >> 2016-01-08T22:45:48.616375+00:00 00-21-28-b2-4f-ea genunix: [ID 812275 >> kern.notice] kernel heap corruption detected >> >> 2016-01-08T22:45:48.616390+00:00 00-21-28-b2-4f-ea unix: [ID 100000 >> kern.notice] #012 >> 2016-01-08T22:45:48.616402+00:00 00-21-28-b2-4f-ea genunix: [ID 802836 >> kern.notice] ffffff0116017a50 fffffffffba2c2f4 () >> 2016-01-08T22:45:48.616412+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff0116017ab0 genunix:kmem_slab_free+c1 () >> 2016-01-08T22:45:48.616420+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff0116017b20 genunix:kmem_magazine_destroy+6e () >> 2016-01-08T22:45:48.616428+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff0116017b60 genunix:kmem_depot_ws_reap+5d () >> 2016-01-08T22:45:48.616435+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff0116017c20 genunix:taskq_thread+2d0 () >> 2016-01-08T22:45:48.616441+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >> kern.notice] ffffff0116017c30 unix:thread_start+8 () >> 2016-01-08T22:45:48.616449+00:00 00-21-28-b2-4f-ea unix: [ID 100000 >> kern.notice] >> 2016-01-08T22:45:48.616456+00:00 00-21-28-b2-4f-ea genunix: [ID 672855 >> kern.notice] syncing file systems... >> 2016-01-08T22:45:48.616464+00:00 00-21-28-b2-4f-ea genunix: [ID 904073 >> kern.notice] done >> 2016-01-08T22:45:48.616472+00:00 00-21-28-b2-4f-ea genunix: [ID 111219 >> kern.notice] dumping to /dev/zvol/dsk/zones/dump, offset 65536, content: >> kernel >> 2016-01-08T22:45:48.616480+00:00 00-21-28-b2-4f-ea genunix: [ID 100000 >> kern.notice] >> 2016-01-08T22:45:48.616488+00:00 00-21-28-b2-4f-ea genunix: [ID 665016 >> kern.notice] #015 36% done: 2962335 pages dumped, >> 2016-01-08T22:45:48.616495+00:00 00-21-28-b2-4f-ea genunix: [ID 495082 >> kern.notice] dump failed: error 28 >> >> >> >> >> >> John Barfield / Sr Principal Engineer >> >> +1 (214) 425-0783/ >> [email protected]<mailto:[email protected]> >> >> BISS, Inc. Office: +1 (214) 506-8354 >> >> 4925 Greenville Ave Suite 900 >> >> Dallas, TX 75206 >> >> support.bissinc.com<http://htmlsig.com/support.bissinc.com> >> >> This e-mail message may contain confidential or legally privileged >> information and is intended only for the use of the intended recipient(s). >> Any unauthorized disclosure, dissemination, distribution, copying or the >> taking of any action in reliance on the information herein is prohibited. >> E-mails are not secure and cannot be guaranteed to be error free as they can >> be intercepted, amended, or contain viruses. Anyone who communicates with us >> by e-mail is deemed to have accepted these risks. Company Name is not >> responsible for errors or omissions in this message and denies any >> responsibility for any damage arising from the use of e-mail. Any opinion >> and other statement contained in this message and any attachment are solely >> those of the author and do not necessarily represent those of the company. >> >> From: John Barfield >> <[email protected]<mailto:[email protected]>> >> Date: Friday, January 8, 2016 at 10:31 AM >> To: >> "[email protected]<mailto:[email protected]>" >> >> <[email protected]<mailto:[email protected]>> >> Subject: joyent_20150108T111855Z Kernel Panic >> >> Greetings List, >> >> Brace yourselves: >> >> I’m writing because I had 3 servers give me kernel panics yesterday and >> reboot themselves around the same time. (Effectively rebooting about 20 VM’s >> as well). >> >> /var/adm/messages available here: >> http://download.txtelecom.net/server-34-36-37.tar.gz for all 3 servers. >> >> If you have the time and patience, please let me know what you think :) >> >> I’m happy to provide more information if needed and potential partial dumps >> if someone can tell me a little bit more about how to get them and If its of >> any use. >> >> Also, let me know if I should post this to the illumos list as well or >> instead. >> >> John Barfield >> >> ________________________________ >> >> >> ### Rambling thoughts and info follow starting here: >> >> All 3 servers are using joyent_20150108T111855Z kernel on Sunfire X4170 M2’s >> with 144GB of RAM each. They have dual hex core Intel’s at 2.66ghz with >> hyper threading enabled. Zpool has L2Arc cache device and mirrored SLOG’s >> (All are SAS drives, all have 1 raidz1 vdev) >> >> Let me say that I am not a kernel engineer at all, but I started following >> Ben Rockwoods coredump guide trying to get a proper dump to analyze with mdb >> and it keeps failing with the following: >> >> 'mdb: failed to read 'platform' string from kernel: No such file or >> directory mdb: failed to initialize target: No such file or directory’ >> >> I’m not sure if I’m doing it wrong or not but besides the error, the logs >> actually had a message that said the coredump was unsuccessful. >> >> Anyway, since I don’t have a viable core dump, I went reading through the >> panic messages in the logs and just figured I would post them here for >> someone else to see what they think from /var/adm/messages. >> I could be way off but I think the kernel panic occurred when a VM wrote to >> a zvol due to Qemu/ZIO errors on servers 1 & 3, but then I see a ZFS buffer >> written to wrong cache on server 2. Could be a cascading event that started >> between VM’s on srv2 in theory. >> >> Anyway, please, if you get a chance, someone tell me what you think >> happened, and if an upgrade could resolve a bug (and which bug??). Unless >> someone thinks its naughty memory or something, blah, I’m just rambling. >> >> My thoughts on Potential bugs: >> https://www.illumos.org/issues/5504 >> http://www.listbox.com/member/archive/182191/2015/01/sort/time_rev/page/1/entry/11:84/20150126102848:0193750E-A570-11E4-93E5-83F5155B1524/ >> >> ### End Rambling. >> >> >> >> Relevant bits in the logs: >> >> Server 1: >> >> 2016-01-08T02:07:55.180609+00:00 00-21-28-b2-24-6a unix: [ID 836849 >> kern.notice] #012#015panic[cpu0]/thread=ffffff011765dc40: >> 2016-01-08T02:07:55.180621+00:00 00-21-28-b2-24-6a genunix: [ID 683410 >> kern.notice] BAD TRAP: type=e (#pf Page fault) rp=ffffff011765d890 >> addr=ffffff0b4d92d000 >> 2016-01-08T02:07:55.180627+00:00 00-21-28-b2-24-6a unix: [ID 100000 >> kern.notice] #012 >> 2016-01-08T02:07:55.180632+00:00 00-21-28-b2-24-6a unix: [ID 839527 >> kern.notice] zpool-zones: >> 2016-01-08T02:07:55.180637+00:00 00-21-28-b2-24-6a unix: [ID 753105 >> kern.notice] #pf Page fault >> 2016-01-08T02:07:55.180642+00:00 00-21-28-b2-24-6a unix: [ID 532287 >> kern.notice] Bad kernel fault at addr=0xffffff0b4d92d000 >> 2016-01-08T02:07:55.180658+00:00 00-21-28-b2-24-6a unix: [ID 243837 >> kern.notice] pid=90, pc=0xfffffffffb85414a, sp=0xffffff011765d988, >> eflags=0x10206 >> 2016-01-08T02:07:55.180669+00:00 00-21-28-b2-24-6a unix: [ID 211416 >> kern.notice] cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: >> 26f8<vmxe,xmme,fxsr,pge,mce,pae,pse,de> >> 2016-01-08T02:07:55.180675+00:00 00-21-28-b2-24-6a unix: [ID 624947 >> kern.notice] cr2: ffffff0b4d92d000 >> 2016-01-08T02:07:55.180680+00:00 00-21-28-b2-24-6a unix: [ID 625075 >> kern.notice] cr3: 1dc00000 >> 2016-01-08T02:07:55.180685+00:00 00-21-28-b2-24-6a unix: [ID 625715 >> kern.notice] cr8: c >> 2016-01-08T02:07:55.180690+00:00 00-21-28-b2-24-6a unix: [ID 100000 >> kern.notice] >> 2016-01-08T02:07:55.180695+00:00 00-21-28-b2-24-6a unix: [ID 592667 >> kern.notice] #011rdi: ffffff0b4d92d000 rsi: ffffff2b4973c000 rdx: >> 1000 >> 2016-01-08T02:07:55.180701+00:00 00-21-28-b2-24-6a unix: [ID 592667 >> kern.notice] #011rcx: 200 r8: 0 r9: >> ffffff26349d4000 >> 2016-01-08T02:07:55.180706+00:00 00-21-28-b2-24-6a unix: [ID 592667 >> kern.notice] #011rax: ffffff2b4d989b30 rbx: ffffff2b3e0f9498 rbp: >> ffffff011765d9b0 >> 2016-01-08T02:07:55.180711+00:00 00-21-28-b2-24-6a unix: [ID 592667 >> kern.notice] #011r10: 6 r11: 0 r12: >> 0 >> 2016-01-08T02:07:55.180716+00:00 00-21-28-b2-24-6a unix: [ID 592667 >> kern.notice] #011r13: 0 r14: 0 r15: >> 0 >> 2016-01-08T02:07:55.180721+00:00 00-21-28-b2-24-6a unix: [ID 592667 >> kern.notice] #011fsb: fffffd7fff071240 gsb: fffffffffbc32600 ds: >> 38 >> 2016-01-08T02:07:55.180726+00:00 00-21-28-b2-24-6a unix: [ID 592667 >> kern.notice] #011 es: 38 fs: 0 gs: >> 0 >> 2016-01-08T02:07:55.180731+00:00 00-21-28-b2-24-6a unix: [ID 592667 >> kern.notice] #011trp: e err: 2 rip: >> fffffffffb85414a >> 2016-01-08T02:07:55.180738+00:00 00-21-28-b2-24-6a unix: [ID 592667 >> kern.notice] #011 cs: 30 rfl: 10206 rsp: >> ffffff011765d988 >> 2016-01-08T02:07:55.180743+00:00 00-21-28-b2-24-6a unix: [ID 266532 >> kern.notice] #011 ss: 38 >> 2016-01-08T02:07:55.180750+00:00 00-21-28-b2-24-6a unix: [ID 100000 >> kern.notice] >> 2016-01-08T02:07:55.180756+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >> kern.notice] ffffff011765d770 unix:real_mode_stop_cpu_stage2_end+9fb3 () >> 2016-01-08T02:07:55.180761+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >> kern.notice] ffffff011765d880 unix:trap+db3 () >> 2016-01-08T02:07:55.180768+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >> kern.notice] ffffff011765d890 unix:cmntrap+e6 () >> 2016-01-08T02:07:55.180791+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >> kern.notice] ffffff011765d9b0 unix:bcopy+55a () >> 2016-01-08T02:07:55.180800+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >> kern.notice] ffffff011765da40 zfs:zio_done+342 () >> 2016-01-08T02:07:55.180807+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >> kern.notice] ffffff011765da70 zfs:zio_execute+78 () >> 2016-01-08T02:07:55.180814+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >> kern.notice] ffffff011765db30 genunix:taskq_thread+2d0 () >> 2016-01-08T02:07:55.180819+00:00 00-21-28-b2-24-6a genunix: [ID 655072 >> kern.notice] ffffff011765db40 unix:thread_start+8 () >> 2016-01-08T02:07:55.180826+00:00 00-21-28-b2-24-6a unix: [ID 100000 >> kern.notice] >> 2016-01-08T02:07:55.180834+00:00 00-21-28-b2-24-6a genunix: [ID 672855 >> kern.notice] syncing file systems... >> 2016-01-08T02:07:55.180840+00:00 00-21-28-b2-24-6a genunix: [ID 733762 >> kern.notice] 17 >> 2016-01-08T02:07:55.180847+00:00 00-21-28-b2-24-6a genunix: [ID 904073 >> kern.notice] done >> 2016-01-08T02:07:55.180854+00:00 00-21-28-b2-24-6a genunix: [ID 111219 >> kern.notice] dumping to /dev/zvol/dsk/zones/dump, offset 65536, content: >> kernel >> 2016-01-08T02:07:55.180860+00:00 00-21-28-b2-24-6a genunix: [ID 100000 >> kern.notice] >> 2016-01-08T02:07:55.180868+00:00 00-21-28-b2-24-6a genunix: [ID 665016 >> kern.notice] #015 51% done: 3210871 pages dumped, >> 2016-01-08T02:07:55.180874+00:00 00-21-28-b2-24-6a genunix: [ID 495082 >> kern.notice] dump failed: error 28 >> >> >> Server2: >> -01-08T03:12:39.904338+00:00 00-10-e0-0f-3f-a0 genunix: [ID 432124 >> kern.notice] buffer freed to wrong cache >> 2016-01-08T03:12:39.904347+00:00 00-10-e0-0f-3f-a0 genunix: [ID 815666 >> kern.notice] buffer was allocated from zio_data_buf_4096, >> 2016-01-08T03:12:39.904352+00:00 00-10-e0-0f-3f-a0 genunix: [ID 530907 >> kern.notice] caller attempting free to zio_data_buf_8192. >> 2016-01-08T03:12:39.904357+00:00 00-10-e0-0f-3f-a0 genunix: [ID 563406 >> kern.notice] buffer=ffffff1397a14000 bufctl=ffffff2a46f33238 cache: >> zio_data_buf_8192 >> 2016-01-08T03:12:39.904362+00:00 00-10-e0-0f-3f-a0 unix: [ID 836849 >> kern.notice] #012#015panic[cpu20]/thread=ffffff0116017c40: >> 2016-01-08T03:12:39.904366+00:00 00-10-e0-0f-3f-a0 genunix: [ID 812275 >> kern.notice] kernel heap corruption detected >> 2016-01-08T03:12:39.904381+00:00 00-10-e0-0f-3f-a0 unix: [ID 100000 >> kern.notice] #012 >> 2016-01-08T03:12:39.904391+00:00 00-10-e0-0f-3f-a0 genunix: [ID 802836 >> kern.notice] ffffff0116017a50 fffffffffba2c2f4 () >> 2016-01-08T03:12:39.904397+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >> kern.notice] ffffff0116017ab0 genunix:kmem_slab_free+c1 () >> 2016-01-08T03:12:39.904402+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >> kern.notice] ffffff0116017b20 genunix:kmem_magazine_destroy+6e () >> 2016-01-08T03:12:39.904406+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >> kern.notice] ffffff0116017b60 genunix:kmem_depot_ws_reap+5d () >> 2016-01-08T03:12:39.904411+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >> kern.notice] ffffff0116017c20 genunix:taskq_thread+2d0 () >> 2016-01-08T03:12:39.904416+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 >> kern.notice] ffffff0116017c30 unix:thread_start+8 () >> 2016-01-08T03:12:39.904420+00:00 00-10-e0-0f-3f-a0 unix: [ID 100000 >> kern.notice] >> >> >> Server3: >> 2016-01-07T23:30:29.392494+00:00 00-21-28-b2-4f-ea unix: [ID 836849 >> kern.notice] #012#015panic[cpu6]/thread=ffffff264340e880: >> 2016-01-07T23:30:29.392507+00:00 00-21-28-b2-4f-ea genunix: [ID 683410 >> kern.notice] BAD TRAP: type=e (#pf Page fault) rp=ffffff01171e0450 >> addr=ffffff1529242000 2016-01-07T23:30:29.392512+00:00 00-21-28-b2-4f-ea >> unix: [ID 100000 kern.notice] #012 2016-01-07T23:30:29.392518+00:00 >> 00-21-28-b2-4f-ea unix: [ID 839527 kern.notice] qemu-system-x86_: >> 2016-01-07T23:30:29.392523+00:00 00-21-28-b2-4f-ea unix: [ID 753105 >> kern.notice] #pf Page fault 2016-01-07T23:30:29.392528+00:00 >> 00-21-28-b2-4f-ea unix: [ID 532287 kern.notice] Bad kernel fault at >> addr=0xffffff1529242000 2016-01-07T23:30:29.392545+00:00 00-21-28-b2-4f-ea >> unix: [ID 243837 kern.notice] pid=4465, pc=0xfffffffff7dc5577, >> sp=0xffffff01171e0540, eflags=0x10282 2016-01-07T23:30:29.392556+00:00 >> 00-21-28-b2-4f-ea unix: [ID 211416 kern.notice] cr0: >> 80050033<pg,wp,ne,et,mp,pe> cr4: 26f8<vmxe,xmme,fxsr,pge,mce,pae,ps >e,de> 2016-01-07T23:30:29.392562+00:00 00-21-28-b2-4f-ea unix: [ID 624947 >kern.notice] cr2: ffffff1529242000 2016-01-07T23:30:29.392568+00:00 >00-21-28-b2-4f-ea unix: [ID 625075 kern.notice] cr3: 1213265000 >2016-01-07T23:30:29.392573+00:00 00-21-28-b2-4f-ea unix: [ID 625715 >kern.notice] cr8: c 2016-01-07T23:30:29.392578+00:00 00-21-28-b2-4f-ea unix: >[ID 100000 kern.notice] 2016-01-07T23:30:29.392583+00:00 00-21-28-b2-4f-ea >unix: [ID 592667 kern.notice] #011rdi: ffffff39b2128c04 rsi: ffffff39b2128c0d >rdx: ffffff1529243ff7 2016-01-07T23:30:29.392589+00:00 00-21-28-b2-4f-ea unix: >[ID 592667 kern.notice] #011rcx: 200000098393bc0 r8: ffffff39b2128c61 r9: >ffffff1529243ff8 2016-01-07T23:30:29.392594+00:00 00-21-28-b2-4f-ea unix: [ID >592667 kern.notice] #011rax: ffffff1529242000 rbx: ffffff1529242000 rbp: >ffffff01171e0610 2016-01-07T23:30:29.392600+00:00 00-21-28-b2-4f-ea unix: [ID >592667 kern.notice] #011r10: ffffff1529244000 r11: ffffff39b2128c59 r12: >ffffff152924200d 2016-01-07T23:30:29.392 >604+00:00 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011r13: df r14: >ffffff39b2128c12 r15: 0 2016-01-07T23:30:29.392610+00:00 00-21-28-b2-4f-ea >unix: [ID 592667 kern.notice] #011fsb: fffffd7fff07e240 gsb: ffffff26016beac0 >ds: 4b 2016-01-07T23:30:29.392615+00:00 00-21-28-b2-4f-ea unix: [ID 592667 >kern.notice] #011 es: 4b fs: 0 gs: 0 2016-01-07T23:30:29.392620+00:00 >00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011trp: e err: 2 rip: >fffffffff7dc5577 2016-01-07T23:30:29.392625+00:00 00-21-28-b2-4f-ea unix: [ID >592667 kern.notice] #011 cs: 30 rfl: 10282 rsp: ffffff01171e0540 >2016-01-07T23:30:29.392630+00:00 00-21-28-b2-4f-ea unix: [ID 266532 >kern.notice] #011 ss: 38 2016-01-07T23:30:29.392635+00:00 00-21-28-b2-4f-ea >unix: [ID 100000 kern.notice] 2016-01-07T23:30:29.392640+00:00 >00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0330 >unix:real_mode_stop_cpu_stage2_end+9fb3 () 2016-01-07T23:30:29.392645+00:00 >00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff0117 >1e0440 unix:trap+db3 () 2016-01-07T23:30:29.392650+00:00 00-21-28-b2-4f-ea >genunix: [ID 655072 kern.notice] ffffff01171e0450 unix:cmntrap+e6 () >2016-01-07T23:30:29.392667+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0610 zfs:LZ4_uncompress_unknownOutputSize+147 () >2016-01-07T23:30:29.392673+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0650 zfs:lz4_decompress+44 () >2016-01-07T23:30:29.392678+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0690 zfs:zio_decompress_data+4b () >2016-01-07T23:30:29.392682+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e06d0 zfs:zio_decompress+56 () >2016-01-07T23:30:29.392687+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0700 zfs:zio_pop_transforms+7a () >2016-01-07T23:30:29.392692+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0790 zfs:zio_done+21a () >2016-01-07T23:30:29.392697+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern. >notice] ffffff01171e07c0 zfs:zio_execute+78 () >2016-01-07T23:30:29.392702+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e07f0 zfs:zio_nowait+21 () >2016-01-07T23:30:29.392706+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e08e0 zfs:arc_read+4fe () >2016-01-07T23:30:29.392711+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0980 zfs:dbuf_read_impl+1a5 () >2016-01-07T23:30:29.392716+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e09f0 zfs:dbuf_read+de () >2016-01-07T23:30:29.392721+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0a50 zfs:dmu_tx_check_ioerr+76 () >2016-01-07T23:30:29.392729+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0b80 zfs:dmu_tx_count_write+3e7 () >2016-01-07T23:30:29.392737+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0bd0 zfs:dmu_tx_hold_write+5a () >2016-01-07T23:30:29.392744+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0c80 zfs:zvol_write+1bc () >2016-01-07T23:30:29.392755+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0cb0 genunix:cdev_write+2d () >2016-01-07T23:30:29.392764+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0d90 specfs:spec_write+4c1 () >2016-01-07T23:30:29.392770+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0e30 genunix:fop_write+8b () >2016-01-07T23:30:29.392777+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0f00 genunix:pwrite+194 () >2016-01-07T23:30:29.392800+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 >kern.notice] ffffff01171e0f10 unix:brand_sys_syscall+1f5 () >2016-01-07T23:30:29.392809+00:00 00-21-28-b2-4f-ea unix: [ID 100000 >kern.notice] 2016-01-07T23:30:29.392814+00:00 00-21-28-b2-4f-ea genunix: [ID >672855 kern.notice] syncing file systems... 2016-01-07T23:30:29.392822+00:00 >00-21-28-b2-4f-ea genunix: [ID 904073 kern.notice] done >> >> >> John Barfield / Sr Principal Engineer >> >> +1 (214) 425-0783/ >> [email protected]<mailto:[email protected]> >> >> BISS, Inc. Office: +1 (214) 506-8354 >> >> 4925 Greenville Ave Suite 900 >> >> Dallas, TX 75206 >> >> support.bissinc.com<http://htmlsig.com/support.bissinc.com> >> > > ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
