It just happened again:
2016-01-08T22:45:48.616347+00:00 00-21-28-b2-4f-ea genunix: [ID 432124 kern.notice] buffer freed to wrong cache 2016-01-08T22:45:48.616355+00:00 00-21-28-b2-4f-ea genunix: [ID 815666 kern.notice] buffer was allocated from zio_data_buf_4096, 2016-01-08T22:45:48.616360+00:00 00-21-28-b2-4f-ea genunix: [ID 530907 kern.notice] caller attempting free to zio_data_buf_8192. 2016-01-08T22:45:48.616365+00:00 00-21-28-b2-4f-ea genunix: [ID 563406 kern.notice] buffer=ffffff14b54a5000 bufctl=ffffff2cd356c020 cache: zio_data_buf_8192 2016-01-08T22:45:48.616370+00:00 00-21-28-b2-4f-ea unix: [ID 836849 kern.notice] #012#015panic[cpu6]/thread=ffffff0116017c40: 2016-01-08T22:45:48.616375+00:00 00-21-28-b2-4f-ea genunix: [ID 812275 kern.notice] kernel heap corruption detected 2016-01-08T22:45:48.616390+00:00 00-21-28-b2-4f-ea unix: [ID 100000 kern.notice] #012 2016-01-08T22:45:48.616402+00:00 00-21-28-b2-4f-ea genunix: [ID 802836 kern.notice] ffffff0116017a50 fffffffffba2c2f4 () 2016-01-08T22:45:48.616412+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff0116017ab0 genunix:kmem_slab_free+c1 () 2016-01-08T22:45:48.616420+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff0116017b20 genunix:kmem_magazine_destroy+6e () 2016-01-08T22:45:48.616428+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff0116017b60 genunix:kmem_depot_ws_reap+5d () 2016-01-08T22:45:48.616435+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff0116017c20 genunix:taskq_thread+2d0 () 2016-01-08T22:45:48.616441+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff0116017c30 unix:thread_start+8 () 2016-01-08T22:45:48.616449+00:00 00-21-28-b2-4f-ea unix: [ID 100000 kern.notice] 2016-01-08T22:45:48.616456+00:00 00-21-28-b2-4f-ea genunix: [ID 672855 kern.notice] syncing file systems... 2016-01-08T22:45:48.616464+00:00 00-21-28-b2-4f-ea genunix: [ID 904073 kern.notice] done 2016-01-08T22:45:48.616472+00:00 00-21-28-b2-4f-ea genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/zones/dump, offset 65536, content: kernel 2016-01-08T22:45:48.616480+00:00 00-21-28-b2-4f-ea genunix: [ID 100000 kern.notice] 2016-01-08T22:45:48.616488+00:00 00-21-28-b2-4f-ea genunix: [ID 665016 kern.notice] #015 36% done: 2962335 pages dumped, 2016-01-08T22:45:48.616495+00:00 00-21-28-b2-4f-ea genunix: [ID 495082 kern.notice] dump failed: error 28 John Barfield / Sr Principal Engineer +1 (214) 425-0783/ [email protected]<mailto:[email protected]> BISS, Inc. Office: +1 (214) 506-8354 4925 Greenville Ave Suite 900 Dallas, TX 75206 support.bissinc.com<http://htmlsig.com/support.bissinc.com> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. From: John Barfield <[email protected]<mailto:[email protected]>> Date: Friday, January 8, 2016 at 10:31 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: joyent_20150108T111855Z Kernel Panic Greetings List, Brace yourselves: I’m writing because I had 3 servers give me kernel panics yesterday and reboot themselves around the same time. (Effectively rebooting about 20 VM’s as well). /var/adm/messages available here: http://download.txtelecom.net/server-34-36-37.tar.gz for all 3 servers. If you have the time and patience, please let me know what you think :) I’m happy to provide more information if needed and potential partial dumps if someone can tell me a little bit more about how to get them and If its of any use. Also, let me know if I should post this to the illumos list as well or instead. John Barfield ________________________________ ### Rambling thoughts and info follow starting here: All 3 servers are using joyent_20150108T111855Z kernel on Sunfire X4170 M2’s with 144GB of RAM each. They have dual hex core Intel’s at 2.66ghz with hyper threading enabled. Zpool has L2Arc cache device and mirrored SLOG’s (All are SAS drives, all have 1 raidz1 vdev) Let me say that I am not a kernel engineer at all, but I started following Ben Rockwoods coredump guide trying to get a proper dump to analyze with mdb and it keeps failing with the following: 'mdb: failed to read 'platform' string from kernel: No such file or directory mdb: failed to initialize target: No such file or directory’ I’m not sure if I’m doing it wrong or not but besides the error, the logs actually had a message that said the coredump was unsuccessful. Anyway, since I don’t have a viable core dump, I went reading through the panic messages in the logs and just figured I would post them here for someone else to see what they think from /var/adm/messages. I could be way off but I think the kernel panic occurred when a VM wrote to a zvol due to Qemu/ZIO errors on servers 1 & 3, but then I see a ZFS buffer written to wrong cache on server 2. Could be a cascading event that started between VM’s on srv2 in theory. Anyway, please, if you get a chance, someone tell me what you think happened, and if an upgrade could resolve a bug (and which bug??). Unless someone thinks its naughty memory or something, blah, I’m just rambling. My thoughts on Potential bugs: https://www.illumos.org/issues/5504 http://www.listbox.com/member/archive/182191/2015/01/sort/time_rev/page/1/entry/11:84/20150126102848:0193750E-A570-11E4-93E5-83F5155B1524/ ### End Rambling. Relevant bits in the logs: Server 1: 2016-01-08T02:07:55.180609+00:00 00-21-28-b2-24-6a unix: [ID 836849 kern.notice] #012#015panic[cpu0]/thread=ffffff011765dc40: 2016-01-08T02:07:55.180621+00:00 00-21-28-b2-24-6a genunix: [ID 683410 kern.notice] BAD TRAP: type=e (#pf Page fault) rp=ffffff011765d890 addr=ffffff0b4d92d000 2016-01-08T02:07:55.180627+00:00 00-21-28-b2-24-6a unix: [ID 100000 kern.notice] #012 2016-01-08T02:07:55.180632+00:00 00-21-28-b2-24-6a unix: [ID 839527 kern.notice] zpool-zones: 2016-01-08T02:07:55.180637+00:00 00-21-28-b2-24-6a unix: [ID 753105 kern.notice] #pf Page fault 2016-01-08T02:07:55.180642+00:00 00-21-28-b2-24-6a unix: [ID 532287 kern.notice] Bad kernel fault at addr=0xffffff0b4d92d000 2016-01-08T02:07:55.180658+00:00 00-21-28-b2-24-6a unix: [ID 243837 kern.notice] pid=90, pc=0xfffffffffb85414a, sp=0xffffff011765d988, eflags=0x10206 2016-01-08T02:07:55.180669+00:00 00-21-28-b2-24-6a unix: [ID 211416 kern.notice] cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: 26f8<vmxe,xmme,fxsr,pge,mce,pae,pse,de> 2016-01-08T02:07:55.180675+00:00 00-21-28-b2-24-6a unix: [ID 624947 kern.notice] cr2: ffffff0b4d92d000 2016-01-08T02:07:55.180680+00:00 00-21-28-b2-24-6a unix: [ID 625075 kern.notice] cr3: 1dc00000 2016-01-08T02:07:55.180685+00:00 00-21-28-b2-24-6a unix: [ID 625715 kern.notice] cr8: c 2016-01-08T02:07:55.180690+00:00 00-21-28-b2-24-6a unix: [ID 100000 kern.notice] 2016-01-08T02:07:55.180695+00:00 00-21-28-b2-24-6a unix: [ID 592667 kern.notice] #011rdi: ffffff0b4d92d000 rsi: ffffff2b4973c000 rdx: 1000 2016-01-08T02:07:55.180701+00:00 00-21-28-b2-24-6a unix: [ID 592667 kern.notice] #011rcx: 200 r8: 0 r9: ffffff26349d4000 2016-01-08T02:07:55.180706+00:00 00-21-28-b2-24-6a unix: [ID 592667 kern.notice] #011rax: ffffff2b4d989b30 rbx: ffffff2b3e0f9498 rbp: ffffff011765d9b0 2016-01-08T02:07:55.180711+00:00 00-21-28-b2-24-6a unix: [ID 592667 kern.notice] #011r10: 6 r11: 0 r12: 0 2016-01-08T02:07:55.180716+00:00 00-21-28-b2-24-6a unix: [ID 592667 kern.notice] #011r13: 0 r14: 0 r15: 0 2016-01-08T02:07:55.180721+00:00 00-21-28-b2-24-6a unix: [ID 592667 kern.notice] #011fsb: fffffd7fff071240 gsb: fffffffffbc32600 ds: 38 2016-01-08T02:07:55.180726+00:00 00-21-28-b2-24-6a unix: [ID 592667 kern.notice] #011 es: 38 fs: 0 gs: 0 2016-01-08T02:07:55.180731+00:00 00-21-28-b2-24-6a unix: [ID 592667 kern.notice] #011trp: e err: 2 rip: fffffffffb85414a 2016-01-08T02:07:55.180738+00:00 00-21-28-b2-24-6a unix: [ID 592667 kern.notice] #011 cs: 30 rfl: 10206 rsp: ffffff011765d988 2016-01-08T02:07:55.180743+00:00 00-21-28-b2-24-6a unix: [ID 266532 kern.notice] #011 ss: 38 2016-01-08T02:07:55.180750+00:00 00-21-28-b2-24-6a unix: [ID 100000 kern.notice] 2016-01-08T02:07:55.180756+00:00 00-21-28-b2-24-6a genunix: [ID 655072 kern.notice] ffffff011765d770 unix:real_mode_stop_cpu_stage2_end+9fb3 () 2016-01-08T02:07:55.180761+00:00 00-21-28-b2-24-6a genunix: [ID 655072 kern.notice] ffffff011765d880 unix:trap+db3 () 2016-01-08T02:07:55.180768+00:00 00-21-28-b2-24-6a genunix: [ID 655072 kern.notice] ffffff011765d890 unix:cmntrap+e6 () 2016-01-08T02:07:55.180791+00:00 00-21-28-b2-24-6a genunix: [ID 655072 kern.notice] ffffff011765d9b0 unix:bcopy+55a () 2016-01-08T02:07:55.180800+00:00 00-21-28-b2-24-6a genunix: [ID 655072 kern.notice] ffffff011765da40 zfs:zio_done+342 () 2016-01-08T02:07:55.180807+00:00 00-21-28-b2-24-6a genunix: [ID 655072 kern.notice] ffffff011765da70 zfs:zio_execute+78 () 2016-01-08T02:07:55.180814+00:00 00-21-28-b2-24-6a genunix: [ID 655072 kern.notice] ffffff011765db30 genunix:taskq_thread+2d0 () 2016-01-08T02:07:55.180819+00:00 00-21-28-b2-24-6a genunix: [ID 655072 kern.notice] ffffff011765db40 unix:thread_start+8 () 2016-01-08T02:07:55.180826+00:00 00-21-28-b2-24-6a unix: [ID 100000 kern.notice] 2016-01-08T02:07:55.180834+00:00 00-21-28-b2-24-6a genunix: [ID 672855 kern.notice] syncing file systems... 2016-01-08T02:07:55.180840+00:00 00-21-28-b2-24-6a genunix: [ID 733762 kern.notice] 17 2016-01-08T02:07:55.180847+00:00 00-21-28-b2-24-6a genunix: [ID 904073 kern.notice] done 2016-01-08T02:07:55.180854+00:00 00-21-28-b2-24-6a genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/zones/dump, offset 65536, content: kernel 2016-01-08T02:07:55.180860+00:00 00-21-28-b2-24-6a genunix: [ID 100000 kern.notice] 2016-01-08T02:07:55.180868+00:00 00-21-28-b2-24-6a genunix: [ID 665016 kern.notice] #015 51% done: 3210871 pages dumped, 2016-01-08T02:07:55.180874+00:00 00-21-28-b2-24-6a genunix: [ID 495082 kern.notice] dump failed: error 28 Server2: -01-08T03:12:39.904338+00:00 00-10-e0-0f-3f-a0 genunix: [ID 432124 kern.notice] buffer freed to wrong cache 2016-01-08T03:12:39.904347+00:00 00-10-e0-0f-3f-a0 genunix: [ID 815666 kern.notice] buffer was allocated from zio_data_buf_4096, 2016-01-08T03:12:39.904352+00:00 00-10-e0-0f-3f-a0 genunix: [ID 530907 kern.notice] caller attempting free to zio_data_buf_8192. 2016-01-08T03:12:39.904357+00:00 00-10-e0-0f-3f-a0 genunix: [ID 563406 kern.notice] buffer=ffffff1397a14000 bufctl=ffffff2a46f33238 cache: zio_data_buf_8192 2016-01-08T03:12:39.904362+00:00 00-10-e0-0f-3f-a0 unix: [ID 836849 kern.notice] #012#015panic[cpu20]/thread=ffffff0116017c40: 2016-01-08T03:12:39.904366+00:00 00-10-e0-0f-3f-a0 genunix: [ID 812275 kern.notice] kernel heap corruption detected 2016-01-08T03:12:39.904381+00:00 00-10-e0-0f-3f-a0 unix: [ID 100000 kern.notice] #012 2016-01-08T03:12:39.904391+00:00 00-10-e0-0f-3f-a0 genunix: [ID 802836 kern.notice] ffffff0116017a50 fffffffffba2c2f4 () 2016-01-08T03:12:39.904397+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 kern.notice] ffffff0116017ab0 genunix:kmem_slab_free+c1 () 2016-01-08T03:12:39.904402+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 kern.notice] ffffff0116017b20 genunix:kmem_magazine_destroy+6e () 2016-01-08T03:12:39.904406+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 kern.notice] ffffff0116017b60 genunix:kmem_depot_ws_reap+5d () 2016-01-08T03:12:39.904411+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 kern.notice] ffffff0116017c20 genunix:taskq_thread+2d0 () 2016-01-08T03:12:39.904416+00:00 00-10-e0-0f-3f-a0 genunix: [ID 655072 kern.notice] ffffff0116017c30 unix:thread_start+8 () 2016-01-08T03:12:39.904420+00:00 00-10-e0-0f-3f-a0 unix: [ID 100000 kern.notice] Server3: 2016-01-07T23:30:29.392494+00:00 00-21-28-b2-4f-ea unix: [ID 836849 kern.notice] #012#015panic[cpu6]/thread=ffffff264340e880: 2016-01-07T23:30:29.392507+00:00 00-21-28-b2-4f-ea genunix: [ID 683410 kern.notice] BAD TRAP: type=e (#pf Page fault) rp=ffffff01171e0450 addr=ffffff1529242000 2016-01-07T23:30:29.392512+00:00 00-21-28-b2-4f-ea unix: [ID 100000 kern.notice] #012 2016-01-07T23:30:29.392518+00:00 00-21-28-b2-4f-ea unix: [ID 839527 kern.notice] qemu-system-x86_: 2016-01-07T23:30:29.392523+00:00 00-21-28-b2-4f-ea unix: [ID 753105 kern.notice] #pf Page fault 2016-01-07T23:30:29.392528+00:00 00-21-28-b2-4f-ea unix: [ID 532287 kern.notice] Bad kernel fault at addr=0xffffff1529242000 2016-01-07T23:30:29.392545+00:00 00-21-28-b2-4f-ea unix: [ID 243837 kern.notice] pid=4465, pc=0xfffffffff7dc5577, sp=0xffffff01171e0540, eflags=0x10282 2016-01-07T23:30:29.392556+00:00 00-21-28-b2-4f-ea unix: [ID 211416 kern.notice] cr0: 80050033<pg,wp,ne,et,mp,pe> cr4: 26f8<vmxe,xmme,fxsr,pge,mce,pae,pse,de> 2016-01-07T23:30:29.392562+00:00 00-21-28-b2-4f-ea unix: [ID 624947 kern.notice] cr2: ffffff1529242000 2016-01-07T23:30:29.392568+00:00 00-21-28-b2-4f-ea unix: [ID 625075 kern.notice] cr3: 1213265000 2016-01-07T23:30:29.392573+00:00 00-21-28-b2-4f-ea unix: [ID 625715 kern.notice] cr8: c 2016-01-07T23:30:29.392578+00:00 00-21-28-b2-4f-ea unix: [ID 100000 kern.notice] 2016-01-07T23:30:29.392583+00:00 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011rdi: ffffff39b2128c04 rsi: ffffff39b2128c0d rdx: ffffff1529243ff7 2016-01-07T23:30:29.392589+00:00 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011rcx: 200000098393bc0 r8: ffffff39b2128c61 r9: ffffff1529243ff8 2016-01-07T23:30:29.392594+00:00 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011rax: ffffff1529242000 rbx: ffffff1529242000 rbp: ffffff01171e0610 2016-01-07T23:30:29.392600+00:00 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011r10: ffffff1529244000 r11: ffffff39b2128c59 r12: ffffff152924200d 2016-01-07T23:30:29.392604+00:00 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011r13: df r14: ffffff39b2128c12 r15: 0 2016-01-07T23:30:29.392610+00:00 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011fsb: fffffd7fff07e240 gsb: ffffff26016beac0 ds: 4b 2016-01-07T23:30:29.392615+00:00 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011 es: 4b fs: 0 gs: 0 2016-01-07T23:30:29.392620+00:00 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011trp: e err: 2 rip: fffffffff7dc5577 2016-01-07T23:30:29.392625+00:00 00-21-28-b2-4f-ea unix: [ID 592667 kern.notice] #011 cs: 30 rfl: 10282 rsp: ffffff01171e0540 2016-01-07T23:30:29.392630+00:00 00-21-28-b2-4f-ea unix: [ID 266532 kern.notice] #011 ss: 38 2016-01-07T23:30:29.392635+00:00 00-21-28-b2-4f-ea unix: [ID 100000 kern.notice] 2016-01-07T23:30:29.392640+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0330 unix:real_mode_stop_cpu_stage2_end+9fb3 () 2016-01-07T23:30:29.392645+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0440 unix:trap+db3 () 2016-01-07T23:30:29.392650+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0450 unix:cmntrap+e6 () 2016-01-07T23:30:29.392667+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0610 zfs:LZ4_uncompress_unknownOutputSize+147 () 2016-01-07T23:30:29.392673+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0650 zfs:lz4_decompress+44 () 2016-01-07T23:30:29.392678+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0690 zfs:zio_decompress_data+4b () 2016-01-07T23:30:29.392682+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e06d0 zfs:zio_decompress+56 () 2016-01-07T23:30:29.392687+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0700 zfs:zio_pop_transforms+7a () 2016-01-07T23:30:29.392692+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0790 zfs:zio_done+21a () 2016-01-07T23:30:29.392697+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e07c0 zfs:zio_execute+78 () 2016-01-07T23:30:29.392702+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e07f0 zfs:zio_nowait+21 () 2016-01-07T23:30:29.392706+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e08e0 zfs:arc_read+4fe () 2016-01-07T23:30:29.392711+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0980 zfs:dbuf_read_impl+1a5 () 2016-01-07T23:30:29.392716+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e09f0 zfs:dbuf_read+de () 2016-01-07T23:30:29.392721+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0a50 zfs:dmu_tx_check_ioerr+76 () 2016-01-07T23:30:29.392729+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0b80 zfs:dmu_tx_count_write+3e7 () 2016-01-07T23:30:29.392737+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0bd0 zfs:dmu_tx_hold_write+5a () 2016-01-07T23:30:29.392744+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0c80 zfs:zvol_write+1bc () 2016-01-07T23:30:29.392755+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0cb0 genunix:cdev_write+2d () 2016-01-07T23:30:29.392764+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0d90 specfs:spec_write+4c1 () 2016-01-07T23:30:29.392770+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0e30 genunix:fop_write+8b () 2016-01-07T23:30:29.392777+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0f00 genunix:pwrite+194 () 2016-01-07T23:30:29.392800+00:00 00-21-28-b2-4f-ea genunix: [ID 655072 kern.notice] ffffff01171e0f10 unix:brand_sys_syscall+1f5 () 2016-01-07T23:30:29.392809+00:00 00-21-28-b2-4f-ea unix: [ID 100000 kern.notice] 2016-01-07T23:30:29.392814+00:00 00-21-28-b2-4f-ea genunix: [ID 672855 kern.notice] syncing file systems... 2016-01-07T23:30:29.392822+00:00 00-21-28-b2-4f-ea genunix: [ID 904073 kern.notice] done John Barfield / Sr Principal Engineer +1 (214) 425-0783/ [email protected]<mailto:[email protected]> BISS, Inc. Office: +1 (214) 506-8354 4925 Greenville Ave Suite 900 Dallas, TX 75206 support.bissinc.com<http://htmlsig.com/support.bissinc.com> ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
