Re: [OmniOS-discuss] [zfs] FMD fails to run

2018-03-19 Thread Schweiss, Chip 
The fault manager is starting now.  However, the disks still as show as
UNAVAIL when running zpool import.

# zpool import
   pool: hcp-arc01
 id: 11579406004081253836
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

hcp-arc01UNAVAIL
insufficient replicas
  raidz3-0   UNAVAIL
insufficient replicas
c0t5000C50093E3BE87d0p0  UNAVAIL
cannot open
c0t5000C50086B52EABd0p0  UNAVAIL
cannot open
c0t5000C50093F046A7d0p0  UNAVAIL
cannot open
c0t5000C50093E3086Fd0p0  UNAVAIL
cannot open
c0t5000C50093E85C07d0p0  UNAVAIL
cannot open
c0t5000C50093E3BED3d0p0  UNAVAIL
cannot open
c0t5000C50093E39267d0p0  UNAVAIL
cannot open
c0t5000C50093E309DBd0p0  UNAVAIL
cannot open
c0t5000C50093E31407d0p0  UNAVAIL
cannot open
c0t5000C50093E3885Bd0p0  UNAVAIL
cannot open
c0t5000C50093E344D7d0p0  UNAVAIL
cannot open
c0t5000C50093E332AFd0p0  UNAVAIL
cannot open
c0t5000C50093F04A2Fd0p0  UNAVAIL
cannot open
c0t5000C50093F04763d0p0  UNAVAIL
cannot open
c0t5000C50086B5DCE3d0p0  UNAVAIL
cannot open
c0t5000C50086B5CD37d0p0  UNAVAIL
cannot open
c0t5000C50086B5E263d0p0  UNAVAIL
cannot open
c0t5000C50086B5CD07d0p0  UNAVAIL
cannot open
c0t5000C50086B5DB3Bd0p0  UNAVAIL
cannot open
c0t5000C50086B5D95Fd0p0  UNAVAIL
cannot open
c0t5000C50086B566BBd0p0  UNAVAIL
cannot open
c0t5000C50086B5F38Fd0p0  UNAVAIL
cannot open
c0t5000C50093E37C97d0p0  UNAVAIL
cannot open
c0t5000C50093E3909Bd0p0  UNAVAIL
cannot open
  raidz3-1   UNAVAIL
insufficient replicas
c0t5000C50093E85C1Fd0p0  UNAVAIL
cannot open
c0t5000C50093E3A29Fd0p0  UNAVAIL
cannot open
c0t5000C50093E342BFd0p0  UNAVAIL
cannot open
c0t5000C50093E359DFd0p0  UNAVAIL
cannot open
c0t5000C50086B5281Fd0p0  UNAVAIL
cannot open
c0t5000C50093E331F7d0p0  UNAVAIL
cannot open
c0t5000C50093E35A93d0p0  UNAVAIL
cannot open
c0t5000C50093E38347d0p0  UNAVAIL
cannot open
c0t5000C50093E8532Bd0p0  UNAVAIL
cannot open
c0t5000C50093E3422Fd0p0  UNAVAIL
cannot open
c0t5000C50093CFA493d0p0  UNAVAIL
cannot open
c0t5000C50093E29DB3d0p0  UNAVAIL
cannot open
c0t5000C50093E3B70Bd0p0  UNAVAIL
cannot open
c0t5000C50093E3946Fd0p0  UNAVAIL
cannot open
c0t5000C50086B5319Bd0p0  UNAVAIL
cannot open
c0t5000C50086B5608Bd0p0  UNAVAIL
cannot open
c0t5000C50086B5D9B7d0p0  UNAVAIL
cannot open
c0t5000C50086B5E1ABd0p0  UNAVAIL
cannot open
c0t5000C50093E85D93d0p0  UNAVAIL
cannot open
c0t5000C50093E85C73d0p0  UNAVAIL
cannot open
c0t5000C50086B5D7CBd0p0  UNAVAIL
cannot open
c0t5000C50093E33F23d0p0  UNAVAIL
cannot open
c0t5000C50093E36A8Fd0p0  UNAVAIL
cannot open
c0t5000C50093E30193d0p0  UNAVAIL
cannot open
  raidz3-2   UNAVAIL
insufficient replicas
c0t5000C50093E34E3Fd0p0  UNAVAIL
cannot open
c0t5000C50093E36DB7d0p0  UNAVAIL
cannot open
c0t5000C50093E2C467d0p0  UNAVAIL
cannot open
c0t5000C50093E3A213d0p0  UNAVAIL
cannot open

Re: [OmniOS-discuss] [zfs] FMD fails to run

2018-03-19 Thread Andy Fiddaman 

Chip,

Try this hot-fix:

# pkg apply-hot-fix --be-name=9317 \
https://downloads.omniosce.org/pkg/r151024/9317-ses2.p5p

If it fixes your problem I'll work on upstreaming this fix to illumos-gate
and it will definitely be in the upcoming OmniOS r151026 as well as
being backported to r151022 & r151024.

Regards,

Andy

-- 
Citrus IT Limited | +44 (0)333 0124 007 | enquir...@citrus-it.co.uk
Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ
Registered in England and Wales | Company number 4899123

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] [zfs] FMD fails to run

2018-03-19 Thread Andy Fiddaman 

On Mon, 19 Mar 2018, Schweiss, Chip wrote:

; You might be on to something.  I was suspicious of Element 96 when
; examining via sg_ses.  This JBOD has 96 slots and no display panel.  The
; vendor suspected other issues.

Well, it's this element:

value='IO Module   PCA-00610-01-A  USE2600052CMK0162B'

which has no PHYs:

ses2_aes_descr_sas1_impl_t {
uint8_t sadsi_n_phy_descriptors = 0
unsigned char _reserved1 :6 = 0
unsigned char sadsi_descriptor_type :2 = 0x1
uint8_t [2] _reserved2 = [ 0, 0 ]
ses2_aes_phy1_descr_impl_t [1] sadsi_phys = [
ses2_aes_phy1_descr_impl_t {
uint8_t sapdi_phy_identifier = 0x16
uint8_t _reserved1 = 0x6e
uint8_t sapdi_connector_element_index = 0
uint8_t sapdi_other_element_index = 0x74
uint64_t sapdi_sas_address = 0xd29300504030
},
]
}

I'm building the hot-fix now - takes about an hour.

Andy

-- 
Citrus IT Limited | +44 (0)333 0124 007 | enquir...@citrus-it.co.uk
Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ
Registered in England and Wales | Company number 4899123

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] [zfs] FMD fails to run

2018-03-19 Thread Schweiss, Chip 
On Mon, Mar 19, 2018 at 9:33 AM, Andy Fiddaman  wrote:

>
> On Mon, 19 Mar 2018, Schweiss, Chip wrote:
>
> ; On Mon, Mar 19, 2018 at 9:19 AM, Andy Fiddaman 
> wrote:
> ;
> ; >
> ; > I'll have a look at this for you and get a hot-fix built. I have the
> core
> ; > file that you made available so just need to go through and work out
> why
> ; > it thinks there are 0 phys somewhere.
> ; >
> ; >
> ; Many thanks!
> ;
> ; In discusscussion with JBOD vendor support, this JBOD has two SAS
> ; expanders, which are linked together.  One is likely incorrectly
> reporting
> ; 0 and should be ignored.
>
> The device is identifying as ESC_ELECTRONICS rather than a SAS_EXPANDER but
> I'll do some more digging.
>
>
You might be on to something.  I was suspicious of Element 96 when
examining via sg_ses.  This JBOD has 96 slots and no display panel.  The
vendor suspected other issues.

sg_ses -p ed /dev/es/ses1
  RAIDINC   96BAY 1715
  Primary enclosure logical identifier (hex): 500093d230938000
Element Descriptor In diagnostic page:
  generation code: 0x1
  element descriptor list (grouped by type):
Element type: Array device slot, subenclosure id: 0 [ti=0]
  Overall descriptor: Array Dev Slot
  Element 0 descriptor: SLOT 01 11
  Element 1 descriptor: SLOT 02 12
  Element 2 descriptor: SLOT 03 13
  Element 3 descriptor: SLOT 04 14
  Element 4 descriptor: SLOT 05 15
  Element 5 descriptor: SLOT 06 16
  Element 6 descriptor: SLOT 07 17
  Element 7 descriptor: SLOT 08 18
  Element 8 descriptor: SLOT 09 19
  Element 9 descriptor: SLOT 10 1A
  Element 10 descriptor: SLOT 11 1B
  Element 11 descriptor: SLOT 12 1C
  Element 12 descriptor: SLOT 13 1D
  Element 13 descriptor: SLOT 14 1E
  Element 14 descriptor: SLOT 15 21
  Element 15 descriptor: SLOT 16 22
  Element 16 descriptor: SLOT 17 23
  Element 17 descriptor: SLOT 18 24
  Element 18 descriptor: SLOT 19 25
  Element 19 descriptor: SLOT 20 26
  Element 20 descriptor: SLOT 21 27
  Element 21 descriptor: SLOT 22 28
  Element 22 descriptor: SLOT 23 29
  Element 23 descriptor: SLOT 24 2A
  Element 24 descriptor: SLOT 25 2B
  Element 25 descriptor: SLOT 26 2C
  Element 26 descriptor: SLOT 27 2D
  Element 27 descriptor: SLOT 28 2E
  Element 28 descriptor: SLOT 29 31
  Element 29 descriptor: SLOT 30 32
  Element 30 descriptor: SLOT 31 33
  Element 31 descriptor: SLOT 32 34
  Element 32 descriptor: SLOT 33 35
  Element 33 descriptor: SLOT 34 36
  Element 34 descriptor: SLOT 35 37
  Element 35 descriptor: SLOT 36 38
  Element 36 descriptor: SLOT 37 39
  Element 37 descriptor: SLOT 38 3A
  Element 38 descriptor: SLOT 39 3B
  Element 39 descriptor: SLOT 40 3C
  Element 40 descriptor: SLOT 41 3D
  Element 41 descriptor: SLOT 42 3E
  Element 42 descriptor: SLOT 43 41
  Element 43 descriptor: SLOT 44 42
  Element 44 descriptor: SLOT 45 43
  Element 45 descriptor: SLOT 46 44
  Element 46 descriptor: SLOT 47 45
  Element 47 descriptor: SLOT 48 46
  Element 48 descriptor: SLOT 49 47
  Element 49 descriptor: SLOT 50 49
  Element 50 descriptor: SLOT 51 4A
  Element 51 descriptor: SLOT 52 4B
  Element 52 descriptor: SLOT 53 4C
  Element 53 descriptor: SLOT 54 4D
  Element 54 descriptor: SLOT 55 4E
  Element 55 descriptor: SLOT 56 51
  Element 56 descriptor: SLOT 57 52
  Element 57 descriptor: SLOT 58 53
  Element 58 descriptor: SLOT 59 54
  Element 59 descriptor: SLOT 60 55
  Element 60 descriptor: SLOT 61 56
  Element 61 descriptor: SLOT 62 57
  Element 62 descriptor: SLOT 63 59
  Element 63 descriptor: SLOT 64 5A
  Element 64 descriptor: SLOT 65 5B
  Element 65 descriptor: SLOT 66 5C
  Element 66 descriptor: SLOT 67 5D
  Element 67 descriptor: SLOT 68 5E
  Element 68 descriptor: SLOT 69 61
  Element 69 descriptor: SLOT 70 62
  Element 70 descriptor: SLOT 71 63
  Element 71 descriptor: SLOT 72 64
  Element 72 descriptor: SLOT 73 65
  Element 73 descriptor: SLOT 74 66
  Element 74 descriptor: SLOT 75 67
  Element 75 descriptor: SLOT 76 68
  Element 76 descriptor: SLOT 77 69
  Element 77 descriptor: SLOT 78 6A
  Element 78 descriptor: SLOT 79 6B
  Element 79 descriptor: SLOT 80 6C
  Element 80 descriptor: SLOT 81 6D
  Element 81 descriptor: SLOT 82 6E
  Element 82 descriptor: SLOT 83 71
  Element 83 descriptor: SLOT 84 72
  Element 84 descriptor: SLOT 85 73
  Element 85 descriptor: SLOT 86 74
  Element 86 descriptor: SLOT 87 75
  Element 87 descriptor: SLOT 88 76
  Element 88 descriptor: SLOT 89 77
  Element 89 descriptor: SLOT 90 78
  Element 90 descriptor: SLOT 91 79
  Element 91 descriptor: SLOT 92 7A
  Element 92 descriptor: SLOT 93 7B
  Element 93 descriptor:

Re: [OmniOS-discuss] [zfs] FMD fails to run

2018-03-19 Thread Andy Fiddaman 

On Mon, 19 Mar 2018, Schweiss, Chip wrote:

; On Mon, Mar 19, 2018 at 9:19 AM, Andy Fiddaman  wrote:
;
; >
; > I'll have a look at this for you and get a hot-fix built. I have the core
; > file that you made available so just need to go through and work out why
; > it thinks there are 0 phys somewhere.
; >
; >
; Many thanks!
;
; In discusscussion with JBOD vendor support, this JBOD has two SAS
; expanders, which are linked together.  One is likely incorrectly reporting
; 0 and should be ignored.

The device is identifying as ESC_ELECTRONICS rather than a SAS_EXPANDER but
I'll do some more digging.

Andy

-- 
Citrus IT Limited | +44 (0)333 0124 007 | enquir...@citrus-it.co.uk
Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ
Registered in England and Wales | Company number 4899123

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] [zfs] FMD fails to run

2018-03-19 Thread Schweiss, Chip 
On Mon, Mar 19, 2018 at 9:19 AM, Andy Fiddaman  wrote:

>
> I'll have a look at this for you and get a hot-fix built. I have the core
> file that you made available so just need to go through and work out why
> it thinks there are 0 phys somewhere.
>
>
Many thanks!

In discusscussion with JBOD vendor support, this JBOD has two SAS
expanders, which are linked together.  One is likely incorrectly reporting
0 and should be ignored.

-Chip
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] [zfs] FMD fails to run

2018-03-19 Thread Andy Fiddaman 

On Mon, 19 Mar 2018, Schweiss, Chip wrote:

; Looks like I'm dead in the water to make this work with Illumos until this
; bug is fixed.

Chip,

I'll have a look at this for you and get a hot-fix built. I have the core
file that you made available so just need to go through and work out why
it thinks there are 0 phys somewhere.

Andy

-- 
Citrus IT Limited | +44 (0)333 0124 007 | enquir...@citrus-it.co.uk
Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ
Registered in England and Wales | Company number 4899123

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] [zfs] FMD fails to run

2018-03-19 Thread Schweiss, Chip 
Even unloading all the modules except 'fmd-self-diagnosis' which will not
unload, fmd still dies as soon as I plug in the JBOD.

# fmadm config
MODULE   VERSION STATUS  DESCRIPTION
fmd-self-diagnosis   1.0 active  Fault Manager Self-Diagnosis

# fmadm unload fmd-self-diagnosis
fmadm: failed to unload fmd-self-diagnosis: module is in use and cannot be
unloaded

Looks like I'm dead in the water to make this work with Illumos until this
bug is fixed.

-Chip

On Fri, Mar 16, 2018 at 3:42 PM, Richard Elling <
richard.ell...@richardelling.com> wrote:

> fmadm allows you to load/unload modules.
>  -- richard
>
> On Mar 16, 2018, at 8:24 AM, Schweiss, Chip  wrote:
>
> I need to get this JBOD working with OmniOS.  Is there a way to get FMD to
> ignore this SES device until this issue is fixed?
>
> It is a RAID, Inc. 4U 96-Bay  http://www.raidinc.
> com/products/object-storage/ability-4u-96-bay
>
> -Chip
>
> On Fri, Mar 16, 2018 at 9:18 AM, Schweiss, Chip 
> wrote:
>
>> While this problem was originally ruled out as an artifact of running as
>> a virtual machine, I've now installed the same HBA and JBOD to a physical
>> server.   The problem is exactly the same.
>>
>> This is on OmniOS CE r151024r
>>
>> -Chip
>>
>> # /usr/lib/fm/fmd/fmd -o fg=true -o client.debug=true
>> fmd: [ loading modules ... ABORT: attempted zero-length allocation:
>> Operation not supported
>> Abort (core dumped)
>>
>> > $C
>> 080462a8 libc.so.1`_lwp_kill+0x15(1, 6, 80462f8, fef42000, fef42000,
>> 8046330)
>> 080462c8 libc.so.1`raise+0x2b(6, 0, 80462e0, feec1b59, 0, 0)
>> 08046318 libc.so.1`abort+0x10e(fead51f0, 0, fede2a40, 30, 524f4241,
>> 61203a54)
>> 08046748 libses.so.1`ses_panic(fdde6758, 8046774, 80467e8, fdb6b67a,
>> 83eb0a8, fdb6c398)
>> 08046768 libses.so.1`ses_realloc(fdde6758, 0, 83f01b8, fdde6130,
>> fddf7000, fdb6658f)
>> 08046788 libses.so.1`ses_alloc+0x27(0, feb8, 6, 10, ee0, 8111627)
>> 080467b8 libses.so.1`ses_zalloc+0x1e(0, 0, 73, fdb6659d, 83f0190, 8)
>> 08046838 ses2.so`elem_parse_aes_misc+0x91(81114f4, 83eb0a8, 8, fdb65d85)
>> 08046888 ses2.so`elem_parse_aes+0xfc(82f1ac8, 83f0288, 80468f8, fdb80eae)
>> 080468a8 ses2.so`ses2_fill_element_node+0x37(82f1ac8, 83f0288, 832e930,
>> 4)
>> 080468d8 ses2.so`ses2_node_parse+0x53(82f1ac8, 83f0288, e, fddf7000)
>> 080468f8 libses.so.1`ses_fill_node+0x22(83f0288, 83f0348, fdde38ae,
>> fdde394c)
>> 08046918 libses.so.1`ses_fill_tree+0x21(83f0288, 82f1c88, 83e4cc8,
>> fdde394c)
>> 08046938 libses.so.1`ses_fill_tree+0x33(82f1d88, 82f1b88, 8046968,
>> fdde394c)
>> 08046958 libses.so.1`ses_fill_tree+0x33(82f1c88, 82ef758, 8046998,
>> fdde394c)
>> 08046978 libses.so.1`ses_fill_tree+0x33(82f1b88, 0, 18, fddf7000)
>> 08046998 libses.so.1`ses_fill_snap+0x22(82f08a0, 80, 0, fdde56eb)
>> 080469e8 libses.so.1`ses_snap_new+0x325(82f1b48, 0, 8046a18, fdde3006)
>> 08046a18 libses.so.1`ses_open_scsi+0xc4(1, 82ef688, 8046aa0, fed71c1b,
>> 81053f8, fede4042)
>> 08046a68 libses.so.1`ses_open+0x98(1, 8046aa0, 0, feecedd3, 43, fde1fc58)
>> 08046eb8 ses.so`ses_process_dir+0x133(fde20159, 83cc348, 0, fed77e40)
>> 08046ee8 ses.so`ses_enum+0xc1(81053f8, 82f21a0, 8386608, 0, 400, 0)
>> 08046f38 libtopo.so.1`topo_mod_enumerate+0xc4(81053f8, 82f21a0, 82fb1c8,
>> 8386608, 0, 400)
>> 08046f88 libtopo.so.1`enum_run+0xe9(8105a18, 83d6f78, a, fed7b1dd)
>> 08046fd8 libtopo.so.1`topo_xml_range_process+0x13e(8105a18, 82eb5b0,
>> 83d6f78, 8047008)
>> 08047028 libtopo.so.1`tf_rdata_new+0x135(8105a18, 82dfde0, 82eb5b0,
>> 82f21a0)
>> 08047088 libtopo.so.1`topo_xml_walk+0x246(8105a18, 82dfde0, 82ebd30,
>> 82f21a0, 8105a18, 83cbac0)
>> 080470e8 libtopo.so.1`topo_xml_walk+0x1b2(8105a18, 82dfde0, 82de080,
>> 82f21a0)
>> 08047128 libtopo.so.1`dependent_create+0x127(8105a18, 82dfde0, 83d3aa0,
>> 82de080, 82f21a0, fed7b1f9)
>> 08047168 libtopo.so.1`dependents_create+0x64(8105a18, 82dfde0, 83d3aa0,
>> 82de300, 82f21a0, 81eb0d8)
>> 08047218 libtopo.so.1`pad_process+0x51e(8105a18, 83ce100, 82de300,
>> 82f21a0, 83ce128, 81d8638)
>> 08047278 libtopo.so.1`topo_xml_range_process+0x31f(8105a18, 82de300,
>> 83ce100, 80472a8)
>> 080472c8 libtopo.so.1`tf_rdata_new+0x135(8105a18, 82dfde0, 82de300,
>> 81eb198)
>> 08047328 libtopo.so.1`topo_xml_walk+0x246(8105a18, 82dfde0, 82d1ca0,
>> 81eb198, 8103f40, fed8c000)
>> 08047358 libtopo.so.1`topo_xml_enum+0x67(8105a18, 82dfde0, 81eb198,
>> feac2000)
>> 08047488 libtopo.so.1`topo_file_load+0x139(8105a18, 81eb198, fe20c127,
>> fe20bda2, 0, 82d2000)
>> 080474b8 libtopo.so.1`topo_mod_enummap+0x26(8105a18, 81eb198, fe20c127,
>> fe20bda2, 8105a18, fe20b11c)
>> 08047508 x86pi.so`x86pi_enum_start+0xc5(8105a18, 8047530, 8047538,
>> fe205580, 8105a18, 8105a18)
>> 08047558 x86pi.so`x86pi_enum+0x55(8105a18, 81eb198, 81d8a90, 0, 0, 0)
>> 080475a8 libtopo.so.1`topo_mod_enumerate+0xc4(8105a18, 81eb198, 80ebf38,
>> 81d8a90, 0, 0)
>> 080475f8 libtopo.so.1`enum_run+0xe9(8105b68, 81f1070, a, fed7b1dd)
>> 08047648