OK, it appears that the solution was some heavy-handed use of the ipcs and ipcrm commands. I'd run out of semaphores.
Sorry for the noise, although this might help someone else in the future, maybe. Ben -- Unix Support, UIS, University of Cambridge, England > -----Original Message----- > From: [email protected] [mailto:linux-poweredge- > [email protected]] On Behalf Of Ben Argyle > Sent: 15 February 2017 16:38 > To: [email protected] > Subject: [Linux-PowerEdge] omreport suddenly not giving information > > I've got a 720xd (revision I) running RHEL6 and OMSA 8.4.0 with the > following firmware: > > BIOS 2.5.4 > iDRAC 2.32.31.30 > > About half an hour ago it started not giving any output for "omreport > chassis". I restarted OMSA with "/opt/dell/srvadmin/sbin/srvadm- > service.sh restart" but still got no joy. I logged into the DRAC via SSH and > did > "racadm racreset soft", waited for it to come back and then tried again (after > restarting OMSA services again). No joy. But now, in addition, "omreport > system" and "omreport storage controller" also don't return information. In > the latter case omreport states "No controllers found". > > Can anyone tell me what's gone wrong? The OS is still working perfectly, > and the DRAC GUI _seems_ to be giving me all the usual information, but > omreport within the OS isn't. > > In addition dsu hangs when doing this: > > # dsu --inventory > Verifying catalog installation ... > Installing catalog from repository ... > Fetching dsucatalog ... > Reading the catalog ... > Installing inventory collector ... > Fetching invcol_WF06C_LN64_16.12.200.896_A00 ... > Verifying inventory collector installation ... > Getting System Inventory ... > > Below is /var/log/messages from when I did the first "srvadm-services.sh > restart". There are no errors or warnings above it. This server has a lot > of FC > mounts and unmounts happening on it. Any thoughts? > > Feb 15 15:49:58 albion dataeng: dsm_sa_snmpd shutdown succeeded > Feb 15 15:50:00 albion dataeng: dsm_sa_eventmgrd shutdown succeeded > Feb 15 15:50:08 albion dataeng: dsm_sa_datamgrd shutdown succeeded > Feb 15 15:50:09 albion instsvcdrv: dell_rbu device driver unloaded > Feb 15 15:50:09 albion instsvcdrv: dell_rbu device driver loaded > Feb 15 15:50:20 albion dataeng: warning: snmpd not started. snmpd must be > started to manage this system using SNMP. > Feb 15 16:01:46 albion kernel: usb 1-1.6: USB disconnect, device number 4 > Feb 15 16:01:46 albion kernel: usb 1-1.6.1: USB disconnect, device number 5 > Feb 15 16:01:56 albion kernel: usb 1-1.6: new high-speed USB device number > 6 using ehci-pci > Feb 15 16:01:57 albion kernel: usb 1-1.6: New USB device found, > idVendor=413c, idProduct=a001 > Feb 15 16:01:57 albion kernel: usb 1-1.6: New USB device strings: Mfr=1, > Product=2, SerialNumber=3 > Feb 15 16:01:57 albion kernel: usb 1-1.6: Product: Gadget USB HUB > Feb 15 16:01:57 albion kernel: usb 1-1.6: Manufacturer: no manufacturer > Feb 15 16:01:57 albion kernel: usb 1-1.6: SerialNumber: 0123456789 > Feb 15 16:01:57 albion kernel: hub 1-1.6:1.0: USB hub found > Feb 15 16:01:57 albion kernel: hub 1-1.6:1.0: 6 ports detected > Feb 15 16:03:00 albion kernel: usb 1-1.6.1: new high-speed USB device > number 7 using ehci-pci > Feb 15 16:03:00 albion kernel: usb 1-1.6.1: New USB device found, > idVendor=0624, idProduct=0249 > Feb 15 16:03:00 albion kernel: usb 1-1.6.1: New USB device strings: Mfr=4, > Product=5, SerialNumber=6 > Feb 15 16:03:00 albion kernel: usb 1-1.6.1: Product: Keyboard/Mouse > Function > Feb 15 16:03:00 albion kernel: usb 1-1.6.1: Manufacturer: Avocent > Feb 15 16:03:00 albion kernel: usb 1-1.6.1: SerialNumber: 20121018 > Feb 15 16:03:00 albion kernel: input: Avocent Keyboard/Mouse Function as > /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1- > 1.6.1:1.0/input/input5 > Feb 15 16:03:00 albion kernel: hid-generic 0003:0624:0249.0004: > input,hidraw0: USB HID v1.00 Keyboard [Avocent Keyboard/Mouse > Function] on usb-0000:00:1a.0-1.6.1/input0 > Feb 15 16:03:00 albion kernel: input: Avocent Keyboard/Mouse Function as > /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1- > 1.6.1:1.1/input/input6 > Feb 15 16:03:00 albion kernel: hid-generic 0003:0624:0249.0005: > input,hidraw1: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] > on usb-0000:00:1a.0-1.6.1/input1 > Feb 15 16:03:00 albion kernel: input: Avocent Keyboard/Mouse Function as > /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1- > 1.6.1:1.2/input/input7 > Feb 15 16:03:00 albion kernel: hid-generic 0003:0624:0249.0006: > input,hidraw2: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] > on usb-0000:00:1a.0-1.6.1/input2 > Feb 15 16:03:01 albion kernel: usb 1-1.6.3: new high-speed USB device > number 8 using ehci-pci > Feb 15 16:03:01 albion kernel: usb 1-1.6.3: New USB device found, > idVendor=413c, idProduct=a102 > Feb 15 16:03:01 albion kernel: usb 1-1.6.3: New USB device strings: Mfr=1, > Product=2, SerialNumber=0 > Feb 15 16:03:01 albion kernel: usb 1-1.6.3: Product: iDRAC Virtual NIC USB > Device > Feb 15 16:03:01 albion kernel: usb 1-1.6.3: Manufacturer: Dell(TM) > Feb 15 16:03:01 albion kernel: cdc_ether 1-1.6.3:1.0 usb0: register > 'cdc_ether' > at usb-0000:00:1a.0-1.6.3, CDC Ethernet Device, ce:7f:94:9c:6a:22 > Feb 15 16:03:01 albion kernel: usbcore: registered new interface driver > cdc_ether > Feb 15 16:03:01 albion kernel: net usb0: 'usb0' renaming to 'idrac' > Feb 15 16:03:06 albion kernel: usb 1-1.6.3: USB disconnect, device number 8 > Feb 15 16:03:06 albion kernel: cdc_ether 1-1.6.3:1.0 idrac: unregister > 'cdc_ether' usb-0000:00:1a.0-1.6.3, CDC Ethernet Device > Feb 15 16:03:20 albion kernel: usb 1-1.6.1: USB disconnect, device number 7 > Feb 15 16:03:21 albion kernel: usb 1-1.6.1: new high-speed USB device > number 9 using ehci-pci > Feb 15 16:03:21 albion kernel: usb 1-1.6.1: New USB device found, > idVendor=0624, idProduct=0249 > Feb 15 16:03:21 albion kernel: usb 1-1.6.1: New USB device strings: Mfr=4, > Product=5, SerialNumber=6 > Feb 15 16:03:21 albion kernel: usb 1-1.6.1: Product: Keyboard/Mouse > Function > Feb 15 16:03:21 albion kernel: usb 1-1.6.1: Manufacturer: Avocent > Feb 15 16:03:21 albion kernel: usb 1-1.6.1: SerialNumber: 20121018 > Feb 15 16:03:21 albion kernel: input: Avocent Keyboard/Mouse Function as > /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1- > 1.6.1:1.0/input/input8 > Feb 15 16:03:21 albion kernel: hid-generic 0003:0624:0249.0007: > input,hidraw0: USB HID v1.00 Keyboard [Avocent Keyboard/Mouse > Function] on usb-0000:00:1a.0-1.6.1/input0 > Feb 15 16:03:21 albion kernel: input: Avocent Keyboard/Mouse Function as > /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1- > 1.6.1:1.1/input/input9 > Feb 15 16:03:21 albion kernel: hid-generic 0003:0624:0249.0008: > input,hidraw1: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] > on usb-0000:00:1a.0-1.6.1/input1 > Feb 15 16:03:21 albion kernel: input: Avocent Keyboard/Mouse Function as > /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1- > 1.6.1:1.2/input/input10 > Feb 15 16:03:21 albion kernel: hid-generic 0003:0624:0249.0009: > input,hidraw2: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] > on usb-0000:00:1a.0-1.6.1/input2 > Feb 15 16:04:21 albion dataeng: dsm_sa_snmpd shutdown succeeded > Feb 15 16:04:22 albion dataeng: dsm_sa_eventmgrd shutdown succeeded > Feb 15 16:04:29 albion dataeng: dsm_sa_datamgrd shutdown succeeded > Feb 15 16:04:30 albion instsvcdrv: dell_rbu device driver unloaded > Feb 15 16:04:30 albion instsvcdrv: dell_rbu device driver loaded > Feb 15 16:04:49 albion dataeng: warning: snmpd not started. snmpd must be > started to manage this system using SNMP. > Feb 15 16:15:23 albion yum[25501]: Erased: dsucatalog > Feb 15 16:15:39 albion yum[25512]: Installed: dsucatalog-17.01.00- > TDDR9.noarch > Feb 15 16:15:58 albion yum[25576]: Installed: > invcol_WF06C_LN64_16.12.200.896_A00-16.12.200.896-WF06C.x86_64 > Feb 15 16:16:14 albion kernel: Initializing USB Mass Storage driver... > Feb 15 16:16:14 albion kernel: usbcore: registered new interface driver usb- > storage > Feb 15 16:16:14 albion kernel: USB Mass Storage support registered. > Feb 15 16:16:16 albion kernel: usb 1-1.6.2: new high-speed USB device > number 10 using ehci-pci > Feb 15 16:16:16 albion kernel: usb 1-1.6.2: New USB device found, > idVendor=0624, idProduct=0250 > Feb 15 16:16:16 albion kernel: usb 1-1.6.2: New USB device strings: Mfr=4, > Product=5, SerialNumber=6 > Feb 15 16:16:16 albion kernel: usb 1-1.6.2: Product: Mass Storage Function > Feb 15 16:16:16 albion kernel: usb 1-1.6.2: Manufacturer: Avocent > Feb 15 16:16:16 albion kernel: usb 1-1.6.2: SerialNumber: 20120731 > Feb 15 16:16:16 albion kernel: scsi3 : usb-storage 1-1.6.2:1.0 > Feb 15 16:16:17 albion kernel: scsi 3:0:0:0: Direct-Access iDRAC SECUPD > 0329 PQ: 0 ANSI: 0 CCS > Feb 15 16:16:17 albion kernel: sd 3:0:0:0: Attached scsi generic sg40 type 0 > Feb 15 16:16:17 albion kernel: sd 3:0:0:0: [sdao] 2112 512-byte logical > blocks: > (1.08 MB/1.03 MiB) > Feb 15 16:16:17 albion kernel: sd 3:0:0:0: [sdao] Write Protect is off > Feb 15 16:16:17 albion kernel: sd 3:0:0:0: [sdao] No Caching mode page > found > Feb 15 16:16:17 albion kernel: sd 3:0:0:0: [sdao] Assuming drive cache: write > through > Feb 15 16:16:18 albion kernel: sd 3:0:0:0: [sdao] No Caching mode page > found > Feb 15 16:16:18 albion kernel: sd 3:0:0:0: [sdao] Assuming drive cache: write > through > Feb 15 16:16:18 albion kernel: sdao: > Feb 15 16:16:18 albion kernel: sd 3:0:0:0: [sdao] No Caching mode page > found > Feb 15 16:16:18 albion kernel: sd 3:0:0:0: [sdao] Assuming drive cache: write > through > Feb 15 16:16:18 albion kernel: sd 3:0:0:0: [sdao] Attached SCSI removable > disk > Feb 15 16:16:18 albion multipathd: sdao: add path (uevent) > Feb 15 16:16:18 albion multipathd: sdao: failed to get path uid > Feb 15 16:16:18 albion multipathd: uevent trigger error > Feb 15 16:16:34 albion kernel: usb 1-1.6.2: USB disconnect, device number 10 > Feb 15 16:16:34 albion multipathd: sdao: remove path (uevent) > Feb 15 16:16:39 albion kernel: usb 1-1.6.2: new high-speed USB device > number 11 using ehci-pci > Feb 15 16:16:39 albion kernel: usb 1-1.6.2: New USB device found, > idVendor=0624, idProduct=0250 > Feb 15 16:16:39 albion kernel: usb 1-1.6.2: New USB device strings: Mfr=4, > Product=5, SerialNumber=6 > Feb 15 16:16:39 albion kernel: usb 1-1.6.2: Product: Mass Storage Function > Feb 15 16:16:39 albion kernel: usb 1-1.6.2: Manufacturer: Avocent > Feb 15 16:16:40 albion kernel: usb 1-1.6.2: SerialNumber: 20120731 > Feb 15 16:16:40 albion kernel: scsi4 : usb-storage 1-1.6.2:1.0 > Feb 15 16:16:41 albion kernel: scsi 4:0:0:0: Direct-Access iDRAC SECUPD > 0329 PQ: 0 ANSI: 0 CCS > Feb 15 16:16:41 albion kernel: sd 4:0:0:0: Attached scsi generic sg40 type 0 > Feb 15 16:16:41 albion kernel: sd 4:0:0:0: [sdao] 2112 512-byte logical > blocks: > (1.08 MB/1.03 MiB) > Feb 15 16:16:41 albion kernel: sd 4:0:0:0: [sdao] Write Protect is off > Feb 15 16:16:41 albion kernel: sd 4:0:0:0: [sdao] No Caching mode page > found > Feb 15 16:16:41 albion kernel: sd 4:0:0:0: [sdao] Assuming drive cache: write > through > Feb 15 16:16:41 albion kernel: sd 4:0:0:0: [sdao] No Caching mode page > found > Feb 15 16:16:41 albion kernel: sd 4:0:0:0: [sdao] Assuming drive cache: write > through > Feb 15 16:16:41 albion kernel: sdao: > Feb 15 16:16:41 albion kernel: sd 4:0:0:0: [sdao] No Caching mode page > found > Feb 15 16:16:41 albion kernel: sd 4:0:0:0: [sdao] Assuming drive cache: write > through > Feb 15 16:16:41 albion kernel: sd 4:0:0:0: [sdao] Attached SCSI removable > disk > Feb 15 16:16:41 albion multipathd: sdao: add path (uevent) > Feb 15 16:16:41 albion multipathd: sdao: failed to get path uid > Feb 15 16:16:41 albion multipathd: uevent trigger error > Feb 15 16:16:56 albion kernel: usb 1-1.6.2: USB disconnect, device number 11 > Feb 15 16:16:56 albion multipathd: sdao: remove path (uevent) > Feb 15 16:16:59 albion kernel: usbcore: deregistering interface driver usb- > storage > Feb 15 16:17:07 albion kernel: dchcfg[31998]: segfault at 0 ip > 000000370813382f sp 00007ffd1346ad88 error 4 in libc- > 2.12.so[3708000000+18a000] > Feb 15 16:17:07 albion abrt[32005]: Saved core dump of pid 31998 > (/opt/dell/dup64/sbin/dchcfg) to /var/spool/abrt/ccpp-2017-02-15-16:17:07- > 31998 (454656 bytes) > Feb 15 16:17:07 albion kernel: dchcfg[32046]: segfault at 0 ip > 000000370813382f sp 00007ffe9dd0ad68 error 4 in libc- > 2.12.so[3708000000+18a000] > Feb 15 16:17:07 albion abrt[32052]: Not saving repeating crash in > '/opt/dell/dup64/sbin/dchcfg' > Feb 15 16:17:07 albion kernel: dchcfg[32090]: segfault at 0 ip > 000000370813382f sp 00007fffb917b1e8 error 4 in libc- > 2.12.so[3708000000+18a000] > Feb 15 16:17:07 albion abrt[32098]: Not saving repeating crash in > '/opt/dell/dup64/sbin/dchcfg' > Feb 15 16:17:07 albion abrtd: Directory 'ccpp-2017-02-15-16:17:07-31998' > creation detected > Feb 15 16:17:07 albion abrtd: Executable '/opt/dell/dup64/sbin/dchcfg' > doesn't belong to any package and ProcessUnpackaged is set to 'no' > Feb 15 16:17:07 albion abrtd: 'post-create' on '/var/spool/abrt/ccpp-2017-02- > 15-16:17:07-31998' exited with 1 > Feb 15 16:17:07 albion abrtd: Deleting problem directory > '/var/spool/abrt/ccpp-2017-02-15-16:17:07-31998' > Feb 15 16:17:15 albion kernel: dchcfg[33467]: segfault at 0 ip > 000000370813382f sp 00007ffc87e3ca98 error 4 in libc- > 2.12.so[3708000000+18a000] > Feb 15 16:17:15 albion abrt[33468]: Not saving repeating crash in > '/opt/dell/dup64/sbin/dchcfg' > Feb 15 16:17:15 albion kernel: dchcfg[33470]: segfault at 0 ip > 000000370813382f sp 00007ffe7e7084e8 error 4 in libc- > 2.12.so[3708000000+18a000] > Feb 15 16:17:15 albion abrt[33471]: Not saving repeating crash in > '/opt/dell/dup64/sbin/dchcfg' > Feb 15 16:17:15 albion kernel: dchcfg[33497]: segfault at 0 ip > 000000370813382f sp 00007ffe1a7c1238 error 4 in libc- > 2.12.so[3708000000+18a000] > Feb 15 16:17:15 albion abrt[33498]: Not saving repeating crash in > '/opt/dell/dup64/sbin/dchcfg' > Feb 15 16:17:15 albion kernel: dchcfg[33573]: segfault at 0 ip > 000000370813382f sp 00007ffea292cf18 error 4 in libc- > 2.12.so[3708000000+18a000] > Feb 15 16:17:15 albion abrt[33574]: Not saving repeating crash in > '/opt/dell/dup64/sbin/dchcfg' > Feb 15 16:17:15 albion kernel: dchcfg[33655]: segfault at 0 ip > 000000370813382f sp 00007ffec47fd148 error 4 in libc- > 2.12.so[3708000000+18a000] > Feb 15 16:17:15 albion abrt[33656]: Not saving repeating crash in > '/opt/dell/dup64/sbin/dchcfg' > Feb 15 16:17:25 albion BMAPI[34316]: ERROR SemCreate() semget() > failed! No space left on device > Feb 15 16:17:25 albion BMAPI[34316]: ERROR BmapiInitialize() > LockCreate() failed! > Feb 15 16:17:25 albion BMAPI[34316]: ERROR BmapiInitialize() > LockCreate() failed! > > Ben > -- > Unix Support, UIS, University of Cambridge, England > > > _______________________________________________ > Linux-PowerEdge mailing list > [email protected] > https://lists.us.dell.com/mailman/listinfo/linux-poweredge _______________________________________________ Linux-PowerEdge mailing list [email protected] https://lists.us.dell.com/mailman/listinfo/linux-poweredge
