Hi Chandra, thanks for the information. Looks like nrpe might be the culprit:
# ipcs -s ------ Semaphore Arrays -------- key semid owner perms nsems 0x00000000 0 root 600 1 0x00000000 32769 root 600 1 0x00000000 1113456642 root 600 1 0x00000000 1113686019 root 600 1 0x00000000 1113948164 root 600 1 0x00000000 1113915397 root 600 1 0x00000000 1113784326 root 600 1 0x00000000 1113817095 root 600 1 0x00000000 1113849864 root 600 1 0x00000000 1113882633 root 600 1 0x00000000 1113980938 root 600 1 0x00000000 1114013707 root 600 1 0x00000000 1114046476 root 600 1 0x00000000 1114079245 root 600 1 0x00000000 1114112014 root 600 1 0x00000000 1114144783 root 600 1 0x00000000 1114177552 root 600 1 0x00000000 1114210321 root 600 1 0x00000000 1117356050 root 600 1 0x00000000 1117519891 root 600 1 0x00000000 1117552660 root 600 1 0x00000000 1116635157 root 600 1 0x00000000 1116504086 root 600 1 0x00000000 1116536855 root 600 1 0x00000000 1116569624 root 600 1 0x00000000 1116602393 root 600 1 0x00000000 1117585434 root 600 1 0x00000000 1117618203 root 600 1 0xd95b59b9 16613404 root 666 2 0x00000000 599719965 nrpe 600 1 0x000003d4 16678942 root 644 1 0x00000000 599752735 nrpe 600 1 0x00000000 316276768 nrpe 600 1 0x00000000 316309537 nrpe 600 1 0x00000000 1608777762 nrpe 600 1 0x00000000 174096419 nrpe 600 1 0x00000000 1608810532 nrpe 600 1 0x00000000 174161957 nrpe 600 1 0x00000000 1608843302 nrpe 600 1 0x00000000 1608876071 nrpe 600 1 0x00000000 1339129896 nrpe 600 1 0x00000000 1339162665 nrpe 600 1 0x00000000 1550483498 nrpe 600 1 0x00000000 1550516267 nrpe 600 1 0x00000000 1339293740 nrpe 600 1 0x00000000 1339326509 nrpe 600 1 0x00000000 1339359278 nrpe 600 1 0x00000000 1339392047 nrpe 600 1 0x00000000 1937965104 nrpe 600 1 0x00000000 1937997873 nrpe 600 1 0x00000000 1783726130 nrpe 600 1 0x00000000 1783758899 nrpe 600 1 0x00000000 1977810996 nrpe 600 1 0x00000000 1977843765 nrpe 600 1 0x00000000 188186678 nrpe 600 1 0x00000000 188219447 nrpe 600 1 0x00000000 1131085880 nrpe 600 1 0x00000000 1131118649 nrpe 600 1 0x00000000 193953850 nrpe 600 1 0x00000000 193986619 nrpe 600 1 0x00000000 194052156 nrpe 600 1 0x00000000 194084925 nrpe 600 1 0x00000000 194150462 nrpe 600 1 0x00000000 194183231 nrpe 600 1 0x00000000 1331134528 nrpe 600 1 0x00000000 1331167297 nrpe 600 1 0x00000000 1504444482 nrpe 600 1 0x00000000 1504477251 nrpe 600 1 0x00000000 1687322692 nrpe 600 1 0x00000000 1687355461 nrpe 600 1 0x00000000 1687388230 nrpe 600 1 0x00000000 1687420999 nrpe 600 1 0x00000000 1723924552 nrpe 600 1 0x00000000 1723957321 nrpe 600 1 0x00000000 1759510602 nrpe 600 1 0x00000000 1759543371 nrpe 600 1 0x00000000 1828454476 nrpe 600 1 0x00000000 1828487245 nrpe 600 1 0x00000000 1889828942 nrpe 600 1 0x00000000 1889861711 nrpe 600 1 0x00000000 1949040720 nrpe 600 1 0x00000000 1949073489 nrpe 600 1 0x00000000 1949106258 nrpe 600 1 0x00000000 1949139027 nrpe 600 1 0x00000000 1968406612 nrpe 600 1 0x00000000 1968439381 nrpe 600 1 0x00000000 1317863510 nrpe 600 1 0x00000000 1317896279 nrpe 600 1 0x00000000 1317929048 nrpe 600 1 0x00000000 1317961817 nrpe 600 1 0x00000000 1306853466 nrpe 600 1 0x00000000 1306886235 nrpe 600 1 0x00000000 1412595804 nrpe 600 1 0x00000000 1412628573 nrpe 600 1 0x00000000 1346568286 nrpe 600 1 0x00000000 1346601055 nrpe 600 1 0x00000000 1346633824 nrpe 600 1 0x00000000 1346666593 nrpe 600 1 0x00000000 481525858 nrpe 600 1 0x00000000 481558627 nrpe 600 1 0x00000000 481591396 nrpe 600 1 0x00000000 481624165 nrpe 600 1 0x00000000 585138278 nrpe 600 1 0x00000000 585171047 nrpe 600 1 0x00000000 585203816 nrpe 600 1 0x00000000 585236585 nrpe 600 1 0x00000000 678527082 nrpe 600 1 0x00000000 678559851 nrpe 600 1 0x00000000 681640044 nrpe 600 1 0x00000000 681672813 nrpe 600 1 0x00000000 683901038 nrpe 600 1 0x00000000 683933807 nrpe 600 1 0x00000000 684097648 nrpe 600 1 0x00000000 684130417 nrpe 600 1 0x00000000 808976498 nrpe 600 1 0x00000000 809009267 nrpe 600 1 0x00000000 819200116 nrpe 600 1 0x00000000 819232885 nrpe 600 1 0x00000000 801800310 nrpe 600 1 0x00000000 801833079 nrpe 600 1 0x00000000 817791096 nrpe 600 1 0x00000000 817823865 nrpe 600 1 0x00000000 860520570 nrpe 600 1 0x00000000 860553339 nrpe 600 1 0x00000000 860586108 nrpe 600 1 0x00000000 860618877 nrpe 600 1 0x00000000 1074528382 nrpe 600 1 0x00000000 1074561151 nrpe 600 1 The server is in production, so a reboot is at the moment not possible. I will try to find a different, where I can debug the problem better. Regards, Stefan ----- Original Message ----- > From: "Chandrasekhar R" <[email protected]> > To: "stefan dietrich" <[email protected]> > Cc: [email protected] > Sent: Friday, April 29, 2016 10:38:42 AM > Subject: RE: OMSA 8.3.0 not starting or segfaulting > Hi Stefan, > > Thanks for providing this. Based on this strace output, I think the number > semaphores has reached the maximum count on your server. If possible, Reboot > the server and see issue resolves. > > Also, send us the output of "ipcs -s" command. > > Regards > Chandra > > -----Original Message----- > From: Dietrich, Stefan [mailto:[email protected]] > Sent: Friday, April 29, 2016 1:45 PM > To: R, Chandrasekhar > Cc: linux-poweredge-Lists > Subject: Re: OMSA 8.3.0 not starting or segfaulting > > Hi Chandra, > > this is the output: > > # strace -f ./dchcfg command=getsystype > execve("./dchcfg", ["./dchcfg", "command=getsystype"], [/* 36 vars */]) = 0 > brk(0) = 0x1dd2000 > mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f60e63c4000 > access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) > open("/etc/ld.so.cache", O_RDONLY) = 3 > fstat(3, {st_mode=S_IFREG|0644, st_size=110432, ...}) = 0 mmap(NULL, 110432, > PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f60e63a9000 > close(3) = 0 > open("/opt/dell/srvadmin/lib64/libdchcfl.so.8", O_RDONLY) = 3 read(3, > "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \230\0\0\0\0\0\0"..., 832) = > 832 fstat(3, {st_mode=S_IFREG|0755, st_size=613566, ...}) = 0 mmap(NULL, > 2271824, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = > 0x7f60e617e000 > mprotect(0x7f60e61a8000, 2097152, PROT_NONE) = 0 mmap(0x7f60e63a8000, 4096, > PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2a000) = > 0x7f60e63a8000 > close(3) = 0 > open("/lib64/libdl.so.2", O_RDONLY) = 3 > read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\r\0@9\0\0\0"..., > 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=22536, ...}) = 0 > mmap(0x3940000000, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, > 0) = 0x3940000000 mprotect(0x3940002000, 2097152, PROT_NONE) = 0 > mmap(0x3940202000, 8192, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x3940202000 > close(3) = 0 > mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f60e617d000 open("/lib64/libpthread.so.0", O_RDONLY) = 3 read(3, > "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000^\300?9\0\0\0"..., 832) = > 832 > fstat(3, {st_mode=S_IFREG|0755, st_size=145936, ...}) = 0 mmap(0x393fc00000, > 2212848, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x393fc00000 > mprotect(0x393fc17000, 2097152, PROT_NONE) = 0 mmap(0x393fe17000, 8192, > PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = > 0x393fe17000 mmap(0x393fe19000, 13296, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x393fe19000 > close(3) = 0 > open("/lib64/libm.so.6", O_RDONLY) = 3 > read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p>\200?9\0\0\0"..., > 832) > = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=599392, ...}) = 0 > mmap(0x393f800000, 2633912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, > 0) = 0x393f800000 mprotect(0x393f883000, 2093056, PROT_NONE) = 0 > mmap(0x393fa82000, 8192, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x82000) = 0x393fa82000 > close(3) = 0 > open("/lib64/libc.so.6", O_RDONLY) = 3 > read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\356A?9\0\0\0"..., > 832) > = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1926520, ...}) = 0 > mmap(0x393f400000, 3750152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, > 0) = 0x393f400000 mprotect(0x393f58a000, 2097152, PROT_NONE) = 0 > mmap(0x393f78a000, 20480, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18a000) = 0x393f78a000 > mmap(0x393f78f000, 18696, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x393f78f000 > close(3) = 0 > mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f60e617c000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f60e617b000 mmap(NULL, 4096, > PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f60e617a000 > arch_prctl(ARCH_SET_FS, 0x7f60e617b700) = 0 mprotect(0x393f78a000, 16384, > PROT_READ) = 0 mprotect(0x393fa82000, 4096, PROT_READ) = 0 > mprotect(0x393fe17000, 4096, PROT_READ) = 0 mprotect(0x3940202000, 4096, > PROT_READ) = 0 mprotect(0x393f21f000, 4096, PROT_READ) = 0 > munmap(0x7f60e63a9000, 110432) = 0 > set_tid_address(0x7f60e617b9d0) = 966829 > set_robust_list(0x7f60e617b9e0, 24) = 0 > futex(0x7ffd9570474c, FUTEX_WAKE_PRIVATE, 1) = 0 futex(0x7ffd9570474c, > FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7f60e617b700) = -1 > EAGAIN (Resource temporarily unavailable) rt_sigaction(SIGRTMIN, > {0x393fc05cb0, > [], SA_RESTORER|SA_SIGINFO, 0x393fc0f7e0}, NULL, 8) = 0 rt_sigaction(SIGRT_1, > {0x393fc05d40, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x393fc0f7e0}, NULL, 8) > = > 0 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 > getrlimit(RLIMIT_STACK, {rlim_cur=10240*1024, rlim_max=RLIM64_INFINITY}) = 0 > prctl(PR_SET_UNALIGN, NOPRINT) = -1 EINVAL (Invalid argument) > rt_sigaction(SIGINT, {0x407f30, [INT], SA_RESTORER|SA_RESTART, 0x393f4326a0}, > {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGTERM, {0x407f30, [TERM], > SA_RESTORER|SA_RESTART, 0x393f4326a0}, {SIG_DFL, [], 0}, 8) = 0 > brk(0) = 0x1dd2000 > brk(0x1df3000) = 0x1df3000 > semget(IPC_PRIVATE, 1, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on > device) > --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} --- > +++ killed by SIGSEGV +++ > Segmentation fault > > Regards, > Stefan > >> Thanks Stefan for these details. Your server is Dell Branded. Can you >> please run the following command and send me the response? >> >> cd /opt/dell/srvadmin/sbin >> strace -f ./dchcfg command=getsystype >> >> Regards >> Chandra >> >> -----Original Message----- >> From: Dietrich, Stefan [mailto:[email protected]] >> Sent: Friday, April 29, 2016 12:52 PM >> To: R, Chandrasekhar >> Cc: linux-poweredge-Lists >> Subject: Re: OMSA 8.3.0 not starting or segfaulting >> >> Hi Chandra, >> >> they should be Dell brand (?). This is from a R430 with a segfault: >> >> # smbios-sys-info-lite >> Libsmbios: 2.2.27 >> System ID: 0x0639 >> Service Tag: >> Express Service Code: >> Aset Tag: >> Product Name: PowerEdge R430 >> BIOS Version: 1.2.6 >> Vendor: Dell Inc. >> Is Dell: 1 >> OEM String 1: Dell System >> OEM String 2: 5[0000] >> OEM String 3: 14[1] >> OEM String 4: 17[537D0FC5205BB363] >> OEM String 5: 17[537BA7D240AC7A6C] >> OEM String 6: 18[0] >> OEM String 7: 19[1] >> OEM String 8: 19[1] >> >> I have removed the service and asset tag information. >> >> Regards, >> Stefan >> >>> Hi Stefan, >>> >>> Are you seeing this segfault issue on Re-branded servers or Dell brand as >>> well? >>> Please get us the following command response on failing system: >>> >>> smbios-sys-info-lite >>> >>> Regards >>> Chandra >>> _______________________________________________ Linux-PowerEdge mailing list [email protected] https://lists.us.dell.com/mailman/listinfo/linux-poweredge
