follow up, another crash today.
On Mon, Nov 30, 2009 at 11:35:07AM +0100, Tru Huynh wrote:
> 1) OS
> SunOS xargos.bis.pasteur.fr 5.10 Generic_141445-09 i86pc i386 i86pc
>
> it's only sharing though NFS v3 to linux clients running
> 20x CentOS-5 x86_64 2.6.18-164.6.1.el5 x86_64/i386
> 78x CentOS-3 x86_64/ia32e/i386
>
> 2) usual logs:
> /var/adm/messages
> -> nothing
still empty
>
> 3) fmdump -ev
> /var/fm/fmd/errlog is empty
same
> 7) not tried yet
> reboot -d to force a dump
failed (not returned from sync)
reboot -dfn
failed at 98% of the dump (I could not catch the reason, screen blanked too
fast)
>
> 9) from the #irc channel, I will keep a screen running with:
[...@xargos ~]$ ps -ef
UID PID PPID C STIME TTY TIME CMD
root 0 0 0 Nov 29 ? 3:16 sched
root 1 0 0 Nov 29 ? 0:00 /sbin/init
root 2 0 0 Nov 29 ? 0:00 pageout
root 3 0 0 Nov 29 ? 20:04 fsflush
root 154 1 0 Nov 29 ? 0:00 /usr/lib/picl/picld
root 7 1 0 Nov 29 ? 0:04 /lib/svc/bin/svc.startd
root 9 1 0 Nov 29 ? 0:08 /lib/svc/bin/svc.configd
daemon 152 1 0 Nov 29 ? 0:03 /usr/lib/crypto/kcfd
tru 2258 2226 0 Nov 30 pts/7 0:00 /usr/bin/bash
root 409 408 0 Nov 29 ? 0:00 /usr/sadm/lib/smc/bin/smcboot
root 142 1 0 Nov 29 ? 0:01 /usr/lib/sysevent/syseventd
root 429 1 0 Nov 29 ? 0:00 sh
/opt/MegaRaidStorageManager/Framework/startup.sh
root 57 1 0 Nov 29 ? 0:00 /sbin/dhcpagent
root 64 1 0 Nov 29 ? 0:00 devfsadmd
root 208 1 0 Nov 29 ? 0:00 /lib/svc/method/iscsid
daemon 306 1 0 Nov 29 ? 0:00 /usr/sbin/rpcbind
root 146 1 0 Nov 29 ? 0:12 /usr/sbin/nscd
root 2228 2226 0 Nov 30 pts/2 0:08 zpool iostat -v 60
root 332 7 0 Nov 29 ? 0:00 /usr/lib/saf/sac -t 300
root 145 1 0 Nov 29 ? 0:00 /usr/lib/power/powerd
root 226 1 0 Nov 29 ? 0:10 /usr/lib/inet/xntpd
root 394 332 0 Nov 29 ? 0:00 /usr/lib/saf/ttymon
root 262 1 0 Nov 29 ? 0:00 /usr/sbin/cron
root 366 1 0 Nov 29 ? 0:00 /usr/lib/utmpd
noaccess 673 1 0 Nov 29 ? 3:04 /usr/java/bin/java -server
-Xmx128m -XX:+UseParallelGC -XX:ParallelGCThreads=4
root 349 7 0 Nov 29 console 0:00 /usr/lib/saf/ttymon -g -d
/dev/console -l console -m ldterm,ttcompat -h -p xarg
daemon 315 1 0 Nov 29 ? 0:00 /usr/lib/nfs/statd
daemon 317 1 0 Nov 29 ? 0:01 /usr/lib/nfs/nfsmapid
root 552 1 0 Nov 29 ? 0:01 /usr/sfw/sbin/snmpd
daemon 324 1 0 Nov 29 ? 0:00 /usr/lib/nfs/lockd
root 431 1 0 Nov 29 ? 0:05 /usr/sbin/syslogd
tru 695 689 0 Nov 29 pts/1 0:00 -bash
root 367 1 0 Nov 29 ? 0:00 /usr/lib/autofs/automountd
root 365 1 0 Nov 29 ? 0:02 /usr/lib/inet/inetd start
root 369 367 0 Nov 29 ? 0:01 /usr/lib/autofs/automountd
root 430 429 0 Nov 29 ? 3:26 ../jre/bin/java -classpath
../jre/lib/rt.jar:../jre/lib/jsse.jar:../jre/lib/jce
root 408 1 0 Nov 29 ? 0:00 /usr/sadm/lib/smc/bin/smcboot
root 410 408 0 Nov 29 ? 0:00 /usr/sadm/lib/smc/bin/smcboot
root 2234 2226 0 Nov 30 pts/5 0:15 intrstat 60
tru 2236 2226 0 Nov 30 pts/6 0:06 vmstat 60
tru 689 688 0 Nov 29 ? 0:01 /usr/lib/ssh/sshd
root 594 1 0 Nov 29 ? 4:25 /usr/sbin/lsi_mrdsnmpagent
-c /etc/sma/snmp/snmpd.conf
tru 2232 2226 0 Nov 30 pts/4 0:13 prstat 60
tru 2225 695 0 Nov 30 pts/1 0:01 screen
root 443 1 0 Nov 29 ? 0:00 /usr/lib/ssh/sshd
root 688 443 0 Nov 29 ? 0:00 /usr/lib/ssh/sshd
root 541 1 0 Nov 29 ? 0:03 /usr/lib/sendmail -bd -q15m
-C /etc/mail/local.cf
smmsp 537 1 0 Nov 29 ? 0:00 /usr/lib/sendmail -Ac -q15m
root 2565 1 0 Nov 30 ? 0:06 /usr/local/bin/mrmonitord
tru 2226 2225 0 Nov 30 ? 0:05 screen
tru 3988 3982 0 15:33:51 pts/11 0:00 prstat
root 498 1 0 Nov 29 ? 0:00 /usr/sbin/vold -f
/etc/vold.conf
root 509 1 0 Nov 29 ? 0:04 /usr/lib/fm/fmd/fmd
tru 2230 2226 0 Nov 30 pts/3 0:06 iostat -xn 60
tru 3967 3966 0 15:33:36 ? 0:00 /usr/lib/ssh/sshd
root 522 1 0 Nov 29 ? 0:00 /usr/lib/nfs/mountd
daemon 524 1 0 Nov 29 ? 9:45 /usr/lib/nfs/nfsd
root 3966 443 0 15:33:36 ? 0:00 /usr/lib/ssh/sshd
tru 3981 3973 0 15:33:50 pts/8 0:00 screen -c .screenrc-crash
tru 3986 3982 0 15:33:51 pts/10 0:00 iostat -xn 1
root 3984 3982 0 15:33:51 pts/9 0:00 zpool iostat -v 1
tru 3982 3981 0 15:33:51 ? 0:00 screen -c .screenrc-crash
root 2584 1 0 Nov 30 ? 0:19 /opt/monit/5.0.3/bin/monit
-c /opt/monit/5.0.3/etc/monitrc
tru 3973 3967 0 15:33:36 pts/8 0:00 -bash
root 3990 3982 0 15:33:51 pts/12 0:01 intrstat 1
tru 3992 3982 0 15:33:51 pts/13 0:00 vmstat 1
tru 3994 3982 0 15:33:55 pts/14 0:00 /usr/bin/bash
tru 3998 3994 0 15:37:43 pts/14 0:00 ps -ef
> prstat
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
3992 tru 2800K 1844K sleep 59 0 0:00:00 0.0% vmstat/1
3990 root 22M 21M sleep 59 0 0:00:00 0.0% intrstat/1
3986 tru 2940K 1984K sleep 59 0 0:00:00 0.0% iostat/1
594 root 5944K 3996K sleep 59 0 0:04:24 0.0% lsi_mrdsnmpagen/5
3994 tru 2988K 1940K sleep 59 0 0:00:00 0.0% bash/1
3973 tru 3000K 1964K sleep 59 0 0:00:00 0.0% bash/1
2584 root 6064K 2272K sleep 59 0 0:00:19 0.0% monit/2
3982 tru 6804K 4940K sleep 59 0 0:00:00 0.0% screen-4.0.3/1
3984 root 5268K 2388K sleep 59 0 0:00:00 0.0% zpool/1
3981 tru 2816K 1676K sleep 59 0 0:00:00 0.0% screen-4.0.3/1
3966 root 4780K 2676K sleep 59 0 0:00:00 0.0% sshd/1
524 daemon 2528K 1688K sleep 60 -20 0:09:45 0.0% nfsd/18
522 root 3516K 2288K sleep 59 0 0:00:00 0.0% mountd/2
3967 tru 6360K 3808K sleep 59 0 0:00:00 0.0% sshd/1
2230 tru 3212K 2256K sleep 59 0 0:00:06 0.0% iostat/1
509 root 13M 9748K sleep 59 0 0:00:03 0.0% fmd/18
498 root 3032K 2272K sleep 59 0 0:00:00 0.0% vold/6
3988 tru 3748K 2924K cpu5 59 0 0:00:00 0.0% prstat/1
2226 tru 7028K 5048K sleep 59 0 0:00:05 0.0% screen-4.0.3/1
2565 root 12M 7536K sleep 59 0 0:00:06 0.0% mrmonitord/6
537 smmsp 8444K 1764K sleep 59 0 0:00:00 0.0% sendmail/1
541 root 8444K 2276K sleep 59 0 0:00:02 0.0% sendmail/1
688 root 4780K 2608K sleep 59 0 0:00:00 0.0% sshd/1
443 root 3944K 1532K sleep 59 0 0:00:00 0.0% sshd/1
2225 tru 2816K 1660K sleep 59 0 0:00:00 0.0% screen-4.0.3/1
2232 tru 3884K 3032K sleep 59 0 0:00:12 0.0% prstat/1
689 tru 6408K 3824K sleep 59 0 0:00:01 0.0% sshd/1
2236 tru 3032K 2076K sleep 59 0 0:00:05 0.0% vmstat/1
[...@xargos ~]$ vmstat 1
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr cd s0 s1 s3 in sy cs us sy id
0 0 0 17000912 20134564 1 2 0 0 0 0 2 1 5 -0 6 1892 235 554 0 0 100
0 1 0 2225340 5364560 20 74 0 0 0 0 0 0 0 0 0 2320 1325 484 0 0 100
0 1 0 2225160 5364460 0 13 0 0 0 0 0 0 0 0 0 2342 1284 492 0 0 100
1 1 0 2225136 5364436 0 0 0 0 0 0 0 0 0 0 0 2300 2027 626 0 0 100
0 1 0 2225108 5364408 0 0 0 0 0 0 0 0 0 0 0 2363 1253 482 0 0 100
0 1 0 2225052 5364352 0 0 0 0 0 0 0 11 0 0 0 2612 1259 711 0 0 100
1 1 0 2225020 5364320 0 0 0 0 0 0 0 0 0 0 0 2369 1334 482 0 0 100
0 1 0 2224936 5364236 0 0 0 0 0 0 0 0 0 0 0 2325 1309 475 0 1 99
2 1 0 2224900 5364200 0 0 0 0 0 0 0 0 0 0 0 2334 2072 634 0 0 100
1 1 0 2224876 5364176 0 0 0 0 0 0 0 0 0 0 0 2334 1323 491 0 0 100
0 1 0 2224844 5364144 0 0 0 0 0 0 0 0 0 0 0 2318 1319 501 0 0 100
0 1 0 2224820 5364120 0 0 0 0 0 0 0 0 0 0 0 2405 1298 482 0 0 100
0 1 0 2224800 5364100 0 0 0 0 0 0 0 0 0 0 0 2507 1282 480 0 0 99
0 1 0 2224660 5363960 0 0 0 0 0 0 0 0 0 0 0 2445 2001 641 0 0 100
2 1 0 2224640 5363940 0 0 0 0 0 0 0 0 0 0 0 2459 1274 477 0 0 100
0 1 0 2224604 5363904 0 0 0 0 0 0 0 0 0 0 0 2466 1281 504 0 0 100
0 1 0 2224568 5363868 0 0 0 0 0 0 0 0 0 0 0 2478 2055 630 0 0 100
0 1 0 2224532 5363832 0 0 0 0 0 0 0 0 0 0 0 2510 1338 501 0 0 100
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr cd s0 s1 s3 in sy cs us sy id
0 1 0 2224496 5363796 0 0 0 0 0 0 0 0 0 0 0 2328 2048 644 0 0 99
0 1 0 2224448 5363748 0 0 0 0 0 0 0 0 0 0 0 2391 1284 496 0 0 100
1 1 0 2224420 5363720 0 0 0 0 0 0 0 0 0 0 0 2381 1359 503 0 1 99
3 1 0 2224404 5363704 0 0 0 0 0 0 0 0 0 0 0 2535 1771 586 0 0 100
0 1 0 2224392 5363692 0 0 0 0 0 0 0 0 0 0 0 2515 1311 534 0 0 100
0 1 0 2224372 5363672 0 0 0 0 0 0 0 0 0 0 0 2464 2042 626 0 0 100
1 1 0 2224344 5363644 0 0 0 0 0 0 0 0 0 0 0 2403 1404 552 0 0 100
0 1 0 2224192 5363612 0 2 0 0 0 0 0 0 0 0 0 2446 1261 514 0 0 100
1 1 0 2224168 5363588 0 0 0 0 0 0 0 0 0 0 0 2459 1298 511 0 0 100
1 1 0 2224144 5363564 0 0 0 0 0 0 0 0 0 0 0 2416 1339 503 0 0 100
0 1 0 2224120 5363540 0 0 0 0 0 0 0 0 0 0 0 2467 2380 640 0 1 99
4 1 0 2224088 5363508 0 0 0 0 0 0 0 0 0 0 0 2379 1330 487 0 0 100
1 1 0 2224072 5363492 0 0 0 0 0 0 0 0 0 0 0 2475 1416 507 0 0 100
1 1 0 2224056 5363476 0 0 0 0 0 0 0 0 0 0 0 2474 1318 524 0 0 99
3 1 0 2224044 5363464 0 0 0 0 0 0 0 0 0 0 0 2605 1320 560 0 0 100
0 1 0 2224024 5363444 0 0 0 0 0 0 0 0 0 0 0 2511 2077 666 0 0 100
[...@xargos ~]$ iostat -xn 1
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t0d0
0.0 0.0 0.0 0.0 0.0 3.0 0.0 0.0 0 100 c4t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
xargos.bis.pasteur.fr:vold(pid498)
...
0.0 0.0 0.0 0.0 0.0 3.0 0.0 0.0 0 100 c4t0d0
in all ouput
[...@xargos ~]$ sudo intrstat 1
device | cpu0 %tim cpu1 %tim cpu2 %tim cpu3 %tim
-------------+------------------------------------------------------------
ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0
igb#0 | 37 0.1 0 0.0 0 0.0 0 0.0
pci-ide#0 | 0 0.0 0 0.0 0 0.0 6 0.0
uhci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
uhci#2 | 0 0.0 0 0.0 0 0.0 0 0.0
uhci#3 | 0 0.0 0 0.0 0 0.0 3 0.0
uhci#4 | 0 0.0 0 0.0 0 0.0 0 0.0
device | cpu4 %tim cpu5 %tim cpu6 %tim cpu7 %tim
-------------+------------------------------------------------------------
ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0
igb#0 | 0 0.0 0 0.0 0 0.0 0 0.0
pci-ide#0 | 0 0.0 0 0.0 0 0.0 0 0.0
uhci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
uhci#2 | 0 0.0 0 0.0 0 0.0 0 0.0
uhci#3 | 0 0.0 0 0.0 0 0.0 0 0.0
uhci#4 | 0 0.0 0 0.0 0 0.0 0 0.0
device | cpu8 %tim cpu9 %tim cpu10 %tim cpu11 %tim
-------------+------------------------------------------------------------
ehci#0 | 0 0.0 3 0.0 0 0.0 0 0.0
ehci#1 | 0 0.0 0 0.0 6 0.0 0 0.0
igb#0 | 0 0.0 0 0.0 0 0.0 0 0.0
pci-ide#0 | 0 0.0 0 0.0 0 0.0 0 0.0
uhci#0 | 0 0.0 0 0.0 0 0.0 1 0.0
uhci#2 | 0 0.0 0 0.0 6 0.0 0 0.0
uhci#3 | 0 0.0 0 0.0 0 0.0 0 0.0
uhci#4 | 0 0.0 3 0.0 0 0.0 0 0.0
device | cpu12 %tim cpu13 %tim cpu14 %tim cpu15 %tim
-------------+------------------------------------------------------------
ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0
igb#0 | 0 0.0 0 0.0 0 0.0 1 0.0
pci-ide#0 | 0 0.0 0 0.0 0 0.0 0 0.0
uhci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
uhci#2 | 0 0.0 0 0.0 0 0.0 0 0.0
uhci#3 | 0 0.0 0 0.0 0 0.0 0 0.0
uhci#4 | 0 0.0 0 0.0 0 0.0 0 0.0
after reboot:
[...@tanatloc-eth2 ~]$ iostat -E
sd0 Soft Errors: 29222307 Hard Errors: 0 Transport Errors: 0
Vendor: LSI Product: MegaRAID 8708ELP Revision: 1.40 Serial No:
Size: 7495.00GB <7494995803648 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 29222307 Predictive Failure Analysis: 0
sd1 Soft Errors: 120784 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: WDC WD3000HLFS-0 Revision: 4V01 Serial No:
Size: 300.07GB <300069051904 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 120784 Predictive Failure Analysis: 0
sd2 Soft Errors: 1 Hard Errors: 0 Transport Errors: 0
Vendor: MATSHITA Product: DVD-ROM UJDA780 Revision: 1.50 Serial No:
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
sd3 Soft Errors: 29221515 Hard Errors: 0 Transport Errors: 0
Vendor: LSI Product: MegaRAID 8708ELP Revision: 1.40 Serial No:
Size: 7495.00GB <7494995803648 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 29221515 Predictive Failure Analysis: 0
sd38 Soft Errors: 29222176 Hard Errors: 0 Transport Errors: 0
Vendor: LSI Product: MegaRAID 8708ELP Revision: 1.40 Serial No:
Size: 7495.00GB <7494995803648 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 29222176 Predictive Failure Analysis: 0
[...@tanatloc-eth2 ~]$ iostat -xen 1
extended device statistics ---- errors ---
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn tot
device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 29225001 0 0
29225001 c3t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 120786 0 0
120786 c0t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 1 0 0 1
c0t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 29224213 0 0
29224213 c1t0d0
0.0 2.0 0.0 4.0 0.0 0.0 0.0 0.1 0 0 29224869 0 0
29224869 c2t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
tanatloc-eth2.bis.pasteur.fr:vold(pid523)
More informations on similar setup:
a) iostat -xen 1
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn tot
device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 29212456 0 0
29212456 c3t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 120776 0 0
120776 c0t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 1 0 0 1
c0t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 29211660 0 0
29211660 c1t0d0
0.0 2.0 0.0 8.0 0.0 0.0 0.0 0.1 0 0 29212328 0 0
29212328 c2t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
tanatloc-eth2.bis.pasteur.fr:vold(pid523)
b) iostat -E
sd0 Soft Errors: 29222307 Hard Errors: 0 Transport Errors: 0
Vendor: LSI Product: MegaRAID 8708ELP Revision: 1.40 Serial No:
Size: 7495.00GB <7494995803648 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 29222307 Predictive Failure Analysis: 0
sd1 Soft Errors: 120784 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: WDC WD3000HLFS-0 Revision: 4V01 Serial No:
Size: 300.07GB <300069051904 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 120784 Predictive Failure Analysis: 0
sd2 Soft Errors: 1 Hard Errors: 0 Transport Errors: 0
Vendor: MATSHITA Product: DVD-ROM UJDA780 Revision: 1.50 Serial No:
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
sd3 Soft Errors: 29221515 Hard Errors: 0 Transport Errors: 0
Vendor: LSI Product: MegaRAID 8708ELP Revision: 1.40 Serial No:
Size: 7495.00GB <7494995803648 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 29221515 Predictive Failure Analysis: 0
sd38 Soft Errors: 29222176 Hard Errors: 0 Transport Errors: 0
Vendor: LSI Product: MegaRAID 8708ELP Revision: 1.40 Serial No:
Size: 7495.00GB <7494995803648 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 29222176 Predictive Failure Analysis: 0
Thanks
Tru
--
Dr Tru Huynh | http://www.pasteur.fr/recherche/unites/Binfs/
mailto:t...@pasteur.fr | tel/fax +33 1 45 68 87 37/19
Institut Pasteur, 25-28 rue du Docteur Roux, 75724 Paris CEDEX 15 France
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
