Comment below. From: Richard Elling [mailto:[email protected]] Sent: 星期五, 一月 22, 2016 13:51 To: [email protected] Subject: Re: [smartos-discuss] Is zfs dead man timer tunable?
answer far below... On Jan 21, 2016, at 8:44 PM, Fred Liu <[email protected]> wrote: -----Original Message----- From: Richard Elling [mailto:[email protected]] Sent: 星期五, 一月 22, 2016 12:02 To: [email protected] Subject: Re: [smartos-discuss] Is zfs deadm man timer tunable? On Jan 21, 2016, at 4:25 AM, Fred Liu <[email protected]> wrote: zfs deadman timer is tunable. But if you hit it, you've got problems that tuning the deadman won't help. The tunable is zfs_deadman_synctime_ms, which is milliseconds. For example, on a test machine here: [root@elvis ~]# echo zfs_deadman_synctime_ms/D | mdb -k zfs_deadman_synctime_ms: zfs_deadman_synctime_ms: 1000000 FYI, you can check on the state of I/Os in the ZIO pipeline and how long they've been there using the zio_state dcmd. Elvis is not currently busy or broken, but here is an example: [root@elvis ~]# echo ::zio_state | mdb -k ADDRESS TYPE STAGE WAITER TIME_ELAPSED ffffff01ada853e0 NULL OPEN - - ffffff01ada85b10 NULL OPEN - - If you see a large TIME_ELAPSED, you can track down the zio in question for more debugging. -- richard Richard, Many thanks! You have been always helpful since I first touched ZFS many years ago. I am trying intel P3600 NVMe ssd on ZFS. I got several random server reboot every day. And I just captured " panic message: I/O to pool 'zones' appears to be hung" from console. I doubt it is related to NVMe driver or ssd firmware. - SmartOS Live Image v0.147+ build: 20151001T070028Z [root@pluto ~]# echo zfs_deadman_synctime_ms/D | mdb -k zfs_deadman_synctime_ms: zfs_deadman_synctime_ms: 1000000 [root@pluto ~]# echo ::zio_state | mdb -k ADDRESS TYPE STAGE WAITER TIME_ELAPSED fffff08576c15028 NULL OPEN - - fffff08576c153a8 NULL OPEN - - fffff08576c15728 NULL OPEN - - fffff08576c15aa8 NULL OPEN - - fffff08576c15e28 NULL OPEN - - fffff08576c161a8 NULL OPEN - - fffff08576c16528 NULL OPEN - - fffff08576c168a8 NULL OPEN - - fffff08576c16c28 NULL OPEN - - fffff08576c25048 NULL OPEN - - fffff08576c253c8 NULL OPEN - - fffff08576c25748 NULL OPEN - - fffff08576c25ac8 NULL OPEN - - fffff08576c25e48 NULL OPEN - - fffff08576c261c8 NULL OPEN - - fffff08576c26548 NULL OPEN - - fffff08576c268c8 NULL OPEN - - fffff08576c26c48 NULL OPEN - - fffff08576e41028 NULL OPEN - - fffff08576e413a8 NULL OPEN - - fffff08576e41728 NULL OPEN - - fffff08576e428a8 NULL OPEN - - fffff08576e45050 NULL OPEN - - fffff08576e453d0 NULL OPEN - - fffff08576e45750 NULL OPEN - - fffff08576e45ad0 NULL OPEN - - fffff08576e45e50 NULL OPEN - - fffff08576e461d0 NULL OPEN - - fffff08576e46550 NULL OPEN - - fffff08576e468d0 NULL OPEN - - fffff08576e46c50 NULL OPEN - - fffff08576e47038 NULL OPEN - - fffff08576e473b8 NULL OPEN - - fffff08576e47738 NULL OPEN - - fffff08576e47ab8 NULL OPEN - - fffff08576e47e38 NULL OPEN - - fffff08576e481b8 NULL OPEN - - fffff08576e48538 NULL OPEN - - fffff08576e488b8 NULL OPEN - - fffff08576e48c38 NULL OPEN - - fffff08576c4f040 NULL OPEN - - fffff08576c4f3c0 NULL OPEN - - fffff08576c4f740 NULL OPEN - - fffff08576c4fac0 NULL OPEN - - fffff08576c4fe40 NULL OPEN - - fffff08576c501c0 NULL OPEN - - fffff08576c50540 NULL OPEN - - fffff08576c508c0 NULL OPEN - - fffff08576c50c40 NULL OPEN - - fffff08576c53708 NULL OPEN - - fffff08576c53a88 NULL OPEN - - fffff08576c53e08 NULL OPEN - - fffff08576c54188 NULL OPEN - - fffff08576c54508 NULL OPEN - - fffff08576c54888 NULL OPEN - - fffff08576c54c08 NULL OPEN - - fffff08576e5a538 NULL OPEN - - fffff08576e5a8b8 NULL OPEN - - fffff085770bb060 NULL OPEN - - fffff085770bb3e0 NULL OPEN - - fffff085770bb760 NULL OPEN - - fffff085770bbae0 NULL OPEN - - fffff085770bbe60 NULL OPEN - - fffff085770bc1e0 NULL OPEN - - fffff085770bc560 NULL OPEN - - fffff085770bc8e0 NULL OPEN - - fffff085770bcc60 NULL OPEN - - fffff085778c9030 NULL OPEN - - fffff085778c93b0 NULL OPEN - - fffff085778c9730 NULL OPEN - - fffff085778c9ab0 NULL OPEN - - fffff085778c9e30 NULL OPEN - - fffff085778ca1b0 NULL OPEN - - fffff085778ca530 NULL OPEN - - fffff085778ca8b0 NULL OPEN - - fffff085778cac30 NULL OPEN - - fffff085778cd068 NULL OPEN - - fffff085778cd3e8 NULL OPEN - - fffff085778cd768 NULL OPEN - - fffff085778cdae8 NULL OPEN - - fffff085778cde68 NULL OPEN - - fffff085778ce1e8 NULL OPEN - - fffff085778ce568 NULL OPEN - - fffff085778ce8e8 NULL OPEN - - fffff085778cec68 NULL OPEN - - fffff085778d7070 NULL OPEN - - fffff085778d73f0 NULL OPEN - - fffff085778d7770 NULL OPEN - - fffff085778d7af0 NULL OPEN - - fffff085778d7e70 NULL OPEN - - fffff085778d81f0 NULL OPEN - - fffff085778d8570 NULL OPEN - - fffff085778d88f0 NULL OPEN - - fffff085778d8c70 NULL OPEN - - fffff085778d9078 NULL OPEN - - fffff085778d93f8 NULL OPEN - - fffff085778d9778 NULL OPEN - - fffff085778d9af8 NULL OPEN - - fffff085778d9e78 NULL OPEN - - fffff085778da1f8 NULL OPEN - - fffff085778da578 NULL OPEN - - fffff085778da8f8 NULL OPEN - - fffff085778dac78 NULL OPEN - - fffff085772f4570 NULL OPEN - - fffff085772f48f0 NULL OPEN - - fffff08576d57020 NULL OPEN - - fffff08576d58520 NULL OPEN - - fffff08576f77080 NULL OPEN - - fffff08576f77400 NULL OPEN - - fffff08576f77780 NULL OPEN - - fffff08576f77b00 NULL OPEN - - fffff08576f77e80 NULL OPEN - - fffff08576f78200 NULL OPEN - - fffff08576f78580 NULL OPEN - - fffff08576f78900 NULL OPEN - - fffff08576f78c80 NULL OPEN - - fffff085771cf058 NULL OPEN - - fffff085771cf3d8 NULL OPEN - - fffff085771cf758 NULL OPEN - - fffff085771cfad8 NULL OPEN - - fffff085771cfe58 NULL OPEN - - fffff085771d01d8 NULL OPEN - - fffff085771d0558 NULL OPEN - - fffff085771d08d8 NULL OPEN - - fffff085771d0c58 NULL OPEN - - fffff085771d1020 NULL OPEN - - fffff085771d13a0 NULL OPEN - - fffff085771d1720 NULL OPEN - - fffff085771d1aa0 NULL OPEN - - fffff085771d1e20 NULL OPEN - - fffff085771d21a0 NULL OPEN - - fffff085771d2520 NULL OPEN - - fffff085771d28a0 NULL OPEN - - fffff085771d2c20 NULL OPEN - - fffff085771d5000 NULL OPEN - - fffff085771d5380 NULL OPEN - - fffff085771d5700 NULL OPEN - - fffff085771d5a80 NULL OPEN - - fffff085771d5e00 NULL OPEN - - fffff085771d6180 NULL OPEN - - fffff085771d6500 NULL OPEN - - fffff085771d6880 NULL OPEN - - fffff085771d6c00 NULL OPEN - - fffff085771e6c30 NULL OPEN - - Is it possible to trigger a core dump? Coz, I can't get anything from /var/adm/message. If you hit the zfs deadman timer, you should get a core dump. The ::zio_state is very helpful for debugging kernel dumps, too :-) [Fred]: the dump device is zones/dump which is a zvol. How to get the core-dump file? That said, it is unusual that you hit the zfs deadman without another failure of some sort that isn't handled by sd. However, there is such a bug in mptsas driver in the past year. The bug has to do with a deadlock in the driver during reset conditions. These can occur if the IOC decides the target needs to be reset. It should be noted in the FMA ereport log since resets tend to be preceded by timeouts which are logged. You might also see some syslog messages around the same time: 1,000 seconds prior to zfs deadman timeout. If this is the case, then we can take a look at your release and see if it contains the fix. See also https://www.illumos.org/issues/6256 [Fred]: These NVMe ssds directly talk to CPUs via PCIE buses. So there is no HBAs. This case may not apply. Many thanks! Fred ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
