Re: [OmniOS-discuss] Host is not being rebooted if it uses ZFS over iSCSI

[Дмитрий Глушенок]

Thanks for the scripts, but I would like to avoid hardcoding pool names 
anywhere. Due to specific roles of my ZFS servers it is possible to export all 
zpools at iSCSI stop and import all visible pools at iSCSI start. So, I end up 
with following modifications:

/lib/svc/method/iscsi-initiator:
at start:
sleep 5
zpool import -a
at stop:
zpools=$(zpool list | awk '!/^rpool / && !/^NAME / {print $1}')
if [ -z $zpools ]; then
echo "No ZFS pools to export"
return $rc
fi
for pool in $zpools; do
echo "Exporting pool $pool"
zpool export $pool
echo "Done, rc=$?"
done

For zpool command to work from SMF service I've also removed privilege 
restrictions:
svccfg -s iscsi/initiator delprop start/privileges
svccfg -s iscsi/initiator delprop stop/privileges
svcadm refresh iscsi/initiator:default


> 30 нояб. 2016 г., в 22:47, Jim Klimov  написал(а):
> 
> 30 ноября 2016 г. 17:45:53 CET, "Дмитрий Глушенок"  пишет:
>> The 'reboot' works, but it kills all processes at once (including
>> iscsid), leaving no chances for ZFS to flush write cache. And I suspect
>> that with current iscsi/initiator SMF service implementation the same
>> happens (but for some reason sync(), which is being called by 'reboot'
>> after killing all processes, is not hangs).
>> 
>>> 30 нояб. 2016 г., в 18:05, Chris Siebenmann 
>> написал(а):
>>> 
 Does anybody uses ZFS over iSCSI?
 
 There is problem with reboots as iscsi-initiator service does not
 taking care of ZFS while shutting down. It leads to zpool goes into
 UNAVAIL state and then first sync() issued gets blocked with
>> following
 stack:
>>> 
>>> We have a significant ZFS-over-iSCSI fileserver environment and
>>> haven't seen this, but we may be shutting our systems down in a
>>> different way than you are.
>>> 
>>> We're on OmniOS r151014 and we haven't seen this behavior when we
>>> reboot or shut down our systems via 'reboot' or 'poweroff'. For us it
>>> seems to go the other way around; running 'reboot' won't completely
>> shut
>>> down the iSCSI initiator before the machine reboots, apparently
>> because
>>> it's being held busy by our active ZFS pools.
>>> 
>>> It may be relevant that we're using OmniOS multipathing across two
>>> IPs for each target (configured statically); this might change the
>>> device stack inside the kernel.
>>> 
>>> I don't think we've ever tried 'shutdown' itself on a live production
>>> system, so I don't know how that would work.
>>> 
>>> - cks
>> 
>> --
>> Dmitry Glushenok
>> Jet Infosystems
>> 
>> 
>> 
>> 
>> 
>> ___
>> OmniOS-discuss mailing list
>> OmniOS-discuss@lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> 
> Shameless plug: did you try an approach like 
> https://github.com/jimklimov/illumos-smf-zfspools to define an SMF dependency 
> between a pool whose vdevs reside on an iscsi initiator, or an iscsi target 
> residing on a pool, and perhaps other services like zones to depend on the 
> non-rpool zfs pool (SMFized)?
> 
> At least for the case of well ordered and synced graceful shutdowns and 
> startups, this should help you.
> 
> HTH, Jim Klimov
> --
> Typos courtesy of K-9 Mail on my Samsung Android

--
Dmitry Glushenok
Jet Infosystems

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Host is not being rebooted if it uses ZFS over iSCSI

[Jim Klimov]

30 ноября 2016 г. 17:45:53 CET, "Дмитрий Глушенок"  пишет:
>The 'reboot' works, but it kills all processes at once (including
>iscsid), leaving no chances for ZFS to flush write cache. And I suspect
>that with current iscsi/initiator SMF service implementation the same
>happens (but for some reason sync(), which is being called by 'reboot'
>after killing all processes, is not hangs).
>
>> 30 нояб. 2016 г., в 18:05, Chris Siebenmann 
>написал(а):
>> 
>>> Does anybody uses ZFS over iSCSI?
>>> 
>>> There is problem with reboots as iscsi-initiator service does not
>>> taking care of ZFS while shutting down. It leads to zpool goes into
>>> UNAVAIL state and then first sync() issued gets blocked with
>following
>>> stack:
>> 
>> We have a significant ZFS-over-iSCSI fileserver environment and
>> haven't seen this, but we may be shutting our systems down in a
>> different way than you are.
>> 
>> We're on OmniOS r151014 and we haven't seen this behavior when we
>> reboot or shut down our systems via 'reboot' or 'poweroff'. For us it
>> seems to go the other way around; running 'reboot' won't completely
>shut
>> down the iSCSI initiator before the machine reboots, apparently
>because
>> it's being held busy by our active ZFS pools.
>> 
>> It may be relevant that we're using OmniOS multipathing across two
>> IPs for each target (configured statically); this might change the
>> device stack inside the kernel.
>> 
>> I don't think we've ever tried 'shutdown' itself on a live production
>> system, so I don't know how that would work.
>> 
>>  - cks
>
>--
>Dmitry Glushenok
>Jet Infosystems
>
>
>
>
>
>___
>OmniOS-discuss mailing list
>OmniOS-discuss@lists.omniti.com
>http://lists.omniti.com/mailman/listinfo/omnios-discuss

Shameless plug: did you try an approach like 
https://github.com/jimklimov/illumos-smf-zfspools to define an SMF dependency 
between a pool whose vdevs reside on an iscsi initiator, or an iscsi target 
residing on a pool, and perhaps other services like zones to depend on the 
non-rpool zfs pool (SMFized)?

At least for the case of well ordered and synced graceful shutdowns and 
startups, this should help you.

HTH, Jim Klimov
--
Typos courtesy of K-9 Mail on my Samsung Android
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Host is not being rebooted if it uses ZFS over iSCSI

[Дмитрий Глушенок]

The 'reboot' works, but it kills all processes at once (including iscsid), 
leaving no chances for ZFS to flush write cache. And I suspect that with 
current iscsi/initiator SMF service implementation the same happens (but for 
some reason sync(), which is being called by 'reboot' after killing all 
processes, is not hangs).

> 30 нояб. 2016 г., в 18:05, Chris Siebenmann  написал(а):
> 
>> Does anybody uses ZFS over iSCSI?
>> 
>> There is problem with reboots as iscsi-initiator service does not
>> taking care of ZFS while shutting down. It leads to zpool goes into
>> UNAVAIL state and then first sync() issued gets blocked with following
>> stack:
> 
> We have a significant ZFS-over-iSCSI fileserver environment and
> haven't seen this, but we may be shutting our systems down in a
> different way than you are.
> 
> We're on OmniOS r151014 and we haven't seen this behavior when we
> reboot or shut down our systems via 'reboot' or 'poweroff'. For us it
> seems to go the other way around; running 'reboot' won't completely shut
> down the iSCSI initiator before the machine reboots, apparently because
> it's being held busy by our active ZFS pools.
> 
> It may be relevant that we're using OmniOS multipathing across two
> IPs for each target (configured statically); this might change the
> device stack inside the kernel.
> 
> I don't think we've ever tried 'shutdown' itself on a live production
> system, so I don't know how that would work.
> 
>   - cks

--
Dmitry Glushenok
Jet Infosystems

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Host is not being rebooted if it uses ZFS over iSCSI

[Chris Siebenmann]

> Does anybody uses ZFS over iSCSI?
>
> There is problem with reboots as iscsi-initiator service does not
> taking care of ZFS while shutting down. It leads to zpool goes into
> UNAVAIL state and then first sync() issued gets blocked with following
> stack:

 We have a significant ZFS-over-iSCSI fileserver environment and
haven't seen this, but we may be shutting our systems down in a
different way than you are.

 We're on OmniOS r151014 and we haven't seen this behavior when we
reboot or shut down our systems via 'reboot' or 'poweroff'. For us it
seems to go the other way around; running 'reboot' won't completely shut
down the iSCSI initiator before the machine reboots, apparently because
it's being held busy by our active ZFS pools.

 It may be relevant that we're using OmniOS multipathing across two
IPs for each target (configured statically); this might change the
device stack inside the kernel.

 I don't think we've ever tried 'shutdown' itself on a live production
system, so I don't know how that would work.

- cks
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss