Bug#982842: Regarding Debian bug #982842, "open-iscsi: do not use iscsistart in the boot process"

[Lee Duncan]

On 12/2/22 11:23, Eric Mackay wrote:

Greetings Heinrich and Lee,

Reaching out to both of you directly because I'm not sure that replies on the 
Debian bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982842 were 
actually propagated

I made a merge request to the Debian open-iscsi package that adds this feature: 
https://salsa.debian.org/linux-blocks-team/open-iscsi/-/merge_requests/14

But the maintainers don't seem to see the importance of using an iscsi root, 
and/or the shortcomings of iscsistart.

Would appreciate any further comments on the bug or the merge request, as I'm 
hoping to move this along. Even if they're negative comments :)


Hi All:

Sorry it took me a while to get around to replying to this.

First, responses to the bug report, i.e.:


https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982842


Ritesh Raj Sarraf says:


Yes. But usually, you don't need the iscsid daemon in initrd. Because
most usual cases, users would be mapping data LUNs only.


This is not my experience at all. Our distribution only supports having 
root or /usr on iscsi at initrd time. If it is a "data lun" (e.g. 
/usr/local or /alt) there is no reason to mount it at initrd time -- it 
can wait until we have the full root filesystem up, since that's much 
simpler.


he goes on to say:


Only in exceptional cases, where you have root on iSCSI, you need to
establish the connections early. And to do that we used iscsistart,
which would establish only a single connection, effectively making it
prone to hangs if that single connection went down.


Using iscsistart is a "shortcut" to having the daemon present. Many 
iscsiroot users these days are using multipathing, with two independant 
paths to the root disc using iscsi, and these cases get a bit 
complicated for iscsistart, which is completely synchronous. For 
example, iscsiadm/iscsid have the ability to send login requests but not 
wait for them to return, potentially making login to slow targets (or 
over slow networks) faster.


Lastly, he says:


The other reason I can recollect is that you could have a very large
number of LUNs mapped to your initiator along with your root LUN. If
you push everything to be processed in initrd:


This is not how it works in open-iscsi. Only LUNs marked as "onboot" are 
handled in the initrd. Generally, LUNs marked "automatic" are brought up 
when the system comes up, and LUNs marked as "manual" are only brought 
up manually by the administrator.


In the bug report, Heinrich Schuchardt asked:

My current configuration is:

iscsid.conf:40:
# node.startup = automatic

nodes/iqn.2000-01.de.xypron:pine-a64-lts/192.168.0.1,3260,1/default:4:
node.startup = manual

nodes/iqn.2000-01.de.xypron:pine-a64-lts/192.168.0.1,3260,1/default:52:
node.conn[0].startup = manual

File /etc/iscsi/iscsi.initramfs specifies the root device.

HWADDR="01:02:03:04:05:06"
ISCSI_TARGET_NAME="iqn.2000-01.de.xypron:pine-a64-lts"
ISCSI_TARGET_IP="192.168.0.1"
ISCSI_TARGET_PORT="3260"
ISCSI_TARGET_GROUP="1"
ISCSI_USERNAME="user"
ISCSI_PASSWORD="password"

Where would "onboot" fit into the image?
What makes a target an "onboot" target?
Do you simply mean the one specified in /etc/iscsi/iscsi.initramfs?


Debian uses a different method than SUSE when it comes to setting up the 
initrd/initramfs.


Nodes can have 3 "startup" values, as mentioned above: onboot, manual, 
or automatic.


The one that matters most at runtime is "automatic", since those are the 
ones that iscsi.service logs into automatically, for you, when the 
system starts (if you have the service enabled).


So not having the node show up as "startup=onboot" is okay, because the 
initrd knows already it is a boot iscsi target, and because iscsid will 
refuse to terminate the session, since it was started from "firmware". 
(But, side note: it's also helpful to set "safe_logout" in iscsid.conf 
on systems with iscsi root discs.)


As far as the feature request:

https://salsa.debian.org/linux-blocks-team/open-iscsi/-/merge_requests/14


It seems fairly reasonable to me, though I'm not the one that gets to 
implement this feature, if the request is accepted. :)


It seems like it does not change the default behavior, so there is 
little risk of breakage.


I would be more than happy to offer advice and/or help, if needed, since 
I have a special place of dislike in my heart for iscsistart.


--
Lee Duncan
open-iscsi co-maintainer

Bug#982842: open-iscsi: do not use iscsistart in the boot process

[Eric Mackay]

I have a merge request to open-iscsi that would add iscsid/iscsiadm capability 
to initrd:
https://salsa.debian.org/linux-blocks-team/open-iscsi/-/merge_requests/14

The default behavior is unaffected, so users would explicitly enable this 
feature if they want it.

Using an iscsi root in initrd is of significant importance in
cloud computing, and ensuring robust and reliable boot capability
while using iscsi is best done with iscsid/iscsiadm and not iscsistart.

Bug#982842: open-iscsi: do not use iscsistart in the boot process

[Lee Duncan]

On 2/15/21 6:48 AM, Ritesh Raj Sarraf wrote:
> Control: tag -1 +help
> 
> On Mon, 2021-02-15 at 09:46 +0100, Heinrich Schuchardt wrote:
>> An upstream maintainer suggested that Debian should not use
>> iscsistart
>> for booting from an iSCSI volume but instead include iscsid and
>> iscsiadm
>> in the initrd. The same is already done in SUSE.
>>
>> Please, consider such a change for Debian Bookworm.
> 
> Yes. But usually, you don't need the iscsid daemon in initrd. Because
> most usual cases, users would be mapping data LUNs only.
> 
> Only in exceptional cases, where you have root on iSCSI, you need to
> establish the connections early. And to do that we used iscsistart,
> which would establish only a single connection, effectively making it
> prone to hangs if that single connection went down.
> 
> But the time from when iscsistart establishes the connection and
> fetched the root LUN, to the time when real init starts and the actual
> iscsid daemon is run, is not a very large time. That has been the
> assumption so far and the integration was built accordingly.
> 
> The other reason I can recollect is that you could have a very large
> number of LUNs mapped to your initiator along with your root LUN. If
> you push everything to be processed in initrd:
> 
> 1. The boot would be much slower, depending on how many LUNs are mapped
> 2. I don't know how the initiator would behave if some of the LUNs,
> from some of the targets, are temporarily unavailable.
> 
> These days, I only work on storage as a hobby. So this feature should
> be better and early co-ordinated, by users and derivatives that want to
> see it follow the path. And we could definitely leverage on what Suse
> has already done.
> 

I cannot access this bug, so I will reply to all here ...

In general, you do not set up initrd to boot into all iSCSI targets,
only the ones with "startup" set to "onboot". Then, later, as part of
the system coming up, once the real root is established and networking
is up, do you log into all "automatic" targets using open-iscsi.

So the idea isn't to log into all targets at initrd time, just the same
ones you log into now (i.e. "onboot" targets, needed to boot) using
iscsid/iscsiadm instead of iscsistart. Note that SUSE only supports the
root and /usr partitions being remote at boot time. If you have
something like /opt you want to mount, it has to be done later (in our
systems).

I hope this clarification helps.

Bug#982842: open-iscsi: do not use iscsistart in the boot process

[Heinrich Schuchardt]

On 15.02.21 18:02, Lee Duncan wrote:
> On 2/15/21 6:48 AM, Ritesh Raj Sarraf wrote:



> I cannot access this bug, so I will reply to all here ...

By CCing 982...@bugs.debian.org the bug report is updated.

>
> In general, you do not set up initrd to boot into all iSCSI targets,
> only the ones with "startup" set to "onboot". Then, later, as part of
> the system coming up, once the real root is established and networking
> is up, do you log into all "automatic" targets using open-iscsi.
>
> So the idea isn't to log into all targets at initrd time, just the same
> ones you log into now (i.e. "onboot" targets, needed to boot) using
> iscsid/iscsiadm instead of iscsistart. Note that SUSE only supports the
> root and /usr partitions being remote at boot time. If you have
> something like /opt you want to mount, it has to be done later (in our
> systems).
>
> I hope this clarification helps.
>

My current configuration is:

iscsid.conf:40:
# node.startup = automatic

nodes/iqn.2000-01.de.xypron:pine-a64-lts/192.168.0.1,3260,1/default:4:
node.startup = manual

nodes/iqn.2000-01.de.xypron:pine-a64-lts/192.168.0.1,3260,1/default:52:
node.conn[0].startup = manual

File /etc/iscsi/iscsi.initramfs specifies the root device.

HWADDR="01:02:03:04:05:06"
ISCSI_TARGET_NAME="iqn.2000-01.de.xypron:pine-a64-lts"
ISCSI_TARGET_IP="192.168.0.1"
ISCSI_TARGET_PORT="3260"
ISCSI_TARGET_GROUP="1"
ISCSI_USERNAME="user"
ISCSI_PASSWORD="password"

Where would "onboot" fit into the image?
What makes a target an "onboot" target?
Do you simply mean the one specified in /etc/iscsi/iscsi.initramfs?

Best regards

Heinrich

Bug#982842: open-iscsi: do not use iscsistart in the boot process

[Chris Hofstaedtler]

* Ritesh Raj Sarraf  [210215 15:49]:
> On Mon, 2021-02-15 at 09:46 +0100, Heinrich Schuchardt wrote:
> > An upstream maintainer suggested that Debian should not use
> > iscsistart
> > for booting from an iSCSI volume but instead include iscsid and
> > iscsiadm
> > in the initrd. The same is already done in SUSE.
> > 
> > Please, consider such a change for Debian Bookworm.

[..]
 
> These days, I only work on storage as a hobby. So this feature should
> be better and early co-ordinated, by users and derivatives that want to
> see it follow the path. And we could definitely leverage on what Suse
> has already done.

Agreed. Note: while I can spend some time on open-iscsi for work,
"root on iSCSI" is not actaully of interest to me.

Chris

Bug#982842: open-iscsi: do not use iscsistart in the boot process

[Ritesh Raj Sarraf]

Control: tag -1 +help

On Mon, 2021-02-15 at 09:46 +0100, Heinrich Schuchardt wrote:
> An upstream maintainer suggested that Debian should not use
> iscsistart
> for booting from an iSCSI volume but instead include iscsid and
> iscsiadm
> in the initrd. The same is already done in SUSE.
> 
> Please, consider such a change for Debian Bookworm.

Yes. But usually, you don't need the iscsid daemon in initrd. Because
most usual cases, users would be mapping data LUNs only.

Only in exceptional cases, where you have root on iSCSI, you need to
establish the connections early. And to do that we used iscsistart,
which would establish only a single connection, effectively making it
prone to hangs if that single connection went down.

But the time from when iscsistart establishes the connection and
fetched the root LUN, to the time when real init starts and the actual
iscsid daemon is run, is not a very large time. That has been the
assumption so far and the integration was built accordingly.

The other reason I can recollect is that you could have a very large
number of LUNs mapped to your initiator along with your root LUN. If
you push everything to be processed in initrd:

1. The boot would be much slower, depending on how many LUNs are mapped
2. I don't know how the initiator would behave if some of the LUNs,
from some of the targets, are temporarily unavailable.

These days, I only work on storage as a hobby. So this feature should
be better and early co-ordinated, by users and derivatives that want to
see it follow the path. And we could definitely leverage on what Suse
has already done.


-- 
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System


signature.asc
Description: This is a digitally signed message part

Bug#982842: open-iscsi: do not use iscsistart in the boot process

[Heinrich Schuchardt]

Package: open-iscsi
Version: 2.1.3-2
Severity: wishlist

An upstream maintainer suggested that Debian should not use iscsistart
for booting from an iSCSI volume but instead include iscsid and iscsiadm
in the initrd. The same is already done in SUSE.

Please, consider such a change for Debian Bookworm.

 Forwarded Message 
Subject:Re: [open-iscsi/open-iscsi] iscsistart fails with 15 - session
exists with ipv6 (#241)
Date:   Fri, 12 Feb 2021 15:29:41 -0800
From:   Lee Duncan 
Reply-To:   open-iscsi/open-iscsi

To: open-iscsi/open-iscsi 
CC: Heinrich Schuchardt , Mention




@gonzoleeman 

Let me emphasize again: iscsistart is not great. Using
iscsiadm/iscsid is still the recommended way to use open-iscsi.

Debian uses iscsistart when booting from iSCSI. Is this the wrong
way to do it?

Cf.

https://salsa.debian.org/linux-blocks-team/open-iscsi/-/blob/master/debian/extra/initramfs.hook



The proper way to use open-iscsi is to (1) start the daemon, then (2)
use iscsiadm to talk to the daemon.

There are a couple of problems with the iscsistart approach: first, it
has no error handling. All the error handling is built into iscsid.
Sure, some of it might have gotten copied to iscsistart, but not most of
it. For example, once iscsistart exits, there is *no* error handling at
all. The second problem is that iscsistart is not maintained, since it's
not used as much. It also has other issues, like not setting up the
session correctly, so that trying to log out of that session fails on
the first attempt.

I'm fairly sure this approach was taken because it's more complex to
start them daemon then use iscsiadm. But it's not a lot more complex.
That's the approach we take at SUSE. To be fair, there are shortcomings
even when using iscsid/iscsiadm. In our case, when we switch from
virtual root to the real root disc, we stop then restart the iscsid
daemon. So even in our case there is a window when no error handling can
occur.

But iscsistart is still part of the open-iscsi distribution, so until
and unless somebody removes it, it's supported, even if I like to
discourage it's use. (I also discourage use of iscsi NOPs, if that
matters, so this isn't the only windmill I tilt at.)

I hope that answers your question. If you wanted to convert debian to
using iscsid/iscsiadm, I'd be glad to help!