Re: [systemd-devel] [dm-devel] RFC: one more time: SCSI device identification
On Wed, 2021-04-28 at 10:09 +1000, Erwin van Londen wrote: > > On Tue, 2021-04-27 at 16:41 -0400, Ewan D. Milne wrote: > > On Tue, 2021-04-27 at 20:33 +, Martin Wilck wrote: > > > On Tue, 2021-04-27 at 16:14 -0400, Ewan D. Milne wrote: > > > > There's no way to do that, in principle. Because there could > > > > be > > > > other I/Os in flight. You might (somehow) avoid retrying an > > > > I/O > > > > that got a UA until you figured out if something changed, but > > > > other > > > > I/Os can already have been sent to the target, or issued before > > > > you > > > > get to look at the status. > > If something happens on a storage side where a lun gets it's > attributes changed (any, doesn't matter which one) a UA should be > sent. Also all outstanding IO's on that lun should be returning an > Abort as it can no longer warrant the validity of any IO due to these > changes. Especially when parameters are involved like reservations > (PR's) etc. If that does not happen from an array side all bets are > off as the only way to be able to get back in business is to start > from scratch. Perhaps an array might abort I/Os it has received in the Device Server whensomething changes. I have no idea if most or any arrays actually do that. But, what about I/O that has already been queued from the host to thehost bus adapter? I don't see how we can abort those I/Os properly.Most high-performance HBAs have a queue of commands and a queueof responses, there could be lots of commands queued before wemanage to notice an interesting status. And AFAIK there is no conditionalmechanism that could hold them off (and, they could be in- flight on thewire anyway). I get what you are saying about what SAM describes, I just don't see howwe can guarantee we don't send any further commands after the statuswith the UA is sent back, before we can understand what happened. -Ewan > > > > > > Right. But in practice, a WWID change will hardly happen under > > > full > > > IO > > > load. The storage side will probably have to block IO while this > > > happens, at least for a short time period. So blocking and > > > quiescing > > > the queue upon an UA might still work, most of the time. Even if > > > we > > > were too late already, the sooner we stop the queue, the better. > > I think in most cases when something happens on an array side you > will see IO's being aborted. That might be a good time to start doing > TUR's and if these come back OK do a new inquiry. From a host side > there is only so much you can do. > > > > The current algorithm in multipath-tools needs to detect a path > > > going > > > down and being reinstated. The time interval during which a WWID > > > change > > > will go unnoticed is one or more path checker intervals, > > > typically on > > > the order of 5-30 seconds. If we could decrease this interval to > > > a > > > sub- > > > second or even millisecond range by blocking the queue in the > > > kernel > > > quickly, we'd have made a big step forward. > > > > Yes, and in many situations this may help. But in the general case > > we can't protect against a storage array misconfiguration, > > where something like this can happen. So I worry about people > > believing the host software will protect them against a mistake, > > when we can't really do that. > > My thought exactly. > > > All it takes is one I/O (a discard) to make a thorough mess of the > > LUN. > > > > -Ewan > > > > > Regards > > > Martin > > > > > > > -- > > dm-devel mailing list > > dm-de...@redhat.com > > https://listman.redhat.com/mailman/listinfo/dm-devel > > > > --dm-devel mailing listdm-de...@redhat.com > https://listman.redhat.com/mailman/listinfo/dm-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] RFC: one more time: SCSI device identification
On Tue, 2021-04-27 at 20:33 +, Martin Wilck wrote: > On Tue, 2021-04-27 at 16:14 -0400, Ewan D. Milne wrote: > > > > There's no way to do that, in principle. Because there could be > > other I/Os in flight. You might (somehow) avoid retrying an I/O > > that got a UA until you figured out if something changed, but other > > I/Os can already have been sent to the target, or issued before you > > get to look at the status. > > Right. But in practice, a WWID change will hardly happen under full > IO > load. The storage side will probably have to block IO while this > happens, at least for a short time period. So blocking and quiescing > the queue upon an UA might still work, most of the time. Even if we > were too late already, the sooner we stop the queue, the better. > > The current algorithm in multipath-tools needs to detect a path going > down and being reinstated. The time interval during which a WWID > change > will go unnoticed is one or more path checker intervals, typically on > the order of 5-30 seconds. If we could decrease this interval to a > sub- > second or even millisecond range by blocking the queue in the kernel > quickly, we'd have made a big step forward. Yes, and in many situations this may help. But in the general case we can't protect against a storage array misconfiguration, where something like this can happen. So I worry about people believing the host software will protect them against a mistake, when we can't really do that. All it takes is one I/O (a discard) to make a thorough mess of the LUN. -Ewan > > Regards > Martin > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] RFC: one more time: SCSI device identification
On Mon, 2021-04-26 at 13:16 +, Martin Wilck wrote: > On Mon, 2021-04-26 at 13:14 +0200, Ulrich Windl wrote: > > > > > > > > > > While we're at it, I'd like to mention another issue: WWID > > > changes. > > > > > > This is a big problem for multipathd. The gist is that the device > > > identification attributes in sysfs only change after rescanning > > > the > > > device. Thus if a user changes LUN assignments on a storage > > > system, > > > it can happen that a direct INQUIRY returns a different WWID as > > > in > > > sysfs, which is fatal. If we plan to rely more on sysfs for > > > device > > > identification in the future, the problem gets worse. > > > > I think many devices rely on the fact that they are identified by > > Vendor/model/serial_nr, because in most professional SAN storage > > systems you > > can pre-set the serial number to a custom value; so if you want a > > new > > disk > > (maybe a snapshot) to be compatible with the old one, just assign > > the > > same > > serial number. I guess that's the idea behind. > > What you are saying sounds dangerous to me. If a snapshot has the > same > WWID as the device it's a snapshot of, it must not be exposed to any > host(s) at the same time with its origin, otherwise the host may > happily combine it with the origin into one multipath map, and data > corruption will almost certainly result. > > My argument is about how the host is supposed to deal with a WWID > change if it happens. Here, "WWID change" means that a given H:C:T:L > suddenly exposes different device designators than it used to, while > this device is in use by a host. Here, too, data corruption is > imminent, and can happen in a blink of an eye. To avoid this, several > things are needed: > > 1) the host needs to get notified about the change (likely by an UA > of > some sort) > 2) the kernel needs to react to the notification immediately, e.g. > by > blocking IO to the device, There's no way to do that, in principle. Because there could be other I/Os in flight. You might (somehow) avoid retrying an I/O that got a UA until you figured out if something changed, but other I/Os can already have been sent to the target, or issued before you get to look at the status. -Ewan > 3) userspace tooling such as udev or multipathd need to figure out > how > to how to deal with the situation cleanly, and eventually unblock > it. > > Wrt 1), we can only hope that it's the case. But 2) and 3) need work, > afaics. > > Martin > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Antw: [EXT] Re: [dm-devel] RFC: one more time: SCSI device identification
On Tue, 2021-04-27 at 12:52 +0200, Ulrich Windl wrote: > > > > Hannes Reinecke schrieb am 27.04.2021 um 10:21 > > > > in Nachricht > > <2a6903e4-ff2b-67d5-e772-6971db844...@suse.de>: > > On 4/27/21 10:10 AM, Martin Wilck wrote: > > > On Tue, 2021‑04‑27 at 13:48 +1000, Erwin van Londen wrote: > > > > > > > > > > Wrt 1), we can only hope that it's the case. But 2) and 3) > > > > > need work, > > > > > afaics. > > > > > > > > > > > > > In my view the WWID should never change. > > > > > > In an ideal world, perhaps not. But in the dm‑multipath realm, we > > > know > > > that WWID changes can happen with certain storage arrays. See > > > https://listman.redhat.com/archives/dm‑devel/2021‑February/msg00116.html > > > > > > and follow‑ups, for example. > > > > > > > And it's actually something which might happen quite easily. > > The storage array can unmap a LUN, delete it, create a new one, and > > map > > that one into the same LUN number than the old one. > > If we didn't do I/O during that interval upon the next I/O we will > > be > > getting the dreaded 'Power‑On/Reset' sense code. > > _And nothing else_, due to the arcane rules for sense code > > generation in > > SAM. > > But we end up with a completely different device. > > > > The only way out of it is to do a rescan for every POR sense code, > > and > > disable the device eg via DID_NO_CONNECT whenever we find that the > > identification has changed. We already have a copy of the original > > VPD > > page 0x83 at hand, so that should be reasonably easy. > > I don't know the depth of the SCSI or FC protocol, but storage > systems > typically signal such events, maybe either via some unit attention or > some FC > event. Older kernels logged that there was a change, but a manual > SCSI bus scan > is needed, while newer kernels find new devices "automagically" for > some > products. The HP EVA 6000 series wored that way, a 3PAR SotorServ > 8000 series > also seems to work that way, but not Pure Storage X70 R3. FOr the > latter you > need something like a FC LIP to make the kernel detect the new > devices (LUNs). > I'm unsure where the problem is, but in principle the kernel can be > notified... There has to be some command on which the Unit Attention status can be returned. (In a multipath configuration, the path checker commands may do this). In absence of a command, there is no asynchronous mechanism in SCSI to report the status. On FC things related to finding a remote port will trigger a rescan. -Ewan > > > > > I had a rather lengthy discussion with Fred Knight @ NetApp about > > Power‑On/Reset handling, what with him complaining that we don't > > handle > > is correctly. So this really is something we should be looking > > into, > > even independently of multipathing. > > > > But actually I like the idea from Martin Petersen to expose the > > parsed > > VPD identifiers to sysfs; that would allow us to drop sg_inq > > completely > > from the udev rules. > > Talking of VPDs: Somewhere in the last 12 years (within SLES 11)there > was a > kernel change regarding trailing blanks in VPD data. That change blew > up > several configurations being unable to re-recognize the devices. In > one case > the software even had bound a license to a specific device with > serial number, > and that software found "new" devices while missing the "old" ones... > > Regards, > Ulrich > > > > > Cheers, > > > > Hannes > > ‑‑ > > Dr. Hannes Reinecke Kernel Storage Architect > > h...@suse.de +49 911 74053 > > 688 > > SUSE Software Solutions Germany GmbH, 90409 Nürnberg > > GF: F. Imendörffer, HRB 36809 (AG Nürnberg) > > > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
