Hi Bob, I agree, it's a hard problem. I'm just trying to understand that we've done the absolute best we can and that if this condition is hit then the best solution really is to just kill the node. I guess it's also a question of how common this actually ends up being. We have now got customers starting to use GFS2 for VM storage on XenServer so I guess we'll just have to see how many support calls we get in on it.
Thanks, Mark. -----Original Message----- From: Bob Peterson <rpete...@redhat.com> Sent: 17 December 2018 20:20 To: Mark Syms <mark.s...@citrix.com> Cc: firstname.lastname@example.org Subject: Re: [Cluster-devel] [GFS2 PATCH] gfs2: Panic when an io error occurs writing ----- Original Message ----- > I think a reasonable alternative to this is to just sit in an infinite > retry loop until the i/o succeeds. > > Dave > [Mark Syms] I would hope that this code would only trigger after some > effort has been put into retrying as panicing the host on the first > I/O failure seems like a sure fire way to get unhappy users (and in > our case paying customers). As Edvin points out there may be other > filesystems that may be able to cleanly unmount and thus avoid having > to check everything on restart. Hi Mark, Perhaps. I'm not block layer or iscsi expert, but afaik, it's not the file system's job to retry IO, and never has been, right? There are already iscsi tuning parameters, vfs tuning parameters, etc. So if an IO error is sent to GFS2 for a write operation, it means the retry algorithms and operation timeout algorithms built into the layers below us (the iscsi layer, scsi layer, block layer, tcp/ip layer etc.) have all failed and given up on the IO operation. We can't really justify adding yet another layer of retries on top of all that, can we? I see your point, and perhaps the system should stay up to continue other mission-critical operations that may not require the faulted hardware. But what's a viable alternative? As Dave T. suggested, we can keep resubmitting the IO until it completes, but then the journal never gets replayed and nobody can have those locks ever again, and that would cause a potential hang of the entire cluster, especially in cases where there's only one device that's failed and the whole cluster is using it. In GFS2, we've got a concept of marking a resource group "in error" so the other nodes won't try to use it, but the same corruption that affects resource groups could be extrapolated to "hot" dinodes as well. For example, suppose the root (mount-point) dinode was in the journal. Now the whole cluster is hung rather than just the one node. Regards, Bob Peterson Red Hat File Systems