Hi Bob,

I agree, it's a hard problem. I'm just trying to understand that we've done the 
absolute best we can and that if this condition is hit then the best solution 
really is to just kill the node. I guess it's also a question of how common 
this actually ends up being. We have now got customers starting to use GFS2 for 
VM storage on XenServer so I guess we'll just have to see how many support 
calls we get in on it.



-----Original Message-----
From: Bob Peterson <rpete...@redhat.com> 
Sent: 17 December 2018 20:20
To: Mark Syms <mark.s...@citrix.com>
Cc: cluster-devel@redhat.com
Subject: Re: [Cluster-devel] [GFS2 PATCH] gfs2: Panic when an io error occurs 

----- Original Message -----
> I think a reasonable alternative to this is to just sit in an infinite 
> retry loop until the i/o succeeds.
> Dave
> [Mark Syms] I would hope that this code would only trigger after some 
> effort has been put into  retrying as panicing the host on the first 
> I/O failure seems like a sure fire way to get unhappy users (and in 
> our case paying customers). As Edvin points out there may be other 
> filesystems that may be able to cleanly unmount and thus avoid having 
> to check everything on restart.

Hi Mark,

Perhaps. I'm not block layer or iscsi expert, but afaik, it's not the file 
system's job to retry IO, and never has been, right?

There are already iscsi tuning parameters, vfs tuning parameters, etc. So if an 
IO error is sent to GFS2 for a write operation, it means the retry algorithms 
and operation timeout algorithms built into the layers below us (the iscsi 
layer, scsi layer, block layer, tcp/ip layer etc.) have all failed and given up 
on the IO operation. We can't really justify adding yet another layer of 
retries on top of all that, can we?

I see your point, and perhaps the system should stay up to continue other 
mission-critical operations that may not require the faulted hardware.
But what's a viable alternative? As Dave T. suggested, we can keep resubmitting 
the IO until it completes, but then the journal never gets replayed and nobody 
can have those locks ever again, and that would cause a potential hang of the 
entire cluster, especially in cases where there's only one device that's failed 
and the whole cluster is using it.

In GFS2, we've got a concept of marking a resource group "in error" so the 
other nodes won't try to use it, but the same corruption that affects resource 
groups could be extrapolated to "hot" dinodes as well. For example, suppose the 
root (mount-point) dinode was in the journal. Now the whole cluster is hung 
rather than just the one node.


Bob Peterson
Red Hat File Systems

Reply via email to