So my question is what allocations do we attempt to recover from? And what
does that recovery look like? And how do we make sure Ganesha is actually
running in a sane way if we do recover? And are we just kicking the can to
the next allocation which chooses not to recover? It seems like if we are
going to try and keep running, we should do so in almost all cases, using
abort only for those cases that are just way too complex to recover from
(for example, there is an out of memory condition in unlock if lock owners
aren't supported where it can become impossible to get to a correct set of
locks).

 

Frank

 

From: Marc Eshel [mailto:es...@us.ibm.com] 
Sent: Wednesday, October 28, 2015 7:38 PM
To: Frank Filz <ffilz...@mindspring.com>
Cc: nfs-ganesha-devel@lists.sourceforge.net
Subject: Re: [Nfs-ganesha-devel] Topic for discussion - Out of Memory
Handling

 

I don't believe that we need to restart Ganesha on every out of memory calls
for many reasons, but I will agree that we can have two types or calls one
that can accept no memory rc and one that terminate Ganesha if the call is
not successful.   
Marc. 



From:        "Frank Filz" <ffilz...@mindspring.com
<mailto:ffilz...@mindspring.com> > 
To:        <nfs-ganesha-devel@lists.sourceforge.net
<mailto:nfs-ganesha-devel@lists.sourceforge.net> > 
Date:        10/28/2015 11:55 AM 
Subject:        [Nfs-ganesha-devel] Topic for discussion - Out of Memory
Handling 

  _____  




We have had various discussions over the years as to how to best handle out
of memory conditions.

In the meantime, our code is littered with attempts to handle the situation,
however, it is not clear to me these really solve anything. If we don't have
100% recoverability, likely we just delay the crash. Even if we manage to
avoid crashing, we may wobble along not really handling things well, causing
retry storms and such (that just dig us in deeper). Another possibility is
we return an error to the client that gets translated into EIO or some other
error the application isn't prepared to handle.

If instead, we just aborted, the HA systems most of us run under would
restart Ganesha. The clients would see some delay, but there should be no
visible errors to the clients. Depending on how well grace period/state
recovery is implemented (and in particular how well it's integrated with
other file servers such as CIFS/SMB or across a cluster), there could be
some openings for lock violation (someone is able to steal a lock from one
of our clients while Ganesha is down).

Aborting would have several advantages. First, it would immediately clear up
any memory leaks. Second, if there was some transient activity that resulted
in high memory utilization, that might also be cleared up. Third, it would
avoid retry storms and such that might just aggravate the low memory
condition. In addition, it would force the sysadmin to deal with a workload
that overloaded the server, possibly by adding additional nodes in a
clustered environment, or adding memory to the server.

No matter what we decide to do, another thing we need to look at is more
memory throttling. Cache inode has a limit on the number of inodes. This is
helpful, but is incomplete. Other candidates for memory throttling would be:

Number of clients
Number of state (opens, locks, delegations, layouts) (per client and/or
global)
Size of ACLs and number of ACLs cached

I'm sure there's more, discuss.

Frank


---
This email has been checked for viruses by Avast antivirus software.
 <https://www.avast.com/antivirus> https://www.avast.com/antivirus


----------------------------------------------------------------------------
--
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
<mailto:Nfs-ganesha-devel@lists.sourceforge.net> 
 <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel>
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel






---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
------------------------------------------------------------------------------
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Reply via email to