Re: [OmniOS-discuss] OmniOS NFS fileserver hanging under sustained high write loads

2015-05-05 Thread Dan McDonald
On May 4, 2015, at 9:03 PM, Dan McDonald dan...@omniti.com wrote: I swear I've seen someone try to address this before. Maybe it's from my Nexenta days. I will be querying the illumos developer's list (as I suspect this affects the other distros as well if they haven't fixed it in their

Re: [OmniOS-discuss] OmniOS NFS fileserver hanging under sustained high write loads

2015-05-05 Thread Chris Siebenmann
On May 4, 2015, at 9:03 PM, Dan McDonald dan...@omniti.com wrote: I swear I've seen someone try to address this before. Maybe it's from = my Nexenta days. I will be querying the illumos developer's list (as I = suspect this affects the other distros as well if they haven't fixed it = in

Re: [OmniOS-discuss] OmniOS NFS fileserver hanging under sustained high write loads

2015-05-05 Thread Dan McDonald
On May 5, 2015, at 11:48 AM, Chris Siebenmann c...@cs.toronto.edu wrote: On May 4, 2015, at 9:03 PM, Dan McDonald dan...@omniti.com wrote: I swear I've seen someone try to address this before. Maybe it's from = my Nexenta days. I will be querying the illumos developer's list (as I =

Re: [OmniOS-discuss] OmniOS NFS fileserver hanging under sustained high write loads

2015-05-05 Thread Chris Siebenmann
The hard part will be testing this. I'm not sure I have the HW in-house to do it. I may need illumos community help. Since we have a test environment where we can reproduce this and a high interest in seeing it fixed, we can test new kernel packages and so on. (If given specific

Re: [OmniOS-discuss] OmniOS NFS fileserver hanging under sustained high write loads

2015-05-05 Thread Doug Hughes
I managed to get my system in a state with dd test across a bunch of client nodes (4k writes, many nodes in parallel, all to the same file -- by mistake, I meant to do many files), that all of the ttys except for /dev/console are stuck. It was showing signs of desparation swapping a few times, but

Re: [OmniOS-discuss] OmniOS NFS fileserver hanging under sustained high write loads

2015-05-04 Thread Doug Hughes
Yes, absolutely. We've run into this same problem, exactly as you describe, in Solaris10 (all versions) You can catch it with a kernel dump, but you have to be wary and quick. keep a vmstat 3 open (or similar), and when free mem drops below 5GB or so, be ready. As soon you start seeing PO or

[OmniOS-discuss] OmniOS NFS fileserver hanging under sustained high write loads

2015-05-04 Thread Chris Siebenmann
We now have a reproducable setup with OmniOS r151014 where an OmniOS NFS fileserver will experience memory exhaustion and then hang in the kernel if it receives sustained NFS write traffic from multiple clients at a rate faster than its local disks can sustain. The machine will run okay for a

Re: [OmniOS-discuss] OmniOS NFS fileserver hanging under sustained high write loads

2015-05-04 Thread Dan McDonald
On May 4, 2015, at 5:45 PM, Chris Siebenmann c...@cs.toronto.edu wrote: Is there any way to limit the number of pending NFS requests that the system will accept? Allowing 270,000 strikes me as kind of absurd. I swear I've seen someone try to address this before. Maybe it's from my