Hi, On Tue, Feb 20, 2018 at 8:12 AM, William Allen Simpson <william.allen.simp...@gmail.com> wrote: > On 2/18/18 2:47 PM, Matt Benjamin wrote: >> >> On Fri, Feb 16, 2018 at 11:23 AM, William Allen Simpson >>> >>> But the planned 2.7 improvements are mostly throughput related, not IOPS. >> >> >> Not at all, though I am trying to ensure that we get async FSAL ops >> in. There are people working on IOPs too. >> > async FSAL ops are not likely to further improve IOPS. > > As I've pointed out many times in the past, async only allows > the same number of threads to handle more concurrent operations. > > But it's actually slower. It basically doubles the number of > system calls. System calls are one of the reasons that Ganesha is > slower than knfsd. It also ruins CPU cache coherency. > > If we're CPU bound or network bandwidth constrained, it won't help.
Not everything being worked on is a direct answer to this problem. > > The effort that I've been waiting for *3* years -- add io vectors to > the FSAL interface, so that we can zero-copy buffers -- is more > likely to improve throughput. Which, as you've noted, also is happening. > > Moreover, going through the code and removing locking other such > bottlenecks is more likely to improve IOPS. No one is disputing this. It is not a new discovery, however. > > >>> If Ganesha is adding 6 ms to every read operation, we have a serious >>> problem, and need to profile immediately! >>> >> >> That's kind of what our team is doing. I look forward to your work >> with rpc-ping-mt. >> > Well, you were able to send me a copy after 6 pm on a Friday, so I'm > now taking a look at it. Hopefully I'll have something by Friday. It came up in a meeting on Friday, that's why I sent it Friday. I got busy in meetings and issues, that's why after 6 pm. > > But I really wish you'd posted it 3 years ago. It doesn't really test > IOPS, other than whatever bandwidth limit is imposed by the interface, > but it does test the client call interface. It measures minimal request latency all the way to nfs-ganesha's, or the Linux kernel's, rpcnull handler--the "top" of the request handling stack, in the given client/server/network configuration. Scaled up, it can report the max such calls/s, which is a kind of best-possible value for iops, taking FSAL ops to have 0 latency. It was posted to this list by Tigran, iirc, in 2011 or 2012. > > We've been using Jeff Layton's delegation callback work to test, and > this test would have been better and easier. > > But a unit test is not what we need. I wrote "profile". We need to > know where the CPU bottlenecks are in Ganesha itself. You also wrote unit test. Matt -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
