Hi William, On 2015/9/28 22:55, William Allen Simpson wrote: > On 9/28/15 4:48 AM, Wei Fang wrote: >> On 2015/9/25 2:06, William Allen Simpson wrote: >>> Do any of these make a system call? Then they are not useful. >>> This thread pool has been designed to have only one system call >>> per invocation -- or better yet continue processing the next >>> task without a system call. >> >> Yes, there has a system call in shed_setaffinity. But I don't understand >> why they can't make a system call. > > This is a userland program, the whole point is to run with limited > system calls. Otherwise, we'd just use the Linux nfsd.... > >> Do you mean that it may lower >> performance of threads? I think it depends on how much work the thread >> is going to do. >> > There seems to be some confusion here. Every system call limits > the performance. It has nothing what-so-ever to do with how much > work the thread is going to do. > > If the thread has a little to do, you'll have serious latency > issues and cap the ops-per-second. This is already the case, > and we're trying to fix it. > > If the thread has a lot to do, the system calls may be amortized > over the longer work load. But the latency remains. > >> I do have tested it on Linux, and obtained some performance benefits. >> > What's the increase in ops-per-second? What environment?
One of our testcases is simple. We used two hosts with Intel Xeon CPU (2 nodes, 6 cores per node), one for server, one for client, with a 10 Gbps network connection between them. ganesha.nfsd was running in one node. We ran fio on client to test 4k random reads IOPS: * if not bind: 107k IOPS * bind the decoder/worker/sender threads to different CPUs: 124k IOPS Other information may be cared: * Linux kernel version: 4.2.0 * transport layer protocol: TCP * nfs protocol: nfs v3 * local filesystem: xfs * fio options: psync, direct, 256 jobs I'm willing to share more details if you are interesting. thanks, Wei >> May those users use the same work pool simultaneously? I thought every >> user has it's own pool, and every pool can set it's own CPU affinity >> separately. >> > All users share this same work pool. So each connection will > grab a worker. And each request over the connection will > grab another worker, although that worker should continue > using the same thread for many subsequent requests. > > What happened is you tested without RDMA (the primary user of > this pool). Moreover, likely many requests were over one > connection. So there was only one main user. But that's not a > typical workload. > > >>> Your current code has them oddly counting themselves. No idea >>> how this would work. >> >> I count the threads created in the pool and use it to split the threads to >> different CPUs. There's problem I mentioned on the TODO list in my patch, >> and should do more work on this later. >> > Looks like you're depending on serendipity for affinity. > > As I mentioned in the next message, perhaps combined with > Malahal's lanes gives us some affinity per connection. > > But my biggest objection is the additional system overhead. > Pin the CPU assignment algorithmically at thread creation. > Minimize system calls. > > > . > ------------------------------------------------------------------------------ _______________________________________________ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
