Hi, I think we're not staying completely on-point.
I agree with Wei's suggestion that different transports likely need different affinity strategies, and presumably you do too. There is probably no big issue with affinity syscalls--we should not do them inline with work, but different pools (e.g.) could resolve that. The RPC subsystem does has at least one group of threads (sender) that TCP consumers care about, so, it's legit to look at affinity for it, I think. Matt ----- "William Allen Simpson" <william.allen.simp...@gmail.com> wrote: > On 9/28/15 4:48 AM, Wei Fang wrote: > > On 2015/9/25 2:06, William Allen Simpson wrote: > >> Do any of these make a system call? Then they are not useful. > >> This thread pool has been designed to have only one system call > >> per invocation -- or better yet continue processing the next > >> task without a system call. > > > > Yes, there has a system call in shed_setaffinity. But I don't > understand > > why they can't make a system call. > > This is a userland program, the whole point is to run with limited > system calls. Otherwise, we'd just use the Linux nfsd.... > > > > Do you mean that it may lower > > performance of threads? I think it depends on how much work the > thread > > is going to do. > > > There seems to be some confusion here. Every system call limits > the performance. It has nothing what-so-ever to do with how much > work the thread is going to do. > > If the thread has a little to do, you'll have serious latency > issues and cap the ops-per-second. This is already the case, > and we're trying to fix it. > > If the thread has a lot to do, the system calls may be amortized > over the longer work load. But the latency remains. > > > > I do have tested it on Linux, and obtained some performance > benefits. > > > What's the increase in ops-per-second? What environment? > > > > May those users use the same work pool simultaneously? I thought > every > > user has it's own pool, and every pool can set it's own CPU > affinity > > separately. > > > All users share this same work pool. So each connection will > grab a worker. And each request over the connection will > grab another worker, although that worker should continue > using the same thread for many subsequent requests. > > What happened is you tested without RDMA (the primary user of > this pool). Moreover, likely many requests were over one > connection. So there was only one main user. But that's not a > typical workload. > > > >> Your current code has them oddly counting themselves. No idea > >> how this would work. > > > > I count the threads created in the pool and use it to split the > threads to > > different CPUs. There's problem I mentioned on the TODO list in my > patch, > > and should do more work on this later. > > > Looks like you're depending on serendipity for affinity. > > As I mentioned in the next message, perhaps combined with > Malahal's lanes gives us some affinity per connection. > > But my biggest objection is the additional system overhead. > Pin the CPU assignment algorithmically at thread creation. > Minimize system calls. -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 ------------------------------------------------------------------------------ _______________________________________________ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
