Re: RFC how to use kernel procs/threads efficiently
On 10/10/17 8:33 pm, Rick Macklem wrote: Julian Elischer wrote: [stuff snipped] On 10/10/17 4:25 am, Rick Macklem wrote: --> As such, having a fixed reasonable # of threads is probably the best that can be done. - The current patch has the # of threads as a sysctl with a default of 32. why not set it to ncpu or something? Well, each of these threads will do an RPC, which means a couple of short bursts of CPU and then sleep the rest of the time waiting for the RPC reply to come back from the Data Server. As such, it would seem to me that you would want a lot more threads than CPUs on the machine? However, setting the default to "N * ncpu" seems better than just a fixed "32" to me. (For nfsd, the current default is 8 * ncpu, so maybe that is a good default for this too?) yeah I really just meant "some function of ncpu".. not specifically "ncpu x 1" What do you think? Thanks for the comment, rick ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: RFC how to use kernel procs/threads efficiently
Julian Elischer wrote: [stuff snipped] >On 10/10/17 4:25 am, Rick Macklem wrote: >> --> As such, having a fixed reasonable # of threads is probably the best >>that can be done. >>- The current patch has the # of threads as a sysctl with a default >> of 32. >why not set it to ncpu or something? Well, each of these threads will do an RPC, which means a couple of short bursts of CPU and then sleep the rest of the time waiting for the RPC reply to come back from the Data Server. As such, it would seem to me that you would want a lot more threads than CPUs on the machine? However, setting the default to "N * ncpu" seems better than just a fixed "32" to me. (For nfsd, the current default is 8 * ncpu, so maybe that is a good default for this too?) What do you think? Thanks for the comment, rick ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: RFC how to use kernel procs/threads efficiently
On 10/10/17 4:25 am, Rick Macklem wrote: Ian Lepore wrote: [stuff snipped] taskqueue(9) is an existing mechanism to enqueue functions to execute asynch using a pool of threads, but it doesn't answer the scalability questions. In fact it may make them harder, inasmuch as I don't think there's a mechanism to dynamically adjust the number of threads after first calling taskqueue_start_threads(). I've coded it using taskqueue and it seems to work ok. The patch is here, in case anyone would like to review it: https://www.reviews.freebsd.org/D12632 I don't know what the overheads are (or even how to measure/compare them), but I suspect it is less than a kproc_create()/kproc_exit() for every RPC. I also don't think having a fixed # of threads is a problem. Since NFS I/O is so bursty, recent I/O activity doesn't give a good indication of how many threads will be needed soon. In other words, it can go from no I/O to heavy I/O and back to no I/O rapidly. --> As such, having a fixed reasonable # of threads is probably the best that can be done. - The current patch has the # of threads as a sysctl with a default of 32. why not set it to ncpu or something? Thanks for your comments and feel free to review it, if you'd like, rick ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: RFC how to use kernel procs/threads efficiently
Ian Lepore wrote: [stuff snipped] >taskqueue(9) is an existing mechanism to enqueue functions to execute >asynch using a pool of threads, but it doesn't answer the scalability >questions. In fact it may make them harder, inasmuch as I don't think >there's a mechanism to dynamically adjust the number of threads after >first calling taskqueue_start_threads(). I've coded it using taskqueue and it seems to work ok. The patch is here, in case anyone would like to review it: https://www.reviews.freebsd.org/D12632 I don't know what the overheads are (or even how to measure/compare them), but I suspect it is less than a kproc_create()/kproc_exit() for every RPC. I also don't think having a fixed # of threads is a problem. Since NFS I/O is so bursty, recent I/O activity doesn't give a good indication of how many threads will be needed soon. In other words, it can go from no I/O to heavy I/O and back to no I/O rapidly. --> As such, having a fixed reasonable # of threads is probably the best that can be done. - The current patch has the # of threads as a sysctl with a default of 32. Thanks for your comments and feel free to review it, if you'd like, rick ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: RFC how to use kernel procs/threads efficiently
Ian Lepore wrote: >On Fri, 2017-10-06 at 19:02 +, Rick Macklem wrote: >> Hi, >> >> I have now dropped the client side of Flexible File Layout for pNFS into head >> and I believe it is basically working. >> Currently when talking to mirrored DS servers, it does the Write and Commit >> RPCs to the mirrors serially. This works, but is inefficient w.r.t. elapsed >> to to >> completion. >> >> To do them concurrently, I need separate kernel processes/threads to do them. >> I can think of two ways to do this: >> 1 - The code that I have running in projects/pnfs-planb-server for the pNFS >> server >> side does a kproc_create() to create a kernel process that does the >> RPC and >> then krpc_exit()s. >> - This was easy to code and works. However, I am concerned that there >> is >> going to be excessive overheads from doing all the kproc_create()s >> and >> kproc_exit()s? >>Anyone know if these calls will result in large overheads? >> 2 - I haven't coded this, but the other way I can think of to do this is to >> create a pool of threads (kthread_create() is sufficient in this case, >> I >> think?) and then hand each RPC to an available thread so it can do the >> RPC. >> - Other than a little more complex coding, the main issue I see with >> this one >> is "How many threads and when to create more/less of them.". >> >> Anyhow, any comments w.r.t. the merits of either of the above approaches >> (or a suggestion of other ways to do this) would be appreciated, rick > >taskqueue(9) is an existing mechanism to enqueue functions to execute >asynch using a pool of threads, but it doesn't answer the scalability >questions. In fact it may make them harder, inasmuch as I don't think >there's a mechanism to dynamically adjust the number of threads after >first calling taskqueue_start_threads(). Hmm, yes. Thanks for the pointer. I hadn't read "man taskqueue" until now. The kernel RPC doesn't use this and I suspect that it is because of what you said w.r.t. dynamically adjusting the # of threads. However, it does save "hand coding" the queues for #2 and I'm lazy (plus don't believe reinventing the wheel is the best plan). I think I will try using taskqueue and just have a sysctl for #of-threads. (Actually most of the code ends up the same, because basically they all end up with a function with a single argument that does the RPC. The only difference is what call starts the RPC.) Anyone else have comments? rick ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: RFC how to use kernel procs/threads efficiently
On Fri, 2017-10-06 at 19:02 +, Rick Macklem wrote: > Hi, > > I have now dropped the client side of Flexible File Layout for pNFS into head > and I believe it is basically working. > Currently when talking to mirrored DS servers, it does the Write and Commit > RPCs to the mirrors serially. This works, but is inefficient w.r.t. elapsed > to to > completion. > > To do them concurrently, I need separate kernel processes/threads to do them. > I can think of two ways to do this: > 1 - The code that I have running in projects/pnfs-planb-server for the pNFS > server > side does a kproc_create() to create a kernel process that does the RPC > and > then krpc_exit()s. > - This was easy to code and works. However, I am concerned that there is > going to be excessive overheads from doing all the kproc_create()s and > kproc_exit()s? > Anyone know if these calls will result in large overheads? > 2 - I haven't coded this, but the other way I can think of to do this is to > create a pool of threads (kthread_create() is sufficient in this case, I > think?) and then hand each RPC to an available thread so it can do the > RPC. > - Other than a little more complex coding, the main issue I see with > this one > is "How many threads and when to create more/less of them.". > > Anyhow, any comments w.r.t. the merits of either of the above approaches > (or a suggestion of other ways to do this) would be appreciated, rick taskqueue(9) is an existing mechanism to enqueue functions to execute asynch using a pool of threads, but it doesn't answer the scalability questions. In fact it may make them harder, inasmuch as I don't think there's a mechanism to dynamically adjust the number of threads after first calling taskqueue_start_threads(). -- Ian ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
RFC how to use kernel procs/threads efficiently
Hi, I have now dropped the client side of Flexible File Layout for pNFS into head and I believe it is basically working. Currently when talking to mirrored DS servers, it does the Write and Commit RPCs to the mirrors serially. This works, but is inefficient w.r.t. elapsed to to completion. To do them concurrently, I need separate kernel processes/threads to do them. I can think of two ways to do this: 1 - The code that I have running in projects/pnfs-planb-server for the pNFS server side does a kproc_create() to create a kernel process that does the RPC and then krpc_exit()s. - This was easy to code and works. However, I am concerned that there is going to be excessive overheads from doing all the kproc_create()s and kproc_exit()s? Anyone know if these calls will result in large overheads? 2 - I haven't coded this, but the other way I can think of to do this is to create a pool of threads (kthread_create() is sufficient in this case, I think?) and then hand each RPC to an available thread so it can do the RPC. - Other than a little more complex coding, the main issue I see with this one is "How many threads and when to create more/less of them.". Anyhow, any comments w.r.t. the merits of either of the above approaches (or a suggestion of other ways to do this) would be appreciated, rick ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
