Hello Willy, Thank you for your feedbacks.
On Sat, May 9, 2020 at 6:39 AM Willy Tarreau <[email protected]> wrote: > Your work could be extended to other files, like errorfiles, maps, ACLs, > etc. However with the config parser being very linear, it's extremely > unlikely that we could actually benefit from this in places other than > certs. What makes certs specific is that a single config line can induce > hundreds of thousands of file accesses and that these ones do benefit in > being performed in parallel. Yes, I take your comment as also something to improve in my commit message explanation. > Despite this I'm wondering if we should think about a generic approach > like an async file open with a callback on what to call once the file > is opened, or optionally an automatic async read and a callback set up > for received data. This would bring the benefit of allowing to perform > runtime FS accesses without destroying performance: right now, if you > try to access the file system from Lua or maybe from within one of the > device detection libs, or if you'd like to reload some files from the > CLI, the whole thread totally freezes until the I/O is completed, which > is why such accesses are absolutely forbidden. But done asynchronously > it could be different. We could imagine an async read API to be used > from Lua. We could even imagine being able to trigger reloading certain > files (ACL patterns or error files maybe) from the CLI, or even to serve > static files. This will of course not solve the security issues that > come with doing this, but at least it could solve the performance issue. wow I did not thought that far, but we could make things more generic indeed. I will try to keep that in mind and improve things along the way. Regarding the CLI, I'm however a bit worried about the `chroot` support though, which is important from my point of view. > Probably that the work should be split into a functional and an > implementation layer. The functional layer is the ability to perform > async file operations. The implementation is done using uring. But for > other OSes we could imagine using an emulation layer based on AIO or > even a bunch of threads, which could still possibly provide some > benefits during startup when loading tons of certificates, especially > when that's from spinning disks (not sure if those are still in use > though, last time I saw one was when I tried to repair my old Ultra5 > a few months ago). ok will try to keep that in mind. not sure I will be able to come up with a 100% nice interface but I will at least try to have something we can iterate on. > From day one I've been totally convinced it's possible to create a new > poller entirely based on io-uring. We could avoid a number of the ugly > tricks we use in epoll to avoid useless epoll_ctl() calls. What's nice > about our pollers is that they're entirely modular so it is perfectly > possible to create your own and experiment with it. yes totally. For now I've duplicated to a new poller, but depending on the result and diff, I might simply propose a similar #ifdef IO_URING in ev_poll.c to activate it or not. I will see the final result. Regarding ev_epoll, there is also an hybrid implementation to use `epoll_wait` but uring for all `epoll_ctl` calls. Thanks a lot for your inputs, -- William
