On Mon, Mar 28, 2011 at 11:04 PM, Dmitry Matveev <[email protected]> wrote: > Hello everybody!
Hello Dmitry, Thanks for writing up this proposal. It seems pretty complete and work on this project would deliver a much more usable desktop (and server!) NetBSD system. > I am a 5th year student of Nizhny Novgorod State Technical University, > Russia. > > Please excuse me for CCing tech-kern@ list in this message, I have > asked some questions regarding the project in tech-pkg@ but there was > no answer. Sorry for the delay; I really meant to ask your original email but I have been away from the computer until now :-) As a general comment, cross-posting to different NetBSD mailing lists is not generally well seen. It is better to repost your message several times, once per list, to prevent followups from spreading to multiple lists. > Here is my proposal for "Add kqueue support to GIO" GSoC project. Any > comments are welcome. Comments inline. > What is the goal of the project? > -------------------------------- > > The goal of the project is to add kqueue support to GIO. GIO, the Glib > IO layer, currently uses inotify interface to monitor changes in the > filesystem, so it does work on Linux and does not work on NetBSD. The > importance of the file monitoring grows with the evolution of modern > desktop environments. Semantic desktops, local file indexers and even > usual file managers, such as Nautilus, use it to provide an updated > information to user. GIO is used in many applications, and providing > a full-featured file monitoring to it will significantly increase > the end-user convenience as well as application compatibility. As mentioned above, this does not only help the desktop; it helps servers as well. You may want to do a bit more research in this area and explain your findings in your proposal, because it will affect your design decisions. As an example, several server packages have file-notification functionality. One that I'm using is Mediatomb: the lack of proper file notifications makes it pretty much unusable because polling is a very slow alternative when the trees to monitor are huge. I'm sure I have also seen imap servers offering file notification functionality, but I can't remember which these are off the top of my head. (For these packages, an inotify-emulation layer may be more suitable if they do not use glib, instead of patching their code to add direct kqueue support.) > If the time will permit (I assume it will), it would be also possible > to implement an userspace library that will emulate the inotify > interface on the top of kqueue. It will help applications that do not > use GIO but use inotify directly. As you mention, time should permit to implement both approaches; it'd be nice to plan both of them in your proposal. If things turn out to be more complex than they seem now during the development of the project, that's fine, we can readjust the goals to focus on one implementation only. But it's better to plan for a longer term ;-) > What will be the deliverables of the project? > --------------------------------------------- > > In the form in which I see the project, the final deliverables are > expected to be: > 1. A set of patches for GIO; Try to elaborate a bit on how you plan to integrate these upstream. > 2. [probably] Userspace inotify-over-kqueue library sources; Elaborate on how you think this library should be distributed. I.e. as a separate software package, or bundled right into NetBSD. Each approach has its advantages/disadvantages, and you should explain what these are in the proposal. > 3. Documentation files describing general design decisions; > 4. A set of sample programs to ensure that is the introduced > functionality does work (tests). Cool; tests are a very nice part of this. For GIO, the tests should be implemented using whatever framework the glib guys are using in their codebase. Such tests need to be sent upstream so that the developers can periodically ensure that your new code works, and the tests need to be readily runnable from NetBSD (make test in pkgsrc). As for the compatibility library: take a look at ATF (http://www.netbsd.org/~jmmv/atf/). It's what we are using in NetBSD for our tests and this library should integrate seamlessly. > Milestones and schedule > ----------------------- > > 1. Community bonding period > * Getting familiar with GIO file monitoring implementation > * Researching the way to isolate it from inotify > * Probably studying kqueue/inotify implementations on the kernel > side > * Discussing the ways of implementation with mentor. > > 2. May 23 - June 23: the core work > * Isolating GIO from inotify with a compile-time abstration level > * Implementing the kqueue backend > > 3. June 24 - July 10: wrap up for this milestone > * Reviews > * Bug fixes > * Tests Don't postpone tests until a late stage. Develop tests as you go. When you are writing code, you can see more easily what the tricky code paths are. If you postpone the development of tests until a very later stage, you will still be able to write simple tests, but you may not remember what the tricky cases are. Also, having tests along the way ensure that your code is easily testable and that you don't introduce regressions early on. > 4. July 11 - August 1: the rest of the work > * Implementing the user-space inotify interface over kqueue > > 5. August 2 - August 7: offline > * Probably I will miss this days > > 6. August 8 - August 22: final stage > * Reviews > * Bug fixes > * Tests > * Delivery Same thing re. tests here. Also, in general, don't postpone reviews until late. It is nice if you keep your mentor involved while you develop your code. Reviewing a ton of code on the deadline is painful; reviewing the code incrementally is much easier, and you may get some interesting comments along the way! > Some implementation notes > ------------------------- > > Obviously my project will use kqueue(). The abstraction layer for GIO > will be implemented as a compile-time Strategy pattern. > > There is a tricky moment with inotify-over-kqueue emulation. The > problem is that inotify provides some types of notifications that > kqueue does not - i.e. IN_MOVED_FROM/IN_MOVED_TO with the additional > information and so on. For it I am going to use the standard file > system routines such as readdir(), stat() and so on. Would be nice to provide a detailed description of what kqueue lacks during the early stages of the project. It might be the case that it's impossible to extend kqueue to implement some important functionality... in which case we could be looking at a different project altogether (e.g. implement inotify in the kernel)! > However the frequent usage of these system calls will result in a > negative impact on the performance, so I consider the possibility to > extend kqueue to support additional events. This moment (as well as > others :)) is open for discussion. Extending kqueue would be nice/preferred, if it is possible. Thanks for your proposal and keep up the good work! -- Julio Merino / @jmmv
