Dear List, I forget to CC the mailing list while replying to kre. Sorry for the confusion! Also I am using the @icloud address hereafter as per kre’s preference.
It just came to me that we can do in-kernel machine learning by making the kernel talk to a userland process capable of doing the heavy lifting with BLAS/LAPACK or even mlpack. I don’t know much about OS, but my understanding is that kernel/userland context switches are expansive, and that the whole point of having Lua in kernel (as opposed to implementing the heuristics with a userland Lua interpreter that calls into the kernel) is to eliminate that overhead, so I would suggest leaving machine learning for now and focusing on implementing a ring-buffer-analyzing toy example first. Bests, Qingyao > On Mar 1, 2023, at 10:27, Qingyao Sun <sunqingyao19970...@gmail.com> wrote: > > Dear kre, > > Thanks for the reply! See inline below: > > > On Wed, Mar 1, 2023 at 7:23 AM Robert Elz <k...@netbsd.org> wrote: > > > > I like this idea a lot more than the inetd/rc.d idea. I am not sure that > "Improvements on Lua Support" is a good title, unless you're actually planning > on working on the kernel Lua implementation, and it doesn't sound like that. > > I also find this idea more interesting. People seem to have different > preferences/opinions on the init system, so I guess it would be hard for me > to propose something everybody is happy with. You are right that I don't plan > to work on the kernel Lua implementation itself, but I would need to expose > some kernel functionalities to Lua for the kernel heuristics to use, or > perhaps even modify it to add floating-point support (see below). > > > > I'd suggest something more like "Using Lua scripts to improve kernel > > heuristics" > or something like that. That is, if that is what you are really proposing, > using your machine learning knowledge in areas where it might benefit the > kernel. > > The title you are suggesting is definitely more suitable. I'm not sure > whether we should use machine learning though. AFAIK the in-kernel Lua > interpreter does not support floating-point arithmetic because context > switches of the floating-point unit are overly expensive, without which we > cannot even do linear regression. Moreover, ML algorithms tend to be > computationally extensive, but of course, some benchmarking is required. I > was thinking something like collecting some historical data into a ring > buffer and checking it periodically to see if we are too optimistic or > pessimistic. Anyway, I would say the core idea behind this proposal is to > make the kernel heuristics more sophisticated with the expressiveness of Lua. > > > > I cannot help with Lua, I know it exists, but that's the extent of my > > knowledge > on that front. I know something about filesystems, but not the details of our > current implementation. I would however like to see this idea applied to CPU > scheduelling, particularly when there are different classes of CPUs available > (ie: they're not all identical to each other - while any can do any work, they > don't all do it equally well). I think we're currently lacking in that > area. > > That sounds interesting as well. Many ARM chips seem to have that big.LITTLE > technology, including Apple Silicon. They can definitely benefit from this > improvement. > > > > For filesystems, my gut feeling is that a better task to take on might be > working out which cached data ought be ejected, and which retained, rather > than (or perhaps just before) working on prefetching algorithms (particularly > as "drives" get much faster, and delays to fetch blocks are reducing > considerably). > > That makes sense. Just for the record, we can improve kernel heuristics in > other areas like I/O latency prediction and page warmth classification (see > the LAKE paper I linked in OP). I don't have a strong opinion on which > specific subsystem to work on. > > > > So, I'm prepared to help with this, with two caveats. First, you need to > > also > find someone knowledgable (enough) with the kernel Lua implementation and use > to assist with that (and Lua scripting) who is willing, and available, to > assist > with this. Some occasional filesystem/uvm and schedueller assistance might be > useful as well. And second, you need to receive e-mail somewhere other than > at > gmail - gmail refuses mail from me (mail I normally send, rather than mail > this > way, from a NetBSD host, which is not at all convenient to send), and even > forwarding > mail from me to them from some other mailbox is likely to fail. I have zero > interest > in bowing to gmail's stupid requirements to avoid that problem, so if I am to > assist > you need to be able to receive mail somewhere else. Probably at UT I'd > guess. > > Thanks for your help! Let's wait for some Lua wizard to jump in then. > Regarding email, I will probably receive a @utexas.edu email address in a few > months (I am enrolling in 2023 fall), and I would be happy to use that. > Meanwhile, we can communicate through sunqingyao19970...@icloud.com if gmail > does not work for you. > > > > kre
