Re: Moving Lua source codes
Am 21.10.13 21:22, schrieb Alexander Nasonov: Marc Balmer wrote: Yes, this is an issue. Dunno if we need a 'kluac' or so, at the moment I'd say loading code from source form is ok. Supporting binary chunks is more challenging because binary format can change completely in a new Lua version. Source code is more stable, there are often small changes in the language too (like new keywords) but I'd say they are manageable. And then the language only changes very slowly, the developers are very careful not to change the language in drastic ways (and after 20 years I hope one can assume some stability in the language, anyways).
Re: Moving Lua source codes
Am 21.10.13 07:57, schrieb Artem Falcon: [...] Doing w/o a diff, i don't have a sane mail environment now to send it inline. There's few more things. First: you forgot to add MODULE_CLASS_LUA_BINDING to a sys/sys/module.h. FWIW, I removed the MODULE_CLASS_LUA_BINDING and use the existing MODULE_CLASS_MISC for Lua bindings. require 'foo' will look for a kernel module with the name luafoo and load it, if existing. It is the up to luafoo to register whatever it has to offer. In the unlike case that luafoo was something else than a Lua binding, it would be loaded, but provide no functionality. That is the only side-effect I can think of right now. So I think require can not be abused to load arbitrary modules, since they must be prefixed by lua.
Re: Moving Lua source codes
2013/10/21 Marc Balmer m...@msys.ch: Am 20.10.13 22:02, schrieb Artem Falcon: [...] Ok, here i have a clean diff. Sorry for the attach instead of inline. By the way, do you want to move from malloc(9) to kmem(9) as recommended? I may provide a draft version, if you're interested. Thank you for your diff, I will look at it in a few days (I am not at my office the next few days, being abroad for @work stuff). As for malloc(9) switching to kmem(9), I don't want to do that because of the way how Lua allocates memory. An allocator function is called with some number of bytes to be allocated. We have no control over how many bytes it wants us to allocate. If using kmem(9), we would have to keep track of this allocations, since kmem_free() takes as a second argument the number of bytes to free. kmem(9) is not very usable here imo, since we have to pass the number of bytes to free. Now i see. Thanks for clearing it out. btw, you can also CC tech-kern@, no need to keep this discussion private... Doing w/o a diff, i don't have a sane mail environment now to send it inline. There's few more things. First: you forgot to add MODULE_CLASS_LUA_BINDING to a sys/sys/module.h. Second: kernel side Lua will reject to load bytecode produced by luac cause the first built FPless. A simple solution is to build luac and liblua feeded with FPless luaconf.h and put them under different names. Maybe a more elegant solution is possible?
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Date: Sat, 19 Oct 2013 09:47:14 +0200 From: Marc Balmer m...@msys.ch And now to give you a practical example what I personally do with lua(4) right now: In the past I wrote several tty line disciplines to decode various serial formats. Now I have a need for that again. Doing this in C is of course possible, but I want something more dynamic. So I wrote a tty line discipline that uses Lua to do all the decoding. That works like a charm: Load the script, test, change the script and reload. Really practical. I will release this code once I sorted out a few remaining details. And in the course of this work, I also found deficencies in slattach(8). Please show us the code. It doesn't have to be perfect, but it has to be there in order to make a case.
Re: Moving Lua source codes
Am 21.10.13 07:57, schrieb Artem Falcon: [...] There's few more things. First: you forgot to add MODULE_CLASS_LUA_BINDING to a sys/sys/module.h. I merely forgot to commit it, thats fixed now. Second: kernel side Lua will reject to load bytecode produced by luac cause the first built FPless. A simple solution is to build luac and liblua feeded with FPless luaconf.h and put them under different names. Maybe a more elegant solution is possible? Yes, this is an issue. Dunno if we need a 'kluac' or so, at the moment I'd say loading code from source form is ok.
Re: Moving Lua source codes
Marc Balmer wrote: Yes, this is an issue. Dunno if we need a 'kluac' or so, at the moment I'd say loading code from source form is ok. Supporting binary chunks is more challenging because binary format can change completely in a new Lua version. Source code is more stable, there are often small changes in the language too (like new keywords) but I'd say they are manageable. Alex
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
On Sat, 19 Oct 2013, Marc Balmer wrote: The inclusion and use of Lua in base, for use in userland and the kernel, [...] has, last but not least, core's blessing. Would you please either present some evidence for that claim, or stop making the claim. To the best of my knowledge, userland Lua was approved by core in 2010, but kernel Lua has never been approved by core. Can we now please stop this useless discussion? People will continue to ask questions until they receive some satisfactory answers. --apb (Alan Barrett)
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
On Fri, 18 Oct 2013, Lourival Vieira Neto wrote: I have to point out that interesting work is commonly used as a sort of euphemism to refer to highly experimental work with unclear future. Yes. But I'm talking about interesting *user* work. I'm not claiming that they should be in the kernel. I'm just saying that, IMHO, we should incorporate a small device driver that facilitates this kind of development (outside the tree). You seem to want the lua device driver to be inside the tree, to facilitate experimental work outside the tree. Other people have asked why the lua(4) device driver itself can't be developed outside the tree (with a view to importing it later, if it ever proves to be more than an experiment), and I have seen no good answer to that. --apb (Alan Barrett)
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Am 19.10.13 09:03, schrieb Alan Barrett: On Sat, 19 Oct 2013, Marc Balmer wrote: The inclusion and use of Lua in base, for use in userland and the kernel, [...] has, last but not least, core's blessing. Would you please either present some evidence for that claim, or stop making the claim. I am not making a claim. And what is this, a trial, that you ask me to present evidence? You were not a core team member at the time, so I really can't blame you that you don't remember it. But I blame you for making this up as if it was sweeping kernel change or so. It's a tiny device driver that uses source code that is already in the tree since about three years. I will eventually dig out the email exchange, but that will have to wait, I am at a trade show right now. To the best of my knowledge, userland Lua was approved by core in 2010, but kernel Lua has never been approved by core. Can we now please stop this useless discussion? People will continue to ask questions until they receive some satisfactory answers. Me and others are giving answers. It just seem so that it is not the answers some people want to hear. What Lua is good for as been discussed many times, there are presentation slides around etc.
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Am 19.10.13 09:12, schrieb Alan Barrett: On Fri, 18 Oct 2013, Lourival Vieira Neto wrote: I have to point out that interesting work is commonly used as a sort of euphemism to refer to highly experimental work with unclear future. Yes. But I'm talking about interesting *user* work. I'm not claiming that they should be in the kernel. I'm just saying that, IMHO, we should incorporate a small device driver that facilitates this kind of development (outside the tree). You seem to want the lua device driver to be inside the tree, to facilitate experimental work outside the tree. Other people have asked why the lua(4) device driver itself can't be developed outside the tree (with a view to importing it later, if it ever proves to be more than an experiment), and I have seen no good answer to that. I can answer that. Lua, the library in userland, the bindings to libararies like sqlite or gpio, and lua(4) plus luactl(8) serve a specific purpose, namely to allow NetBSD _users_ to work with the system in an easy to learn and more dynamic approach. They are tools for users. This has been the goal that I stated in almost any talk about the subject matter. It's like gpio: That is also something for users. Hasn't Unix alwas provided the tools, but it was up to the user to do with them what he wanted? Lua is a scripting language in the original sense of the word: It allows a user to script or change the behaviour of software. It is not even seen as a freestanding language by it's inventors, but rather a hosted language, a small library to be incorporated in other software to make it extensible and more dynamic. Yes, there is a lua command, but that is merely an example use of the Lua library (as stated by the Lua developers). More and more people that were aware of the benefits of Lua, both in userspace, but also in the form of a driver, have pushed me to finally commit this. Also from a marketing perspective it's a good thing (and I speaking as a member of marketing@ now): It makes NetBSD more attractive for newcomers, for hardware tinkerers and such. The can now use their Soekris or Alix board and use GPIO lines to do play with hardware. I showed NetBSD with Lua at a local ham (radio amateur) meeting and they were thrilled. A sofar Linux only community all of a sudden created new NetBSD users. And that is good. There will always be folks who don't like Lua for various reasons. Unfortunately they are very vocal and sometimes I do think that they not always understand what Lua is and what it's general goals are. They should just not use it, then. And now to give you a practical example what I personally do with lua(4) right now: In the past I wrote several tty line disciplines to decode various serial formats. Now I have a need for that again. Doing this in C is of course possible, but I want something more dynamic. So I wrote a tty line discipline that uses Lua to do all the decoding. That works like a charm: Load the script, test, change the script and reload. Really practical. I will release this code once I sorted out a few remaining details. And in the course of this work, I also found deficencies in slattach(8). In previous work I used Lua to create a software gpio device, a modified version of gpiosim(4) that uses a Lua script to mimick a real device. Also handy. In one email thread I once specifically said that Lua should not necesseraly be used to write system software, e.g. startup scripts or so. Even when you see no use for Lua for your personal work, I hope that you understand that there are users that have a valid and sound use for Lua. Or do you really thing we do a GSoC project, go to conferences, talk about, bring new developers on board just to annoy anyone? No, it's because we (the Lua users) want it, not for the sake of it, but to work with it. - Marc
Re: Why do we need lua in-tree again? Yet another call for actualevidence, please. (was Re: Moving Lua source codes)
marc@ wrote: Me and others are giving answers. It just seem so that it is not the answers some people want to hear. Probably many people want to see an actual sample implementation, like a dumb device driver that blinks LEDs via GPIO etc. using integrated Lunatik APIs, like computer textbooks. What Lua is good for as been discussed many times, there are presentation slides around etc. People who don't use lua (including me) won't see what Lua is good for without studying lua... --- Izumi Tsutsui
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
On Oct 19, 2013, at 12:26 AM, Marc Balmer m...@msys.ch wrote: Am 19.10.13 09:03, schrieb Alan Barrett: On Sat, 19 Oct 2013, Marc Balmer wrote: The inclusion and use of Lua in base, for use in userland and the kernel, [...] has, last but not least, core's blessing. Would you please either present some evidence for that claim, or stop making the claim. I am not making a claim. And what is this, a trial, that you ask me to present evidence? You were not a core team member at the time, so I really can't blame you that you don't remember it. But I blame you for making this up as if it was sweeping kernel change or so. It's a tiny device driver that uses source code that is already in the tree since about three years. I will eventually dig out the email exchange, but that will have to wait, I am at a trade show right now. Well, I've been on core a lot longer (over a decade now) and I don't remember approving in-kernel Lua either. I checked my mail archives. The relevant mail is from around October 24th, 2010. The only kernel references are for things like exec_script support and to make sure userland Lua does not conflict with the kernel Lua from his [Lourival Neto] GSoC project. That strongly implies that you were only asking for userland lua support and that's what core granted permission for. Looking through past mail, it saddens me to note that the bozohttpd changes took nearly 4 years to get into the tree.
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
On Sat, 19 Oct 2013, Marc Balmer wrote: And now to give you a practical example what I personally do with lua(4) right now: In the past I wrote several tty line disciplines to decode various serial formats. Now I have a need for that again. Doing this in C is of course possible, but I want something more dynamic. So I wrote a tty line discipline that uses Lua to do all the decoding. That works like a charm: Load the script, test, change the script and reload. Really practical. I will release this code once I sorted out a few remaining details. And in the course of this work, I also found deficencies in slattach(8). In previous work I used Lua to create a software gpio device, a modified version of gpiosim(4) that uses a Lua script to mimick a real device. Also handy. Thank you. Those seem like useful example. --apb (Alan Barrett)
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Le 19/10/13 00:47, Marc Balmer a écrit : Am 19.10.13 00:14, schrieb Aleksej Saushev: [...] I'm of opinion that this device driver can and should stay outside the tree until its utility can be demonstrated without this much strain. At last this is one of the reasons why we support kernel modules. The inclusion and use of Lua in base, for use in userland and the kernel, has been the subject to public discussion, it has been the topic of a GSoC project, it has been presented at many conferences, it is well received by the community at large, it has users, it has attracted new users to NetBSD, I would like to underline the attracted new users to NetBSD with an e-mail thread I've just read on DragonFly BSD kernel mailing list: http://lists.dragonflybsd.org/pipermail/kernel/2013-October/063013.htm It seems true that Lua support is attractive to users and developers :) Best regards, -- Yann Sionneau
Re: Moving Lua source codes
Mouse mouse at Rodents-Montreal.ORG writes: ... I think the biggest reasons I haven't tried to push any of those into NetBSD are (1) a perception that NetBSD doesn't want my changes, combined with (2) a lack of motivation to, a sort of NetBSD no longer cares about me; why should I care about it? feeling. (Note that I'm not saying anything about how justifiable - or correct! - these feelings are.) That said I'd certainly be happy to offer any-to-all of my changes to anyone who wants to bring them into NetBSD. ... If anyone's interested, of course. My expectation has been that nobody is, but this email makes me think that could well be wrong. I can easily pull a full list of changes I've made and mail it wherever. Mouse, I'm not an EdgeBSD advocacy, but when i've heard of it and briefly looked at i got a feeling that they're kinda open to bringing of various experimental things, even kernel-side ones. I don't know their relations with NetBSD project, but there was an info that they will try to push well-tried bits back to the NetBSD. If all of this is true, maybe it's a more suitable place for your stuff now?
Re: Moving Lua source codes
Date: Mon, 14 Oct 2013 22:46:53 +0200 From: Marc Balmer m...@msys.ch It is entirely plausible to me that we could benefit from using Lua in base, or sysinst, or maybe even in the kernel. But that argument must be made by showing evidence of real, working code that has compelling benefits, together with confidence in its robustness -- not by saying that if we let users do it then it will happen. There is real word, real working code. In userland and in kernel space. There are developers waiting for the kernel parts to be committet, so they can continue their work as well. Where is the real, working application code?
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Date: Thu, 17 Oct 2013 19:16:16 -0300 From: Lourival Vieira Neto lourival.n...@gmail.com Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. This is not a chicken-and-egg problem. You can make an experimental kernel with Lua support and make an experimental application in Lua, all before anything has to be committed to HEAD[*]. Then you can show that the application serves a useful function, has compelling benefits over writing it in C, and can offer confidence in robustness. [*] You could do this in a branch, you could do this in a private Git repository, or you could even just do this in a local CVS checkout (since kernel Lua requires no invasive changes, right?). That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). Prototyping and experimentation is great! Show examples! What hurts is getting bitrotten code that nobody actually maintains or uses (when was the last Lua update in src?) and provides a new Turing machine with device access in the kernel for attack vectors. [1] https://github.com/dergraf/PacketScript [2] http://www.pdsw.org/pdsw12/papers/grawinkle-pdsw12.pdf In the two links you gave, I found precisely five lines of Lua code, buried in the paper, and those five lines seemed to exist only for the purpose of measuring how much overhead Lua adds to the existing pNFS code or something.
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Hi, The linked research was performed on Linux, which has NFsv4.1 and pNFS client implementations. Evidently, you can do this kind of thing with an out-of-tree Lua kernel extension. Matt - Taylor R Campbell riastr...@netbsd.org wrote: [1] https://github.com/dergraf/PacketScript [2] http://www.pdsw.org/pdsw12/papers/grawinkle-pdsw12.pdf In the two links you gave, I found precisely five lines of Lua code, buried in the paper, and those five lines seemed to exist only for the purpose of measuring how much overhead Lua adds to the existing pNFS code or something. -- Matt Benjamin The Linux Box 206 South Fifth Ave. Suite 150 Ann Arbor, MI 48104 http://linuxbox.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309
Re: Moving Lua source codes
[...] If all of this is true, maybe [EdgeBSD i]s a more suitable place for your stuff now? Possibly. They're welcome to my changes if they want them too. I'll send a ping thataway. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Hello, Lourival Vieira Neto lourival.n...@gmail.com writes: On Thu, Oct 17, 2013 at 1:26 PM, Jeff Rizzo r...@tastylime.net wrote: On 10/14/13 1:46 PM, Marc Balmer wrote: There is real word, real working code. In userland and in kernel space. There are developers waiting for the kernel parts to be committet, so they can continue their work as well. *Where* is this code? The pattern I see happening over and over again is: NetBSD Community: Please show us the real working code that needs this mbalmer: the code is there! (pointer to actual code not in evidence) I do not doubt that something exists, but the onus is on the person proposing the import to convince the skeptics, or at least to make an actual effort. I see lots of handwaving, and little actual code. YEARS after the import of lua into the main tree, I see very little in-tree evidence of its use. In fact, what I see is limited to : 1) evidence of lua bindings for netpgp. 2) evidence of some tests in external/bsd/lutok 3) the actual lua arc in external/mit/lua 4) gpio and sqlite stuff in liblua 5) some lua bindings in libexec/httpd (bozohttpd) 6) two example files in share/examples/lua 7) the luactl/lua module/lua(4) stuff you imported yesterday ...and counting. There is also ongoing working happening =). As Jeff points what is counting is support code. Am I missing something major here? The only actual usage I see is netpgp and httpd; the rest is all in support of lua itself. I do not see evidence that anyone is actually using lua in such a way that requires it in-tree. When you originally proposed importing lua back in 2010, you talked a lot about how uses would materialize. It's now been 3 years, and I just don't see them. If I am wrong about this, I would love some solid pointers to evidence of my wrongness. Now you're using very similar arguments for bringing lua into the kernel; I would very much like to see some real, practical, *useful* code demonstrating just why this is a good thing. Beyond the 'gee, whiz' factor, I just don't see it. Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. The problem with your approach is that such chicken-and-egg problems are to be solved _at_once_ rather than laying eggs everywhere around and have everyone else wait till at least one chicken appears. Sure, we do not *need* a script language interpreter embedded in the kernel, as we do not need a specific file system. But I do not get why we should not. There is current development of applications being done right now. Also, there is a few interesting works that used Lunatik in Linux [1, 2] that could be done more easily now in NetBSD just because we have the right environment for that. That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). I think that is why we *should* (not need) have this on the tree. IMHO. I have to point out that interesting work is commonly used as a sort of euphemism to refer to highly experimental work with unclear future. You tell that there's interesting work using Lua in Linux. Was it accepted in any experimental Linux distribution like Fedora? What was the outcome of discussion among linux kernel developers? Currently there's no indication that it was accepted anywhere. I doubt very much that we want such unreliable development practices like agile ones in the kernel, and experimentation work can be done easier and better in a branch or a personal repository. And last. The appeal to why not is defective. NetBSD is not your personal playground, there exist other people who have to deal with the inadvertent mess you can leave after you. That's why you ought to present solid arguments that justify why other people should tolerate your experimentations. -- BCE HA MOPE!
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
On Fri, 18 Oct 2013 20:31:05 +0400 Aleksej Saushev a...@inbox.ru wrote: I doubt very much that we want such unreliable development practices like agile ones in the kernel, and experimentation work can be done easier and better in a branch or a personal repository. I think I agree with your sentiment but it seems like you are misusing the word agile here. The term refers to a very specific programming methodology and would be very beneficial if applied to kernel programming. Throwing random code into a system does not constitute agile development. http://en.wikipedia.org/wiki/Agile_software_development -- D'Arcy J.M. Cain da...@netbsd.org http://www.NetBSD.org/ IM:da...@vex.net
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Hi, On Fri, Oct 18, 2013 at 11:09 AM, Matt W. Benjamin m...@linuxbox.com wrote: Hi, The linked research was performed on Linux, which has NFsv4.1 and pNFS client implementations. Evidently, you can do this kind of thing with an out-of-tree Lua kernel extension. Matt Evidently. I'm not arguing that we need that. I'm just arguing that I see benefits and none harm. Regards, -- Lourival Vieira Neto
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
On Fri, Oct 18, 2013 at 9:08 AM, Taylor R Campbell riastr...@netbsd.org wrote: Date: Thu, 17 Oct 2013 19:16:16 -0300 From: Lourival Vieira Neto lourival.n...@gmail.com Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. This is not a chicken-and-egg problem. You can make an experimental kernel with Lua support and make an experimental application in Lua, all before anything has to be committed to HEAD[*]. Then you can show that the application serves a useful function, has compelling benefits over writing it in C, and can offer confidence in robustness. [*] You could do this in a branch, you could do this in a private Git repository, or you could even just do this in a local CVS checkout (since kernel Lua requires no invasive changes, right?). Yes, but how do we do device driver development? We are branching the tree for each non-intrusive and disabled-by-default device driver? If we have developed a device driver for an uncommon device, we have to put it in a branch? (Please, note I'm friendly asking that). That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). Prototyping and experimentation is great! Show examples! What hurts is getting bitrotten code that nobody actually maintains or uses (when was the last Lua update in src?) and provides a new Turing machine with device access in the kernel for attack vectors. I don't see how an optional module could be used for attacks. If users enable that, they should know what they are doing (such as loading a kernel module). [1] https://github.com/dergraf/PacketScript [2] http://www.pdsw.org/pdsw12/papers/grawinkle-pdsw12.pdf In the two links you gave, I found precisely five lines of Lua code, buried in the paper, and those five lines seemed to exist only for the purpose of measuring how much overhead Lua adds to the existing pNFS code or something. I'm just showing examples of how it could be useful for user applications. I understand that you do not agree with that. But I'm not arguing that we have to add these applications into the tree. I'm arguing that we could benefit users with such a tool. Regards, -- Lourival Vieira Neto
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Hi, On Fri, Oct 18, 2013 at 1:31 PM, Aleksej Saushev a...@inbox.ru wrote: (...) Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. The problem with your approach is that such chicken-and-egg problems are to be solved _at_once_ rather than laying eggs everywhere around and have everyone else wait till at least one chicken appears. No. I'm talking about put just one egg, just a device driver. Sure, we do not *need* a script language interpreter embedded in the kernel, as we do not need a specific file system. But I do not get why we should not. There is current development of applications being done right now. Also, there is a few interesting works that used Lunatik in Linux [1, 2] that could be done more easily now in NetBSD just because we have the right environment for that. That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). I think that is why we *should* (not need) have this on the tree. IMHO. I have to point out that interesting work is commonly used as a sort of euphemism to refer to highly experimental work with unclear future. Yes. But I'm talking about interesting *user* work. I'm not claiming that they should be in the kernel. I'm just saying that, IMHO, we should incorporate a small device driver that facilitates this kind of development (outside the tree). You tell that there's interesting work using Lua in Linux. Was it accepted in any experimental Linux distribution like Fedora? What was the outcome of discussion among linux kernel developers? Currently there's no indication that it was accepted anywhere. Really don't know. I'm not a member of these communities neither I'm claiming to incorporate such works here. However, I think that there was a discussion about PacketScript on OpenWRT, but I don't know how it evolved. I doubt very much that we want such unreliable development practices like agile ones in the kernel, and experimentation work can be done easier and better in a branch or a personal repository. I agree with you in this point: experimental work should be done aside from the tree. And last. The appeal to why not is defective. NetBSD is not your personal playground, there exist other people who have to deal with the inadvertent mess you can leave after you. That's why you ought to present solid arguments that justify why other people should tolerate your experimentations. I guess you misunderstood that. I'm not arguing that we should do it just because there is no contrary argument. I sincerely asked 'why not?' trying to understand the contrary argumentation. Also, I'm not saying that you should tolerate my experimentation. Far away from that. I haven't committed anything nor tried to impose nothing. I'm just trying to make a point of view and understand yours. When I talked about experimentation, I was trying to say that providing support for that kind of experimentation for users sounds a good idea for me and I don't see how it is prejudicial. Which doesn't mean that I'm proposing that my personal experimentation should be in tree. Regards, -- Lourival Vieira Neto
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Hello, Lourival Vieira Neto lourival.n...@gmail.com writes: On Fri, Oct 18, 2013 at 1:31 PM, Aleksej Saushev a...@inbox.ru wrote: (...) Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. The problem with your approach is that such chicken-and-egg problems are to be solved _at_once_ rather than laying eggs everywhere around and have everyone else wait till at least one chicken appears. No. I'm talking about put just one egg, just a device driver. Sorry, but this is not just one egg. And counting was your reaction to complaints that almost all the code related to Lua is the code to support Lua itself rather than anything else. Sure, we do not *need* a script language interpreter embedded in the kernel, as we do not need a specific file system. But I do not get why we should not. There is current development of applications being done right now. Also, there is a few interesting works that used Lunatik in Linux [1, 2] that could be done more easily now in NetBSD just because we have the right environment for that. That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). I think that is why we *should* (not need) have this on the tree. IMHO. I have to point out that interesting work is commonly used as a sort of euphemism to refer to highly experimental work with unclear future. Yes. But I'm talking about interesting *user* work. I'm not claiming that they should be in the kernel. I'm just saying that, IMHO, we should incorporate a small device driver that facilitates this kind of development (outside the tree). I'm of opinion that this device driver can and should stay outside the tree until its utility can be demonstrated without this much strain. At last this is one of the reasons why we support kernel modules. You tell that there's interesting work using Lua in Linux. Was it accepted in any experimental Linux distribution like Fedora? What was the outcome of discussion among linux kernel developers? Currently there's no indication that it was accepted anywhere. Really don't know. I'm not a member of these communities neither I'm claiming to incorporate such works here. However, I think that there was a discussion about PacketScript on OpenWRT, but I don't know how it evolved. This demonstrates that Lua isn't actually useful in the kernel. I doubt very much that we want such unreliable development practices like agile ones in the kernel, and experimentation work can be done easier and better in a branch or a personal repository. I agree with you in this point: experimental work should be done aside from the tree. And last. The appeal to why not is defective. NetBSD is not your personal playground, there exist other people who have to deal with the inadvertent mess you can leave after you. That's why you ought to present solid arguments that justify why other people should tolerate your experimentations. I guess you misunderstood that. I'm not arguing that we should do it just because there is no contrary argument. I sincerely asked 'why not?' trying to understand the contrary argumentation. Also, I'm not saying that you should tolerate my experimentation. Far away from that. I haven't committed anything nor tried to impose nothing. On my side it sounded like that, sorry, if I'm wrong. I'm just trying to make a point of view and understand yours. When I talked about experimentation, I was trying to say that providing support for that kind of experimentation for users sounds a good idea for me and I don't see how it is prejudicial. Which doesn't mean that I'm proposing that my personal experimentation should be in tree. The problem as I see it is that we have one developer (two at most) pushing hard for Lua in base and in kernel and providing no satisfactory arguments why this is to be done at all. Lack of any real code for years reinforces such doubts. Why not sounds as an argument for highly experimental work in this context. And I wouldn't have anything against this why not if all the work were dressed accordingly. For now I'd say that Lua support hasn't demonstrated any benefit. I'd say that it should be removed and the work continued in a branch until benefits become more clear. -- BCE HA MOPE!
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Hello, Lourival Vieira Neto lourival.n...@gmail.com writes: On Fri, Oct 18, 2013 at 9:08 AM, Taylor R Campbell riastr...@netbsd.org wrote: Date: Thu, 17 Oct 2013 19:16:16 -0300 From: Lourival Vieira Neto lourival.n...@gmail.com Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. This is not a chicken-and-egg problem. You can make an experimental kernel with Lua support and make an experimental application in Lua, all before anything has to be committed to HEAD[*]. Then you can show that the application serves a useful function, has compelling benefits over writing it in C, and can offer confidence in robustness. [*] You could do this in a branch, you could do this in a private Git repository, or you could even just do this in a local CVS checkout (since kernel Lua requires no invasive changes, right?). Yes, but how do we do device driver development? We are branching the tree for each non-intrusive and disabled-by-default device driver? If we have developed a device driver for an uncommon device, we have to put it in a branch? (Please, note I'm friendly asking that). We didn't import yet another programming language interpreter for driver development previously. Besides, what are drivers developed in Lua so far? If I understand it correctly, the only driver is the Lua interpreter itself. That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). Prototyping and experimentation is great! Show examples! What hurts is getting bitrotten code that nobody actually maintains or uses (when was the last Lua update in src?) and provides a new Turing machine with device access in the kernel for attack vectors. I don't see how an optional module could be used for attacks. If users enable that, they should know what they are doing (such as loading a kernel module). Was anything done to warn users? [1] https://github.com/dergraf/PacketScript [2] http://www.pdsw.org/pdsw12/papers/grawinkle-pdsw12.pdf In the two links you gave, I found precisely five lines of Lua code, buried in the paper, and those five lines seemed to exist only for the purpose of measuring how much overhead Lua adds to the existing pNFS code or something. I'm just showing examples of how it could be useful for user applications. I understand that you do not agree with that. But I'm not arguing that we have to add these applications into the tree. I'm arguing that we could benefit users with such a tool. The problem is that the number of such users is negligible and all of them are developers that are able to build their kernel module outside the tree. -- BCE HA MOPE!
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Am 19.10.13 00:14, schrieb Aleksej Saushev: [...] I'm of opinion that this device driver can and should stay outside the tree until its utility can be demonstrated without this much strain. At last this is one of the reasons why we support kernel modules. The inclusion and use of Lua in base, for use in userland and the kernel, has been the subject to public discussion, it has been the topic of a GSoC project, it has been presented at many conferences, it is well received by the community at large, it has users, it has attracted new users to NetBSD, it has attracted users that are now developers, and it has, last but not least, core's blessing. Can we now please stop this useless discussion? Lua is part of NetBSD. [...]
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. The problem with your approach is that such chicken-and-egg problems are to be solved _at_once_ rather than laying eggs everywhere around and have everyone else wait till at least one chicken appears. No. I'm talking about put just one egg, just a device driver. Sorry, but this is not just one egg. And counting was your reaction to complaints that almost all the code related to Lua is the code to support Lua itself rather than anything else. And counting == there is ongoing work happening outside the tree. Sure, we do not *need* a script language interpreter embedded in the kernel, as we do not need a specific file system. But I do not get why we should not. There is current development of applications being done right now. Also, there is a few interesting works that used Lunatik in Linux [1, 2] that could be done more easily now in NetBSD just because we have the right environment for that. That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). I think that is why we *should* (not need) have this on the tree. IMHO. I have to point out that interesting work is commonly used as a sort of euphemism to refer to highly experimental work with unclear future. Yes. But I'm talking about interesting *user* work. I'm not claiming that they should be in the kernel. I'm just saying that, IMHO, we should incorporate a small device driver that facilitates this kind of development (outside the tree). I'm of opinion that this device driver can and should stay outside the tree until its utility can be demonstrated without this much strain. At last this is one of the reasons why we support kernel modules. Understand. You tell that there's interesting work using Lua in Linux. Was it accepted in any experimental Linux distribution like Fedora? What was the outcome of discussion among linux kernel developers? Currently there's no indication that it was accepted anywhere. Really don't know. I'm not a member of these communities neither I'm claiming to incorporate such works here. However, I think that there was a discussion about PacketScript on OpenWRT, but I don't know how it evolved. This demonstrates that Lua isn't actually useful in the kernel. I don't think so. It could even evince that, but not demonstrate. And last. The appeal to why not is defective. NetBSD is not your personal playground, there exist other people who have to deal with the inadvertent mess you can leave after you. That's why you ought to present solid arguments that justify why other people should tolerate your experimentations. I guess you misunderstood that. I'm not arguing that we should do it just because there is no contrary argument. I sincerely asked 'why not?' trying to understand the contrary argumentation. Also, I'm not saying that you should tolerate my experimentation. Far away from that. I haven't committed anything nor tried to impose nothing. On my side it sounded like that, sorry, if I'm wrong. It could sound as you want, but it wasn't what I meant. I'm just trying to make a point of view and understand yours. When I talked about experimentation, I was trying to say that providing support for that kind of experimentation for users sounds a good idea for me and I don't see how it is prejudicial. Which doesn't mean that I'm proposing that my personal experimentation should be in tree. The problem as I see it is that we have one developer (two at most) pushing hard for Lua in base and in kernel and providing no satisfactory arguments why this is to be done at all. Lack of any real code for years reinforces such doubts. Why not sounds as an argument for highly experimental work in this context. And I wouldn't have anything against this why not if all the work were dressed accordingly. For now I'd say that Lua support hasn't demonstrated any benefit. I'd say that it should be removed and the work continued in a branch until benefits become more clear. Understand. Regards, -- Lourival Vieira Neto
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. This is not a chicken-and-egg problem. You can make an experimental kernel with Lua support and make an experimental application in Lua, all before anything has to be committed to HEAD[*]. Then you can show that the application serves a useful function, has compelling benefits over writing it in C, and can offer confidence in robustness. [*] You could do this in a branch, you could do this in a private Git repository, or you could even just do this in a local CVS checkout (since kernel Lua requires no invasive changes, right?). Yes, but how do we do device driver development? We are branching the tree for each non-intrusive and disabled-by-default device driver? If we have developed a device driver for an uncommon device, we have to put it in a branch? (Please, note I'm friendly asking that). We didn't import yet another programming language interpreter for driver development previously. Besides, what are drivers developed in Lua so far? If I understand it correctly, the only driver is the Lua interpreter itself. I meant traditional device driver, but never mind. That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). Prototyping and experimentation is great! Show examples! What hurts is getting bitrotten code that nobody actually maintains or uses (when was the last Lua update in src?) and provides a new Turing machine with device access in the kernel for attack vectors. I don't see how an optional module could be used for attacks. If users enable that, they should know what they are doing (such as loading a kernel module). Was anything done to warn users? The code is not linked yet. Regards, -- Lourival Vieira Neto
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Date: Fri, 18 Oct 2013 18:12:29 -0300 From: Lourival Vieira Neto lourival.n...@gmail.com On Fri, Oct 18, 2013 at 9:08 AM, Taylor R Campbell riastr...@netbsd.org wrote: [*] You could do this in a branch, you could do this in a private Git repository, or you could even just do this in a local CVS checkout (since kernel Lua requires no invasive changes, right?). Yes, but how do we do device driver development? We are branching the tree for each non-intrusive and disabled-by-default device driver? If we have developed a device driver for an uncommon device, we have to put it in a branch? (Please, note I'm friendly asking that). Device drivers usually have trivially demonstrable useful functions related to physical devices that one encounters on the market. Example: I wrote uatp(4) because the trackpad in my MacBook didn't work very well. I also developed uatp(4) in a local Git branch because at first it was an experiment which I expected to throw away. In the two links you gave, I found precisely five lines of Lua code, buried in the paper, and those five lines seemed to exist only for the purpose of measuring how much overhead Lua adds to the existing pNFS code or something. I'm just showing examples of how it could be useful for user applications. I understand that you do not agree with that. But I'm not arguing that we have to add these applications into the tree. I'm arguing that we could benefit users with such a tool. I don't disagree that Lua could be useful for user applications, and I'm not asking you to propose applications to add to the tree. All I'm asking for is examples of applications at all, which I couldn't find in either of the links you gave. Where is the Lua code?
Re: Moving Lua source codes
On Tue, 15 Oct 2013, Marc Balmer wrote: Well, you are in contradiction to our guide, which under http://www.netbsd.org/releases/release-map.html#current states that NetBSD-current is the main development branch. NetBSD-current the main development branch for things that we know we want, and that we are prepared to support for a long time, and that mostly work. If any of those tests fail, then I'd say that the code should not be in -current, but could be in a branch or in pkgsrc or in some third party tree. In the case of kernel Lua, some people are not convinced that we want it, and some people are not convinced that the API is stable enough that we should commit to long-term compatibility for it. Although I think that developing in the main -current tree is acceptable (especially if users are told not to expect as much future compatibility as for most other parts of NetBSD), I would have preferred to see development in a branch. It's certainly not the clear-cut no need for a branch situation that you seem to think. --apb (Alan Barrett)
Re: Moving Lua source codes
Well, you are in contradiction to our guide, which under http://www.netbsd.org/releases/release-map.html#current states that NetBSD-current is the main development branch. So, you're saying I should have done my AF_TIMER work in -current? Labeled control structure? diskwatch? pfw? WSEMUL_MTERM? tun(4)-for-IPv6? v6-capable SLIP? netstat -T? touch -d and -i? In short... NetBSD-current the main development branch for things that we know we want, and that we are prepared to support for a long time, and that mostly work. ...that. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: Moving Lua source codes
On Thu, Oct 17, 2013 at 02:44:03AM -0400, Mouse wrote: Well, you are in contradiction to our guide, which under http://www.netbsd.org/releases/release-map.html#current states that NetBSD-current is the main development branch. So, you're saying I should have done my AF_TIMER work in -current? Labeled control structure? diskwatch? pfw? WSEMUL_MTERM? tun(4)-for-IPv6? v6-capable SLIP? netstat -T? touch -d and -i? To me, some of those seem like they'd be fine to have done in NetBSD-current. Why didn't you do things like WSEMUL_MTERM there? Is it mostly because you track an older version of NetBSD? In short... NetBSD-current the main development branch for things that we know we want, and that we are prepared to support for a long time, and that mostly work. Which isn't always something that can be determined ahead of time, and especially the support for a long time phase I'm not entirely in agreement with. IMO, a little more appetite for removing things that are barely used would not be a bad thing, especially if it's something that's only been present in -current and not in a release. Eric
Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
On 10/14/13 1:46 PM, Marc Balmer wrote: It is entirely plausible to me that we could benefit from using Lua in base, or sysinst, or maybe even in the kernel. But that argument must be made by showing evidence of real, working code that has compelling benefits, together with confidence in its robustness -- not by saying that if we let users do it then it will happen. There is real word, real working code. In userland and in kernel space. There are developers waiting for the kernel parts to be committet, so they can continue their work as well. *Where* is this code? The pattern I see happening over and over again is: NetBSD Community: Please show us the real working code that needs this mbalmer: the code is there! (pointer to actual code not in evidence) I do not doubt that something exists, but the onus is on the person proposing the import to convince the skeptics, or at least to make an actual effort. I see lots of handwaving, and little actual code. YEARS after the import of lua into the main tree, I see very little in-tree evidence of its use. In fact, what I see is limited to : 1) evidence of lua bindings for netpgp. 2) evidence of some tests in external/bsd/lutok 3) the actual lua arc in external/mit/lua 4) gpio and sqlite stuff in liblua 5) some lua bindings in libexec/httpd (bozohttpd) 6) two example files in share/examples/lua 7) the luactl/lua module/lua(4) stuff you imported yesterday Am I missing something major here? The only actual usage I see is netpgp and httpd; the rest is all in support of lua itself. I do not see evidence that anyone is actually using lua in such a way that requires it in-tree. When you originally proposed importing lua back in 2010, you talked a lot about how uses would materialize. It's now been 3 years, and I just don't see them. If I am wrong about this, I would love some solid pointers to evidence of my wrongness. Now you're using very similar arguments for bringing lua into the kernel; I would very much like to see some real, practical, *useful* code demonstrating just why this is a good thing. Beyond the 'gee, whiz' factor, I just don't see it. +j
Re: Moving Lua source codes
So, you're saying I should have done my AF_TIMER work in -current? Labeled control structure? diskwatch? pfw? WSEMUL_MTERM? tun(4)-for-IPv6? v6-capable SLIP? netstat -T? touch -d and -i? To me, some of those seem like they'd be fine to have done in NetBSD-current. Why didn't you do things like WSEMUL_MTERM there? Is it mostly because you track an older version of NetBSD? At this point, yes. I don't recall the reasons behind them for most of those - for example, I don't recall whether I was still supping -current when I did WSEMUL_MTERM, though I feel reasonably sure I was no longer routinely building and using it. There are also some (which I didn't mention) which I don't think NetBSD _should_ adopt, such as changing the default /etc/localtime link target to .../Canada/Eastern. I think the biggest reasons I haven't tried to push any of those into NetBSD are (1) a perception that NetBSD doesn't want my changes, combined with (2) a lack of motivation to, a sort of NetBSD no longer cares about me; why should I care about it? feeling. (Note that I'm not saying anything about how justifiable - or correct! - these feelings are.) That said I'd certainly be happy to offer any-to-all of my changes to anyone who wants to bring them into NetBSD. Except for v6-capable SLIP and the touch(1) options, all the things I listed are things I've been using in personal production use for long enough I'm confident they work fairly well, though I have a suspicion there's a kernel memory leak lurking in the AF_TIMER code[%]. Those exceptions I haven't used much, but I think they may well be integration-ready anyway - that would be something to discuss. [%] There's certainly a kernel memory leak _somewhere_ in one of my versions' kernel, and what vague data I have on it seems to correlate its striking with AF_TIMER socket use. If anyone's interested, of course. My expectation has been that nobody is, but this email makes me think that could well be wrong. I can easily pull a full list of changes I've made and mail it wherever. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Hi Jeff, On Thu, Oct 17, 2013 at 1:26 PM, Jeff Rizzo r...@tastylime.net wrote: On 10/14/13 1:46 PM, Marc Balmer wrote: It is entirely plausible to me that we could benefit from using Lua in base, or sysinst, or maybe even in the kernel. But that argument must be made by showing evidence of real, working code that has compelling benefits, together with confidence in its robustness -- not by saying that if we let users do it then it will happen. There is real word, real working code. In userland and in kernel space. There are developers waiting for the kernel parts to be committet, so they can continue their work as well. *Where* is this code? The pattern I see happening over and over again is: NetBSD Community: Please show us the real working code that needs this mbalmer: the code is there! (pointer to actual code not in evidence) I do not doubt that something exists, but the onus is on the person proposing the import to convince the skeptics, or at least to make an actual effort. I see lots of handwaving, and little actual code. YEARS after the import of lua into the main tree, I see very little in-tree evidence of its use. In fact, what I see is limited to : 1) evidence of lua bindings for netpgp. 2) evidence of some tests in external/bsd/lutok 3) the actual lua arc in external/mit/lua 4) gpio and sqlite stuff in liblua 5) some lua bindings in libexec/httpd (bozohttpd) 6) two example files in share/examples/lua 7) the luactl/lua module/lua(4) stuff you imported yesterday ...and counting. There is also ongoing working happening =). Am I missing something major here? The only actual usage I see is netpgp and httpd; the rest is all in support of lua itself. I do not see evidence that anyone is actually using lua in such a way that requires it in-tree. When you originally proposed importing lua back in 2010, you talked a lot about how uses would materialize. It's now been 3 years, and I just don't see them. If I am wrong about this, I would love some solid pointers to evidence of my wrongness. Now you're using very similar arguments for bringing lua into the kernel; I would very much like to see some real, practical, *useful* code demonstrating just why this is a good thing. Beyond the 'gee, whiz' factor, I just don't see it. Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. Sure, we do not *need* a script language interpreter embedded in the kernel, as we do not need a specific file system. But I do not get why we should not. There is current development of applications being done right now. Also, there is a few interesting works that used Lunatik in Linux [1, 2] that could be done more easily now in NetBSD just because we have the right environment for that. That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). I think that is why we *should* (not need) have this on the tree. IMHO. [1] https://github.com/dergraf/PacketScript [2] http://www.pdsw.org/pdsw12/papers/grawinkle-pdsw12.pdf +j Regards, -- Lourival Vieira Neto
Re: Moving Lua source codes
Marc Balmer marc at msys.ch writes: Hi All I am in the progress of ... - Commiting the kernel parts from our GSoC project, Lua in the NetBSD Kernel ... - mbalmer Hi, Marc Had you managed to fix broken things i mentioned here? [1] Thank you! [1] http://marc.info/?l=netbsd-tech-kernm=137111030727302w=2
Re: Moving Lua source codes
On Mon, Oct 14, 2013 at 03:26:19PM -0700, Paul Goyette wrote: -current is where development should take place. I disagree. That way - doing development in the master tree - lies the madness that has given Linux some of its worst problems. Development should take place on branches, or, preferably, outside the master tree entirely. When something is developed and is past initial testing, _then_ is the time to bring it into the tree. For example: AF_TIMER sockets. I developed them on my systems, never going near NetBSD's main tree with them. I'd've suggested bringing them in long since, except I think they're not ready. Not, as always, that anyone has to agree with me, or care what I think I agree with Mouse. It seems to me that this is why branches exist. aolme too/aol (am I dating myself with that expression?) -- David A. Holland dholl...@netbsd.org
Re: Moving Lua source codes
Am 15.10.13 00:21, schrieb Mouse: -current is where development should take place. I disagree. That way - doing development in the master tree - lies the madness that has given Linux some of its worst problems. Development should take place on branches, or, preferably, outside the master tree entirely. When something is developed and is past initial testing, _then_ is the time to bring it into the tree. For example: AF_TIMER sockets. I developed them on my systems, never going near NetBSD's main tree with them. I'd've suggested bringing them in long since, except I think they're not ready. Not, as always, that anyone has to agree with me, or care what I think Well, you are in contradiction to our guide, which under http://www.netbsd.org/releases/release-map.html#current states that NetBSD-current is the main development branch. Branches may be ok when a development takes place that touches large parts of the system, a compiler change e.g. (but we don't even use a branch for that) or when something is done that takes longer time to implement and touches many places. Merging branches using cvs is a different story, though. As Lua triggered this dicussion I should probably explain why a branch is not needed for it: It is a device driver and a small userspace command only: simple, small, safe, and isolated. It's even a module. Don't want it? Don't laod it. It does not - as some wrongly assumed - allow a script to access arbitrary memory in kernel space or call arbitarty funcions there. It is a highly (user) controlled environment.
Re: Moving Lua source codes
Taylor, I found out that several programs already have Lua bindings, but for unknown reasons that code is not enabled by default, which is kind of weird given that the Lua library is in base... So we have Lua bindings in our code, whe have Lua as well, but yet we don't enable it and continue to whine about nothing using Lua, that's weirder than weird ;) Bindings do not an application make. For years, many people have been asking you for real, working Lua code that (a) serves some nontrivial purpose beyond mapping between the C world and the Lua world, and (b) has a compelling reason to rely on Lua in base rather than Lua in pkgsrc. But so far, all I've seen is a few bindings to C libraries. Lua is a scripting language in the original sense of the word, a language to be embedded in a hosting program, to make this program extensible. As such, Lua bindings in hosting programs are key. But not only there, it also important to provide access to key functionality in the form of Lua bindings. That is why Lua modules like sqlite, pgsql, json, gpio, or whatever exist. Lua provides an easy to use, easy to learn, yet powerful interface to the system. It is a tool not only for developers, but also for users of NetBSD. It's like a shell: We provide it to our users so that they can do magic with it. It's a tool. httpd(8) uses Lua by default. You can use Lua to create dynamic content, like you do with a CGI script, but the startup time is faster. You can access SQLite3 database files from it, creating e.g. a fast blog or whatever web application. Lua is a tool, it's your imagination that sets the limits. It is entirely plausible to me that we could benefit from using Lua in base, or sysinst, or maybe even in the kernel. But that argument must be made by showing evidence of real, working code that has compelling benefits, together with confidence in its robustness -- not by saying that if we let users do it then it will happen. There is real word, real working code. In userland and in kernel space. There are developers waiting for the kernel parts to be committet, so they can continue their work as well. I myself use Lua since years in the kernel, sysinst, and now also in httpd(8). I did not commit all uses of Lua, because they are very specific to _our_ (i.e. my company) uses. And that is what Lua is about: I can use it to taylor (no pun intended) a certain piece of software to my needs. I agree that the sysinst Lua integration should be commited. We use Lua in sysinst since ever to install pkgsrc packages during install time, btw. I use Lua in the kernel to simulate certain types of devices. Why don't you experiment with some more radical changes for Lua in a branch, mbalmer-lua? You needn't worry about anyone else's approval, you can break anything you want, and you can let others review and play with your work. There is nothing radical about Lua. In userland we already have it, no need for a branch. Use it, if you like. Adding Lua to your software is easy and straightforward and will not bloat it. In the kernel space, all of Lua we did so far are loadable modules, no existing kernel subsystem needed to be changed source code wise. So it's perfectly self contained and does not interfere with the existing kernel code. Again no branch needed. Kernel mode Lua is a few kernel modules plus a utility luactl(8) to control matters from userland. A new tty line discipline attachment program will probably make use of kernel mode Lua. I see no need for a branch. A branch would probably only mean that no one will ever look at this code anyways. -current is where development should take place. And I am careful enough that this development does not interfere with the work of others. - Marc
Re: Moving Lua source codes
Am 11.10.13 09:19, schrieb David Holland: lua(4) comes with examples. Such as? - Several examples om how to use kernel mode Lua, like how to provide and use modules, call module functions etc. - tty(4) line disciplines written in Lua, decoding serial data - Obscure GPIO stuff that I did not plan to commit, because it only helps me and my oscilloscope... - How to write extension modules (C bindings) and making them available to kernel mode Lua
Re: Moving Lua source codes
-current is where development should take place. I disagree. That way - doing development in the master tree - lies the madness that has given Linux some of its worst problems. Development should take place on branches, or, preferably, outside the master tree entirely. When something is developed and is past initial testing, _then_ is the time to bring it into the tree. For example: AF_TIMER sockets. I developed them on my systems, never going near NetBSD's main tree with them. I'd've suggested bringing them in long since, except I think they're not ready. Not, as always, that anyone has to agree with me, or care what I think /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: Moving Lua source codes
On Mon, 14 Oct 2013, Mouse wrote: -current is where development should take place. I disagree. That way - doing development in the master tree - lies the madness that has given Linux some of its worst problems. Development should take place on branches, or, preferably, outside the master tree entirely. When something is developed and is past initial testing, _then_ is the time to bring it into the tree. For example: AF_TIMER sockets. I developed them on my systems, never going near NetBSD's main tree with them. I'd've suggested bringing them in long since, except I think they're not ready. Not, as always, that anyone has to agree with me, or care what I think I agree with Mouse. It seems to me that this is why branches exist. - | Paul Goyette | PGP Key fingerprint: | E-mail addresses: | | Customer Service | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com| | Network Engineer | 0786 F758 55DE 53BA 7731 | pgoyette at juniper.net | | Kernel Developer | | pgoyette at netbsd.org | -
Re: Moving Lua source codes
Date: Wed, 09 Oct 2013 11:08:51 +0200 From: Marc Balmer m...@msys.ch I found out that several programs already have Lua bindings, but for unknown reasons that code is not enabled by default, which is kind of weird given that the Lua library is in base... So we have Lua bindings in our code, whe have Lua as well, but yet we don't enable it and continue to whine about nothing using Lua, that's weirder than weird ;) Bindings do not an application make. For years, many people have been asking you for real, working Lua code that (a) serves some nontrivial purpose beyond mapping between the C world and the Lua world, and (b) has a compelling reason to rely on Lua in base rather than Lua in pkgsrc. But so far, all I've seen is a few bindings to C libraries. It is entirely plausible to me that we could benefit from using Lua in base, or sysinst, or maybe even in the kernel. But that argument must be made by showing evidence of real, working code that has compelling benefits, together with confidence in its robustness -- not by saying that if we let users do it then it will happen. Why don't you experiment with some more radical changes for Lua in a branch, mbalmer-lua? You needn't worry about anyone else's approval, you can break anything you want, and you can let others review and play with your work.
Re: Moving Lua source codes
On Wed, Oct 09, 2013 at 11:00:53AM +0200, Marc Balmer wrote: My question was about source code location, the other issues raised have been discussed already in the past, but for the record: Where and when? Several people have gone looking and not found this discussion in the archives. lua(4) comes with examples. Such as? -- David A. Holland dholl...@netbsd.org
Re: Moving Lua source codes
Am 06.10.13 21:59, schrieb Marc Balmer: I am in the progress of - Updating the Lua code in base from version 5.1 to version 5.2 - Commiting the kernel parts from our GSoC project, Lua in the NetBSD Kernel mrg@ suggestest that I move the Lua source code from src/external/ to src/sys/external, which imo makes sense. Comments requested. The plan is to import Lua 5.2 sources in src/sys/external, then adjust userland and kernel parts to uses sources from there, and once all works, remove the Lua 5.1 sources from src/external. The transition from 5.1 to 5.2 is more complicated that one would think, especially from a C module programmers perspective (the C API changed in a way that makes it necessary to adjust most modules). So if no one really objects the plan is as follows: - Import Lua 5.2 to src/sys/external/ - Remove Lua 5.1 from src/external/ Before this hit's the repository I will check everything locally and bake a diff. - mb
Re: Adding Lua to the kernel and moving Lua source codes
On Mon, Oct 07, 2013 at 12:05:26PM +0200, Alan Barrett wrote: If the same source code will be compiled into both the kernel and userland, then I think that src/common/external/licence/name would be the place for that. Let's please not import new sources from 3rd parties into src/common. src/sys/external is really good enough. Joerg
Re: Moving Lua source codes
On Wed, Oct 9, 2013 at 9:26 AM, Thomas Klausner w...@netbsd.org wrote: On Wed, Oct 09, 2013 at 08:37:23AM +0200, Marc Balmer wrote: So if no one really objects the plan is as follows: - Import Lua 5.2 to src/sys/external/ - Remove Lua 5.1 from src/external/ apb suggested using src/common/external/licence/name and lots of Trying to describe a license in a filename is an exercise in futility; I can see the case for gpl as a generic license class maybe, but in general you need to read the licenses for all the code, there are lots of them. people asked for working examples first. What's your reply to that? Various people have posted things they are working on. While there is no proposal to remove it, upgrading makes sense; certainly the stuff I am working on targets 5.2 as a preference. Also the 5.1 series is no longer getting bug fixes, and there is a hostile code fix in 5.2 that Wikipedia found in their code audit when they added user facing Lua to Wikipedia. Justin
Re: Moving Lua source codes
Am 09.10.13 10:26, schrieb Thomas Klausner: On Wed, Oct 09, 2013 at 08:37:23AM +0200, Marc Balmer wrote: So if no one really objects the plan is as follows: - Import Lua 5.2 to src/sys/external/ - Remove Lua 5.1 from src/external/ apb suggested using src/common/external/licence/name and lots of people asked for working examples first. What's your reply to that? phone said that src/common/external was a mistake and should not be prolifered at all. Joerg also said so. I think that settles it. My question was about source code location, the other issues raised have been discussed already in the past, but for the record: lua(4) comes with examples. and just mentioning it: many more people asked me to go on with this because they like the concept and want finally be able to use it than people asked for more examples. Having Lua in the kernel allows for experimentation that is otherwise not possible. So it's all about what _you_ do with it. But kernel Lua is not really the issue here, the real issue is the update to 5.2, that one is important. And moving the code at the same time is sensible (and does not break anything).
Re: Moving Lua source codes
Am 09.10.13 10:53, schrieb Justin Cormack: On Wed, Oct 9, 2013 at 9:26 AM, Thomas Klausner w...@netbsd.org wrote: On Wed, Oct 09, 2013 at 08:37:23AM +0200, Marc Balmer wrote: So if no one really objects the plan is as follows: - Import Lua 5.2 to src/sys/external/ - Remove Lua 5.1 from src/external/ apb suggested using src/common/external/licence/name and lots of Trying to describe a license in a filename is an exercise in futility; I can see the case for gpl as a generic license class maybe, but in general you need to read the licenses for all the code, there are lots of them. people asked for working examples first. What's your reply to that? Various people have posted things they are working on. While there is no proposal to remove it, upgrading makes sense; certainly the stuff I am working on targets 5.2 as a preference. Also the 5.1 series is no longer getting bug fixes, and there is a hostile code fix in 5.2 that Wikipedia found in their code audit when they added user facing Lua to Wikipedia. I updated all my code to 5.2 by now (I have lots of modules written in C, so the API changes were a real issue for me). I am currently updating Lua in my dev machine to 5.2 to see whether everything still works. I found out that several programs already have Lua bindings, but for unknown reasons that code is not enabled by default, which is kind of weird given that the Lua library is in base... So we have Lua bindings in our code, whe have Lua as well, but yet we don't enable it and continue to whine about nothing using Lua, that's weirder than weird ;) fwiw, personally I use Lua in sysinst and to use gpio and sqlite (which is the reason why there are gpio and sqlite modules...)
Re: Moving Lua source codes
Justin Cormack jus...@specialbusservice.com wrote: On Wed, Oct 9, 2013 at 9:26 AM, Thomas Klausner w...@netbsd.org wrote: On Wed, Oct 09, 2013 at 08:37:23AM +0200, Marc Balmer wrote: So if no one really objects the plan is as follows: - Import Lua 5.2 to src/sys/external/ - Remove Lua 5.1 from src/external/ apb suggested using src/common/external/licence/name and lots of Trying to describe a license in a filename is an exercise in futility; We already use such structure without trouble. See README: http://nxr.netbsd.org/xref/src/external/ I can see the case for gpl as a generic license class maybe, but in general you need to read the licenses for all the code, there are lots of them. You need to read the licenses anyway, unless you do not care about the legal implications. We want to know what are we shipping. We want to conveniently manage this knowledge and provide it to our users. -- Mindaugas
Re: Adding Lua to the kernel and moving Lua source codes
Jean-Yves Migeon wrote: Le 07/10/2013 12:05, Alan Barrett a écrit : I still haven't seen a use case for in-kernel Lua. I mean, an example (preferably a working example) of something useful that could not easily be done without in-kernel Lua. I'd prefer not to see it added to the base system without a use case. I second the use case. Not something as polished or finished as possible, but at least shows that it is useful (I am well aware of the cause/effect circle ie. you cannot prove it without having Lua first available, but breaking that vicious circle with a few examples can help). In the early days of bpfjit when I didn't yet know of sljit, I was considering ripping off lua code for generating machine instructions from LuaJIT2 code. I still believe that rewriting bpfjit in Lua would improve readability. I even started a rewrite mostly as a good use-case for Lua bindings for sljit but it's low priority project for me. Alex
Moving Lua source codes
Hi All I am in the progress of - Updating the Lua code in base from version 5.1 to version 5.2 - Commiting the kernel parts from our GSoC project, Lua in the NetBSD Kernel mrg@ suggestest that I move the Lua source code from src/external/ to src/sys/external, which imo makes sense. Comments requested. The plan is to import Lua 5.2 sources in src/sys/external, then adjust userland and kernel parts to uses sources from there, and once all works, remove the Lua 5.1 sources from src/external. The transition from 5.1 to 5.2 is more complicated that one would think, especially from a C module programmers perspective (the C API changed in a way that makes it necessary to adjust most modules). - mbalmer
