Re: any help for your BSD project
On Tue, May 27, 2014 at 01:56:45AM -0400, Yizheng Jiao wrote: do you still need any help for you BSD projects? I am a phd students. I want to some programming project to enhance my understanding of operating system. A number of projects are posted here: http://wiki.netbsd.org/projects/ -- David A. Holland dholl...@netbsd.org
Re: API/ABI rank of headers in /usr/include/isofs/cd9660
On Wed, May 28, 2014 at 06:22:16PM +1000, matthew green wrote: On Wed, May 14, 2014 at 08:00:56AM +1000, matthew green wrote: thanks for looking at this. we've gotten better since early days with these sorts of issues, but there are still a lot of headers that are installed for no good reason but no one has had the chance/motivation to clean them up yet. It's not that so much as cleaning it all up properly requires cvs rename, which unfortunately fails to exist. Otherwise I'd have done it years ago. i don't see why that matters in the vast majority of cases that are just like this one -- headers that were never useful for userland were installed because of history (the original scripts were find ... -name \*.h, with some filters, IIRC), not because we need to shuffle their contents around to make it so. all we have to do is, like this case, simply not install them. Well... at least in the case of ufs (dunno about cd9660) the definitions are all mixed together. And, worse, ffs doesn't even distinguish the in-memory and on-disk superblock structures. If there are some that can just be punted from userland, then we should do that, but between kmem grovelers and other mess I'm not sure there are really that many. For fixing things like ufs the best way to do it is to pick up the minimum set of decls that actually public and move them to a new file; for best long-term results that file should be in a reserved place for such files (so among other things it doesn't end up part of the supported API) ...and setting up such a place and populating it really wants rename. That doesn't mean we couldn't start with things like the ufs headers and do more incrementally or even leave things halfway until we solve the SCM problem. But I haven't been entirely inclined to push ahead, especially since creating src/sys/include isn't universally popular. :-) Since maybe there are passersby who haven't seen the plan before: userland-only includes in src/include, src/include/sys; kernel-only includes in src/sys/include, src/sys/include/sys; shared includes in src/sys/include/kern; src/sys/include/kern/*.h gets installed in /usr/include/kern; no filename in src/sys/include/kern is part of any supported API (all code just includes sys/foo.h or whatever, and that header includes kern/*.h headers as needed); shared MD header files go in src/sys/arch/foo/include/kern with the same rules; standard C headers that can be provided in the kernel environment can be placed in src/sys/include, e.g. stdarg.h. I think I get to claim this is proper after laying out a number of research kernels and getting it wrong the first two or three times, but everyone's of course welcome to disagree with me. The choice of kern for the kernel headers directory is necessitated by the fact that sys is already taken. (In fact, sys/ was originally just the place for for kernel headers; but years of abuse, bad design, standardization, and cruft accumulation has resulted in about 1/3 to 1/2 of sys/ belonging to userland; with the organization we currently have all this stuff sits in the kernel include directory and gets in the way.) Something other than kern could be used but kern seems like a reasonable choice. Note that while in this scheme some or all non-exported kernel things could be migrated to be top-level headers (e.g. sys/proc.h - proc.h) there's probably little benefit to this in the short to medium term. Anyhow, creating this setup requires wholesale moves. I don't think it's really a very good idea to create the file that should be e.g. kern/ufs.h as some other name with the expectation of changing it later. And like I was saying above, I think it isn't a great idea to create src/sys/include/kern and put only a handful of things in it. Partially completed migrations are horrible mudpits for large projects. -- David A. Holland dholl...@netbsd.org
Re: API/ABI rank of headers in /usr/include/isofs/cd9660
Hi, David Holland: at least in the case of ufs (dunno about cd9660) the definitions are all mixed together. In cd9660 it is the full spectrum: What i deem legit userland API cd9660_mount.h : struct iso_args and its flags for mount(2). Somewhat exotic userland API supplements iso.h : Constants and structs describing the byte layout of ISO 9660 storage entities. Educational and re-usable because supposed to stay absolutely stable. cd9660_rrip.h : About the same for ISO 9660 add-on SUSP/RRIP (Rock Ridge). API for binaries intimately related to the kernel cd9660_node.h : In the non _KERNEL part it exposes the filesystem specific struct iso_node which serves as vnode.v_data: - struct iso_node points to the ISO 9660 directory record on disk. I did not yet decide whether i will propose to break this API aspect. Currently i tend to not breaking it. - type ISO_RRIP_INODE representing RRIP POSIX attributes. - Macro to retrieve iso_node from vnode. - Macro to retrieve vnode from iso_node. - a riddling flag bit IN_ACCESS which gets set but not inquired in cd9660 - two riddling macros i_forw, i_back which are unused in cd9660. - To come (in order to keep that API fully usable after my changes to cd9660 and especially iso_node): - Macros to tell presence of multi-extent code in cd9660 and to tell its limitations. - A description of address and section count encoding in ino_t. - Macros to retrieve address and section count. - Macro which defines a C function which can tell the overall byte count of the data file represented by an iso_node. To be implanted and called in programs which cannot keep themselves from groping through VFS. cd9660_extern.h: In non _KERNEL area there are a few entrails of the filesystem driver, and the struct iso_mnt which serves underneath struct mount as .mnt_data. Programs which peek into iso_node might be interested in hopping further to iso_mnt. Internal headers iso_rrip.h : Internal definitions for the RRIP interpreter. No _KERNEL area. This interpreter is unreachable from userland, afaics. -- The cleanup effect of removing only iso_rrip.h would be tiny. A more radical approach would be to demand that the intimately kernel related binaries include headers from /usr/src/sys/fs/cd9660 rather than from /usr/include/isofs/cd9660. This would make clear that they are depending on inofficial kernel entrails. Have a nice day :) Thomas
Re: Lockless IP input queue, the pktqueue interface
On 29/05/2014 12:29 PM, Mindaugas Rasiukevicius wrote: Darren Reed darr...@netbsd.org wrote: On 29/05/2014 5:06 AM, Mindaugas Rasiukevicius wrote: Darren Reed darr...@netbsd.org wrote: No, there is no need to expose the structure. Even if there would be another internal component using the structure(s) one should consider accessors/mutators. Even if that component would have a good reason not to use accessors/mutators, the structure should be placed under #ifdef __SUBSYSTEM_PRIVATE (and certainly #ifdef _KERNEL). However, I strongly discourage to start from the last step without having a necessity first. One of the main reasons why we ended up with our network stack being such a mess is exactly this: the internal structures are exposed and accessed all over the place, there is a lack of strict interfacing, and the information hiding principle is not followed. So if someone were to write a program to grovel through a crash dump or /dev/mem and print out these structures, how would they get the definition of it? This is getting off-topic, but for the sake of wondering readers: Serialize and export the structure, or create a wrapper structure used for data transportation, or implement interface with accessors/mutators. If you think that you should be able to fiddle with any structure in the kernel (as it would be 1980s) then you are plain wrong (do I really need to explain this?). Your justification for including the structure in the .c file is that developers can't betrusted to not use header files that clearly aren't public interfaces. That's what code review is for and puts the focus of what you're arguing as being a human problem and not a technology problem. You can't solve human problems with technology. Putting the structure there won't stop determined people, it will just make it harder and they'll begrudge NetBSD for making their life harder. Yes, and that is absolutely valid justification. Code review ought to prevent from such problems, but that does not always happen (for a variety of reasons - from developers lacking time, or lack of developers who are familiar with that particular code base, to developers trying to reach a compromise; you know the realities). Technology is also a one of the means to address the human problem, at least partially. Do not get me wrong - I am not advocating black and white world of view. You cannot solve these problems with tough restrictions. Otherwise, one could advocate for adopting something like MISRA C standard; which I think is horrible - it takes away flexibility from an engineer, it forces you to produce ugly and tasteless code. However, it sort of does its job for the purpose it was created for, that is, to be idiot proof. I just think that the BSD community can do better than that. As an old open source we can produce code which is above the average quality. At least I hope so. Despite that, the point i.e. the justification still stands. We just try to apply some common sense when judging which restrictions are worth in our case and which are not. If code review would be so efficient solution, we would not have so much spaghetti code, as we do today. Right? :) Even if on average it is much better than a statistical commercial code. Putting structures inside a .c file represents a very short term view of how the implementation willevolve and be used. The method that I've seen used in Solaris (for example) is to use foo_impl.h to providethe details of data structure that are essentially private and those .h filesmay or may notbe shipped as part of the end user system.Using pktqueue_private.h might be suitable. ... It is basically the third phase I described in my original response (and I do use _impl.h headers), except if you read it carefully - you should better have a necessity and provide a good reason for it. Because if you are fiddling with an internal structure - it is a good indicator that you might potentially be doing something wrong; even more so if you fiddle with a lockless data structure as in pktqueue's case. It might be useful some day is a very poor and dangerous reasoning in this case. All of your arguments boil down to can't trust someone else. Why do you need to be so insulting of other developers in your arguments? Do you think you're the only person capable of making good design decisions? Sorry, but I won't be a party to that kind of attitude and want nothing more to do with this. Darren
Re: Lockless IP input queue, the pktqueue interface
On Fri, May 30, 2014 at 12:01:23AM +1000, Darren Reed wrote: On 29/05/2014 12:29 PM, Mindaugas Rasiukevicius wrote: Darren Reed darr...@netbsd.org wrote: On 29/05/2014 5:06 AM, Mindaugas Rasiukevicius wrote: Darren Reed darr...@netbsd.org wrote: No, there is no need to expose the structure. Even if there would be another internal component using the structure(s) one should consider accessors/mutators. Even if that component would have a good reason not to use accessors/mutators, the structure should be placed under #ifdef __SUBSYSTEM_PRIVATE (and certainly #ifdef _KERNEL). However, I strongly discourage to start from the last step without having a necessity first. One of the main reasons why we ended up with our network stack being such a mess is exactly this: the internal structures are exposed and accessed all over the place, there is a lack of strict interfacing, and the information hiding principle is not followed. So if someone were to write a program to grovel through a crash dump or /dev/mem and print out these structures, how would they get the definition of it? This is getting off-topic, but for the sake of wondering readers: Serialize and export the structure, or create a wrapper structure used for data transportation, or implement interface with accessors/mutators. If you think that you should be able to fiddle with any structure in the kernel (as it would be 1980s) then you are plain wrong (do I really need to explain this?). Your justification for including the structure in the .c file is that developers can't betrusted to not use header files that clearly aren't public interfaces. That's what code review is for and puts the focus of what you're arguing as being a human problem and not a technology problem. You can't solve human problems with technology. Putting the structure there won't stop determined people, it will just make it harder and they'll begrudge NetBSD for making their life harder. Yes, and that is absolutely valid justification. Code review ought to prevent from such problems, but that does not always happen (for a variety of reasons - from developers lacking time, or lack of developers who are familiar with that particular code base, to developers trying to reach a compromise; you know the realities). Technology is also a one of the means to address the human problem, at least partially. Do not get me wrong - I am not advocating black and white world of view. You cannot solve these problems with tough restrictions. Otherwise, one could advocate for adopting something like MISRA C standard; which I think is horrible - it takes away flexibility from an engineer, it forces you to produce ugly and tasteless code. However, it sort of does its job for the purpose it was created for, that is, to be idiot proof. I just think that the BSD community can do better than that. As an old open source we can produce code which is above the average quality. At least I hope so. Despite that, the point i.e. the justification still stands. We just try to apply some common sense when judging which restrictions are worth in our case and which are not. If code review would be so efficient solution, we would not have so much spaghetti code, as we do today. Right? :) Even if on average it is much better than a statistical commercial code. Putting structures inside a .c file represents a very short term view of how the implementation willevolve and be used. The method that I've seen used in Solaris (for example) is to use foo_impl.h to providethe details of data structure that are essentially private and those .h filesmay or may notbe shipped as part of the end user system.Using pktqueue_private.h might be suitable. ... It is basically the third phase I described in my original response (and I do use _impl.h headers), except if you read it carefully - you should better have a necessity and provide a good reason for it. Because if you are fiddling with an internal structure - it is a good indicator that you might potentially be doing something wrong; even more so if you fiddle with a lockless data structure as in pktqueue's case. It might be useful some day is a very poor and dangerous reasoning in this case. All of your arguments boil down to can't trust someone else. Why do you need to be so insulting of other developers in your arguments? Do you think you're the only person capable of making good design decisions? Sorry, but I won't be a party to that kind of attitude and want nothing more to do with this. I think that Mindaugas is being pragmatic here. Developers are not equally brilliant[*], observant of the rules, or perceptive of the patterns, layers, or abstractions in the code. He is writing the code in a way that discourages us from casually misusing or breaking it by getting under the abstraction. I don't find that offensive. Dave [*] However, we are all above average at NetBSD.
re: Lockless IP input queue, the pktqueue interface
I think that Mindaugas is being pragmatic here. Developers are not equally brilliant[*], observant of the rules, or perceptive of the patterns, layers, or abstractions in the code. He is writing the code in a way that discourages us from casually misusing or breaking it by getting under the abstraction. I don't find that offensive. i agree 100%. infact, i spent a few weeks earlier this year making every struct that isn't used as part of a direct interface (like, eg struct tm) in an program written in 1990 hidden inside the relevant source. in the process i found about a dozen bugs, a couple that had evaded me for quite a while, and introduced 4 that were fairly quickly fixed.. (that i know about :-) not *all* code can have structures hidden, but if it is easy to do, it is almost certainly the right solution. it encourages modular code by giving you easy to see boundaries. it shouldn't be a hard-fast rule, which is why pragmatic is great way to describe what rmind is saying. regardless of how brilliant developers are, i want my code to be written like this anyway. it helps *everyone*. .mrg.
Re: Lockless IP input queue, the pktqueue interface
On Fri, May 30, 2014 at 12:01:23AM +1000, Darren Reed wrote: [code cleanup] All of your arguments boil down to can't trust someone else. Why do you need to be so insulting of other developers in your arguments? Do you think you're the only person capable of making good design decisions? Sorry, but I won't be a party to that kind of attitude and want nothing more to do with this. I am surprised... no, more like shocked really... that someone as experienced as you are could think this way. -- David A. Holland dholl...@netbsd.org
Re: Lockless IP input queue, the pktqueue interface
On Fri 30 May 2014 at 05:46:37 +1000, matthew green wrote: regardless of how brilliant developers are, i want my code to be written like this anyway. it helps *everyone*. Sometimes I say, on the topic of code, if I don't understand it, it's probably wrong. That's not because I'm so brilliant, but because code should be clear enough to be understood by average programmers. Code is twice as difficult to debug as to write (citation needed!). So if somebody writes brilliantly complex code but it needs to be debugged later, then nobody can do it. And it is still possible that the reason that I don't understand some piece of code is that the author had a mixed up thought process. -Olaf. -- ___ Olaf 'Rhialto' Seibert -- The Doctor: No, 'eureka' is Greek for \X/ rhialto/at/xs4all.nl-- 'this bath is too hot.' pgpsr5VNMpuDW.pgp Description: PGP signature
