Re: [osol-discuss] VMware published under GPL
Would it help to ask for a joint Opensolaris.org/VMware project to port the host OS kernel modules to Opensolaris? The biggest hurdle isn't porting the kernel modules, but rather the gazillion components that make up the UI, daemons, etc. Since these components need to build under the VMware internal toolchain (which isn't externally visible) I'm not sure there are many opportunities to help there. But we have been thinking about it... - Eric This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Project proposal: 64k kernel project
Frank Hofmann wrote: How would this work with network I/O? Do the IOMMUs support 64k pages? AFAIK yes Yes. I remember I/O (both the ingraindness of MAXBSIZE in segmap and the segmap needs to die a slow, miserable death. exposure of PAGESIZE via VOP_GETPAGE) was a big killer argument against The VM/FS interfaces need a good rewhacking anyway. the 64kB pagesize project back in its time. Even these days, I/O sizes are often few kB, especially on networks even with jumbo frames, but also still on disks/filesystems :( IMO 8K (or smaller) blocks still need to be supported for I/O. The fact that the I/O subsystem relies on assumptions of PAGESIZE throughout is unfortunate, and will make an effort such as this difficult. In due time the plan is for the VM2 work to slowly remove these restrictions but it isn't going to happen overnight. Depends a lot on your average I/O chunk size. Which is certainly well below 64K for files and ethernet. Most of the interesting code in the 64K prototype was to fix UFS' assumption of MAXBSIZE = PAGESIZE, and to hack around the PAGESIZE exposure to FS via VOP_GETPAGE(). There was also some code to synthesize 8K TTEs in the ITLB miss handler for CPUs which don't support 64K in the ITLB (Cheetah this means you). Cleaning up this code and bringing it forward to Nevada so it can be CDDL'ed is a very nontrivial piece of work since there is no one expert in all of the areas touched (it would require many engineers to bring it up to date I think). Instead I would think it would be more productive to think about ways to get a prototype working without these hacks: pick a FS, modify it to use a PAGESIZE-agnostic set of VFS interfaces, - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
[osol-discuss] QEMU project is open for business
The QEMU project is live and open for business. http://www.opensolaris.org/os/project/qemu/ The QEMU project provides a centralized place for documentation, support, and further OpenSolaris-related development of the QEMU virtual machine. For those of you who don't know what it is, QEMU is a fast virtual machine capable of running any x86 OS on an x86 or SPARC host. QEMU also has limited support for emulating other processor architectures (most of the other architectures are at least capable of booting a small custom Linux distro). If you want to run Windows or Linux under your favorite OpenSolaris distro, crash (err, create :)) custom kernels, or you even want to single-step the OpenSolaris kernel and examine the machine state, you'll find QEMU to be a useful tool. A particularly interesting feature: when using the kernel accelerator module (available on the download page) x86 guest OSes run at near native speed on an x86 host! Further discussion should be directed toward [EMAIL PROTECTED] Enjoy. The QEMU project team. ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
[osol-discuss] Project proposal: DMA Memory Infrastructure project
I propose a virtual memory project, DMA Memory Infrastructure (DMI for short). This project will provide infrastructure to: - Allocate physical memory in contiguous [1], large page [1], or small page-size chunks within device address constraints - Lock and unlock the allocated memory pages and return bus addresses as a scatter/gather list - Create kmem caches for frequently constructed DMA objects - Reduce external fragmentation on x86/x64 platform for more efficient scatter/gather DMA - Optionally map the memory into process address spaces and unmap it when required - Optionally map the memory into kernel address space and callback to device drivers to unmap it when required - Optionally reserve physical memory at boot for contiguous device allocation (useful for devices like shared memory frame buffers) - Control processor cache attributes for all established mappings This project does NOT provide leaf driver interfaces -- that is out of scope -- it only provides the core kernel infrastructure required for developing new high-performance DDI DMA interfaces with improved RAS. However the interfaces should also be usable by existing driver _frameworks_ such as GLD, SCSA, and SATA. Initial leads will be myself and Mike Corcoran. - Eric [1] When physical memory is not fragmented. ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
[osol-discuss] Re: Community Proposal: Gardeners
I would like to propose a new community called Gardeners based on some recent discussions on opensolaris-code. [1,2] ... I think this is great. +1 - Care and feeding of the build process Any improvements in this area would be greatly appreciated. In particular, my personal nit is that the Solaris build system lags behind some other build systems for large OS-scope projects. For instance, we don't have incremental linking (I can't build one .c file and link in .o files from the gate's build which didn't change), and incremental builds are slow because of relying on make to handle the dependencies dynamically, rather than statically generating the dependency tree. [3] project, our focus will be more on structural changes than on code examination (we're not here to sweep up after your broken project). The Yes please. community expects that it will necessarily have some overlap with the OS/Net community and with the Tools community. I think your proposal clearly defines the scope, and it appears that this is just fine, any overlap would be healthy in this case. Regards, - Eric This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
[osol-discuss] Project proposal: qemu
I propose a qemu project. This project will contribute and support on an on-going basis: - A kernel accelerator module for kqemu - Support for building 64-bit qemu out of the box (kernel accelerator requires this on AMD64) - Support for building qemu with Sun Studio 10 and 11 compilers - Support for ATAPI DMA running OpenSolaris as a guest OS - Performance enhancements for OpenSolaris as a guest including support for -kernel-kqemu with the accelerator - Anything else spiffy we can think of! The leaders initially will be Juergen Zimmermann and myself. - Eric This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Project proposal: qemu
Dennis Clarke wrote: However, here comes the tough part, is anyone upstairs in the corner office doing anything to get VMWare running on Solaris 10 ? I know that Andy Tucker has shuffled over to VMWare and he must be doing some sort of magic there. Or is it a Windows and Linux club only? I just can not think of a better place to run VMWare than on a maxxed out X4600 and I sure as hell don't mean on top of Slep - SUSE Linux Enterprise Poop. I won't speculate, but a lot of us think it would be very cool. The more solutions we have (Zones+BrandZ, Xen, QEMU, VMWare), the better, since each will have its place, and virtual machines are the binary compatibility of tomorrow (i.e. deploy an appliance, not an app on an OS). - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Project proposal: qemu
Martin Bochnig wrote: History: .. Wow this is great information, I'll be sure to archive this. Though the sparc host support currently in qemu's cvs still misses a few things. I will release a new patch as soon as I come to that (after certain other public launches). As well as upgrading my CSWqemu package on Blastwave. I have some patches to send you for building 64-bit binaries and supporting kqemu to get into Blastwave. It probably isn't worth going for sfw integration until we have something more stable, Blastwave is awesome because you can just Plug-N-Run(TM) every time a new feature gets hack^H^H^H^Hadded. :) Here is still my old work, based on what Juergen Keil and Johannes Schindelin have achieved. This was posted to public back then. http://user.cs.tu-berlin.de/~mbeinsx/s_r_c/ http://user.cs.tu-berlin.de/~mbeinsx/s_r_c/qemu_0.7.2_SPARC_HOST_PATCH__FullDiff.diff This is excellent. Hopefully we can get everything under one roof. Martin, would you like to be added as a leader also so you can edit the project pages? Cheers - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Project proposal: qemu
Someone please yes yes ... get Martin involved. The guy rocks. Also .. has anyone heard from Ben Taylor ? Yep, I had a thread offline, he's buried right now after coming back from an extended vacation. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] ONNV - ON community renaming proposal
Danek Duvall wrote: I'd like to propose that we rename the onnv community to be the on community. The original name is more suitable for a project (a proposal for which is to be made shortly), but we didn't have the infrastructure at the time to do the right thing. +1 And while you're at it, would you create a Nevada project which is sponsored by the ON community? Or is everyone content with the current structure as-is? - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Ultra 20 CD/DVD burning probs
Rich Teer wrote: This is a weird one: burning CDs and DVDs in my Ultra 20 is a crapshoot. Sometimes it works, and sometimes it doesn't. The Ultra 20 is using the DVD-RW drive that Sun shipped with it, and I'm using the latest BIOS and b42a of Nevada. The media I'm trying to burn is brand new Maxell branded DVD-Rs and CD-Rs, and they all seem to be in excellent physical condition. I had the same problem -- cdrecord reported that the drive was returning a servo motor failure error.. I replaced the drive and the problem is gone. Your problem may be different but you'll find cdrecord will tell you why it failed whereas cdrw just sticks its tongue out at you after eating a perfectly good blank disc. ;) - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: [ufs-discuss] Generic filesystem code list/community for opensolaris ?
Rainer Orth wrote: * Therefore, an alternative model would be to use the storage community (given that block and object based storage are getting closer recently, cf. the object storage devices work for SCSI) as an umbrella for both block-based storage (e.g. iscsi, svm projets, maybe others?) and filesystems (with the projects mentioned above). i.e. create a generic Filesystems community and grandfather the existing FS-related communities as projects, maybe adding more related projects at the same time. While I agree, there is the not-so-small problem that there currently is no mechanism in the charter to grandfather communities into other communities demoting them to projects in the process. Moreover, the communities being absorbed would have to agree to this occurring... and defining exactly what agree means for an entire community is a bit of a dilemma. The end result of all of this is I don't expect to see any community restructuring soon, because it's sufficiently difficult to drive any sane person away. :/ That was probably me, speaking of Erez Zadok's FiST work on stackable filesystems: http://www.filesystems.org/ Yes that was it, thanks. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: [ufs-discuss] Generic filesystem code list/community for opensolaris ?
Joerg Schilling wrote: Does this mean you don't like the generic fs community because the current comunity structure is wrong and you cannot change that? Let me try to articulate my concerns a little better. I'm 100% in favor of having an FS community. I've also mentioned many times I would like to see a kernel community as well. My concern is regarding overlap in communities, and its resulting potential for conflicts and ambiguity. According to the governance and charter of communities, the core developers within a community are responsible for setting the direction of the community, determining releases, etc., not just establishing projects. If there are overlaps between communities, then it is no longer clear which community has jurisdiction over the direction, roadmaps, release cycles, or even end product names. Hence my comment earlier in this thread of that way lies madness. Having competing projects can be a good thing, especially early on when the various teams involved have different approaches in mind, the projects are in concept phase, and there is little data in hand to determine with any degree of certainty which approach is technically superior. Having competing communities sounds like a recipe for disaster. Perhaps I'm overly paranoid, and having an FS community would be just fine alongside UFS, NFS, and ZFS. In general I'm dissatisfied with the overall community structure because it seems spread far and wide, yet it demonstrably has significant gaps in areas where OpenSolaris needs community involvement and guidance. Furthermore, as it stands right now the governance proposal has no means of addressing the situation. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: [ufs-discuss] Generic filesystem code list/community for opensolaris ?
Frank Hofmann wrote: sorry opensolaris-discuss, this originally bounced. Trying again. We're proposing a generic fs-discuss for OpenSolaris. Any reason why we shouldn't have a generic filesystem community? You're proposing a discussion list, plus there is work afoot by yourself and possibly others to document filesystem interfaces. There are FS ports to OpenSolaris (ext2, FUSE). And, as someone pointed out recently on opensolaris-code there is a kit (albeit outdated now) for writing a simple filesystem for education purposes which is portable across many systems including OpenSolaris kernel. Filesystems seem as alive and well as anything is alive and well in OpenSolaris that has a community now. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: [ufs-discuss] Generic filesystem code list/community for opensolaris ?
Rainer Orth wrote: Alternatively, the following more general proposal http://www.opensolaris.org/jive/thread.jspa?messageID=28219#28219 might be worth consideration. The ultimate outcome of that discussion was that it was decided that opensolaris-code would not be removed as was planned, and that it would be repurposed as a technical discussion list. So far it has not seen a lot of traffic but there has been some. The question of whether a kernel community would include filesystems or not is one of the things which has kept us from proposing a kernel community as we do not want to have significant overlaps with other, established communities. (That way lies madness) - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Sun ce port GLDV3 Drivers
Cathy Zhou wrote: Steven Sim wrote: Hello; I have pushed for the usage of dladm for link aggregation (trunking) at my customer's site only to have egg on my face when I discovered the only network port with GLDV3 interface (necessary for link aggregation) is bge. The ce port is extensively used. May I know when Sun will release the GLDV3 driver for the ce network port? Will the above be included in Solaris 10 Update 2, supposedly due in June '06? One of the goal of the Nemo Unification component of Clearview project is to use the same set of administrative tools to administrate network devices, no mater they are GLDv3 based or not. After this work, you will be able to using dladm to configure trunking on Cassini devices without porting the ce driver to GLDv3. This is good. Given the history of the cassini I would consider a GLD port unlikely, unless someone decides to rewrite the whole thing from scratch. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Network Device Drivers tosibha laptop m105-s3011
Rajeev wrote: Dear All, Does anybody know where to get the drivers for... 1. Intel(R) PRO/Wireless 3945ABG Net To my knowledge nobody has this working with ndisulator yet. I'm trying to hack ndisulator enough to get it working on my Sony SZ-140 since it has the same card, I've gotten close but haven't quite finished it yet. It requires a lot of additional code to the ndisulator over the version which is posted on the laptop community. 2.Intel(R) PRO/100 VM Network Conn Intel Pro/100 is supported by iprb, has been for a long time. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Project proposal: NFS/RDMA
Spencer Shepler wrote: I would like to propose the creation of an NFS/RDMA project to be affiliated with the OpenSolaris NFS community. The purpose of this project is to update the existing NFS/RDMA(Infiniband) support in OpenSolaris to match the most recent Internet Drafts on the topic. It would also undertake performance characterization and tuning of the implementation. Sun and Network Appliance have funded Professor DK Panda and his team at Ohio State University to do the majority of this work. Their current project page may be found at: http://nowlab.cse.ohio-state.edu/projects/nfs-rdma I will act as project leader. +1. Also I'm very interested in the VM interactions of the NFS implementation and what we can do to improve performance. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Project proposal: NTP
Rainer Orth wrote: We propose the creation of an NTP project on OpenSolaris.ORG, affiliated with the Nevada and Device Driver communities. Seconded. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: Distributed File System for Solaris
Andrew Watkins wrote: I don't see why CacheFS can not work in this situation, so that if there is a local version of the file use it and not just hang... I don't think CacheFS is a good match here. There is a disconnected mode where the server isn't consulted, but in my experience you can't ensure that all blocks of all files will be available in the cache. Also, CacheFS doesn't work with NFSv4 :-( As I said at first glance Cachefs looked perfect for the jobs and it was so easy to set up, but by the looks of it it may be better to start using rsync. The NFSv4 point is interesting and it looks like cachfeFS will become another EOL product. Yes work is afoot to get it removed. If folks want a real distributed FS solution we'd be better off writing a completely new implementation. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: Re: technical (kernel?) discussion list progress?
Dan Price wrote: You said: as long as everybody knows to go there for technical questions, spec posts, and RFC submissions. That sounds like you're spreading your project's wings out over the charter of other communities and projects. That certainly wasn't the intent. I haven't seen complaints about opensolaris-code overlapping with anything else so far so moving forward as we try to build volume over there I don't think we will... Well, it isn't the first time, and I guess this discussion is now over, since you've announced your project. As you'll see we have removed the discussion part from the project description just leaving the repository part. opensolaris-code will be repurposed per the thread on cab-discuss. In the end I don't think any of this is in conflict with any of the points you were trying to make? - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: Re: technical (kernel?) discussion list progress?
Moazam Raja wrote: I guess the reference to [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] in the announcement left some people confused and thinking that this was the new list previously being discussed in this thread. Wow...even that sounds confusing. I suppose the confusion isn't surprising, this is such a high volume list it's easy to miss something important. :) So, for those of you who are keeping score at home: OpenSolaris-code: 1 Muskoka-discuss: 0 The OpenSolaris technical discussion alias will be opensolaris-code. Muskoka-discuss will be for suggesting or providing content to post in the repository (and we've had several good suggestions already!). - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: Re: technical (kernel?) discussion list progress?
Shawn Walker wrote: I thought the same thing. The first thing I did when I saw the name muskoka being thrown around was to search for it on wikipedia. While it's cute and all, I relaly think that mailing lists should have a blindingly obvious title :) The suggested name of, Solaris Kernel Mailing List; seems rather appropriate. There is no hidden meaning. I picked muskoka for the project because there is an internal machine within Sun which serves a similar purpose for engineering, it has slideware, home pages, etc. - various random things - hosted there. The name is purposely obscure. It's just meant to be a hodge-podge repository of random techno-babble. :) As for the technical discussion list, as I've said before, I don't care about the means as long as we accomplish the ends. For all I care we can call it [EMAIL PROTECTED], and as long as everybody knows to go there for technical questions, spec posts, and RFC submissions. [EMAIL PROTECTED] does not fit the bill, sorry. Neither does SKML. The intent was to have a broader-reaching technical discussion list. Linux is a kernel; OpenSolaris is an *os*. I would like the mailing list to reflect that difference in reality, although the mission is certainly similar. I suggest that, if folks really want a kernel mailing list, we should add a kernel community. There is plenty of innovation going on in core kernel infrastructure; I've seen everything from research on queued spinlocks and mutexes to a suggestion of a dispatcher redesign to a complete virtual memory system overhaul. Getting back to the topic of this thread, I'll throw a random suggestion out there: tech-discuss. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: technical (kernel?) discussion list progress?
To specifically address your question: Also, the project seems to have mixed together the idea of a discussion, virtual memory changes, and a library. Wouldn't it make more sense to have a library project which is separate, and a VM project which is also separate? The muskoka project is meant to be a technical library. We folded in the discussion list due to an earlier post from Stephen Hahn suggesting that this would be a better option from an infrastructure standpoint than creating a new top-level list to replace opensolaris-code. There wasn't any debate on that thread; if other folks feel that muskoka-discuss isn't the right place for this discussion, and you think we should instead have a top-level mailing list for this, now is the time to speak up. Nothing is cemented yet. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: Re: technical (kernel?) discussion list progress?
[EMAIL PROTECTED] wrote: I believe it adds a barrier to entry for those willin gto participate and not knowing where to look. It seems as if a lot of high technical, interesting, content may be hidden under the muskoka cloak. A name is just a name; we could change it to something less hidden like techlib. There are so many projects with codenames already, are we just fighting the tide? Getting back to the topic of this thread, I'll throw a random suggestion out there: tech-discuss. So is the intention of this mailing list to be: - all things technical, where no other project/community is appropriate. Yes. In addition this would be a good place to raise issues which cross community boundaries, and to post RFCs (requests for comment outside of the project's hosting community or communities). tech just seems to generic; somewhere I am picking up vibes that JDS and install may not be the intended audience. If I wanted to be exclusive I would have just gone for the kernel mailing list since I'm a kernel guy. :) - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] technical (kernel?) discussion list progress?
Peter Buckingham wrote: There was some discussion about having a more technical mailing list/community ala freebsd hackers/lkml/... Was there any progress made on that? I'm definitely interested in discoverying/learning more about the internals of Solaris. It has been discussed, but there is some grumbling that opensolaris-discuss is still OK for this kind of discussion since we haven't reached the threshold of pain yet as far as traffic. There has also been a fair amount of this type of traffic on opensolaris-code, although that list should be going away soon... - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] technical (kernel?) discussion list progress?
Al Hopper wrote: On Tue, 11 Apr 2006, Eric Lowe wrote: Peter Buckingham wrote: There was some discussion about having a more technical mailing list/community ala freebsd hackers/lkml/... Was there any progress made on that? I'm definitely interested in discoverying/learning more about the internals of Solaris. It has been discussed, but there is some grumbling that opensolaris-discuss is still OK for this kind of discussion since we haven't reached the threshold of pain yet as far as traffic. There has also been a fair amount of this type of traffic on opensolaris-code, although that list should be going away soon... I would like to propose a highly technical kernel related mailing list/project - name TBD (chosen via discussion) per the original (failed community) proposal by Eric Lowe. Essentially I am proposing the Eric Lowe project/list of his behalf or by proxy, if you will. The list would be a moderated list to ensure that the signal to noise ratio is maintained over the long term. This would be our first OpenSolaris moderated list and it should be considered an experiment that will be successfull or unsuccessfull based on the number of active participants, the quality of the postings and the ability of the moderators to ensure timely postings and to refer off-topic posts to other, more appropriate, unmoderated, lists. Ultimately the community will decide if they like the list being moderated. And it could be converted into an unmoderated list in the future if the experiment is deemed unsuccessful. IMHO we need a highly technical list with good signal/noise ratio and it will attract those talented developers/contributors based on its technical content and retain them based on the fact that they won't have to wade through tons of off-topic or low-tech posts. In my experience there is a co-relation between telented developers, their available time to participate on mailing lists and their tolerance towards list noise. As an aside, I've noticed that everytime someone proposes a new project they are immediatly punished for the we have too many communities/projects/lists already, previous committed, sins of others. IMO we should not punish the proposer(s) of a new community/project for lists/projects they likely had no part in creating. Dealing with the 'too-many-projects-lists issue is a separate issue which must/will be dealt with in time. PS: I will also make myself available as a moderator for the proposed list. +1 I will also help moderate the list if created, and further propose that the list should maintain a FAQ (maybe on the genunix wiki). - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: RFE: /etc/systemtuneabletosetthedefaultpagesize
Hi, Roland Mainz wrote: serious cycles on this though we'd like to, and the folks who were in involved in 64K simply don't have any interest in working on this in the open. Why ? What do they fear ? Being swampedoverburned with too many emails or what ? This is a community -- it's up to them whether they want to participate. The VOP_GETPAGE() and VOP_PUTPAGE() interfaces were added in the page cache unification which I believe (without looking) dates back to SunOS 4.x. The original vnode interface specification from srk was based on the buffer cache. Who or what is srk ? Steve Kleiman. Looking at the notes at the end of the paper it says that the architecture was designed by Bill Joy but (I'm told) Steve was the original implementor of the vnode interfaces. The major problem with these interfaces is that they expose PAGESIZE to filesystems. UFS in particuar is problematic because (as I've said before) it assumes that PAGESIZE = MAXBSIZE. Yes, but in the x86 case we have PAGESIZE != MAXBSIZE so these locations are likely well-known and we do not have to search anymore... The filesystems appear to be prepared to handle multiple pages per block but not multiple blocks per page. The right way to fix this is to refactor the VM/FS interfaces to remove PAGESIZE from them. That work is getting underway now. What about skipping UFS in the initial pass and only concentrate - as Holger Berger suggested - on NFS booting ? That sounds like a fine idea. BTW: Are the ZFS/zfsboot people aware of the problem ? IMO at least the person(s) working on zfsboot should be warned that relying on pagesize for whatever reason may be bad... ZFS doesn't have any of these issues; when Jeff Bonwick started ZFS he knew more about the VM system than most of us did back then, and specifically he realized how broken the VM/FS interfaces are long before we did. weeds One for instance is to go look at the ZFS implementation of mmap() and compare it to UFS. UFS has all sorts of horrible deadlocks between mmap() fault-driven file I/O and its other data paths due to the page cache, whereas ZFS does not. ZFS doesn't exhibit similar problems partly because it requires transactional interfaces -- from the time the data is checksummed to when the transaction group is committed the data buffers have to be immutable. Pulling this off with a unified page cache is simply impractical, since every write would require a writers-lock plus a global TLB shootdown to prevent subsequent writes. Modern architectures are capable of copying the data in about 1/10th of the time it takes to make a user-mapped buffer immutable on an MP. Logging UFS also has transactional issues which have been band-aided over leading to locking hell and codepaths so twisted they make my head spin. VOP_GETPAGE() and VOP_PUTPAGE() and their reliance on PAGESIZE lead to a complete lack of transactional semantics which more and more modern filesystems are starting to require. /weeds Since extensive file system changes are necessary one way or the other to pull this off, myself and others would prefer to take the tact (since we have to go there eventually ANYWAY) of blowing up VOP_*PAGE(), and introducing a new VOP_IO() interface which does not depend on PAGESIZE at all. Instead of a page_t, it would use a base/bounds pair attached to a different data structure which is associated with the I/O transaction. What about committing the original solution first (excluding UFS (e.g. make UFS module unloadable for now in a 64k kernel) and concentrating on a prototype which can only boot via NFS) ? You're welcome to try to implement partial-page reads and writes in the VOP layer if you want. Myself, I wouldn't waste my time on it. As you can see from the discussion thread I think there is a lot more than throwing the code over the wall, Yes... but at least the hungry wolves can chew on it for some time until complains come back... :-) :) because it implicitly assumes that their approach was the best approach to solving the problem or was even tractable. Ok... I'll try to sync with Holger berger then how to proceed... Sounds good. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: RFE: /etc/system tuneabletosetthedefaultpagesize
Roland Mainz wrote: ... delay the project proposal until it is clear that Sun actually releases the patches for their work. Starting from scratch without help from Sun will be much harder. Just a quick note to say I haven't dropped this on the floor. Please give me some time to look into this. Nobody else on the VM team wanted to help look into this right now Sounds a little bit bitter... ;-( ... is there any reason for that ? Nope. The 64K project wasn't done by the VM team, it was actually done by the high-end platform group. Right now the other folks on the VM team (as well as myself for that matter) are too busy fighting fires to spend any serious cycles on this though we'd like to, and the folks who were in involved in 64K simply don't have any interest in working on this in the open. I managed to pull up a webrev yesterday and took a peek; there are about 7,000 lines of code change spread over 122 files. The rest of the changes were committed over the course of Solaris 10 development as bugfixes so they are already in place. The bulk of the changes are in UFS and in the vnode layer. There's also a big whack to the nexus layer since as I mentioned it also uses paged I/O. The VOP_GETPAGE() and VOP_PUTPAGE() interfaces were added in the page cache unification which I believe (without looking) dates back to SunOS 4.x. The original vnode interface specification from srk was based on the buffer cache. The major problem with these interfaces is that they expose PAGESIZE to filesystems. UFS in particuar is problematic because (as I've said before) it assumes that PAGESIZE = MAXBSIZE. The 64K project modified VOP_GETPAGE() and VOP_PUTPAGE() as well as the direct I/O code to support partial page I/O. It's a pile of code, probably 80% or more of the changes. Since extensive file system changes are necessary one way or the other to pull this off, myself and others would prefer to take the tact (since we have to go there eventually ANYWAY) of blowing up VOP_*PAGE(), and introducing a new VOP_IO() interface which does not depend on PAGESIZE at all. Instead of a page_t, it would use a base/bounds pair attached to a different data structure which is associated with the I/O transaction. To do this correctly requires revisiting the layering. Currently if I call read() or write(), say, the system call goes directly into the VOP() call, and the file system goes into the VM layer which goes BACK into the file system with further VOP() calls. If everything went through the VM first, and then into the filesystem, then things like PAGESIZE would be completely hidden from the filesystem, and we can change the base pagesize to whatever we wish, or even make it different for different processes! and I'm buried in other stuff through next week. Ok... As you can see from the discussion thread I think there is a lot more than throwing the code over the wall, but also deciding whether the previous approach was the right one, and enumerating other possible angles of attack. Not initiating the project without the code seems to me putting the cart before the horse, because it implicitly assumes that their approach was the best approach to solving the problem or was even tractable. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: RFE: /etc/system tuneable to set thedefaultpagesize
Roland Mainz wrote: That's correct, though there were some pretty sticky issues with MAXBSIZE since the filesystems (UFS, probably others) assume (wrongly) that MAXBSIZE is PAGESIZE. Obviously when you up the PAGESIZE this isn't true anymore, so some code had to be added to make the filesystem understand that a page doesn't hold a single block, but can hold multiple blocks. How does UFS on i386 deal with this problem ? AFAIK x86 has 4k pages by default and only half the size of |MAXBSIZE| ... It's different, MAXBSIZE being larger than PAGESIZE works, it's the other way around that presents a problem. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: RFE: /etc/system tuneable tosetthedefaultpagesize
Roland Mainz wrote: Aside from the apps that broke the next largest drawback seen with the 64K prototype kernels was that when you have only 64K pages, if you touch 8K in the middle of a mmap() region you end up writing back 64K to the backing store. If you do this enough (which some apps do) you see a performance degradation. The same factor comes into play with paging, since you lose track of your working set; Erm, no. You still have device buffers and max_*xfer* tuneables which counter-act this. Additionally it may not be as bad as you think since The issue is that a 64K page_t has a mod bit of 64K granularity. If I set the mod bit, I've lost the information on what part of the page is really modified, so I have to write back the whole 64K wad to the backing store. Since some apps (stupidly) do a lot of writes with mmap() this is a bad thing; as I recall there were 20% or larger regressions on some file workloads due to this single issue. Don't get me wrong; if I had my way, we'd deprecate mmap(), and push for as many apps as possible to migrate away from it. It is a nightmare to the VM system -- it exposes sysconf(_SC_PAGESIZE), and it throws away too much useful information. It causes us to have to run a scanner in the background to go push the modified pages out to disk. It doesn't tell us what parts of a big page are really modified when a write is issued. It has horrible race conditions with the page cache including one UFS bug which has been open unresolved for ten years. It's.. my evil nemesis. :) read() and write() are a much better model for doing any volume of file access and are so much easier to make large-page friendly since they have a base and bounds instead of being wired to the granularity of the MMU translations mapping a segment... And overall, the performance gains weren't always stellar due to the reduction in page coloring on systems with direct-mapped or 2-way associative external caches. Which was on SF68k-SF15k and not Niagara. But I think SF68k-SF15k would still benefit from such a change - if you measured a performance gain it's still worth to work on this - remember that the kernel and all tuneables were optimizes for 8k pages, not 64k pages. Further fine-tuning could deliver huge improvements in performace (the 8k-kernel is simply maxed out at this point). Yep yep yep. I've been a champion since the beginning to go 64K all the way on Niagara and for sun4v across the board. Upcoming Niagara revs and later planned sun4v processors all appear to have hardware capabilities in their caches that address the coloring issues. The low thread:TLB entry ratio makes big pages a clear winner. If we can overcome (or ignore) the compatibility hurdles, it's a no-brainer on these machines. As a result of the lessons learned from the 64K project I believe the ability to do 8K protection granular mappings to files at the user level moving forward is probably not a bad thing to keep in our pocket, although it is in fairness an undue constraint. I believe that if we depart from the conventional wisdom of physical memory management in the kernel and make the physical and virtual page sizes are correctly decoupled in the VM system (a problem I've been looking at for awhile now -- it ain't easy) this is not a difficult constraint to retain. Do you have time to explain what you are planning here ? Not right now but it is planned to bring a lot of this out into the open. The disadvantage of doing this is that it would bring back from the dead the 32bit/64bit dual testing scenarios that caused us nightmares before we EOF'ed 32-bit SPARC kernel support. Agreed. But for now only a OpenSolaris.org project was proposed (with a pretty big alliance behind the scene) - to get the 64k kernel project back to life, fix it, making it bootable (via NFS or UFS), tune it and then try to measure where it has it's strengths and weaknesses. May I suggest that you propose a 64K kernel project. You'll get a +1 from me, and you'll be off and running. Neither I nor anyone else within Sun can promise to hurl the work done over the wall, as we still have our day job. But clearly there's interest, and there are some good leads to follow. Yes, but MPSS inflicts serious pain elsewhere - and it's unfortunately not solving many problems. Even with agressive MPSS and kernel tuneables (autoMPSS etc.) you still have so many 8k pages around that it hurts. And the comparisation MPSS+kernel-tuning vs. 64-kernel is little bit unfair since the first option was heavily fine-tuned - and the 2nd option AFAIK was not. Intentionally so. I think a fixed page size per architecture approach would be better received. I think the Linux people may disagree here. And there is always Niagara which could benefit from this much better than (auto)MPSS will ever be able to do. What exactly is Linux doing here? It sounds like there is a lot of research and investigation that has been
Re: [osol-discuss] Re: RFE: /etc/system tuneabletosetthedefaultpagesize
David S. Miller wrote: The filesystem issues are the worst part of a larger default page size, for sure. You can waste more than half of your ram when doing something as simple as grepping around in a big source tree with 64K vs 8K pages. I posted some example numbers for doing that in the Linux source tree on sparclinux@vger.kernel.org We're looking at ways to decouple the cache from PAGESIZE entirely. The unified page cache model is looking more and more like a historical relic every day. Doing so requires cooperation from mmap() and from read()/write() we currently don't have to track modified ranges to write back to store. The new mostly-baked proposal is to re-implement the unified cache at the vnode layer instead of the page layer. If we pull it off, it's significantly less code than the current page cache, it's PAGESIZE agnostic, and it'll be really freaking fast since we won't be traversing big global hash tables and grabbing contended locks anymore. Now, the 8K materialization issue I think is not such a big deal. You already have to do it for 256MB TTEs on panther since the Panther I-TLB doesn't support such large page sizes, only the dual indexed D-TLB does. Yes and the code was there way back when to handle the same issues. In fact you can be lazy and just load one 8K I-tlb entry each 64K I-TLB miss on Cheetah, Cheetah+, Jaguar, Panther, Jalapeno, and Serrano. There are several ways to approach this problem, so it's actually the least of concern wrt. going to a 64K default page size. Indeed it never was a big concern. Jonathan hit on the two big order issues that led to the death of the 64K project inside Sun: - sysconf(_SC_PAGESIZE) and broken apps - The increased load on the I/O side from using 64K as the I/O currency for everything using mmap() or direct I/O which are both implemented based on the MOD bit in the page_t. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: RFE: /etc/system tuneabletosetthedefaultpagesize
We really ought to ship a debugging C library people could run for development purposes, a C library which: Yep, the equivalent of kmem_flags for userland. I like it. - dies if the qcmp function passed to qsort() isn't correct. - doesn't have any of the backward compatibility workarounds in malloc (e.g., the allocator can be completely different) One which randomly selects among several *different* algorithms on each call. Now I dare you to depend on the behavior of this interface!! bwahahaha ;) - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Features found in other OS you'd like to see in Solaris
okay, we've just seen some strange problems without swap. i haven't had time to look into why, so we are just going to use swap anyway for now. Please let us know you perceive to be strange issues are. Solaris virtual memory is designed to work just fine without swap space. It's not pretty when you get really close to being out of memory, but that's true whether you have swap space configured or not. When there isn't enough food to go around, there's a famine. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: Re: RFE: /etc/system tuneable tosetthedefaultpagesize
Holger Berger wrote: I wasn't directly involved in the 64K prototype but only 64K and larger were used for user applications, and the page_t was 64K in span (PAGESIZE=65536). There may have been some 8K mappings in the kernel due to OBP handing off translation lists with holes -- I don't remember the details there. Who was involved in this project? I don't believe any of the folks involved directly are active in OpenSolaris yet, so to avoid having vegetables thrown at me I won't say. :) Maybe the project lead at Sun is willing to comment here. Could you ask him, please? OK I'll ask *her* if she wants to chime in. :) First I'd like to ensure that Sun is willing to contribute the code. A project without a basis is useless. Willing, yes. Finding someone with the bandwidth is a different story, as we all still have day jobs. (It is a pile of code, too. This isn't nearly as easy as you would believe it is.) The kernel I/O subsystem does not need to be large file aware. The minimum page size changes from 8k to 64k. By design and definition 64k pages will no longer be large pages at all. This is why I think much less work needs to be done as you think. UFS may need work, MMU core The SPARC IOMMU driver needs significant changes to get this working, since it assumes 8K pages. The hardware is capable of 64K translations at least though I don't recall if there were any hardware bugs there. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [perf-discuss] Re: [osol-discuss] Re: Re: RFE: /etc/system tuneable tosetthedefaultpagesize
Eric Lowe wrote: Holger Berger wrote: I wasn't directly involved in the 64K prototype but only 64K and larger were used for user applications, and the page_t was 64K in span (PAGESIZE=65536). There may have been some 8K mappings in the kernel due to OBP handing off translation lists with holes -- I don't remember the details there. OK I asked for you -- OBP does everything with 8K pages, so we run into trouble when trying to remap OBP's translations into the kernel (which includes a good portion of the kernel itself). As a result, with a 64K pagesize kernel you still end up with a bunch of 8K translations in the kernel. Since these hunks of memory don't have page_t's -- they exist long before the kernel initializes the virtual memory system -- it's mainly a problem for US-III+/IV/IV+ processors since we have to keep one of the big TLBs programmed for 8K in nucleus (kernel) context. That leads to greater TLB pressure on the 64K kernel pages and in a loss of associativity which generates more conflict misses on the 8K translations. The SPARC boot rewrite (to use GRUB to boot the kernel instead of relying on the 1275 interfaces for early system services) should help alleviate this problem, but it's still a ways off. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: Features found in other OS you'd like to see in Solaris
Eric Boutilier wrote: I'd like to have virtual consoles like the ones Linux has (at least on x86). Of course one could use screen, but generally CTRL+A hotkey interferes with e.g. bash hotkey to move to the first character in the cmd-line (yes, ctrl+a, a is a workaround - but nothing more than a workaround). Nino -- Here's some background on virtual consoles in Solaris: http://www.opensolaris.org/os/community/desktop/communities/jds/roadmap/solaris_gaps/#Virtual_Consoles One very annoying gap here is that when my Opteron desktop goes casters-up (yes I run my own kernel bits on my desktop before I put them back) I'm left staring at what appears to be a locked-up GUI screen. The system drops into the debugger, but the video is still in X! I'm hoping that the console stuff will fix that, too. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: Re: Re: Features found in other OS you'd like to see in Solaris
Having general memory protection in the kernel (such that a crashed driver can be known not to have damaged anything else) sounds like an interesting project, but nowhere near where we are today. .. and on some architectures like SPARC (sun4u at least) it's virtually impossible to pull off anyway. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: Re: RFE: /etc/system tuneable tosetthedefaultpagesize
Holger Berger wrote: I wasn't directly involved in the 64K prototype but only 64K and larger were used for user applications, and the page_t was 64K in span (PAGESIZE=65536). There may have been some 8K mappings in the kernel due to OBP handing off translation lists with holes -- I don't remember the details there. Who was involved in this project? I don't believe any of the folks involved directly are active in OpenSolaris yet, so to avoid having vegetables thrown at me I won't say. :) Did you receive any answers yet? I'd like to write a project proposal for a 64k kernel project - assuming Sun is willing to release the sources of their original prototype. Great -- by all means, go for it! We'll need to figure out which community should own this project long-term. We've been reluctant to build a VM community because several of us feel that a broader kernel community would better facilitate open discussion and collaboration, yet such a community would have significant overlap with existing communities, so that still needs some sorting out. For the time being, I think the Performance community would be a good home for it -- check with the leaders. I did check into the source issue a little but haven't had the bandwidth to investigate the details yet. The old project gate is based on S9, so we can't just toss it over the wall. If you're open to a partially complete starter solution to seed the work, i.e. which doesn't quite boot and run on all machines but has most of the right areas at least fleshed out, that would probably be much easier for me to push for on my end. It depends on the application. For the majority of smaller and medium sized applications such as FEM 64k pages have a serious advantage. I/O operations are limited by dwarfpages, too. MPSS buys nothing in this case. Our whole kernel I/O subsystem is not large page aware, which is something that clearly needs work, you're right about that. (And, the fact that the bus nexus internal interfaces are all page-based is a perfect example of why this is so hard to change!) - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
[osol-discuss] Re: [request-sponsor] Re: Contributing Code
Would there be a way to get a list of bugs that are older than some number (say a year or so), getting basically no attention, and are likely to be relatively manageable? If there is a particular area you are interested in contributing, you could request that someone who works in that area go off and do this for you. In general, I don't think this would be productive to do across the board due to the unbelievable quantity of bug reports that would be generated. ;) - Eric This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: Community proposal: solaris-internals
I disagree about the project docs. I think project docs should live with the projects. That's where the conversations and collaboration are taking place. I do agree that there should be a top-level link to docs from the opensolaris.org home page. As one of the proposers let me add my thoughts here: 1) If projects have project docs and they have their own communities and project pages established, that's great. The proposed community could provide a home to projects which don't have their own community or project home established yet. Depending on the size and scope of the effort, they may or may not move out of the house to their own pages. 2) We want a central repository to hold internals documentation in its final form. This will include to a large extent, releasing into the open documentation about project internals of projects which were completed years ago. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Community proposal: solaris-internals
Jim, Operating System is the entire WOS, from kernel through desktop and servers, all the consolidations, so presumably you're thinking of the subset of ON that's not networking. Maybe Core OS or something? Even at that, I think it's really quite vague and likely implicitly refers to internal Sun organization more than it does anything about the code. I agree OpenSolaris should not reflect the structure of Sun's organization . Shouldn't STREAMS be part of any definition of OS? Why would (at a guess) procfs be part of the OS but devfs not? Other than considering historical divisions within Sun, and distinctions that likely mean little or nothing to external developers, it's not at all clear to me what the proposed OS group would cover and what it would not. I agree core OS is really too nebulous. Do you have any ideas on how to break this down? A really rough strawman might look something like the following community structure in place of core OS: kernel user commands libraries but that's not perfect either. (For what it's worth, I'm not happy with the existing networking group. Why wouldn't X.25, SNA, ATM, and other basic networking technologies be part of that group? Why is it just a subset of TCP/IP? But, then, if I were happy ... ;-}) If the existence of a networking community isn't the right thing, do you believe that networking could be merged into kernel and user commands communities? Taking Linux-kernel as an example (though you can argue their model is too broad) filesystems, networking code, and even drivers are part of one community. What about file systems? Why have ZFS and UFS communities (and possibly NFS in the future) instead of one file system community with three file system projects under it? What about drivers? There's no home for them yet, and I (in my little corner of the world) expect drivers to be one of the biggest areas for contributing code as it is in other open source OSes. It sounds like there is a grumbling that the bigger picture of community structure needs to be re-thought, and that isn't what we signed up for with our proposal :), but on the other hand I certainly can see the need to rethink the whole structure. OpenSolaris has grown a lot lately, and the time might be right for a revolution. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Community proposal: solaris-internals
Rainer Orth wrote: Eric Lowe [EMAIL PROTECTED] writes: What about file systems? Why have ZFS and UFS communities (and possibly NFS in the future) instead of one file system community with three file system projects under it? Speaking of which, it occured to me recently, when I posted a question about the interaction of zfs, automount/autofs and nfs to nfs-discuss and zfs-discuss, but got no reply whatsoever http://www.opensolaris.org/jive/thread.jspa?threadID=6717tstart=0 that a different organization of the filesystem and storage related communities might be much better than what we currently have: I can think of two models here: * Create a new filesystems community to subsume the existing zfs, ufs and nfs communities as projects. There might be a new autofs/automount community as well, and the recent proposal for a cifs community would naturally fall in as a cifs project, as has already been suggested. The problem is what to do with svm, which doesn't fall under filesystems. * Therefore, an alternative model would be to use the storage community (given that block and object based storage are getting closer recently, cf. the object storage devices work for SCSI) as an umbrella for both block-based storage (e.g. iscsi, svm projets, maybe others?) and filesystems (with the projects mentioned above). Thoughts? After going through this thread I've had similar thoughts. Since the postings are dwindling off I think we may have more or less reached a group concensus. I'm going to go off and draft up a set of proposals to replace our original single proposal, and submit them as separate threads to facilitate further discussion. Thanks for all the great feedback everybody. I think we have a great atmosphere set here for positive change. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] opensolaris-{code, rfe, bugs} (was Community proposal: solaris-internals)
Al Hopper wrote: On Thu, 16 Mar 2006, Mike Kupfer wrote: I think opensolaris-code can just go away. It was originally created Agreed. during the Pilot program, when there was a single consolidation and a lot fewer people. When I proposed it, I thought of it as a stopgap measure to deal with the high traffic, much of it non-technical, on opensolaris-discuss. Now that we have multiple consolidations, each with (at least) its own discuss list, and a whole lot more people, opensolaris-code serves only to confuse people about where they should post. I'd like to see opensolaris-bugs and opensolaris-rfe go away, too. Bugs and RFEs should get posted to the bug database, not mailing lists. And they can be discussed on opensolaris-discuss@opensolaris.org, if necessary. It sounds like we have agreement that opensolaris-code, opensolaris-bugs, and opensolaris-rfe should go away. Who will take ownership of this and make it happen? - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Community proposal: solaris-internals
I believe this proposal needs to provide further contrasts against existing communities and projects to make aspects more clear. opensolaris-discuss is too broad an audience for internals discussions. opensolaris-code I thought was meant to cover code topics and questions, but has been overloaded to handle broader topics due to lack of a better general forum. Perhaps we could promote opensolaris-code to be the official discussion forum for internals, but I personally don't feel this is the right thing. Regardless of the discussion forum outcome, none of the existing discussion lists has a community page to host system internals documentation. We still need to fill that gap. Docs seems focused on end-user feature documentation. As for scope I'm on the fence as far as kernel-internals versus .*-internals. As long as there is something in between opensolaris-code and opensolaris-discuss which is far reaching, I'm fine with it as one of the leaders, though I prefer to err on the side of keeping it broad. 1. What is the relationship between this community and existing, demonstrably technical communities, like the networking, zones, and zfs communities? Networking, zones, zfs, etc. are focused on specific areas of the system, and a good portion of their discussion (the vast majority in fact) centers around learning about or using the specific features rather than learning about the nitty-gritty technical details. We want to create a forum which is 100% purely technical in nature. 2. What is the relationship between this community and the existing ON (Nevada) community? Why is that alias, or a second alias (or project) not appropriate for hosting this content? (Why not [EMAIL PROTECTED]) Same argument as above. 3. What is the relationship between this community and the existing opensolaris-code and opensolaris-rfe aliases (which are discussing technical topics regarding ON components)? Already covered above. 4. We're examining communities commenting on new community proposals and community-to-project demotion processes on cab-discuss; no aspect of that discussion really suggests that one community should propose its existence based on the subsumption of others. I certainly don't think that that's a suitable mechanism for the alleged proliferation problem. The communities which exist today have their place because there are aspects around learning about and experimenting with new features. Continuing down that path without providing a more general technical forum means that if I want to keep up on the state of the art of the system internals, I have to drown in non-technical information and discussion. Linux, NetBSD, FreeBSD, and others have kernel discussion lists for this reason. Beyond the discussion list we want to build a community page to capture system internals documentation and provide a one-stop shop for system internals (aside from buying the book). -- Eric Lowe Solaris Kernel Development Austin, Texas Sun Microsystems. We make the net work. x40577/+1(512)366-9080 ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Community proposal: solaris-internals
Things were setup that way before projects existed and everything had to be a community. The Nevada community (which is misnamed to begin with), should become a ONNV project of an ON community which could also host your internals information. ... and proposed discussion list(s). Agreed, this proposed structure does sound a lot more sensible than the current way it's set up. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
[osol-discuss] Re: opensolaris-{code, rfe, bugs} (was Community proposal: solaris-internals)
Mike Kupfer wrote: I think opensolaris-code can just go away. It was originally created .. I'd like to see opensolaris-bugs and opensolaris-rfe go away, too. Bugs and RFEs should get posted to the bug database, not mailing lists. Agreed. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Community proposal: solaris-internals
Then you'ld have to define OS. The official definition of the Solaris Operating System is the entire WOS, from kernel through desktop and servers, all the consolidations, so presumably you're thinking of the subset of ON that's not networking. Maybe Core OS or something? Yes, core OS - kernel, utilities, and libraries. X, freeware, etc. I would imagine would eventually have their own communities on OpenSolaris, or communities outside of Sun already? - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Unifying configuration files management / use
Yann POUPET wrote: why not having a lib function that would help3 parsing/reading/writing such a file ? With a configuration file such as the example above, one could think about a function that would return the value of arg1 arg = cfgfile_readparam(/file/location,param1) We could also have a function to write the conf cfgfile_writeparam(/file/location,param1,newarg1) cfgfile_replaceparam(/file/location,param2,newarg2) cfgfile_addcomment() Having a config file library which parses config files and works with name-value pairs (see libnvpair(3LIB)) seems like an excellent idea to me. One big advantage I can see over the current proliferation of parsers is that such a library would provide a centralized place for future expansion. For instance, I would like to replace the configuration files stored on each of my machines with an LDAP respository, so that I can administer my systems via a database rather than at the individual machine level. A libconfig or whatever would be a logical stepping stone toward adding such features. Furthermore, as I said before, there may be a lot of config files. Why not have ALL configuration into a single file ? That could be a hash, a berkeleydb file (I think E17 uses such files for conf), or whatever. (isn't it the way MS Win does with is registry ?) For files needed by everyone, this file would be /config (or /etc/config), for personnal config that could be ~/config. Thus only two locations for all configs. Many of these config files have stable-enough commitment levels that this will be challenging. Plus, I don't see any advantage to such an architecture, other than the fact that going that route would force our hand in preventing bad applications from directly parsing and manipulating config files. ;) And there's also the drawback of putting your eggs in one basket (if the central respository gets corrupted, you're toast). - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: Re: RFE: /etc/system tuneable tosetthedefaultpagesize
Hello, [...] Comparing SF68k/SF15k with Niagara is problematic. The broken MMU design in the US3/4 CPU models used in these machines is not able to use a significant amount of 64k pages. If you still got a small performance win there then this would prove that an all-64k kernel has significant performance advanges over the stock version with it's 8k dwarf page size. Please explain what you mean by broken. If you mean US-III, I would agree, as this chip only supported 8K pages in the T8 (the only big 512 entry) TLB. 64K performance was, well, problematic on that machine because all the entries fight for the fully associative T16 TLB which only has around 9 available unlocked entries. Not a pretty picture! For this reason the autoMPSS code is turned off on US-III platforms. We tried it, and it was miserable for some apps. On the other hand, US-III+, US- IIIi, and later chips support programming both large 512-entry data TLBs so there is no loss of associativity nor capacity, so 64K pages work great within the DTLB. The performance analysis was, as I recall, done mostly using US-III+. The instruction TLB on some of these chips only supports 8K, but that has a straightforward workaround in the form of TTE synthesis/replication. US-IV+ supports 8K or 64K in the ITLB so you're set there. The fact that one of the dual large TLBs is always hard wired to 8K in Solaris today on US-III+/IV/IV+ is an artifact of the implementation and doesn't reflect any limitation of the hardware. Could Sun get the project code released into Opensolaris, please? I agree with both David Miller and Roland Mainz that a kernel which uses 64k pages by default will have significant performance advantages over the kernel which uses dwarfpages. The 8k page size is a significant limitation. I'm sure the archives are lying around somewhere. As I stated earlier, I'd be interested in a thorough analysis of how 64K base page size kernel stacks up against our current autoMPSS code on the T2000. If there was a platform where the TLB-vs-.* tradeoffs are obvious, Niagara would be it. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: Re: RFE: /etc/system tuneable tosetthedefaultpagesize
Holger Berger wrote: US3 only has one TLB set with 512 entries for 8k pages. US3+ improved this by the addition of another TLB set with 512 entries for 4M pages - anything between these points - 64k and 512k pages - was ignored. Today this design shows it's drawbacks as automatic MPSS has only limited use on such CPU models. US-III+ and later can handle 8K, 64K, 512K, or 4M in either 512 entry TLB, so you could program them 8K/4M, 8K/64K, 64K/64K, etc. 64K/64K was the default setting used for the 64K kernel prototype, AIR, and was only switched to 64K/XXX when other page sizes were used. Ok - this is new information for me (Linux just assumes the 2nd TLB set with 512 entries is for 4M pages). It's easier to do things that way. We discovered in the autoMPSS stuff it's best to stick with 8K/4M for a couple of reasons; first SysV intimate shared memory uses 4M pages, so anything else leads to complications if you're using a different size for your heap other than 4M. Second, there is a loss of associativity when you reprogram one of the two big TLBs since the chip treats the 2x512s as a 4-way associative TLB when they're both programmed to the same size. Hence maximizing the reach (and utility) of the second TLB is paramount when you are mixing up page sizes on those machines. The performance analysis was, as I recall, done mostly using US-III+. Did this include the concept that dwarfpages (8k) are no longer available to both kernel and user land applications? I wasn't directly involved in the 64K prototype but only 64K and larger were used for user applications, and the page_t was 64K in span (PAGESIZE=65536). There may have been some 8K mappings in the kernel due to OBP handing off translation lists with holes -- I don't remember the details there. Can Sun release the code? I'd be more than happy to write a project proposal then. I assume we will receive some help from the HPC community. The next generation of vector supercomputers may use similar large page sizes (= 64k) as minimum configurable value and Opensolaris may be a good testbed implementation - assuming we can completely eradicate dwarfpages from kernel and user land. I'll ask around. Keep in mind though that just flipping page_t's to be 64K isn't a magic pill; most applications in the HPC space, etc. still need huge pages to fit their working set in the TLB, and spend most or all of their TLB misses on the working data set, so 64K is not likely to be a win at all in that space. That's what MPSS was designed for (madv.so.1, ppgsz, memcntl(MC_HAT_ADVISE), etc to allow use of 4M or larger if the machine has them). Due to the 80/20% rule 80% of the apps are small enough to fit in the TLB with 8K pages, so again no gain there. So you have to ask, is it worth all of the pain for 20% of the apps, none of which are in the big database or HPC space (since they already use MPSS, or should be using MPSS explicitly)? And again, will any of this do better than our autoMPSS policies can? Of course I don't know, I'm just trying to shed light on the tradeoffs. :) In the last twenty years the page size for SPARC CPUs did only double (from 4k to 8k, both which I call dwarfpages as they are no longer the optimum - in fact they are far below the optimum) while disk and memory throughput increased many many times. Even latency was reduced by a factor larger than 8. One of the remaining factors is the default page size which did not increase to reflect the new conditions. The default reach of the TLB today is what the size of physical memory was in the machine back when the current VM system was designed. A lot has changed since then! - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: RFE: /etc/system tuneable to set thedefaultpagesize
Roland Mainz wrote: Userland. The problem is that mmap() exposed too much detail and as a result assumptions about sysconf(_SC_PAGESIZE) or even the range of possible values resulted in programs dying when sysconf returns 64K for the pagesize. In other words, the value of sysconf(_SC_PAGESIZE) has sort of defacto ended up baked into the Solaris ABI in an unintentional way. ... but how ? I've read the ABI specs and mmap(2) several times and the only thing which AFAIK may be possible is trouble with MAP_FIXED and rounding. But that's again not in the ABI, it's just that the applications make special assumptions about the page size. Yes, exactly. Once enough applications make assumptions about pagesize the platform pagesize might as well have been part of the ABI.. as I recall it was more than one application that broke, so we can't reasonable change it now and claim to remain binary compatible. I assume this also means a (hypothetical) SPARC CPU which only supports 16k and 256k pages cannot be supported by Solaris, right ? Sure, there would be potential problems with that, but I would guess it would be easier to swallow a few apps not working correctly on a new machine as opposed to a new release of Solaris breaking compatibility on existing hardware. But that's my personal opinion. ;) FWIW, the sun4v specification requires CPUs to support 8K and 4M, which the Fujitsu CPUs also support, so I would imagine there wouldn't be any deviation from these page sizes in the forseeable future. -- Eric Lowe Solaris Kernel Development Austin, Texas Sun Microsystems. We make the net work. x40577/+1(512)366-9080 ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: RFE: /etc/system tuneable to setthedefaultpagesize
Roland Mainz wrote: This could be very unfortunate since it limits future development (at least for Solaris... other operating systems like Linux are likely not affected, right ?). The decision may be acceptable today - but in twenty years it _may_ become a real problem - assuming the optimum page size continues to grow like it did in the last twenty years (remember VAX had a page size of 512bytes, now 4k/8k are very common and in the future a larger default page size may become usefull). Fixing the default page size in the ABI to 8k may be disastrous then. It's not baked into the ABI; I was simply pointing out that it might as well be at the moment as far as our ability to change it goes. ;) We still have the freedom to change this down the road, it's just a matter of making the decision that the benefits outweigh the loss of backward compatibility with some apps which assume protections are 8K-granular. In regards to DaveM's remarks I agree completely that it's best to have as few constraints as possible in the design of any system, and that this is a constraint that we don't like to have. Aside from the apps that broke the next largest drawback seen with the 64K prototype kernels was that when you have only 64K pages, if you touch 8K in the middle of a mmap() region you end up writing back 64K to the backing store. If you do this enough (which some apps do) you see a performance degradation. The same factor comes into play with paging, since you lose track of your working set; TANSTAAFL. And overall, the performance gains weren't always stellar due to the reduction in page coloring on systems with direct-mapped or 2-way associative external caches. As a result of the lessons learned from the 64K project I believe the ability to do 8K protection granular mappings to files at the user level moving forward is probably not a bad thing to keep in our pocket, although it is in fairness an undue constraint. I believe that if we depart from the conventional wisdom of physical memory management in the kernel and make the physical and virtual page sizes are correctly decoupled in the VM system (a problem I've been looking at for awhile now -- it ain't easy) this is not a difficult constraint to retain. Also note keeping support for 8K user mappings doesn't necessarily mean we need to constrain ourselves to using 8K everywhere in the VM, or even keeping PAGESIZE at 8K (excepting some tricky issues with MAXBSIZE and filesystems); only that we should (at least for the moment) keep supporting 8K pages for user mappings. We have discussed in the past, and am still considering, changing the basic currency of physical memory in the kernel to be considerably larger than 8K, and sucking it up when it comes to fragmentation loss since segments on average are relatively large. The nice side effect of that would be that every 64K aligned 64K sized span would automatically be promotable to a 64K mapping should we choose without the page relocation and renaming overhead that MPSS has to go through today. BTW: The discussion was about a _tuneable_ which could be set to a value used as default page size (used by kernel and returned by |getpagesize()|co.) - the default for this tuneable should remain 8k. The disadvantage of doing this is that it would bring back from the dead the 32bit/64bit dual testing scenarios that caused us nightmares before we EOF'ed 32-bit SPARC kernel support. This was a problem not just for Sun but also made life more difficult for our ISVs also. If the gains were on average double digit, we might be able to justify such a move, but the typical application gains were a lot more modest, and we can in most cases approach the maximum gains simply by instituting more aggressive automatic MPSS policies. I think a fixed page size per architecture approach would be better received. It would allow people to switch to 64k pages on demand and even allows them to return to the 8k size if something breaks (e.g. setting the tunable is not mandatory). Additionally a shared library similar to /usr/lib/[EMAIL PROTECTED] could be provided to switch to the old page size if individual userland applications cause trouble. And there would be a way to fix all those broken applications out there. Without having such a tuneable it is almost impossible to fix the applications which makes the situation even worse (which may backfire at some point in the future). If we get to the point where we can support sysconf(_SC_PAGESIZE) of 8K or 64K depending on which library you have in your LD_LIBRARY_PATH and manage memory underneath with 64K this would be a great thing; this is a fine long-term goal. It will take a lot of work to get the system into that kind of shape though. - Eric ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Re: RFE: /etc/system tuneable tosetthedefaultpagesize
David S. Miller wrote: The only thing that breaks is if apps don't call sysconf(_SC_PAGESIZE) or some similar function such as getpagesize() to obtain that information portably. .. or they make assumptions about the possible range of values. ;) Or did Solaris accidently return 8K always in some version of the Solaris libc? I don't see how this is possible, as applications No that wasn't the case. The case Bart mentioned was one, an app was creating a unmapped zone around a segment and the segment ended up all being unmapped because they were subtracting 64K off of each end of a 128K segment. It was clearly a bug but since the app was statically linked to library code containing the bug there wasn't a good way to fix it in the field. There were the other drawbacks I mentioned as well, which we could get around by only upping the pagesize on newer platforms with better cache associativity and larger memory. This approach may be tenable. I also disagree with Eric Lowe about the usefulness of increasing the base page size. It's very useful, and that's why we have several platforms under Linux which have moved up to a default page size of 64K or larger (IA64, PowerPC 64-bit). We even use 256MB TLB entries for the Linux kernel on Niagara, and if the chip supported 16GB TLB entries we'd use those too, it's a huge issue. In the case of Niagara the tradeoffs are clearly in favor of optimizing for the TLB. Our auto-MPSS policies are very aggressive on that platform and result in most of the heap and stack mapped with 64K and larger pages, up to 256M, and large pages are also automatically selected for suitably sized text on that platform. It would be interesting to try native 64K PAGESIZE support on a Niagara and see how much of a win it is over what is currently available in Nevada... When the 64K prototype was done a few years back, most of the performance analysis was done on machines of the SunFire 6800-15K class since they have the biggest memories; this was before Niagara silicon even taped out! There are so many applications which are TLB miss bound, and this is especially true with the tiny 64-entry Data and Instruction TLBs in the Niagara cores, which btw must be shared with the real-physical TLB mappings the hypervisor uses to implement domain isolation and also the locked TLB entries setup by OBP and the OS. The sun4v implementation does not use client-side locked TLB entries; the client-locked entries are simulated by hypervisor, which runs completely with address translation off. The real-physical translation for each partition is done using a base/bounds register rather than a TLB. Even with that, having all 64 entries available for four threads isn't exactly ideal. ;) The TSB only helps so much, even with the Hypervisor or hw doing the TSB walk and per-address-space dynamically sized TSBs. Certainly; TLB misses are still slow even with hardware walk, and are really doggone slow with a software TSB walk. -- Eric Lowe Solaris Kernel Development Austin, Texas Sun Microsystems. We make the net work. x40577/+1(512)366-9080 ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [request-sponsor] Re: [osol-discuss] Contributing Code
That seems broken, but I suppose that's my non-ON point of view, coming from consolidations where the engineer marks the bug fix available when they submit the fix for code review and integrated when they putback to the master gate. Perhaps ON needs to reconsider how it uses the fields. (Of course, having the bugs.o.o website show Fix in progress for anything from Bug filed, but being ignored to Fixed but not yet integrated makes this doubly hard for external contributors.) ON allows Fix available to be set after integration, but most folks don't do this; we could make this a best practice and key off of it in the bug view, but it would still be up to each engineer to do this manually since there isn't a automated tool to handle this. Setting the bug to Fix integrated is only done by the gatekeepers after the close of the build as noted earlier. One other problem that I've noticed is that for bugs which have an external contributor assigned you can't see the external contributor in the bug report view. Instead you have to search the list of sponsored bugs looking there, which isn't intuitive and is more work. - Eric -- Eric Lowe Solaris Kernel Development Austin, Texas Sun Microsystems. We make the net work. x40577/+1(512)366-9080 ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
