Re: Announcing: nuOS 0.0.9.1b1 - a whole NEW FreeBSD distro, NOT a fork
On 7/9/13 6:12 AM, Teske, Devin wrote: On Jul 8, 2013, at 11:43 AM, Chad J. Milios wrote: We also had to put one file into the etc directory on the / "beneath" the /etc mount so that /sbin/init can read it before /etc is mounted. There were two or three ways we could do that and each has a tradeoff. I've been bitten by that. Getting access to that file that's "beneath" once you've booted the system can be ... less than easy. if it's hardlinked to another copy that is not "beneath" then you can just edit it. I once had a system at vicor where I had a temporary "beneath" /etc that had all its files linked to files of the same name in /etc.boot/ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Announcing: nuOS 0.0.9.1b1 - a whole NEW FreeBSD distro, NOT a fork
On Jul 8, 2013, at 11:43 AM, Chad J. Milios wrote: > On 07/08/13 00:12, Teske, Devin wrote: >> On Jul 7, 2013, at 2:58 PM, Chad J. Milios wrote: >> [snip] >> >>>/etc is now a ZFS dataset of its own >>>How did we do it? >>>Decades of conventional wisdom says /etc must be on /. >>>Check it out, discuss the whys and the trade-offs. >> Well, I see in nu_install on GitHub how you're doing it: >> >> You added: >> >> init_script="/boot/init.sh" >> >> to /boot/loader.conf, wich -- among other things -- does these two >> interesting things (variable names changed to make things more clear): >> >> zfs rollback -r $zfs/swap/host@blank >> NOTE: $zfs is equal to $( /bin/kenv vfs.root.mountfrom ) minus the leading >> "zfs:" >> >> and >> >> zfs mount $zpool/etc >> NOTE: $zpool is equal to $zfs from above, leading up to (but not including) >> the first slash (/). >> >> Cute. Have to say I wasn't aware of the init_script feature of >> loader.conf(5). Not bad. > > We also had to put one file into the etc directory on the / "beneath" the > /etc mount so that /sbin/init can read it before /etc is mounted. There were > two or three ways we could do that and each has a tradeoff. > I've been bitten by that. Getting access to that file that's "beneath" once you've booted the system can be ... less than easy. I'm interested in your cost/benefit points of having /etc a separate filesystem. On the face of it, I want to say that "/etc" is (or at least contains) the "core identity" of the machine (and to a lesser extent -- because this is BSD after-all -- /usr/local/etc). In my mind, /etc and /usr/local/etc *are* the machine (metaphorically speaking), so the merits of having it as a separate filesystem are weighed against your desired topology. If you want to bunch of machines to look and/or act differently, then a shared /etc is precisely what you want. However, without allowing minor changes (ala ZFS clone/snapshot or by way of UnionFS), you'll quickly find that the only way to cope is with role-based scripting in /etc/rc.conf (it is after-all a shell script) or complicated abstraction layers (for example, using netgraph eiface devices with the jail-name inside them so that rc.conf have have jail-specific ifconfig_* lines). But I digress. I think the better solution to your loading of files "beneath" the eventual /etc filesystem is to throw away the ZFS snapshot/clone method and instead move to a UnionFS approach for /etc. If you use UnionFS for your /etc, then what you do is for each of the machines that you want *that* /etc to appear, you do something like: (as root) mount_unionfs -o below /etc /other/etc Now /other/etc (assuming it was empty before) looks exactly like /etc. Pros: With "rm -f ; rm -W " (in /other/etc) you can reclaim a file from the underlying /etc. ZFS does not allow you to revert a single file (you can revert the entire volume or filesystem, but not a single file). Cons: The advantage of having /etc as a ZFS filesystem is probably going to be the compressratio. Using something like lzjb compression on your /etc directory is beneficial (not as beneficial has say /var/log, but by means of having mostly text files, /etc should compress nicely). But... if you *really* need to compress your /etc (that is to say, you're hard-up enough for space that you need the little-savings that you'll gain from compressing /etc), then you're also hard-up enough that you should just set compression on the entire filesystem (nullifying your need to make /etc a separate filesystem -- /etc would get the compression feature from the underlying root filesystem; whatever that is -- zfs filesystem, zpool, zvol, etc.). So again, UnionFS looks like a win unless you *really* want to set separate filesystem features for /etc that you don't set elsewhere. Were you perhaps after a zfs-/etc for some other reason? because there are other reasons that I'm not getting into. For example, using sysutils/zxfer to make backups of the /etc directory of an entire cloud of machines to a single host. If you don't have /etc as a separate filesystem (and all you want is /etc) then a ZFS stream is of course out of the question and you'll have to resort to rsync. I personally think zxfer is more efficient than rsync but I haven't done the calculations yet to prove it (but it feels like it -- incremental snapshot transfers are pretty darned quick). > What we did (mv /etc/login.conf.db /boot/etc; ln -s ../boot/etc/login.conf.db > /etc/login.conf.db) has the undesirable effect that one must remember to (or > be reminded/automated) run cap_mkdb anytime /etc is rolled to a different > snapshot or a backup is restored to it (if that changes login.conf). > *nods* > With our customers at ccsys.com we have a proprietary management thing in > userland (and you could lose out on that important event hook if you used >
Re: Announcing: nuOS 0.0.9.1b1 - a whole NEW FreeBSD distro, NOT a fork
On 07/08/13 00:12, Teske, Devin wrote: On Jul 7, 2013, at 2:58 PM, Chad J. Milios wrote: [snip] /etc is now a ZFS dataset of its own How did we do it? Decades of conventional wisdom says /etc must be on /. Check it out, discuss the whys and the trade-offs. Well, I see in nu_install on GitHub how you're doing it: You added: init_script="/boot/init.sh" to /boot/loader.conf, wich -- among other things -- does these two interesting things (variable names changed to make things more clear): zfs rollback -r $zfs/swap/host@blank NOTE: $zfs is equal to $( /bin/kenv vfs.root.mountfrom ) minus the leading "zfs:" and zfs mount $zpool/etc NOTE: $zpool is equal to $zfs from above, leading up to (but not including) the first slash (/). Cute. Have to say I wasn't aware of the init_script feature of loader.conf(5). Not bad. We also had to put one file into the etc directory on the / "beneath" the /etc mount so that /sbin/init can read it before /etc is mounted. There were two or three ways we could do that and each has a tradeoff. What we did (mv /etc/login.conf.db /boot/etc; ln -s ../boot/etc/login.conf.db /etc/login.conf.db) has the undesirable effect that one must remember to (or be reminded/automated) run cap_mkdb anytime /etc is rolled to a different snapshot or a backup is restored to it (if that changes login.conf). With our customers at ccsys.com we have a proprietary management thing in userland (and you could lose out on that important event hook if you used anything other than our interface for zfs rollbacks and restoring backups, which we forbid). Since our goals at nuos.org are different, i'd like to implement that trigger somewhere better, ideally as-needed and immediate as possible. if anyone with more intimate knowledge of how and exactly when login.conf.db gets accessed has any thoughts... It could be a disaster for an admin to think their /etc is in a certain state and have that one file be out of sync. If better minds could chip in, I'm wondering if we're better off editing /sbin/init to run init_script _before_ loading the daemon class from login.conf.db (or explain why thats a bad idea) or if i should just add some sort of hook to run cap_mkdb right when needed using a DTrace script or auditd? Does anyone think this issue is moot? (Can't we just document this particular specific "gotcha" instance? I don't think so, I abhor any "gotcha" that deviates from behavior people expect from "upstream" fbsd.) Does anyone agree it's important we come as close to perfect a solution as we can? Is a separate /etc even worth it to people? Should i scrap that feature because of this issue? I think we can tighten this up so theres no twisted ankles and no one falling in this rare case but certainly potential manhole. (the manhole i'm talking about is login.conf and login.conf.db being out of sync because the later is a symlink to /boot/etc and someone might rollback to a more restrictive login.conf and think they're covered without running cap_mkdb again but their login.conf.db is actually out of sync and less restrictive in a way that burns them) Devin, thank you IMMENSELY for bsdinstall and especially bsdconfig. I use them both at work and they make life so much better. And thank you for the simplification using kenv. I was unaware of it ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Announcing: nuOS 0.0.9.1b1 - a whole NEW FreeBSD distro, NOT a fork
On Jul 7, 2013, at 2:58 PM, Chad J. Milios wrote: [snip] >/etc is now a ZFS dataset of its own >How did we do it? >Decades of conventional wisdom says /etc must be on /. >Check it out, discuss the whys and the trade-offs. Well, I see in nu_install on GitHub how you're doing it: You added: init_script="/boot/init.sh" to /boot/loader.conf, wich -- among other things -- does these two interesting things (variable names changed to make things more clear): zfs rollback -r $zfs/swap/host@blank NOTE: $zfs is equal to $( /bin/kenv vfs.root.mountfrom ) minus the leading "zfs:" and zfs mount $zpool/etc NOTE: $zpool is equal to $zfs from above, leading up to (but not including) the first slash (/). Cute. Have to say I wasn't aware of the init_script feature of loader.conf(5). Not bad. -- Devin NOTE: Paring down on the cross-posting (bad OP). Posting only to -hackers@ (as it seems to be the right place to post disseminating analysis). _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Announcing: nuOS 0.0.9.1b1 - a whole NEW FreeBSD distro, NOT a fork
On 07/07/13 22:05, "C. Bergström" wrote: omg you've created Solaris If you're going to spam commercial stuff with absolutely no technically interesting details - please keep it brief at the least. Generally people will be curious about What are you actually adding to the ISO which FBSD-current can't do? If it's not upstream already - will it be contributed upstream? Please reply further on freebsd-chat, I'd like to consolidate any discussion this may garner. This doesn't provide anything to the core OS that can't already be done, albeit with many more keystrokes and the peril of possible confusion and misconfiguration. The main thing here is a collaboration of what we consider best practices and consolidating the more useful configurations into consistent recipes with useful simplification of parameters. We don't mean to add yet another layer in the name of simplicity that obscures or hides the real nuts and bolt beneath and limits your options. We want to make things more flexible and easier at the same time by using the sanctioned FreeBSD ways of doing things, simply allowing the ones with most merit to rise to the top, hopefully through community involvement. We've had a lot of success using this in our production deployments and hope that we don't have to be the only ones to maintain it forever. It is an open offer of contribution to The FreeBSD Project but it probably doesn't exactly belong there yet. It's a layer above, so to speak, and we think we have a place in the community working side by side. It's a distro around FreeBSD, think picoBSD or maybe FreeNAS. It's not going to be a fork like PC-BSD or Dragonfly. I'm hoping we can be a proving ground for the more advanced features of FreeBSD, by allowing more users to jump on board with them sooner, and then offer the applicable bits and pieces back upstream while continually pushing the innovation envelope in a way that more people and companies can participate in. The tool nu_install is basically sysinstall on steroids. It doesn't do all the things that sysinstall does and you may still use sysinstall to configure a system or a jail you've provisioned with nu_install or nu_jail. nu_install automates a process of building a ZFS only FreeBSD system and offers a default dataset layout featuring best practices we've deduced from using ZFS on FreeBSD since its infancy and reading and considering many various differing and conflicting ZFS on root how-tos. For instance, many ZFS on root tutorials use a UFS /boot partition and/or mountpoint=legacy and entries in /etc/fstab. We suffer neither of those holdovers. Another feature I've not yet found in any tutorial is /etc having its own dataset. nu_jail creates cloned datasets and jail.conf entries along the school of thought set out by our nu_install base system. Jails in FreeBSD allow many use cases that were never dreamed of on other platforms and we don't seek to force any particular cookie-cutter way of provisioning a jail, just simplifying the uses that we've found most common. We wanted ease and simplicity but refused to give up less-common possibilities or give up the simplicity just to tweak something a little differently to do something that's never been done. Thank you for reading and offering your thoughts. LOL @ the Solaris comment, as I am a long-time Solaris user and fan but always been a bigger fan of the BSDs and FreeBSD mostly in particular for the last decade. In short, we seek to do with FreeBSD something like what Joyent has done with illumos in their SmartOS but then continue further with that idea. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: [SPAM] Announcing: nuOS 0.0.9.1b1 - a whole NEW FreeBSD distro, NOT a fork
On 07/ 8/13 05:28 AM, Alfred Perlstein wrote: On 7/7/13 3:05 PM, "C. Bergström" wrote: On 07/ 8/13 04:58 AM, Chad J. Milios wrote: Outline of features: Extends plain old FreeBSD 9.1 (RELEASE or STABLE) and maintains total compatibility We seek to remain nimble Expect a production-ready seal of approval to lag behind releases by no more than a week or two and prebuilt images and packages e.g. releases like 9.2 and 10.0, et al Someone should be able to build it and use all applicable features on 8.4 with ease we simply haven't the time or inclination to even try Default full ZFS filesystem layout, completely legacy-free Boot from ZFS, boot to ZFS If you'd like use all 100.0% of all your drives for one large zpool Use one large zpool for all of your filesystems block volumes alternate boot environments, including one called "rescue" which is included NO partitions, not some tiny /, not even a /boot Just ZFS datasets in their infinite flexibility /etc is now a ZFS dataset of its own How did we do it? Decades of conventional wisdom says /etc must be on /. Check it out, discuss the whys and the trade-offs. nu_jail - provision all sorts of jails No guesswork Yet no cookie-cutter limitations Clean-room jails provisioned almost instantly ZFS clone of /etc and /var give you almost no storage overhead nullfs and/or unionfs mounts of /, /usr, /usr/local give you almost no memory overhead Run 1,000 jails and 10,000 Apache instances they safely access the same executable memory pages they securely know not of one-another's existence Advanced intra-host networking with VIMAGE kernel by default, simplified Made for developers who want robustness, power and flexibility streamlined for Unlimited development, testing, staging and production environments Uses all of the new jail and vnet features of FreeBSD 9.1 We cleaned out all of the cruft left over from earlier versions omg you've created Solaris If you're going to spam commercial stuff with absolutely no technically interesting details - please keep it brief at the least. Generally people will be curious about What are you actually adding to the ISO which FBSD-current can't do? If it's not upstream already - will it be contributed upstream? It seems pretty obvious to me that the contribution is that all this stuff works out of the box. That is pretty nice. Ok so I repeat my question If the current ISO doesn't do this - why not? (bugs fixed, different configuration. etc) If it's not upstream already - will it be contributed upstream? (clearly someone is interested in this) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: [SPAM] Announcing: nuOS 0.0.9.1b1 - a whole NEW FreeBSD distro, NOT a fork
On 7/7/13 3:05 PM, "C. Bergström" wrote: On 07/ 8/13 04:58 AM, Chad J. Milios wrote: Outline of features: Extends plain old FreeBSD 9.1 (RELEASE or STABLE) and maintains total compatibility We seek to remain nimble Expect a production-ready seal of approval to lag behind releases by no more than a week or two and prebuilt images and packages e.g. releases like 9.2 and 10.0, et al Someone should be able to build it and use all applicable features on 8.4 with ease we simply haven't the time or inclination to even try Default full ZFS filesystem layout, completely legacy-free Boot from ZFS, boot to ZFS If you'd like use all 100.0% of all your drives for one large zpool Use one large zpool for all of your filesystems block volumes alternate boot environments, including one called "rescue" which is included NO partitions, not some tiny /, not even a /boot Just ZFS datasets in their infinite flexibility /etc is now a ZFS dataset of its own How did we do it? Decades of conventional wisdom says /etc must be on /. Check it out, discuss the whys and the trade-offs. nu_jail - provision all sorts of jails No guesswork Yet no cookie-cutter limitations Clean-room jails provisioned almost instantly ZFS clone of /etc and /var give you almost no storage overhead nullfs and/or unionfs mounts of /, /usr, /usr/local give you almost no memory overhead Run 1,000 jails and 10,000 Apache instances they safely access the same executable memory pages they securely know not of one-another's existence Advanced intra-host networking with VIMAGE kernel by default, simplified Made for developers who want robustness, power and flexibility streamlined for Unlimited development, testing, staging and production environments Uses all of the new jail and vnet features of FreeBSD 9.1 We cleaned out all of the cruft left over from earlier versions omg you've created Solaris If you're going to spam commercial stuff with absolutely no technically interesting details - please keep it brief at the least. Generally people will be curious about What are you actually adding to the ISO which FBSD-current can't do? If it's not upstream already - will it be contributed upstream? It seems pretty obvious to me that the contribution is that all this stuff works out of the box. That is pretty nice. -- Alfred Perlstein VP Software Engineering, iXsystems ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: [SPAM] Announcing: nuOS 0.0.9.1b1 - a whole NEW FreeBSD distro, NOT a fork
On 07/ 8/13 04:58 AM, Chad J. Milios wrote: Outline of features: Extends plain old FreeBSD 9.1 (RELEASE or STABLE) and maintains total compatibility We seek to remain nimble Expect a production-ready seal of approval to lag behind releases by no more than a week or two and prebuilt images and packages e.g. releases like 9.2 and 10.0, et al Someone should be able to build it and use all applicable features on 8.4 with ease we simply haven't the time or inclination to even try Default full ZFS filesystem layout, completely legacy-free Boot from ZFS, boot to ZFS If you'd like use all 100.0% of all your drives for one large zpool Use one large zpool for all of your filesystems block volumes alternate boot environments, including one called "rescue" which is included NO partitions, not some tiny /, not even a /boot Just ZFS datasets in their infinite flexibility /etc is now a ZFS dataset of its own How did we do it? Decades of conventional wisdom says /etc must be on /. Check it out, discuss the whys and the trade-offs. nu_jail - provision all sorts of jails No guesswork Yet no cookie-cutter limitations Clean-room jails provisioned almost instantly ZFS clone of /etc and /var give you almost no storage overhead nullfs and/or unionfs mounts of /, /usr, /usr/local give you almost no memory overhead Run 1,000 jails and 10,000 Apache instances they safely access the same executable memory pages they securely know not of one-another's existence Advanced intra-host networking with VIMAGE kernel by default, simplified Made for developers who want robustness, power and flexibility streamlined for Unlimited development, testing, staging and production environments Uses all of the new jail and vnet features of FreeBSD 9.1 We cleaned out all of the cruft left over from earlier versions omg you've created Solaris If you're going to spam commercial stuff with absolutely no technically interesting details - please keep it brief at the least. Generally people will be curious about What are you actually adding to the ISO which FBSD-current can't do? If it's not upstream already - will it be contributed upstream? ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Announcing: nuOS 0.0.9.1b1 - a whole NEW FreeBSD distro, NOT a fork
PLEASE reply to this only in freebsd-chat. I have posted this announcement to five freebsd mailing lists, I hope I am not overstepping. Hello everybody. My name's Chad J. Milios. Long-time lurker, sparse rare sporadic poster. TL;DR? -- Skip below to our summary of features in an outline format then grab it at http://nuos.org . I would like to enthusiastically announce the release of an open-source project of much pride and passion of my good friend Scott C. Ziegler and myself which we have brought forth thanks to the support and contributions of about 15 others. I believe it is solidly ready to be shared with the world in the hopes that others may help build out the software and community in a way that promotes quality, innovation and collaboration much like FreeBSD has led the open-source community at doing. The nuOS project ( http://nuos.org ) is about bringing back the power to the people! Currently, technical software, hardware and networking power. Ultimately, the power of personal communication and community self-organization. Currently made by geeks/nerds/hackers for geeks/nerds/hackers, our intent is to create an entirely new software ecosystem that promotes quality, easy to use software that is for any-and-every man woman and child yet without lassoing us all into one herd of sheeple. ;) Simple, common things should always be EASY. Complex, amazing or never-before imagined things should always be POSSIBLE. We have a live image for download from our site. (Fully functional at 189 MB, just cat or dd to your 4 GB or larger usb drive or select it as a flat-file virtual disk in your hypervisor of choice. It is not an ISO and nuOS does not work well from optical media.) Or grab our source (currently hosted by GitHub at https://github.com/CropCircleSys/nuOS ) and build the entire system from any FreeBSD 9.1 system with one simple yet deeply customizable command. (We only build/test on amd64 and would like that to change in the future.) It is my belief that our software is PRODUCTION READY with our new beta release. It might just be the answer to the management headaches you may be having. Take the plunge tonight and find yourself breezing through your day-job with "nu"-found ease tomorrow morning. If you're the comfortable yet cautious type, watch the discussion for a week or two first instead. Either way, we intend to cause a positive large and lasting motion in the FreeBSD community. I hope you will give nuOS a look and offer your assessments and ask any questions you have. Please tear it and us apart in discussion with the goal of a better FreeBSD for us all! Documentation is one area that is sorely lacking though it is mostly because Scott and I consider most of our code clear enough to have been pretty self-documenting [for our purposes we've had until now]. It is our hope that with the community's help we will bring more and more of this platform to the high standard of quality that FreeBSD is known for. We aren't trying to create our own new garden. We offer this code with hopes that it, in part or in whole, might be some day included in canonical FreeBSD releases. We have NO intention on forking FreeBSD and are instead developing a very lightweight suite of tools which hopefully capture and collect modern best practices while providing a testing and proving ground for advanced FreeBSD features. We want to bring computing to more people, bring more computer users to open source, bring more high-value and responsible open-source users to FreeBSD and bring more current FreeBSD users guidance and enlightenment regarding advanced features in the face of FreeBSD's typical adherence to maximal backward compatibility, legacy support and solid ground yet sometimes daunting array of intimately detailed configuration choices. We do not seek to limit those choices or to shift the ground beneath current FreeBSD users' feet. We seek to offer an alternative flavor of default system for those interested in taking a step back from their current perspective in order to take a giant flying leap forward. This doesn't mean giving up anything in terms of compatibility or configurabilty, quite the contrary. Throughout our evolution, we seek to always maintain the environment that FreeBSD users have come to know and love while reducing the issues that sometimes irk them. We simply seek to provide a better way to structure, provision and maintain production systems and development processes. Outline of features: Extends plain old FreeBSD 9.1 (RELEASE or STABLE) and maintains total compatibility We seek to remain nimble Expect a production-ready seal of approval to lag behind releases by no more than a week or two and prebuilt images and packages e.g. releases like 9.2 and 10.0, et al Someone should be able to build it and use all applicable features on 8.4 with ease we simply haven't the tim