Re: ZFS Boot patch
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Dec 18, 2013, at 9:34 AM, Allan Jude wrote: On 2013-12-18 12:27, Allan Jude wrote: An issue we thought we had fixed, was not actually fixed. When doing a GELI based Root-on-ZFS install, the 'bootpool' is not always properly mounted in the installed system. The lines added to loader.conf to make it use the zpool.cache file and learn of the existence of the 2nd pool are required, and have the desired effect. However, it seems that the 2nd pool is not always listed in the cache file. The attached patch should fix this issue. Hopefully this can get MFCd in time for the next RC A note for the release notes. If you have an existing install, it can be 'repaired' easily: zpool import -f bootpool Yes, I too had noticed this. Figured it was a minor annoyance. zpool set cachefile=/boot/zfs/zpool.cache bootpool I'll have to re-test, but I didn't find this to be necessary. After importing the bootpool and rebooting, I noticed it always came back. The explanation I told myself was: 1. You boot into the new system 2. You import the bootpool 3. That adds an entry to /boot/zfs/zpool.cache 4. Next time you reboot, that entry is still in there But what I was trying to figure out was how exactly to get the bootpool to auto-import into the newline installed system -- I think the above line you shared shows me precisely how we can accomplish that (I see it in the patch you submitted). This will add the bootpool to the existing zpool.cache (which contains the data for your main pool) This only applies to users who opted to encrypt their zpool. Minor nit... not only for geli; now all MBR layouts use a bootpool. I found this to be required for the edge-case of MBR+NOSWAP (which I fiddled with for almost a week and got nowhere without using a bootpool). - -- Devin -BEGIN PGP SIGNATURE- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJSsgTUAAoJEKrMn5R9npq5SwEH/2iCbpkLaYcKOpn8ibMAxY7k CUvpP8A5r3LEIxrf6lSIuiF0UpavTBQDZLS0n4mjAsQDQh2nyvXp8URpbjFUoHtS CKCP8MlaNit5xEnnIx3nCOuU7yelBkTUUqzSGfKFZc0GHO1uhV9OTvuwNvyrNce2 y4/6l5ieUH4deoLTNQDIS4p6BoJr94JIopWLx/A+jF3SOlt34Z/LxeVEvQGeUaZ7 7YeAfg/NxeAGSnmRBNPox+HloGJvrHXzAhZLmXZI0cUUwvue/icmYH/hth9bmNYZ NGMJckMyAZm600Vi9OoDqJf8y+sHjrYmLAuyXu1hfDb6CBfZD9S49CsTWV40W54= =rP4m -END PGP SIGNATURE- _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Defaults in 10.0 ZFS through bsdinstall
On Nov 14, 2013, at 12:34 PM, Marcus Reid wrote: On Thu, Nov 14, 2013 at 06:35:43PM +, Teske, Devin wrote: On Nov 14, 2013, at 10:21 AM, Allan Jude wrote: On 2013-11-14 9:34, Reid, Marcus wrote: Hi, I noticed a couple of things with the ZFS defaults that result from using the new installer in 10.0-BETA3. One, atime is turned off everywhere by default. There was a thread on this list on June 8 with a subject of 'Changing the default for ZFS atime to off?', and from what I can tell the idea of turning off atime by default was not a popular one. Two, and probably less controversial, is that fletcher4 is specified exlicitly on the root pool, even though it is default (wouldn't you just want to go with the default, in case it changes?) Marcus I have never heard a good argument for having atime on. Can you address some of the issues that people brought up in the thread I mentioned earlier? I'll summarize some: - breaks some software (MTAs were mentioned), and the admin should know when to turn atime on in those cases. - any mail program using mbox mail folders uses them to correctly report which mailboxes have not been read yet I'm looking at HEAD and I don't see atime=off for the /var dataset. Knowing that most folks (accepting the defaults) will store their mail in /var/mail ... does the concern over atime=off still exist? However, I did notice that before we go creating the /var dataset, we do set atime=off for the boot pool/dataset. Is inheritance at-play here? and /var is turning up with atime=off even though it's not specified? In that case, I certainly agree we should remove atime=off from the last place it is used -- the boot pool (nowhere else). - Of course it can't be turned off by default. It is specified by POSIX. - If the overhead were a real problem then file systems would use a special atime cache The performance penalty on ZFS is quite large This is subjective, and if it were that bad it seems like other ZFS-based systems would have turned it off as well, or done something like caching/logging or relatime it to mitigate the performance hit. I do recall discussions about bringing over relatime and making it the default. Allan was the one that brought up the performance hit... Similarly where I $work, we turn it off where need performance (large storage arrays and such). I gather it's only a matter of time until relatime is default. and it also makes your snapshots grow constant. This does not seem to be the case. For example, if I snapshot zroot/usr/src and then tar it up, I generate a bunch of writes and the snapshot grows to 70.5MB. However, when I tar it up a second time, there are writes but the snapshot remains 70.5MB. It seems to use the same blocks for subsequent atime updates. If you made another snapshot, you would see more space used for that one, of course. I'll defer to Allan to expand (was his recollection on snapshots) If you have a use for it, you can turn it on I guess. This would be solved by having the dataset editor we're planning for 10.1 Seems easy enough to turn on once installed. And like Allan says, the editor for 10.1 would allow changing of the default. The same case could be made for turning it off, make that an option in the installer instead. Fair enough. Good idea. Some people don't seem to use atime on a regular basis, but I use it a lot when troubleshooting things, and I think others do to when they get to the point where they realize how useful it is. I'll add that if scripting it, you can (in current state for 10.0) change the dataset options at-will. The fletcher4 thing is a leftover, from older versions of ZFS where the default was fletcher2, I suppose we can remove it While I'm picking nits: There are some other locally set things that are default (exec set to on on zroot/var/tmp is one), that might just inherit that from zroot/var. Need a +1 from Allan on that. Also, zroot doesn't appear to be mounted on /zroot, as it is on other ZFS-based systems. This has the side-effect of newly created datasets not having a mountpoint by default, instead of /zroot/dataset. I don't know of the reason behind this change either. Hmm, wonder if that's been fixed in HEAD already. I'm going to cross-post this to -current, this is a topic that I think warrants a wider audience. Cool. Thanks. -- Devin -BEGIN PGP SIGNATURE- Comment: GPGTools - https://urldefense.proofpoint.com/v1/url?u=https://gpgtools.org/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=wXwxswJCpc3K%2Bud90aYI%2B82v%2Fyfs1mrTBzzeCe5XIXw%3D%0As=a9e0697e43ce4744f130997648d0104319221bba6924c28a9493ffa1cf6c3719 iQEcBAEBCgAGBQJShRf/AAoJEKrMn5R9npq5TYwH/j8RMceM4STpCAoMTAz8zkKn gpftMrdf37TnAPHxsXJaiZejlp+D85/a9lglYbB7e4WdgoDX9ZXA0QLtCEmEtN4y jz7KNa5oABq1GjDtwm2xQISDIe8yYI
Re: Defaults in 10.0 ZFS through bsdinstall
On Nov 14, 2013, at 12:34 PM, Marcus Reid wrote: On Thu, Nov 14, 2013 at 06:35:43PM +, Teske, Devin wrote: On Nov 14, 2013, at 10:21 AM, Allan Jude wrote: On 2013-11-14 9:34, Reid, Marcus wrote: Hi, I noticed a couple of things with the ZFS defaults that result from using the new installer in 10.0-BETA3. One, atime is turned off everywhere by default. There was a thread on this list on June 8 with a subject of 'Changing the default for ZFS atime to off?', and from what I can tell the idea of turning off atime by default was not a popular one. Two, and probably less controversial, is that fletcher4 is specified exlicitly on the root pool, even though it is default (wouldn't you just want to go with the default, in case it changes?) Marcus I have never heard a good argument for having atime on. Can you address some of the issues that people brought up in the thread I mentioned earlier? I'll summarize some: - breaks some software (MTAs were mentioned), and the admin should know when to turn atime on in those cases. - any mail program using mbox mail folders uses them to correctly report which mailboxes have not been read yet I'm looking at HEAD and I don't see atime=off for the /var dataset. Knowing that most folks (accepting the defaults) will store their mail in /var/mail ... does the concern over atime=off still exist? However, I did notice that before we go creating the /var dataset, we do set atime=off for the boot pool/dataset. Is inheritance at-play here? and /var is turning up with atime=off even though it's not specified? In that case, I certainly agree we should remove atime=off from the last place it is used -- the boot pool (nowhere else). - Of course it can't be turned off by default. It is specified by POSIX. - If the overhead were a real problem then file systems would use a special atime cache The performance penalty on ZFS is quite large This is subjective, and if it were that bad it seems like other ZFS-based systems would have turned it off as well, or done something like caching/logging or relatime it to mitigate the performance hit. I do recall discussions about bringing over relatime and making it the default. Allan was the one that brought up the performance hit... Similarly where I $work, we turn it off where need performance (large storage arrays and such). I gather it's only a matter of time until relatime is default. and it also makes your snapshots grow constant. This does not seem to be the case. For example, if I snapshot zroot/usr/src and then tar it up, I generate a bunch of writes and the snapshot grows to 70.5MB. However, when I tar it up a second time, there are writes but the snapshot remains 70.5MB. It seems to use the same blocks for subsequent atime updates. If you made another snapshot, you would see more space used for that one, of course. I'll defer to Allan to expand (was his recollection on snapshots) If you have a use for it, you can turn it on I guess. This would be solved by having the dataset editor we're planning for 10.1 Seems easy enough to turn on once installed. And like Allan says, the editor for 10.1 would allow changing of the default. The same case could be made for turning it off, make that an option in the installer instead. Fair enough. Good idea. Some people don't seem to use atime on a regular basis, but I use it a lot when troubleshooting things, and I think others do to when they get to the point where they realize how useful it is. I'll add that if scripting it, you can (in current state for 10.0) change the dataset options at-will. The fletcher4 thing is a leftover, from older versions of ZFS where the default was fletcher2, I suppose we can remove it While I'm picking nits: There are some other locally set things that are default (exec set to on on zroot/var/tmp is one), that might just inherit that from zroot/var. Need a +1 from Allan on that. Also, zroot doesn't appear to be mounted on /zroot, as it is on other ZFS-based systems. This has the side-effect of newly created datasets not having a mountpoint by default, instead of /zroot/dataset. I don't know of the reason behind this change either. Hmm, wonder if that's been fixed in HEAD already. I'm going to cross-post this to -current, this is a topic that I think warrants a wider audience. Cool. Thanks. -- Devin -BEGIN PGP SIGNATURE- Comment: GPGTools - https://urldefense.proofpoint.com/v1/url?u=https://gpgtools.org/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=wXwxswJCpc3K%2Bud90aYI%2B82v%2Fyfs1mrTBzzeCe5XIXw%3D%0As=a9e0697e43ce4744f130997648d0104319221bba6924c28a9493ffa1cf6c3719 iQEcBAEBCgAGBQJShRf/AAoJEKrMn5R9npq5TYwH/j8RMceM4STpCAoMTAz8zkKn gpftMrdf37TnAPHxsXJaiZejlp+D85/a9lglYbB7e4WdgoDX9ZXA0QLtCEmEtN4y jz7KNa5oABq1GjDtwm2xQISDIe8yYI
Re: [CFT] bsdinstall and zfsboot enhancements
On Nov 12, 2013, at 4:02 AM, Johan Hendriks wrote: Teske, Devin schreef: Hi all, Another Call For Testing... This one is for bsdinstall. Two patchsets are required for this CFT: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_debug/ http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/ The enhancements are: + Add a `-D FILE command-line option for overriding the path to the bsdinstall log file (BSDINSTALL_LOG env var). + Document new `-D FILE' in the man page for bsdinstall. + If FILE in `-D FILE' begins with a +, debug output goes to stdout (interleaved between dialog(1) invocations/output) as well as to FILE (minus the leading + of course). + If BSDINSTALL_LOG cannot be written, then debugging is disabled (except in the case of a leading + in the pathname, wherein debug will still be printed to stdout). + Update source code format to approximate a future shstyle(9) + Fix a dangling participle (Begun ... - Began ...) + Rewrite the docsinstall script (was necessary to abate direct dependency on BSDINSTALL_LOG (instead, use fault-tolerant bsdconfig framework which displays appropriate errors for package management). + Add additional debug output for dhclient/rtsol/wpa_cliscan + Display script errors in a textbox rather than just on stdout + Update many coments. + Add new f_show_err() API call (like f_show_msg but changes the dialog title to Error)(see bsdconfig's `common.subr'). + Add new f_eval_catch() API call for executing a command via eval but not before logging the command to debug. Several example cases documented in API header for function in bsdconfig's `common.subr'. + Fix dialog auto-sizing when launched as an rvalue to a pipe for indirected scripts (previously would default to 80x24 sizing in this case, now it can autosize to full size even when in a pipe chain). + Fix a bug in f_snprintf wherein if the $format argument began with a - or -- it would be misinterpreted as a flag to printf. (this is in bsdcofig's strings.subr) + Add accompanying f_sprintf() and f_vsprintf() to go along with already existing f_snprintf() and f_vsnprintf() (see bsdconfig's strings.subr). + Update bsdinstall's config script to adjust ttyu* entries in /etc/ttys when it is determined that we are in-fact doing an install over serial (e.g. bhyve). + Remove some unnecessary default ZFS datasets from the automatic zfsboot script. Such as: /usr/ports/distfiles /usr/ports/packages /usr/obj /var/db /var/empty /var/mail and /var/run (these can all be created as-needed once the system is installed). + Remove setuid=off for /usr/home (as discussed with others from last round of CFT). + Fix some i18n string violations in zfsboot + Bolster debugging output in zfsboot + Fix some string quoting issues in zfsboot + Fix some variable scope issues in zfsboot + Only display the ZFS vdev type selection menu when running interactively (duh). + Change Create to Install in zfsboot main menu + Increase pedantic error checking in zfsboot (type-check arguments and such). + Add a call to graid destroy to kill automatic metadata (part of the series of pedantic destructions we do when bootstrapping a new/naked disk). + Make judicious use of new f_eval_catch() in zfsboot + More comments (zfsboot). + Fixup some variable names for consistency (zfsboot). I did not try this installer myself, but one question, can i seperate the swap space from the pool? I have seen to many strange hangs on a server where i use swap on zfs! If I understand correctly, it does this by default. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] bsdinstall and zfsboot enhancements
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Nov 11, 2013, at 11:46 AM, Michael Dexter wrote: Hello all, I have been experimenting with various BSD and GNU/Linux boot media under bhyve and noticed that we may want to accommodate the LiveCD mode of the installer, which in turn requires the correct console. Currently, one is prompted for VT100 for installation but this does not appear to work/stick for LiveCD mode. Can anyone verify this? While I developed this patch... http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/usr.sbin%3A%3Absdinstall%3A%3Ascripts%3A%3Aconfig.patch?revision=1.10view=markup Reasons exist to search for a better solution, see here: http://lists.freebsd.org/pipermail/freebsd-current/2013-November/046148.html (and messages that follow it) Is modifying init(8) still the way to go? What modification do we want to make? I'll do the work if we can come to consensus. Or should we touch up the patch in some way to address the original concerns? - -- Devin -BEGIN PGP SIGNATURE- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJSgTuCAAoJEKrMn5R9npq5M90H/RrIhOqhCAPj9lyG9ElM6NlC G2EDyajzIqntzOQY6O9AZXl+CrKndMkOAthOAYBfWTXCIKV0J5tlV167Gmfm25tZ GQpJkneOiHwMuhj7Fe0WWAKaXWwLj8ASTI6bj2bFz3Xls/5p0Q6YA0TMBraUp9K3 NVJGbscp4S+BG6wYQjoAEzwek2Kz5yc4jHSJhfraLTEWmUwHczPV7Levp37rLeiW n/OQT1p3w5OJVgnRFy9ptQCST/MaYU9KJWP360bXnqGYg6NrrrlFdIejnIIsSkxX u21Mu+sF76S3rg0D+L+ByZgbP957azcJ9NeDTooi3Gb+/TqgWL9j4Qyp7DkR2no= =T2FX -END PGP SIGNATURE- _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] bsdinstall and zfsboot enhancements
(disabling default gpg-signing until they fix a bug with the quoting) On Nov 11, 2013, at 11:46 AM, Michael Dexter wrote: Hello all, I have been experimenting with various BSD and GNU/Linux boot media under bhyve and noticed that we may want to accommodate the LiveCD mode of the installer, which in turn requires the correct console. Currently, one is prompted for VT100 for installation but this does not appear to work/stick for LiveCD mode. Can anyone verify this? Sorry, I mistook your issue in the previous e-mail to be on-going with the thread it was inline with. LiveCD changes things a bit. (thinks) I would expect that a prompt could do: 1. modify /etc/ttys on the boot media 2. run init q But there's a couple of assumptions... 3. Can we even write to /etc/ttys? NB: We can write to /tmp because it's an md0 swap device So do we have to get fancy with slipping a unionfs layer backed by another md swap device above the root? That would make every file on the boot media writable (writes would go to the swap-backed md device and if you execute rm -fW file you can get back files that have been previously unlinked -- unlinks are stored as whiteouts in the swap backed md device). For all intents and purposes, the read-only file- system becomes writable and we could then munge /etc/ttys to enable serial only when a menu item is chosen. If I'm off-base, let me know... sounds like a lot of trouble. The alternative being that you enable serial by default but then I have to tell field engineers to unplug barcode readers before they do an install??? (that's a question, it may be entirely safe, but I've never tried, seems unsafe) Question is, how would you disable it? Goes back to writable filesystem. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] bsdinstall and zfsboot enhancements
On Nov 11, 2013, at 12:35 PM, Nathan Whitehorn wrote: On 11/11/13 14:30, Teske, Devin wrote: (disabling default gpg-signing until they fix a bug with the quoting) On Nov 11, 2013, at 11:46 AM, Michael Dexter wrote: Hello all, I have been experimenting with various BSD and GNU/Linux boot media under bhyve and noticed that we may want to accommodate the LiveCD mode of the installer, which in turn requires the correct console. Currently, one is prompted for VT100 for installation but this does not appear to work/stick for LiveCD mode. Can anyone verify this? Sorry, I mistook your issue in the previous e-mail to be on-going with the thread it was inline with. LiveCD changes things a bit. (thinks) I would expect that a prompt could do: 1. modify /etc/ttys on the boot media 2. run init q But there's a couple of assumptions... 3. Can we even write to /etc/ttys? NB: We can write to /tmp because it's an md0 swap device So do we have to get fancy with slipping a unionfs layer backed by another md swap device above the root? That would make every file on the boot media writable (writes would go to the swap-backed md device and if you execute rm -fW file you can get back files that have been previously unlinked -- unlinks are stored as whiteouts in the swap backed md device). For all intents and purposes, the read-only file- system becomes writable and we could then munge /etc/ttys to enable serial only when a menu item is chosen. If I'm off-base, let me know... sounds like a lot of trouble. The alternative being that you enable serial by default but then I have to tell field engineers to unplug barcode readers before they do an install??? (that's a question, it may be entirely safe, but I've never tried, seems unsafe) Question is, how would you disable it? Goes back to writable filesystem. This is why I don't think we want to modify /etc/ttys. Instead we want to have init do the right thing (follow the boot console) with a single unchanged /etc/ttys. Let me see if I've got it right... Already-knowns: + The boot variables are influenced by loader.conf which is read-only media. + But Michael is loading the media from bhyve. Question: Does bhyve set kern.console irrespective of loader.conf values? If so, then yeah... I'm all for supporting your stated Option-A in the other thread. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] bsdinstall and zfsboot enhancements
On Nov 11, 2013, at 12:30 PM, Nathan Whitehorn wrote: On 11/11/13 14:18, Teske, Devin wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Nov 11, 2013, at 11:46 AM, Michael Dexter wrote: Hello all, I have been experimenting with various BSD and GNU/Linux boot media under bhyve and noticed that we may want to accommodate the LiveCD mode of the installer, which in turn requires the correct console. Currently, one is prompted for VT100 for installation but this does not appear to work/stick for LiveCD mode. Can anyone verify this? While I developed this patch... https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/usr.sbin%253A%253Absdinstall%253A%253Ascripts%253A%253Aconfig.patch?revision%3D1.10%26view%3Dmarkupk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=CRY%2BFNOe6Q01s5oKeCFeGOhE0bW%2BvEmw6GTjyy5eHFo%3D%0As=3e5345ea6b13f84e719068bb993b66978f1971b648b430edfde9165677c279de Reasons exist to search for a better solution, see here: https://urldefense.proofpoint.com/v1/url?u=http://lists.freebsd.org/pipermail/freebsd-current/2013-November/046148.htmlk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=CRY%2BFNOe6Q01s5oKeCFeGOhE0bW%2BvEmw6GTjyy5eHFo%3D%0As=9ebff0354adb2634006e52cb4c0048287b7518bd7f4defe420ce5f34675ce5cd (and messages that follow it) Is modifying init(8) still the way to go? What modification do we want to make? I'll do the work if we can come to consensus. Or should we touch up the patch in some way to address the original concerns? I think modifying init is the way to go -- it keeps the install system from interfering with the installed one, as well as fixing this kind of issue with moved hard drives or PXE booting or what have you. If we can provide a guarantee that any system that displays text has a working console (unless explicitly configured not to), useability is improved. I would propose one of the following (and volunteer to write the code): Option A 1. init checks if there is an entry in /etc/ttys for the terminal[s] corresponding to the value[s] in kern.console Is kern.console set by bhyve while NULL or unset when booting from a physical PC (bare metal)? 2. If an entry for each console terminal exists in /etc/ttys, enable it 3. If not, invent one with a terminal type of ansi The one issue here is that someone may want to force a particular entry to off and still have it be the kernel console. This is tricky. We could invent a new status field that is not on or off (auto, maybe, or ifconsole?). Which brings us to: Trying to think of an edge-case where they would want to force it to off. If the kern.console setting is only expected to be set via ... help me out here... + bhyve ? + pxe ? + what else ? Then maybe the secret sauce shouldn't be in /etc/ttys, but in the value that is passed in via kern.console (that is, ... of the above ruminated technologies which allow you to customize the value?) Option B --- Very similar to Option A, except only provide an automatic console using (2) and (3) if the console terminal is marked on. This would increase the magic attached to console in /etc/ttys, but fix the problem with (A). It's possible another approach would work as well. Does anyone have thoughts on this? If I'm not wrong, there's a third Option C that skirts the issue by bolstering the ability to change things for different needs. That prior change that I had made to bsdinstall/scripts/config (iirc) was a big agressive because it tried to automatically figure things out. If we simply made it a choice (a scriptable one), then that would solve the problem for the installation. But for the LiveCD issue (Michael wants to bring up the installer's LiveCD via serial), then perhaps we could solve it by using unionfs in the installer to make the install environment writable (and again, change on-the-fly for different needs). Just a thought. Tell me if it's not an option ;D I won't mind. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] bsdinstall and zfsboot enhancements
On Nov 11, 2013, at 12:44 PM, Peter Grehan wrote: Hi Devin, Question: Does bhyve set kern.console irrespective of loader.conf values? The kernel sets it based on what it determines the console to be. Bhyve influences that by requesting a serial console. This is no different than booting on a headless machine with a serial console. Well, for a headless meachine, I would set console=comconsole,vidconsole in loader.conf(5), then our Forth code slurps it in via loader.4th + support.4th routines... When boot is executed, I know I can see kenv console, but hadn't realized that there were/are a host of others that are slurped into the kernel for later (very purposeful) fetching. So when you say that bhyve requests a serial console... I assume now it's setting variables... but via raw Forth? C code? loader.conf(5)? I've seen my menu come up under bhyve, and I noticed that it only has a 5-second count- down instead of the usual 9 -- but I'm curious how you're exporting the variables. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] bsdinstall and zfsboot enhancements
On Nov 11, 2013, at 12:54 PM, Nathan Whitehorn wrote: On 11/11/13 14:30, Nathan Whitehorn wrote: On 11/11/13 14:18, Teske, Devin wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Nov 11, 2013, at 11:46 AM, Michael Dexter wrote: Hello all, I have been experimenting with various BSD and GNU/Linux boot media under bhyve and noticed that we may want to accommodate the LiveCD mode of the installer, which in turn requires the correct console. Currently, one is prompted for VT100 for installation but this does not appear to work/stick for LiveCD mode. Can anyone verify this? While I developed this patch... https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/usr.sbin%253A%253Absdinstall%253A%253Ascripts%253A%253Aconfig.patch?revision%3D1.10%26view%3Dmarkupk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=aZg%2BxwqLTX6Zcpf3Rc44iPtAQhFrpqoS4FC5B8ykxJQ%3D%0As=6d54d337ea5472f5713ddbe7788d3248c45feefd4b291eab0a976c39e9d40428 Reasons exist to search for a better solution, see here: https://urldefense.proofpoint.com/v1/url?u=http://lists.freebsd.org/pipermail/freebsd-current/2013-November/046148.htmlk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=aZg%2BxwqLTX6Zcpf3Rc44iPtAQhFrpqoS4FC5B8ykxJQ%3D%0As=5f8128901747346f937ffc4478eadb4bc39059504258dfb9b36f0fb9e7a61b79 (and messages that follow it) Is modifying init(8) still the way to go? What modification do we want to make? I'll do the work if we can come to consensus. Or should we touch up the patch in some way to address the original concerns? I think modifying init is the way to go -- it keeps the install system from interfering with the installed one, as well as fixing this kind of issue with moved hard drives or PXE booting or what have you. If we can provide a guarantee that any system that displays text has a working console (unless explicitly configured not to), useability is improved. I would propose one of the following (and volunteer to write the code): Option A 1. init checks if there is an entry in /etc/ttys for the terminal[s] corresponding to the value[s] in kern.console 2. If an entry for each console terminal exists in /etc/ttys, enable it 3. If not, invent one with a terminal type of ansi The one issue here is that someone may want to force a particular entry to off and still have it be the kernel console. This is tricky. We could invent a new status field that is not on or off (auto, maybe, or ifconsole?). Which brings us to: One easy way to accomplish this is just to only implement (1) and (3), then comment out the ttyu0 entry in /etc/ttys on x86 instead of marking it off. Then the behavior is just that a tty marked off stays off, one marked on stays on, and one not present spawns login with a terminal type corresponding to console (by default unknown) if it happens to be the console. I will implement this over the next few days and then send patches unless anyone has an objection. I love it. (smiles) Excellent idea and that's the winner I think. +1 -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] bsdinstall and zfsboot enhancements
On Nov 11, 2013, at 12:56 PM, Nathan Whitehorn wrote: On 11/11/13 14:52, Teske, Devin wrote: On Nov 11, 2013, at 12:44 PM, Peter Grehan wrote: Hi Devin, Question: Does bhyve set kern.console irrespective of loader.conf values? The kernel sets it based on what it determines the console to be. Bhyve influences that by requesting a serial console. This is no different than booting on a headless machine with a serial console. Well, for a headless meachine, I would set console=comconsole,vidconsole in loader.conf(5), then our Forth code slurps it in via loader.4th + support.4th routines... When boot is executed, I know I can see kenv console, but hadn't realized that there were/are a host of others that are slurped into the kernel for later (very purposeful) fetching. So when you say that bhyve requests a serial console... I assume now it's setting variables... but via raw Forth? C code? loader.conf(5)? I've seen my menu come up under bhyve, and I noticed that it only has a 5-second count- down instead of the usual 9 -- but I'm curious how you're exporting the variables. I think you've misunderstood. kern.console isn't set by loader. It reflects the state of the kernel, which decides what to do autonomously based on a number of driver and platform-dependent things including, but not limited to, kenv (loader variables, for instance). I was thinking it, but didn't express it properly... yes... kern.console is influenced by loader variables. slurped is definitely not the technical explanation I should be striving for ;D (that is, if I'm trying to be clear). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] bsdinstall and zfsboot enhancements
On Nov 11, 2013, at 12:59 PM, Peter Grehan wrote: Hi Devin, When boot is executed, I know I can see kenv console, but hadn't realized that there were/are a host of others that are slurped into the kernel for later (very purposeful) fetching. So when you say that bhyve requests a serial console... I assume now it's setting variables... but via raw Forth? C code? loader.conf(5)? I've seen my menu come up under bhyve, and I noticed that it only has a 5-second count- down instead of the usual 9 -- but I'm curious how you're exporting the variables. bhyve uses the loader facility to set env variables directly from loader code. This is no different than other arch ports of the loader that set env variables. The forced setting of boot_serial is just a convenience for users since that is the only supported console type. Thank you much for the explanation. I see it from end-to-end now. It doesn't touch the timeout code - should be whatever FreeBSD loader forth scripts tell it to do. Ah, must have been something Michael did. I noticed this whilst getting demos from him on his laptop. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Default MBR boot manager
Topic: Lenovo Laptops and bsdinstall zfsboot with MBR layout... Should we do the quick patch to change the default from /boot/boot0 to /boot/mbr: Index: zfsboot === --- zfsboot (revision 258016) +++ zfsboot (working copy) @@ -764,7 +764,7 @@ zfs_create_diskpart() # f_eval_catch $funcname gpart $GPART_CREATE mbr \$disk || return $FAILURE - f_eval_catch $funcname gpart $GPART_BOOTCODE /boot/boot0 \ + f_eval_catch $funcname gpart $GPART_BOOTCODE /boot/mbr \ \$disk || return $FAILURE # That would fix things for Lenovo laptops for the next release until I finish up the bootcode selection menu. I'd like to take my time in making sure Allan and I design a worthy bootcode selection menu. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Default MBR boot manager
On Nov 11, 2013, at 1:32 PM, Nathan Whitehorn wrote: On 11/11/13 15:19, Teske, Devin wrote: Topic: Lenovo Laptops and bsdinstall zfsboot with MBR layout... Should we do the quick patch to change the default from /boot/boot0 to /boot/mbr: Index: zfsboot === --- zfsboot (revision 258016) +++ zfsboot (working copy) @@ -764,7 +764,7 @@ zfs_create_diskpart() # f_eval_catch $funcname gpart $GPART_CREATE mbr \$disk || return $FAILURE - f_eval_catch $funcname gpart $GPART_BOOTCODE /boot/boot0 \ + f_eval_catch $funcname gpart $GPART_BOOTCODE /boot/mbr \ \$disk || return $FAILURE # That would fix things for Lenovo laptops for the next release until I finish up the bootcode selection menu. I'd like to take my time in making sure Allan and I design a worthy bootcode selection menu. This patch looks good (I don't remember why it was boot0 in the first place). I think gpart automatically installs something like /boot/mbr by default, so I'd be interested to know if making the diff purely negative still works. On another note, I think we should move away from a selector. Right now, we have three kinds of boot code: 1. ZFS boot code 2. UFS boot code 3. boot0 Unifying 1 and 2 would help a lot -- I don't know of any reason we need both except for tradition. #3 is probably best done as a post-install config step (Install FreeBSD boot manager or something), which also means it works for UFS systems. Well, I'm sensitive to the fact that sysinstall offered none and even explained why in an F1 dialog that brought up drives.hlp to explain that you might want to keep whatever (alternate) boot manager you may be using already. In a proposed selector, the full breadth of options that I was envisioning was: GPT + gptboot GPT + none (use your existing boot manager... syslinux?) MBR + mbr MBR + boot0 MBR + none (again, BYOBM) Hadn't got around to zfsboot yet. Where would that go? at the top? GPT + zfsboot ? (and of course, this is x86 specific... I was gleaning from sysinstall that for systems like pc98, they call it an IPL and there's only two options... a standard IPL or bring your own boot manager, aka none). I imagine that there would be architectures that are like the ol' pc98, wherein they don't have all these options (is, for example? sparc64 GPT only?) -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Default MBR boot manager
On Nov 11, 2013, at 1:43 PM, Nathan Whitehorn wrote: On 11/11/13 15:39, Teske, Devin wrote: On Nov 11, 2013, at 1:32 PM, Nathan Whitehorn wrote: On 11/11/13 15:19, Teske, Devin wrote: Topic: Lenovo Laptops and bsdinstall zfsboot with MBR layout... Should we do the quick patch to change the default from /boot/boot0 to /boot/mbr: Index: zfsboot === --- zfsboot (revision 258016) +++ zfsboot (working copy) @@ -764,7 +764,7 @@ zfs_create_diskpart() # f_eval_catch $funcname gpart $GPART_CREATE mbr \$disk || return $FAILURE - f_eval_catch $funcname gpart $GPART_BOOTCODE /boot/boot0 \ + f_eval_catch $funcname gpart $GPART_BOOTCODE /boot/mbr \ \$disk || return $FAILURE # That would fix things for Lenovo laptops for the next release until I finish up the bootcode selection menu. I'd like to take my time in making sure Allan and I design a worthy bootcode selection menu. This patch looks good (I don't remember why it was boot0 in the first place). I think gpart automatically installs something like /boot/mbr by default, so I'd be interested to know if making the diff purely negative still works. On another note, I think we should move away from a selector. Right now, we have three kinds of boot code: 1. ZFS boot code 2. UFS boot code 3. boot0 Unifying 1 and 2 would help a lot -- I don't know of any reason we need both except for tradition. #3 is probably best done as a post-install config step (Install FreeBSD boot manager or something), which also means it works for UFS systems. Well, I'm sensitive to the fact that sysinstall offered none and even explained why in an F1 dialog that brought up drives.hlp to explain that you might want to keep whatever (alternate) boot manager you may be using already. In a proposed selector, the full breadth of options that I was envisioning was: GPT + gptboot GPT + none (use your existing boot manager... syslinux?) MBR + mbr MBR + boot0 MBR + none (again, BYOBM) Hadn't got around to zfsboot yet. Where would that go? at the top? GPT + zfsboot ? (and of course, this is x86 specific... I was gleaning from sysinstall that for systems like pc98, they call it an IPL and there's only two options... a standard IPL or bring your own boot manager, aka none). I imagine that there would be architectures that are like the ol' pc98, wherein they don't have all these options (is, for example? sparc64 GPT only?) This would be super-unportable. SPARC uses VTOC8, for example, and doesn't support GPT at all. PC98 has the differences you mentioned. PowerPC uses MBR sometimes, sometimes APM, sometimes GPT. You never have a choice. No platforms except x86 have any analog to boot0. Etc, etc. This is why I'd like to pull ZFS into partedit, which already knows how to set up everything and does the right thing everywhere. For the only system (x86) where there is a real choice (do you want to replace whatever you have already with boot0?), it makes sense to do this as a post-install config. Two migration paths before us, and I do rather like the idea of benefiting from all your work there. My biggest concern is how to maximize functionality in the migration of the ZFS stuff to partedit. You know the code better there better than I, ... have you given much thought to how you might integrate what we've done? It's sad that we would be giving up i18n, X11, and discrete scripting (surely there are more parts to ZFS than what partedit supports now -- e.g., datasets, etc.). Naturally, the scripting can be solved. i18n is a bit harder to solve as it's a start from the bottom venture. And I fear X11 is a lost cause in its current state for partedit. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Default MBR boot manager
On Nov 11, 2013, at 2:57 PM, Nathan Whitehorn wrote: On 11/11/13 15:51, Teske, Devin wrote: On Nov 11, 2013, at 1:43 PM, Nathan Whitehorn wrote: On 11/11/13 15:39, Teske, Devin wrote: On Nov 11, 2013, at 1:32 PM, Nathan Whitehorn wrote: On 11/11/13 15:19, Teske, Devin wrote: Topic: Lenovo Laptops and bsdinstall zfsboot with MBR layout... Should we do the quick patch to change the default from /boot/boot0 to /boot/mbr: Index: zfsboot === --- zfsboot (revision 258016) +++ zfsboot (working copy) @@ -764,7 +764,7 @@ zfs_create_diskpart() # f_eval_catch $funcname gpart $GPART_CREATE mbr \$disk || return $FAILURE - f_eval_catch $funcname gpart $GPART_BOOTCODE /boot/boot0 \ + f_eval_catch $funcname gpart $GPART_BOOTCODE /boot/mbr \ \$disk || return $FAILURE # That would fix things for Lenovo laptops for the next release until I finish up the bootcode selection menu. I'd like to take my time in making sure Allan and I design a worthy bootcode selection menu. This patch looks good (I don't remember why it was boot0 in the first place). I think gpart automatically installs something like /boot/mbr by default, so I'd be interested to know if making the diff purely negative still works. On another note, I think we should move away from a selector. Right now, we have three kinds of boot code: 1. ZFS boot code 2. UFS boot code 3. boot0 Unifying 1 and 2 would help a lot -- I don't know of any reason we need both except for tradition. #3 is probably best done as a post-install config step (Install FreeBSD boot manager or something), which also means it works for UFS systems. Well, I'm sensitive to the fact that sysinstall offered none and even explained why in an F1 dialog that brought up drives.hlp to explain that you might want to keep whatever (alternate) boot manager you may be using already. In a proposed selector, the full breadth of options that I was envisioning was: GPT + gptboot GPT + none (use your existing boot manager... syslinux?) MBR + mbr MBR + boot0 MBR + none (again, BYOBM) Hadn't got around to zfsboot yet. Where would that go? at the top? GPT + zfsboot ? (and of course, this is x86 specific... I was gleaning from sysinstall that for systems like pc98, they call it an IPL and there's only two options... a standard IPL or bring your own boot manager, aka none). I imagine that there would be architectures that are like the ol' pc98, wherein they don't have all these options (is, for example? sparc64 GPT only?) This would be super-unportable. SPARC uses VTOC8, for example, and doesn't support GPT at all. PC98 has the differences you mentioned. PowerPC uses MBR sometimes, sometimes APM, sometimes GPT. You never have a choice. No platforms except x86 have any analog to boot0. Etc, etc. This is why I'd like to pull ZFS into partedit, which already knows how to set up everything and does the right thing everywhere. For the only system (x86) where there is a real choice (do you want to replace whatever you have already with boot0?), it makes sense to do this as a post-install config. Two migration paths before us, and I do rather like the idea of benefiting from all your work there. My biggest concern is how to maximize functionality in the migration of the ZFS stuff to partedit. You know the code better there better than I, ... have you given much thought to how you might integrate what we've done? Some, though not as much as it deserves. The general idea would be to treat ZFS as a kind of partitioning. I don't think we can realistically expose all ZFS features through the installer. It's sad that we would be giving up i18n, X11, and discrete scripting (surely there are more parts to ZFS than what partedit supports now -- e.g., datasets, etc.). Naturally, the scripting can be solved. i18n is a bit harder to solve as it's a start from the bottom venture. And I fear X11 is a lost cause in its current state for partedit. This is a pity. I'm not sure what to do there. Would be nice to have something libdialog-alike for X, but that leaves all these other issues too. No matter which way I look at it, that sounds like a lot of work. One way is that it would just fork-exec Xdialog and return the results. Another way would be interfacing with an actual UI library like Tk, Gtk, Gtk2 or your favorite. But then, do we risk the app not looking like the rest of the functions if they use Xdialog(1)? But I'm thinking... We'd have to use dlopen(3) for conditional support anyway, because whatever dependency-chain we choose for X11, we can't link the binary (which will go into base) against the port dependency. Has to be runtime value-add. One option would be to factor out
Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu
On Nov 6, 2013, at 1:18 AM, Lars Engels wrote: Am 05.11.2013 18:06, schrieb Kurt Lidl: Well, I'd probably be in support of this change - it sure beats having to interrupt the normal boot sequence and typing: unload load /boot/kernel.old/kernel load /boot/kernel.old/opensolaris.ko load /boot/kernel.old/zfs.ko boot To load an older kernel I always just type boot kernel.old Doesn't that unload the currently loaded kernel automatically? Actually... it does. Thanks for pointing that out (forgot about that). I think we still want the loader_delay feature added by the last patch I shared on-list. Yeah? I think it's still a good (optional) value-add. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
[Review] bsdconfig pkgng integration
Subject says it all. I need a review on the attached patch. Come on, you know you want to... -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. Index: http.subr === --- http.subr (revision 256550) +++ http.subr (working copy) @@ -79,6 +79,7 @@ f_dialog_menu_media_http() local menu_list= '$msg_main_site' 'ftp.freebsd.org' 'URL' '$msg_specify_some_other_http_site' + 'pkg $msg_main_site' 'pkg.freebsd.org' # END-QUOTE local hline=$msg_select_a_site_thats_close signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [Review] bsdconfig pkgng integration
On Nov 6, 2013, at 3:02 PM, Allan Jude wrote: On 2013-11-06 13:32, Teske, Devin wrote: Subject says it all. I need a review on the attached patch. Come on, you know you want to... I think you missed something, you attached a 1 line diff Oh I see... too much to review. Hey, don't all review it at once now... Seriously... come on... I'm asking for some feedback on the most important part of the patchset... The user-facing element. The least everyone can do is chime in with their favorite mirrors. So... please... Tell me if the mirror list is incomplete (I *know* there are more mirrors than that, and I *know* we want pkg.eu and pkg.us-* ... but I don't actually know what people want to see ... so please... feedback!) -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [CFT] bsdinstall and zfsboot enhancements
On Nov 5, 2013, at 7:54 PM, Craig Rodrigues wrote: On Sat, Nov 2, 2013 at 10:59 AM, Teske, Devin devin.te...@fisglobal.com wrote: + Update bsdinstall's config script to adjust ttyu* entries in /etc/ttys when it is determined that we are in-fact doing an install over serial (e.g. bhyve). I think this is useful. In earlier versions of BHyve, the default was to use console over a PCI serial device. With this commit from Neel Natu: http://lists.freebsd.org/pipermail/svn-src-all/2013-October/076027.html it is now possible to specify a different device for the console, such as COM1. Unfortunately, that's getting the axe. I agree with Nathan... the installer shouldn't make this assumption. I think init should be fixed instead. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [Review] bsdconfig pkgng integration
On Nov 6, 2013, at 3:19 PM, Allan Jude wrote: On 2013-11-06 18:15, Teske, Devin wrote: On Nov 6, 2013, at 3:02 PM, Allan Jude wrote: On 2013-11-06 13:32, Teske, Devin wrote: Subject says it all. I need a review on the attached patch. Come on, you know you want to... I think you missed something, you attached a 1 line diff Oh I see... too much to review. Hey, don't all review it at once now... Seriously... come on... I'm asking for some feedback on the most important part of the patchset... The user-facing element. The least everyone can do is chime in with their favorite mirrors. So... please... Tell me if the mirror list is incomplete (I *know* there are more mirrors than that, and I *know* we want pkg.eu and pkg.us-* ... but I don't actually know what people want to see ... so please... feedback!) pkg.freebsd.org is a SVR record This matters not... see SVN r257755 that lists the mirrors, We do not ever want to display the contents of the SVR record in the list. There are multiple pkg*.freebsd.org SVR zones. and pkg automatically picks the correct one, pkg does no such thing. It picks the first one that works. The decisioning for which order and what content gets returned from the SVR query is on the back-end, not on the pkg-end; pkg just takes the first one that works. For example, host -t SVR _http._tcp.pkg.freebsd.org (for me at least) returns 3 entries. pkg goes after the first one. I'd be shocked if you told me that pkg was doing geo-mapping with the 3 names it returns and determining which was was closest. Rather, pkg just does a DNS query and goes through the list of hosts. so it is best to list only that 1 mirror. No. You're thinking that I'm talking about the contents of the SVR query. I'm not. I'm instead talking about... *actual* and *real* multiple names that you can query SVR data from. pkg.freebsd.org is only one There are multiples, and I believe that we should add them. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [Review] bsdconfig pkgng integration
On Nov 6, 2013, at 3:32 PM, Glen Barber wrote: On Wed, Nov 06, 2013 at 11:28:25PM +, Teske, Devin wrote: We do not ever want to display the contents of the SVR record in the list. There are multiple pkg*.freebsd.org SVR zones. You do not need to show any servers except pkg.FreeBSD.org. Bapt *just* got finished telling me in IRC that we should also list: pkg.eu pkg.us-* Which I assume to be the same make/model of pkg.f.o (does not return an A or but only SVR). If that is the case (that pkg.eu.f.o is just like pkg.f.o -- and it just happens to be a name that returns SVR names for servers local to Europe), then I don't see why we would want to omit this. I'm not the only one you have to convince... bapt was the one that brought up the existence of the other names. In fact, you can use that by default, and not show any servers at all. Do not add more complexity to this than there needs to be. By adding explicit listing of servers here, you are putting a dependency on clusteradm that is not necessary at all. I want to get some confusion out of the way... I do not want to add even one single name that resolves to an A or . I want *only* names that resolve to SVR. I've just heard from bapt that pkg.f.o is not the only such name, that there are other such names that similarly do not return A or . I don't see the increase in complexity or maintenance. I'm asking for locale-specific CDN names. The user ought to be able to choose the master CDN or a locale-specific CDN. This is not rocket science. -- Devin Glen _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [Review] bsdconfig pkgng integration
On Nov 6, 2013, at 3:44 PM, Glen Barber wrote: On Wed, Nov 06, 2013 at 11:37:43PM +, Teske, Devin wrote: You do not need to show any servers except pkg.FreeBSD.org. Bapt *just* got finished telling me in IRC that we should also list: pkg.eu pkg.us-* Why? Which I assume to be the same make/model of pkg.f.o (does not return an A or but only SVR). And if we need to change something with the infrastructure, we now have to tell people that this was done. You are putting hard-coded limitations on us. Don't. I disagree. You're making the argument for what if a CDN goes away. Well... people stop using that entry because it's broken and in the next release you remove it. What sounds like a maintenance issue will actually save you support events. And for whatever hours that this proposed maintenance actually cost you, you'll more than make up that in prevented support events. I'm talking about saving time (because we're all volunteers, right?) Let me be very explicit... With a single entry (pkg.FreeBSD.org), we do not have to worry about what is broken if (when) we need to change something. Let's say that the one entry doesn't do what you want (which requires trusting the CDN maintenance on the back-end)... You're telling me that it's more efficient to immediately defer every single user to the support forums the moment there's a single hiccup in the ONE and ONLY CDN name that we're providing? Ok... but let's look at one alternative... If we listed more than one CDN, and one breaks... the wise and tenacious user simply ... tries another? But if there are no other choices... then you'll be headed strait to the forums or mailing lists and generating what I call a support event. I'm sorry, but I don't agree that listing only one single entry is going to be beneficial to the end-user; only to the person that thinks having a single CDN to maintain is going to actually save them work (which may only be true if they are not the ones that have to manage support events). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [Review] bsdconfig pkgng integration
On Nov 6, 2013, at 3:53 PM, Teske, Devin wrote: On Nov 6, 2013, at 3:44 PM, Glen Barber wrote: On Wed, Nov 06, 2013 at 11:37:43PM +, Teske, Devin wrote: You do not need to show any servers except pkg.FreeBSD.org. Bapt *just* got finished telling me in IRC that we should also list: pkg.eu pkg.us-* Why? Which I assume to be the same make/model of pkg.f.o (does not return an A or but only SVR). And if we need to change something with the infrastructure, we now have to tell people that this was done. You are putting hard-coded limitations on us. Don't. I disagree. You're making the argument for what if a CDN goes away. Well... people stop using that entry because it's broken and in the next release you remove it. What sounds like a maintenance issue will actually save you support events. And for whatever hours that this proposed maintenance actually cost you, you'll more than make up that in prevented support events. I'm talking about saving time (because we're all volunteers, right?) Let me be very explicit... With a single entry (pkg.FreeBSD.org), we do not have to worry about what is broken if (when) we need to change something. Let's say that the one entry doesn't do what you want (which requires trusting the CDN maintenance on the back-end)... You're telling me that it's more efficient to immediately defer every single user to the support forums the moment there's a single hiccup in the ONE and ONLY CDN name that we're providing? Ok... but let's look at one alternative... If we listed more than one CDN, and one breaks... the wise and tenacious user simply ... tries another? But if there are no other choices... then you'll be headed strait to the forums or mailing lists and generating what I call a support event. I'm sorry, but I don't agree that listing only one single entry is going to be beneficial to the end-user; only to the person that thinks having a single CDN to maintain is going to actually save them work (which may only be true if they are not the ones that have to manage support events). I think there's a lot of misunderstanding going on because of the way that clusteradm has set up our CDN. It is true that right now, pkg.f.o returns SRV records for all known mirrors. It is also true that previously mentioned pkg.eu.f.o returns the one and only SRV record for the one mirror in Europe. However, I am planning for a day when our CDN is administered in a more traditional fashion. That is... I envisage pkg.f.o not returning all known mirrors (because there will be too many as the CDN grows), but instead pkg.f.o would return only the 3 closest mirrors. Meanwhile, pkg.eu.f.o would grow to return records for the top-3 closest servers in Europe. Just because the amount of machines that we have available for our CDN is small, doesn't mean that it will be small forever. Eventually (as is the plan), the list of servers should far exceed the number of SRV records you will want to return. And when we get to that point, I highly recommend that we allow the user to choose the CDN specific to their country (at the very least). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [Review] bsdconfig pkgng integration
On Nov 6, 2013, at 4:01 PM, Glen Barber wrote: On Wed, Nov 06, 2013 at 11:53:28PM +, Teske, Devin wrote: And if we need to change something with the infrastructure, we now have to tell people that this was done. You are putting hard-coded limitations on us. Don't. I disagree. You're making the argument for what if a CDN goes away. No, you are thinking about it backwards. If a node goes down (there is no CDN), we take it out of DNS. A single entry, removed from the pkg.FreeBSD.org SRV record. We do not have to redirect anything, we do not have to update various configuration files in various places in the tree. Please help me to understand why on Earth you think: [I am] putting hard-coded limitations on [you] The facts: 1. The DNS names I am talking about: 1.a. pkg.f.o 1.b. pkg.eu.f.o 2. Do not resolve to an A or record So what hard-coding are you talking about? You talk about how if a node goes down we take it out of DNS but that has absolutely nothing to do with me because I'm not putting A/-resolving names in the menu. You do realize don't you that pkg.eu.f.o is a locale-specific name that will eventually hold potentially many-more European server names, right? You do realize that the actual European server is *NOT* pkg.eu, right? You do realize that while the name pkg.f.o may wholly encompass all the mirrors, that this will not always be true, right? I have no idea what you're talking about with the updating of config files. The only reason you would ever have to update a config file is if pkg.eu.f.o went away, but ... Now I will repeat... pkg.eu.f.o is NOT A REAL SERVER It is a name, that resolves to the SRV record of the European server. That being said... that name should not go away unless we no longer have even one single server in Europe. Oh, and by the way... The name pkg.f.o does *NOT* resolve to pkg.eu... pkg.eu is a sibling name that is disassociated -- it was actually *designed* to be used for this. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [Review] bsdconfig pkgng integration
On Nov 6, 2013, at 4:22 PM, Glen Barber wrote: On Thu, Nov 07, 2013 at 12:13:49AM +, Teske, Devin wrote: So what hard-coding are you talking about? You are trying to hard-code hostnames for a service in the FreeBSD src/ tree, when it is *absolutely* unnecessary. Not all hostnames are equal. As you know, pkg.freebsd.org is one that you would never deny. So just as pkg.freebsd.org is different from a normal hostname, pkg.eu.freebsd.org and pkg.us-east.freebsd.org and pkg.us-west.freebsd.org are similarly different (in the same way). I'm not trying to hard-code any ol' hostnames... I'm trying to hardcode the hostnames which can be administered to point to one or more grouped servers. You talk about how pkg.freebsd.org is unique because we can simply change where it points... I'm talking about doing the same thing for the other 3 names we have which were designed for the same thing. I think that you're thinking that: pkg.eu.freebsd.org pkg.us-east.freebsd.org pkg.us-west.freebsd.org Will somehow change? The only way they would change is in the same way that you would change pkg.freebsd.org -- I don't know how much clearer I can make this. You talk about how if a node goes down we take it out of DNS but that has absolutely nothing to do with me because I'm not putting A/-resolving names in the menu. I did not once say anything about A or records. You talked about a node going down. A node has a A or quad-A record. The name for that node is served via the SRV record. Read the above paragraph again carefully. I said A/-resolving names That is equivalent to node name and you were talking about what if a node goes down. And my answer to that was... I don't care if a node goes down, because I'm not using node names in the menu. You do realize don't you that pkg.eu.f.o is a locale-specific name that will eventually hold potentially many-more European server names, right? So? So, the name pkg.eu is never going to change. It, like pkg.freebsd.org will just have modified SRV records to track the European nodes. You do realize that the actual European server is *NOT* pkg.eu, right? *Sigh*... Your sign leads me to be concerned that there is a false assumption that the DNS name that serves the SRV records should be a one-to-one mapping. There's absolutely no reason whatsoever to dedicate an entire locale specific domain to one SRV record. That's silly. As we grow, this domain -- which is decidedly locale-specific by way of its actual name -- should be used for what it was intended... a locale abstraction of many-to-one. You do realize that while the name pkg.f.o may wholly encompass all the mirrors, that this will not always be true, right? No, you are wrong. I surely hope I am not. You're saying that: + We'll never have more than one server in Europe. + We'll never have more than one server in the West Coast of the USA. + We'll never have more than one server in the East Coast of the USA. That makes me very sad. Very sad indeed. I envisaged a growth that would have had dozens upon dozens of servers all throughout, and the locale-specific domains would then return geographically based SRV records (of which we administer on the back-end). I have no idea what you're talking about with the updating of config files. Clearly. If you hard-code anything other than pkg.FreeBSD.org in bsdconfig, now clusteradm has to become aware of it, and make sure that record *always* exists, no matter what the endpoint is. http://lists.freebsd.org/pipermail/freebsd-pkg/2013-October/000107.html According to that mail, clusteradm *should* already be away of the three I mentioned... pkg.eu, pkg.us-east, and pkg.us-west This nonsense happened with sysinstall, and anything else that used the FTP mirror list. And when a node disappears, for whatever reason, it is an absolute nightmare to sort out. I have to continue to beat the drum... You had to do that in sysinstall, because the names that were used there were A/quad-A resolving names (node names). Just as pkg.freebsd.org is not one of those names... pkg.eu and pkg.us-* are not like the names used in sysinstall. So please... tell me again, how or why you would *ever* have to modify pkg.eu or pkg.us-* ??? For the last time, you do not need to have *any* host entries other than 'pkg.FreeBSD.org'. Period. Until you acknowledge that pkg.eu and pkg.us-* are of the same pedigree as pkg.f.o, I assume that you are still confusing this list of names with a traditional list of names like sysinstall had. That being said... that name should not go away unless we no longer have even one single server in Europe. This has nothing to do with bsdconfig. Try to see my larger point. Oh, I agree... I'm looking forward to the big picture when pkg.eu returns more than one SRV record. I'm looking forward to the day that pkg.us-east and pkg.us-west return more than one
Re: [Review] bsdconfig pkgng integration
On Nov 6, 2013, at 5:02 PM, Glen Barber wrote: On Thu, Nov 07, 2013 at 12:48:02AM +, Teske, Devin wrote: I think that you're thinking that: pkg.eu.freebsd.org pkg.us-east.freebsd.org pkg.us-west.freebsd.org Will somehow change? I think you're thinking they *won't* change. Don't assume. Well, what *kind* of change? Changes to what they serve should be no problem... but what I was assuming that wouldn't change was their actual names. (e.g. pkg.eu stays pkg.eu). Couldn't care about any other change, because that would percolate at runtime off the DNS servers. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [Review] bsdconfig pkgng integration
On Nov 6, 2013, at 5:13 PM, Teske, Devin wrote:
On Nov 6, 2013, at 5:02 PM, Glen Barber wrote:
On Thu, Nov 07, 2013 at 12:48:02AM +, Teske, Devin wrote:
I think that you're thinking that:
pkg.eu.freebsd.org
pkg.us-east.freebsd.org
pkg.us-west.freebsd.org
Will somehow change?
I think you're thinking they *won't* change. Don't assume.
Well, what *kind* of change?
Changes to what they serve should be no problem...
but what I was assuming that wouldn't change was
their actual names. (e.g. pkg.eu stays pkg.eu).
Couldn't care about any other change, because that
would percolate at runtime off the DNS servers.
We concluded that despite the fact that pkg.* are officially
supported names, a POLA issue has been brought to the
top of the discussion.
That is to say, that the concern is that someone may choose
pkg.eu or pkg.us-* and then become astonished if/when
that name mirror is down (precisely because we do not yet
have geodns and because the pkg.{eu,us-*} names are
currently one-to-one mappings).
So in that spirit, we move forward with only the pkg.f.o entry.
--
Devin
_
The information contained in this message is proprietary and/or confidential.
If you are not the intended recipient, please: (i) delete the message and all
copies; (ii) do not disclose, distribute or use the message in any manner; and
(iii) notify the sender immediately. In addition, please be aware that any
message addressed to our domain is subject to archiving and review by persons
other than the intended recipient. Thank you.
signature.asc
Description: Message signed with OpenPGP using GPGMail
Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu
On Nov 5, 2013, at 9:06 AM, Kurt Lidl wrote: You can try enabling the beastie menu on sparc64 by editing /boot/loader.rc: === Change #1 in /boot/loader.rc to enable beastie menu === Find: \ Reads and processes loader.conf variables \ NOTE: Change to `initialize' if you enable the below boot menu start Change start to initialize as shown below: \ Reads and processes loader.conf variables \ NOTE: Change to `initialize' if you enable the below boot menu initialize === Change #2 in [same file] to enable beastie menu === Find: \ Uncomment to enable boot menu \ include /boot/beastie.4th \ beastie-start Uncomment beastie-start as shown below: \ Uncomment to enable boot menu \ include /boot/beastie.4th beastie-start == If you find that making those two trivial changes, that you are able to load the menu... then maybe it's time for us to start thinking about enabling the beastie menu by-default for the sparc64 architecture. Seems to work just fine. I tested by booting, toggling through the different kernel choices (/boot/kernel/kernel /boot/kernel.old/kernel) and both worked correctly. (Although I uncommented the include /boot/beastie.4th line too.) Oops, heh, good eye ;D and sorry if that caused any pain for you. Does anybody else have any thoughts on enabling it for sparc64? Well, I'd probably be in support of this change - it sure beats having to interrupt the normal boot sequence and typing: unload load /boot/kernel.old/kernel load /boot/kernel.old/opensolaris.ko load /boot/kernel.old/zfs.ko boot When I need to get back to the prior version of the kernel. We'll have to draw up a clean patch to draw this in for sparc64. For i386 and amd64, they have a dedicated separate loader.rc stashed in sys/boot/i386/loader/ Not sure of the cleanest way to do this for sparc64 (dup the loader.rc and make sparc64 drop it's own? or perhaps make sparc64 drop the i386 loader.rc? I think amd64 does the latter). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu
On Nov 5, 2013, at 9:28 AM, Nathan Whitehorn wrote: On 11/05/13 11:06, Kurt Lidl wrote: You can try enabling the beastie menu on sparc64 by editing /boot/loader.rc: === Change #1 in /boot/loader.rc to enable beastie menu === Find: \ Reads and processes loader.conf variables \ NOTE: Change to `initialize' if you enable the below boot menu start Change start to initialize as shown below: \ Reads and processes loader.conf variables \ NOTE: Change to `initialize' if you enable the below boot menu initialize === Change #2 in [same file] to enable beastie menu === Find: \ Uncomment to enable boot menu \ include /boot/beastie.4th \ beastie-start Uncomment beastie-start as shown below: \ Uncomment to enable boot menu \ include /boot/beastie.4th beastie-start == If you find that making those two trivial changes, that you are able to load the menu... then maybe it's time for us to start thinking about enabling the beastie menu by-default for the sparc64 architecture. Seems to work just fine. I tested by booting, toggling through the different kernel choices (/boot/kernel/kernel /boot/kernel.old/kernel) and both worked correctly. (Although I uncommented the include /boot/beastie.4th line too.) Does anybody else have any thoughts on enabling it for sparc64? Well, I'd probably be in support of this change - it sure beats having to interrupt the normal boot sequence and typing: unload load /boot/kernel.old/kernel load /boot/kernel.old/opensolaris.ko load /boot/kernel.old/zfs.ko boot When I need to get back to the prior version of the kernel. Is there a way to make this work even without the beastie menu? A way to interrupt the boot before kernel load (even holding down a key) would be really valuable, even on systems that do not support fancy terminals with colors and such. Nathan, Can you drop into your /boot/loader.conf: loader_delay=3 And reboot? I'm trying to think how we could use that to our advantage. I'm interested in creative applications thereof. For more skinny on what that does... man delay.4th, it spills the beans on a delayed command execution module that I added a few years ago. For example... Here's my idea of making things easier on the user that wants to load a different kernel, but *without* using a menu... 1. We have loader_delay default to 3 2. Dots are displayed for 3 seconds before we do anything (like load a kernel) 3. User presses ENTER during those 3 seconds, and the delay is truncated 4. User presses Ctrl-C during those 3 seconds (or ESC) and they get a prompt (before the kernel has loaded). Then here's what I would do (as to not have to load separate modules)... set kernel=kernel.old boot + The forth code figures out that the kernel is in /boot + If loader.conf had opensolaris_load=YES and + zfs_load=YES then the forth code also loads those from kernel.old But let's say loader.conf was bare, you could do: set kernel=kernel.old set opensolaris_load=YES set zfs_load=YES boot NB: The reason this works is because you would not call start in your loader.rc (which actually loads things), but instead initialize (which only loads loader.conf et. al.) followed by dc_execute to fire off autoboot in 3 seconds (with the aforementioned allowed interrupt ability). Is that what you were thinking? Will that work? Bonus: It would be extremely trivial to implement. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu
On Nov 5, 2013, at 9:53 AM, Teske, Devin wrote: On Nov 5, 2013, at 9:28 AM, Nathan Whitehorn wrote: On 11/05/13 11:06, Kurt Lidl wrote: You can try enabling the beastie menu on sparc64 by editing /boot/loader.rc: === Change #1 in /boot/loader.rc to enable beastie menu === Find: \ Reads and processes loader.conf variables \ NOTE: Change to `initialize' if you enable the below boot menu start Change start to initialize as shown below: \ Reads and processes loader.conf variables \ NOTE: Change to `initialize' if you enable the below boot menu initialize === Change #2 in [same file] to enable beastie menu === Find: \ Uncomment to enable boot menu \ include /boot/beastie.4th \ beastie-start Uncomment beastie-start as shown below: \ Uncomment to enable boot menu \ include /boot/beastie.4th beastie-start == If you find that making those two trivial changes, that you are able to load the menu... then maybe it's time for us to start thinking about enabling the beastie menu by-default for the sparc64 architecture. Seems to work just fine. I tested by booting, toggling through the different kernel choices (/boot/kernel/kernel /boot/kernel.old/kernel) and both worked correctly. (Although I uncommented the include /boot/beastie.4th line too.) Does anybody else have any thoughts on enabling it for sparc64? Well, I'd probably be in support of this change - it sure beats having to interrupt the normal boot sequence and typing: unload load /boot/kernel.old/kernel load /boot/kernel.old/opensolaris.ko load /boot/kernel.old/zfs.ko boot When I need to get back to the prior version of the kernel. Is there a way to make this work even without the beastie menu? A way to interrupt the boot before kernel load (even holding down a key) would be really valuable, even on systems that do not support fancy terminals with colors and such. Nathan, Can you drop into your /boot/loader.conf: loader_delay=3 And reboot? I'm trying to think how we could use that to our advantage. I'm interested in creative applications thereof. For more skinny on what that does... man delay.4th, it spills the beans on a delayed command execution module that I added a few years ago. For example... Here's my idea of making things easier on the user that wants to load a different kernel, but *without* using a menu... 1. We have loader_delay default to 3 2. Dots are displayed for 3 seconds before we do anything (like load a kernel) 3. User presses ENTER during those 3 seconds, and the delay is truncated 4. User presses Ctrl-C during those 3 seconds (or ESC) and they get a prompt (before the kernel has loaded). Then here's what I would do (as to not have to load separate modules)... set kernel=kernel.old boot + The forth code figures out that the kernel is in /boot + If loader.conf had opensolaris_load=YES and + zfs_load=YES then the forth code also loads those from kernel.old But let's say loader.conf was bare, you could do: set kernel=kernel.old set opensolaris_load=YES set zfs_load=YES boot NB: The reason this works is because you would not call start in your loader.rc (which actually loads things), but instead initialize (which only loads loader.conf et. al.) followed by dc_execute Err, dc_execute is what I called it nearly a decade ago when I first wrote it at $work. I forgot that I renamed it to delay_execute (used to be dc_execute for delayed-command execute). to fire off autoboot in 3 seconds (with the aforementioned allowed interrupt ability). Is that what you were thinking? Will that work? Bonus: It would be extremely trivial to implement. I wipped up a patch... (and tested it about 12 different ways) See attached patch.txt. It should achieve what you're looking for. I was able to trim down Kurt's process from 6 steps: 1. Escape 10-second autoboot 2. unload 3. load /boot/kernel.old/kernel 4. load /boot/kernel.old/opensolaris.ko 5. load /boot/kernel.old/zfs.ko 6. boot To simply 2 (with loader_delay=3 in loader.conf(5)): 1. Escape new 3-second delay 2. boot kernel.old No beastie menu. Will this work? -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. Index: sys/boot/forth/beastie.4th === --- sys/boot/forth/beastie.4th (revision 257650) +++ sys/boot/forth/beastie.4th (working copy) @@ -28,8 +28,6 @@ marker task-beastie.4th -include /boot/delay.4th - only forth definitions also support
Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu
On Nov 4, 2013, at 2:43 PM, Kurt Lidl wrote: On Nov 2, 2013, at 1:45 AM, Sam Fourman Jr. wrote: On Sat, Nov 2, 2013 at 7:09 AM, Teske, Devin Devin.Teske at fisglobal.com wrote: Hi all, Here's a chance to test out the kernel selection menu enhancements to the boot loader menu before they go into HEAD. Discussion welcome, feedback desired. No recompile needed, just drop the new forth files onto a HEAD or stable/9 box and reboot. -- Cheers, Devin where are the forth files in question? D'Oh! Here they are: http://druidbsd.cvs.sf.net/viewvc/druidbsd/forth_zfs/ Supplied as patches to either stable/9 or head. -- Devin Hmmm. I saw no appreciable changes to behavior when I patched all the files in /boot with these versions. This was on a sparc64 host running 10-BETA3 (compile this morning). Excellent! Thank you for testing. NB: That's what *should* happen on sparc64 since that architecture doesn't actually enable the beastie menu (sad, I know... I wish that the beastie menu was active on all platforms by default). Notably, the kernel and modules still loaded before presenting me with the option to tell it which kernel to load: Executing last command: boot /pci@1f,0/pci@1/scsi@8/disk@0,0:a Boot device: /pci@1f,0/pci@1/scsi@8/disk@0,0:a File and args: FreeBSD/sparc64 ZFS boot block Boot path: /pci@1f,0/pci@1/scsi@8/disk@0,0:a Consoles: Open Firmware console \ FreeBSD/sparc64 ZFS enabled bootstrap loader, Revision 1.0 (r...@snap.freebsd.org, Sun Oct 27 07:20:42 UTC 2013) bootpath=zfs:sys/ROOT/default: Loading /boot/defaults/loader.conf /boot/kernel/kernel data=0x8d54c0+0x66180 syms=[0x8+0x954f0+0x8+0x8d7ef] /boot/kernel/zfs.ko text=0x223328 data=0xa4e0+0x17fc0 syms=[0x8+0x197b8+0x8+0x143f8] loading required module 'opensolaris' /boot/kernel/opensolaris.ko text=0x3130 data=0x2c8+0x2030 syms=[0x8+0xd98+0x8+0x929] /boot/kernel/geom_mirror.ko text=0x38430 data=0x5b0+0x20 syms=[0x8+0x16b0+0x8+0x119e] Hit [Enter] to boot immediately, or any other key for command prompt. Booting [/boot/kernel/kernel] in 9 seconds... You can try enabling the beastie menu on sparc64 by editing /boot/loader.rc: === Change #1 in /boot/loader.rc to enable beastie menu === Find: \ Reads and processes loader.conf variables \ NOTE: Change to `initialize' if you enable the below boot menu start Change start to initialize as shown below: \ Reads and processes loader.conf variables \ NOTE: Change to `initialize' if you enable the below boot menu initialize === Change #2 in [same file] to enable beastie menu === Find: \ Uncomment to enable boot menu \ include /boot/beastie.4th \ beastie-start Uncomment beastie-start as shown below: \ Uncomment to enable boot menu \ include /boot/beastie.4th beastie-start == If you find that making those two trivial changes, that you are able to load the menu... then maybe it's time for us to start thinking about enabling the beastie menu by-default for the sparc64 architecture. Does anybody else have any thoughts on enabling it for sparc64? -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [CFT] bsdinstall and zfsboot enhancements
On Nov 2, 2013, at 8:40 PM, Nathan Whitehorn wrote: On 11/02/13 12:59, Teske, Devin wrote: Hi all, Another Call For Testing... This one is for bsdinstall. Will look at the rest later... + Update bsdinstall's config script to adjust ttyu* entries in /etc/ttys when it is determined that we are in-fact doing an install over serial (e.g. bhyve). I think this is the wrong solution. Thank you for your feedback. Sincerely. The installer is run in a lot of circumstances, and tying it to the boot environment is a mistake. If we want serial consoles to default to on for x86, they should default to on (they do for most other architectures). The magic should never ever be in the installer. Well, I would say we *don't* want serial to be on by default... That would actually hurt me at $work where we actually use the serial ports for barcode readers. Setting the terminal type to vt100 unconditionally is also questionable. Using kbdcontrol also doesn't do the right thing, since it will turn on serial consoles if you install to, say, a disk image from an xterm or if you use newcons. Kk. I'm CC'ing Peter Grehan, because we only arrived at this solution based on a quick discussion with Michael Dexter at vBSDcon with respect to bhyve. Peter, can you restate the problem for Nathan so that we can maybe find a better home for this change? or perhaps more clearly define (than I) how we arrived at the code for the bhyve work? -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu
On Nov 2, 2013, at 8:27 PM, Artem Belevich wrote: On Sat, Nov 2, 2013 at 9:47 AM, Teske, Devin devin.te...@fisglobal.com wrote: where are the forth files in question? D'Oh! Here they are: https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/forth_zfs/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=LikaP7fhAuWFwcCB34idLlWshVBNdWkfLMLvZ5MEnX8%3D%0As=3d2d4cfda3daa8bd25f165bb1622cf30975b89ae1bbcaea301da0b8e2f5b00f8 Supplied as patches to either stable/9 or head. -- Devin Thanks for the patches. Tried on stable/9 and they worked great for me. Do you have anything similar to select boot environment as well? You're right to assume that these patches are actually an incremental step in that direction (that is to say, that it's silly to load a kernel before the menu if the menu will allow you to select an ad hoc root device). So indeed, the next step is to allow selecting your root device (be it a ZFS snapshot or alternate root for whatever purpose). I've been giving some thought to this one... and I'm actually thinking about writing this one as an FICL call-out (that is to say, written in the C code of the loader which is then invoked by the Forth code). I have already done mockups for this, but it will take some time before you see a worthy patchset (think 10.1). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 21, 2013, at 10:18 AM, Allan Jude wrote: On 2013-10-21 13:14, Teske, Devin wrote: On Oct 21, 2013, at 10:12 AM, Allan Jude wrote: On 2013-10-21 12:19, Teske, Devin wrote: On Oct 21, 2013, at 9:06 AM, Allan Jude wrote: On 2013-10-21 12:04, Teske, Devin wrote: On Oct 21, 2013, at 8:50 AM, Allan Jude wrote: On 2013-10-21 11:45, Johan Broman wrote: Hi! Sorry for the delayed answer. I've patched zfsboot and rebuilt the release. I now get an error message that ada2 can't be used, which is correct. Good stuff! :) ( I've recreated the test environment using a KVM guest with four SATA drives instead of the server I was using. I makes it easier to test stuff. ) Here's the screenshot: https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.pngk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=f6fc4ec7c9d8b9486e897156dac5af7c3f64e3fae746dcf6c66ad91564a8ce99 Maybe one should be unable to select drives that are part of a graid in the first place? Or is that out-of-scope for bsdinstall at this point? (As I guess that requires too many changes/new lines) Cheers Johan On 19/10/13 22:20, Teske, Devin wrote: On Oct 19, 2013, at 10:07 AM, Johan Broman wrote: I recreated the graid mirror on ada2 and ada3 and reran the installation. I'm unable to scroll the msgbox using PgDn or arrow keys. There is no indication that the action failed and I'm returned to the ZFS setup screen if I hit OK. I have screen shots (taken with my phone) of the msgbox and ps auxwww output. Let me know what kind of debug info you would like. I've put the screen shots here: https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstallk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=6322000e13ed155bda748698c4a0d54c9de7c29f5566affe202d7c5a29917cd1 I've added a patch to fix debugging in the zfsboot script... https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=1dc96a8b5450d27fe8210b361c9ed736ccd448f78df6aabe170bb80e31bca6d9 Feedback welcome. Johan,... Can you see if the patch sheds some better light as to what's failing? The patch won't fix the problem, but it should give us an accurate error message so that we can learn what precisely is returning an error status. Thanks in advance. I do notice that Devin's manually prefixing the error message with the tool name, is partially redundancy when the tool does it it self, but we can't always be sure it will do that. The next patchset will fix that. I'm dropping the tool name from the msgbox contents and putting it in the title (e.g., 'Error: gpart) that way... even if the tool spits out its own name (or not), we'll know what exactly what was going on by looking at the title. the graid thing is rather hard to detect, especially when it is a faulted array that doesn't even appear in graid status etc. I believe the idea behind the script is that whatever you tell it to use will be destroyed. Allan, maybe perhaps we could add some code that attempts to dis- assemble a graid to make the disk usable? Johan, what would you be more apt to expect? That it killed your graid or that it gave an error? (/me thinks what the recourse to the error might entail -- going to partedit?) Your recourse would be switching to the shell (control+alt+f4) and destroying the graid. I am a little hesitant to go destroying graids unprompted. If we had the geom.confxml parsing, we might be able to detect it and ask the user what to do Well, my concern with going and asking about each/every configuration before we clear it... Hasn't the user already answered a Last Chance! dialog giving the go- ahead to nuke any/all data *and* configurations on a drive? I guess that makes sense, We offer the dialog to allow the user to investigate their disk in detail so they can be sure they picked the correct ones. the graid thing is the biggest gotcha because it picks up metadata written by the motherboard raid system, so you can have a graid configuration even if you have never booted freebsd before. Hmmm... so what does one do in *that* situation? Go into the motherboard BIOS and disable on-board RAID features for the controller providing the disks-in-question? You can destroy the metadata with the graid delete command. I've only encountered once instance where 'graid status' didn't show any devices, but graid was blocking zfs from using the device until the graid was destroyed (the graid part was only discovered after generating an image out of the geom.confdot sysctl) Allan, Do we want to add
[CFT] Kernel-Selection Enhancemnt to Boot Menu
Hi all, Here's a chance to test out the kernel selection menu enhancements to the boot loader menu before they go into HEAD. Discussion welcome, feedback desired. No recompile needed, just drop the new forth files onto a HEAD or stable/9 box and reboot. -- Cheers, Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu
On Nov 2, 2013, at 1:45 AM, Sam Fourman Jr. wrote: On Sat, Nov 2, 2013 at 7:09 AM, Teske, Devin devin.te...@fisglobal.com wrote: Hi all, Here's a chance to test out the kernel selection menu enhancements to the boot loader menu before they go into HEAD. Discussion welcome, feedback desired. No recompile needed, just drop the new forth files onto a HEAD or stable/9 box and reboot. -- Cheers, Devin where are the forth files in question? D'Oh! Here they are: http://druidbsd.cvs.sf.net/viewvc/druidbsd/forth_zfs/ Supplied as patches to either stable/9 or head. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
[CFT] bsdinstall and zfsboot enhancements
Hi all, Another Call For Testing... This one is for bsdinstall. Two patchsets are required for this CFT: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_debug/ http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/ The enhancements are: + Add a `-D FILE command-line option for overriding the path to the bsdinstall log file (BSDINSTALL_LOG env var). + Document new `-D FILE' in the man page for bsdinstall. + If FILE in `-D FILE' begins with a +, debug output goes to stdout (interleaved between dialog(1) invocations/output) as well as to FILE (minus the leading + of course). + If BSDINSTALL_LOG cannot be written, then debugging is disabled (except in the case of a leading + in the pathname, wherein debug will still be printed to stdout). + Update source code format to approximate a future shstyle(9) + Fix a dangling participle (Begun ... - Began ...) + Rewrite the docsinstall script (was necessary to abate direct dependency on BSDINSTALL_LOG (instead, use fault-tolerant bsdconfig framework which displays appropriate errors for package management). + Add additional debug output for dhclient/rtsol/wpa_cliscan + Display script errors in a textbox rather than just on stdout + Update many coments. + Add new f_show_err() API call (like f_show_msg but changes the dialog title to Error)(see bsdconfig's `common.subr'). + Add new f_eval_catch() API call for executing a command via eval but not before logging the command to debug. Several example cases documented in API header for function in bsdconfig's `common.subr'. + Fix dialog auto-sizing when launched as an rvalue to a pipe for indirected scripts (previously would default to 80x24 sizing in this case, now it can autosize to full size even when in a pipe chain). + Fix a bug in f_snprintf wherein if the $format argument began with a - or -- it would be misinterpreted as a flag to printf. (this is in bsdcofig's strings.subr) + Add accompanying f_sprintf() and f_vsprintf() to go along with already existing f_snprintf() and f_vsnprintf() (see bsdconfig's strings.subr). + Update bsdinstall's config script to adjust ttyu* entries in /etc/ttys when it is determined that we are in-fact doing an install over serial (e.g. bhyve). + Remove some unnecessary default ZFS datasets from the automatic zfsboot script. Such as: /usr/ports/distfiles /usr/ports/packages /usr/obj /var/db /var/empty /var/mail and /var/run (these can all be created as-needed once the system is installed). + Remove setuid=off for /usr/home (as discussed with others from last round of CFT). + Fix some i18n string violations in zfsboot + Bolster debugging output in zfsboot + Fix some string quoting issues in zfsboot + Fix some variable scope issues in zfsboot + Only display the ZFS vdev type selection menu when running interactively (duh). + Change Create to Install in zfsboot main menu + Increase pedantic error checking in zfsboot (type-check arguments and such). + Add a call to graid destroy to kill automatic metadata (part of the series of pedantic destructions we do when bootstrapping a new/naked disk). + Make judicious use of new f_eval_catch() in zfsboot + More comments (zfsboot). + Fixup some variable names for consistency (zfsboot). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. signature.asc Description: Message signed with OpenPGP using GPGMail
[RFC] libdispatch (aka Grand Central Dispatch) in base
Hi all, I'd like to bring up the discussion for topic.. Importing libdispatch (aka Apple's Grand Central Dispatch) into base (contrib?). Specifically into HEAD then MFC'd only as far back as stable/10. Here's the reason why: http://devinteske.com/freebsd-installer-enhancements Summary: For the purpose of providing a concurrency model better than pthreads for the expressed desire to bring about concurrent data processing (applicable directly to distributions, packages, signing and more). Multiple people have confirmed with me with respect to the above blog article that the concurrency model would be most efficient with libdispatch. Since the tool mentioned in the blog is a. Compiling with clang b. Requires newest dialog(3) that is only in stable/10 or higher I'd say that it looks like a match made in heaven. But of course, there's that one hang-up... dispatch is not available in base yet. Is anyone working on getting dispatch into base? -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [RFC] libdispatch (aka Grand Central Dispatch) in base
On Oct 29, 2013, at 7:17 PM, Freddie Cash wrote: On Oct 29, 2013 7:07 PM, Teske, Devin devin.te...@fisglobal.com wrote: Hi all, I'd like to bring up the discussion for topic.. Importing libdispatch (aka Apple's Grand Central Dispatch) into base (contrib?). Hasn't this been done already? There's mention of it in the archives from 2009/2010. There was a port by Robert Watson et al to 8.1 mentioned in the quarterly status reports, and indications it would be imported after 8.1-release. I knew about the port, and I had heard about previous efforts, but when I go scrounging around HEAD, I can't find it. Let alone older branches. Do you know where we should be looking? I already checked contrib, and lib. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [RFC] libdispatch (aka Grand Central Dispatch) in base
On Oct 29, 2013, at 9:01 PM, Nathan Whitehorn wrote: On 10/29/13 21:04, Teske, Devin wrote: Hi all, I'd like to bring up the discussion for topic.. Importing libdispatch (aka Apple's Grand Central Dispatch) into base (contrib?). Specifically into HEAD then MFC'd only as far back as stable/10. Here's the reason why: http://devinteske.com/freebsd-installer-enhancements Summary: For the purpose of providing a concurrency model better than pthreads for the expressed desire to bring about concurrent data processing (applicable directly to distributions, packages, signing and more). Multiple people have confirmed with me with respect to the above blog article that the concurrency model would be most efficient with libdispatch. Since the tool mentioned in the blog is a. Compiling with clang b. Requires newest dialog(3) that is only in stable/10 or higher I'd say that it looks like a match made in heaven. But of course, there's that one hang-up... dispatch is not available in base yet. Is anyone working on getting dispatch into base? I have no opinion on GCD in base -- probably a good idea -- but I was hoping you could explain further what you are trying to do here. Parallelism in these steps is usually of very limited utility. For checksum evaluation, it could speed things up on multicore systems. For fetch and extract, however, it either has no effect (if you are bandwidth limited during fetch) or slows things down significantly by causing extra seeks. This problem is especially bad for CD installs. For extract, it can actually cause lasting problems on the installed system by increasing disk fragmentation. The parallel concurrency was actually an after-thought. First... I started with this... http://pastebin.com/LvDtJNGh A straight-forward attempt to see if I could add X11 support by finding a feature that dialog(1) and Xdialog(1) agree upon. They agree that when they are reading pipe data for the gauge widget, they will update their prompt text when the input looks like: XXX New prompt text XXX NB: Literally XXX on a line by itself, sandwiching the new prompt. I then quickly rewrote that into C and added dozens of features to get where the fdpv prototype is at now. I'm glad we're having this discussion, because I acknowledge that I very well can push ahead without concurrency (and we may want to do that, because the utility brings other benefits that are foremost on my list behind parallelism). Trying to achieve: 1. X11 support 2. Get rid of the need for temporary files (without sacrificing signing) 3. Internationalization So would you recommend just moving forward without the parallelism so we could add that as a value-add down the line? That would actually speed up the development of fdpv as a new base utility that can take either stdin or a list of fifo's with size and signature data, and verify the data before forking off a tar command, or pkg etc. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 21, 2013, at 8:50 AM, Allan Jude wrote: On 2013-10-21 11:45, Johan Broman wrote: Hi! Sorry for the delayed answer. I've patched zfsboot and rebuilt the release. I now get an error message that ada2 can't be used, which is correct. Good stuff! :) ( I've recreated the test environment using a KVM guest with four SATA drives instead of the server I was using. I makes it easier to test stuff. ) Here's the screenshot: https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.pngk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=f6fc4ec7c9d8b9486e897156dac5af7c3f64e3fae746dcf6c66ad91564a8ce99 Maybe one should be unable to select drives that are part of a graid in the first place? Or is that out-of-scope for bsdinstall at this point? (As I guess that requires too many changes/new lines) Cheers Johan On 19/10/13 22:20, Teske, Devin wrote: On Oct 19, 2013, at 10:07 AM, Johan Broman wrote: I recreated the graid mirror on ada2 and ada3 and reran the installation. I'm unable to scroll the msgbox using PgDn or arrow keys. There is no indication that the action failed and I'm returned to the ZFS setup screen if I hit OK. I have screen shots (taken with my phone) of the msgbox and ps auxwww output. Let me know what kind of debug info you would like. I've put the screen shots here: https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstallk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=6322000e13ed155bda748698c4a0d54c9de7c29f5566affe202d7c5a29917cd1 I've added a patch to fix debugging in the zfsboot script... https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=1dc96a8b5450d27fe8210b361c9ed736ccd448f78df6aabe170bb80e31bca6d9 Feedback welcome. Johan,... Can you see if the patch sheds some better light as to what's failing? The patch won't fix the problem, but it should give us an accurate error message so that we can learn what precisely is returning an error status. Thanks in advance. I do notice that Devin's manually prefixing the error message with the tool name, is partially redundancy when the tool does it it self, but we can't always be sure it will do that. The next patchset will fix that. I'm dropping the tool name from the msgbox contents and putting it in the title (e.g., 'Error: gpart) that way... even if the tool spits out its own name (or not), we'll know what exactly what was going on by looking at the title. the graid thing is rather hard to detect, especially when it is a faulted array that doesn't even appear in graid status etc. I believe the idea behind the script is that whatever you tell it to use will be destroyed. Allan, maybe perhaps we could add some code that attempts to dis- assemble a graid to make the disk usable? Johan, what would you be more apt to expect? That it killed your graid or that it gave an error? (/me thinks what the recourse to the error might entail -- going to partedit?) -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 21, 2013, at 8:45 AM, Johan Broman wrote: Hi! Sorry for the delayed answer. I've patched zfsboot and rebuilt the release. I now get an error message that ada2 can't be used, which is correct. Good stuff! :) ( I've recreated the test environment using a KVM guest with four SATA drives instead of the server I was using. I makes it easier to test stuff. ) Here's the screenshot: http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.png Thank you so much, definitely says we're headed in the right direction. It was bothering me that the debugging wasn't up to-par. Maybe one should be unable to select drives that are part of a graid in the first place? Or is that out-of-scope for bsdinstall at this point? (As I guess that requires too many changes/new lines) Perhaps we just need to do a better job of clearing previous configurations from the disks that are to be used in the pool. -- Devin On 19/10/13 22:20, Teske, Devin wrote: On Oct 19, 2013, at 10:07 AM, Johan Broman wrote: I recreated the graid mirror on ada2 and ada3 and reran the installation. I'm unable to scroll the msgbox using PgDn or arrow keys. There is no indication that the action failed and I'm returned to the ZFS setup screen if I hit OK. I have screen shots (taken with my phone) of the msgbox and ps auxwww output. Let me know what kind of debug info you would like. I've put the screen shots here: https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstallk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=HwEpYSg1%2Bqv6o4vps1DECabNoHOTpZj5VxD1rbHI5jU%3D%0As=a27573c7279ac5bc926ca2dc872a18550ee8244421b91d85b9f51b202b23e168 I've added a patch to fix debugging in the zfsboot script... https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=HwEpYSg1%2Bqv6o4vps1DECabNoHOTpZj5VxD1rbHI5jU%3D%0As=195830666030a264348cc1f37355789cd9ada04e0851b8c5769ce6426403fa28 Feedback welcome. Johan,... Can you see if the patch sheds some better light as to what's failing? The patch won't fix the problem, but it should give us an accurate error message so that we can learn what precisely is returning an error status. Thanks in advance. _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 21, 2013, at 9:06 AM, Allan Jude wrote: On 2013-10-21 12:04, Teske, Devin wrote: On Oct 21, 2013, at 8:50 AM, Allan Jude wrote: On 2013-10-21 11:45, Johan Broman wrote: Hi! Sorry for the delayed answer. I've patched zfsboot and rebuilt the release. I now get an error message that ada2 can't be used, which is correct. Good stuff! :) ( I've recreated the test environment using a KVM guest with four SATA drives instead of the server I was using. I makes it easier to test stuff. ) Here's the screenshot: https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.pngk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=f6fc4ec7c9d8b9486e897156dac5af7c3f64e3fae746dcf6c66ad91564a8ce99 Maybe one should be unable to select drives that are part of a graid in the first place? Or is that out-of-scope for bsdinstall at this point? (As I guess that requires too many changes/new lines) Cheers Johan On 19/10/13 22:20, Teske, Devin wrote: On Oct 19, 2013, at 10:07 AM, Johan Broman wrote: I recreated the graid mirror on ada2 and ada3 and reran the installation. I'm unable to scroll the msgbox using PgDn or arrow keys. There is no indication that the action failed and I'm returned to the ZFS setup screen if I hit OK. I have screen shots (taken with my phone) of the msgbox and ps auxwww output. Let me know what kind of debug info you would like. I've put the screen shots here: https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstallk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=6322000e13ed155bda748698c4a0d54c9de7c29f5566affe202d7c5a29917cd1 I've added a patch to fix debugging in the zfsboot script... https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=1dc96a8b5450d27fe8210b361c9ed736ccd448f78df6aabe170bb80e31bca6d9 Feedback welcome. Johan,... Can you see if the patch sheds some better light as to what's failing? The patch won't fix the problem, but it should give us an accurate error message so that we can learn what precisely is returning an error status. Thanks in advance. I do notice that Devin's manually prefixing the error message with the tool name, is partially redundancy when the tool does it it self, but we can't always be sure it will do that. The next patchset will fix that. I'm dropping the tool name from the msgbox contents and putting it in the title (e.g., 'Error: gpart) that way... even if the tool spits out its own name (or not), we'll know what exactly what was going on by looking at the title. the graid thing is rather hard to detect, especially when it is a faulted array that doesn't even appear in graid status etc. I believe the idea behind the script is that whatever you tell it to use will be destroyed. Allan, maybe perhaps we could add some code that attempts to dis- assemble a graid to make the disk usable? Johan, what would you be more apt to expect? That it killed your graid or that it gave an error? (/me thinks what the recourse to the error might entail -- going to partedit?) Your recourse would be switching to the shell (control+alt+f4) and destroying the graid. I am a little hesitant to go destroying graids unprompted. If we had the geom.confxml parsing, we might be able to detect it and ask the user what to do Well, my concern with going and asking about each/every configuration before we clear it... Hasn't the user already answered a Last Chance! dialog giving the go- ahead to nuke any/all data *and* configurations on a drive? -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 21, 2013, at 10:18 AM, Allan Jude wrote: On 2013-10-21 13:14, Teske, Devin wrote: On Oct 21, 2013, at 10:12 AM, Allan Jude wrote: On 2013-10-21 12:19, Teske, Devin wrote: On Oct 21, 2013, at 9:06 AM, Allan Jude wrote: On 2013-10-21 12:04, Teske, Devin wrote: On Oct 21, 2013, at 8:50 AM, Allan Jude wrote: On 2013-10-21 11:45, Johan Broman wrote: Hi! Sorry for the delayed answer. I've patched zfsboot and rebuilt the release. I now get an error message that ada2 can't be used, which is correct. Good stuff! :) ( I've recreated the test environment using a KVM guest with four SATA drives instead of the server I was using. I makes it easier to test stuff. ) Here's the screenshot: https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.pngk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=f6fc4ec7c9d8b9486e897156dac5af7c3f64e3fae746dcf6c66ad91564a8ce99 Maybe one should be unable to select drives that are part of a graid in the first place? Or is that out-of-scope for bsdinstall at this point? (As I guess that requires too many changes/new lines) Cheers Johan On 19/10/13 22:20, Teske, Devin wrote: On Oct 19, 2013, at 10:07 AM, Johan Broman wrote: I recreated the graid mirror on ada2 and ada3 and reran the installation. I'm unable to scroll the msgbox using PgDn or arrow keys. There is no indication that the action failed and I'm returned to the ZFS setup screen if I hit OK. I have screen shots (taken with my phone) of the msgbox and ps auxwww output. Let me know what kind of debug info you would like. I've put the screen shots here: https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstallk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=6322000e13ed155bda748698c4a0d54c9de7c29f5566affe202d7c5a29917cd1 I've added a patch to fix debugging in the zfsboot script... https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=1dc96a8b5450d27fe8210b361c9ed736ccd448f78df6aabe170bb80e31bca6d9 Feedback welcome. Johan,... Can you see if the patch sheds some better light as to what's failing? The patch won't fix the problem, but it should give us an accurate error message so that we can learn what precisely is returning an error status. Thanks in advance. I do notice that Devin's manually prefixing the error message with the tool name, is partially redundancy when the tool does it it self, but we can't always be sure it will do that. The next patchset will fix that. I'm dropping the tool name from the msgbox contents and putting it in the title (e.g., 'Error: gpart) that way... even if the tool spits out its own name (or not), we'll know what exactly what was going on by looking at the title. the graid thing is rather hard to detect, especially when it is a faulted array that doesn't even appear in graid status etc. I believe the idea behind the script is that whatever you tell it to use will be destroyed. Allan, maybe perhaps we could add some code that attempts to dis- assemble a graid to make the disk usable? Johan, what would you be more apt to expect? That it killed your graid or that it gave an error? (/me thinks what the recourse to the error might entail -- going to partedit?) Your recourse would be switching to the shell (control+alt+f4) and destroying the graid. I am a little hesitant to go destroying graids unprompted. If we had the geom.confxml parsing, we might be able to detect it and ask the user what to do Well, my concern with going and asking about each/every configuration before we clear it... Hasn't the user already answered a Last Chance! dialog giving the go- ahead to nuke any/all data *and* configurations on a drive? I guess that makes sense, We offer the dialog to allow the user to investigate their disk in detail so they can be sure they picked the correct ones. the graid thing is the biggest gotcha because it picks up metadata written by the motherboard raid system, so you can have a graid configuration even if you have never booted freebsd before. Hmmm... so what does one do in *that* situation? Go into the motherboard BIOS and disable on-board RAID features for the controller providing the disks-in-question? You can destroy the metadata with the graid delete command. I've only encountered once instance where 'graid status' didn't show any devices, but graid was blocking zfs from using the device until the graid was destroyed (the graid part was only discovered after generating an image out of the geom.confdot sysctl) Would you know where precisely
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 21, 2013, at 10:12 AM, Allan Jude wrote: On 2013-10-21 12:19, Teske, Devin wrote: On Oct 21, 2013, at 9:06 AM, Allan Jude wrote: On 2013-10-21 12:04, Teske, Devin wrote: On Oct 21, 2013, at 8:50 AM, Allan Jude wrote: On 2013-10-21 11:45, Johan Broman wrote: Hi! Sorry for the delayed answer. I've patched zfsboot and rebuilt the release. I now get an error message that ada2 can't be used, which is correct. Good stuff! :) ( I've recreated the test environment using a KVM guest with four SATA drives instead of the server I was using. I makes it easier to test stuff. ) Here's the screenshot: https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.pngk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=f6fc4ec7c9d8b9486e897156dac5af7c3f64e3fae746dcf6c66ad91564a8ce99 Maybe one should be unable to select drives that are part of a graid in the first place? Or is that out-of-scope for bsdinstall at this point? (As I guess that requires too many changes/new lines) Cheers Johan On 19/10/13 22:20, Teske, Devin wrote: On Oct 19, 2013, at 10:07 AM, Johan Broman wrote: I recreated the graid mirror on ada2 and ada3 and reran the installation. I'm unable to scroll the msgbox using PgDn or arrow keys. There is no indication that the action failed and I'm returned to the ZFS setup screen if I hit OK. I have screen shots (taken with my phone) of the msgbox and ps auxwww output. Let me know what kind of debug info you would like. I've put the screen shots here: https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstallk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=6322000e13ed155bda748698c4a0d54c9de7c29f5566affe202d7c5a29917cd1 I've added a patch to fix debugging in the zfsboot script... https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=1dc96a8b5450d27fe8210b361c9ed736ccd448f78df6aabe170bb80e31bca6d9 Feedback welcome. Johan,... Can you see if the patch sheds some better light as to what's failing? The patch won't fix the problem, but it should give us an accurate error message so that we can learn what precisely is returning an error status. Thanks in advance. I do notice that Devin's manually prefixing the error message with the tool name, is partially redundancy when the tool does it it self, but we can't always be sure it will do that. The next patchset will fix that. I'm dropping the tool name from the msgbox contents and putting it in the title (e.g., 'Error: gpart) that way... even if the tool spits out its own name (or not), we'll know what exactly what was going on by looking at the title. the graid thing is rather hard to detect, especially when it is a faulted array that doesn't even appear in graid status etc. I believe the idea behind the script is that whatever you tell it to use will be destroyed. Allan, maybe perhaps we could add some code that attempts to dis- assemble a graid to make the disk usable? Johan, what would you be more apt to expect? That it killed your graid or that it gave an error? (/me thinks what the recourse to the error might entail -- going to partedit?) Your recourse would be switching to the shell (control+alt+f4) and destroying the graid. I am a little hesitant to go destroying graids unprompted. If we had the geom.confxml parsing, we might be able to detect it and ask the user what to do Well, my concern with going and asking about each/every configuration before we clear it... Hasn't the user already answered a Last Chance! dialog giving the go- ahead to nuke any/all data *and* configurations on a drive? I guess that makes sense, We offer the dialog to allow the user to investigate their disk in detail so they can be sure they picked the correct ones. the graid thing is the biggest gotcha because it picks up metadata written by the motherboard raid system, so you can have a graid configuration even if you have never booted freebsd before. Hmmm... so what does one do in *that* situation? Go into the motherboard BIOS and disable on-board RAID features for the controller providing the disks-in-question? -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 21, 2013, at 9:04 AM, Teske, Devin wrote: On Oct 21, 2013, at 8:50 AM, Allan Jude wrote: On 2013-10-21 11:45, Johan Broman wrote: Hi! Sorry for the delayed answer. I've patched zfsboot and rebuilt the release. I now get an error message that ada2 can't be used, which is correct. Good stuff! :) ( I've recreated the test environment using a KVM guest with four SATA drives instead of the server I was using. I makes it easier to test stuff. ) Here's the screenshot: https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.pngk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=f6fc4ec7c9d8b9486e897156dac5af7c3f64e3fae746dcf6c66ad91564a8ce99 Maybe one should be unable to select drives that are part of a graid in the first place? Or is that out-of-scope for bsdinstall at this point? (As I guess that requires too many changes/new lines) Cheers Johan On 19/10/13 22:20, Teske, Devin wrote: On Oct 19, 2013, at 10:07 AM, Johan Broman wrote: I recreated the graid mirror on ada2 and ada3 and reran the installation. I'm unable to scroll the msgbox using PgDn or arrow keys. There is no indication that the action failed and I'm returned to the ZFS setup screen if I hit OK. I have screen shots (taken with my phone) of the msgbox and ps auxwww output. Let me know what kind of debug info you would like. I've put the screen shots here: https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstallk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=6322000e13ed155bda748698c4a0d54c9de7c29f5566affe202d7c5a29917cd1 I've added a patch to fix debugging in the zfsboot script... https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0As=1dc96a8b5450d27fe8210b361c9ed736ccd448f78df6aabe170bb80e31bca6d9 Feedback welcome. Johan,... Can you see if the patch sheds some better light as to what's failing? The patch won't fix the problem, but it should give us an accurate error message so that we can learn what precisely is returning an error status. Thanks in advance. I do notice that Devin's manually prefixing the error message with the tool name, is partially redundancy when the tool does it it self, but we can't always be sure it will do that. The next patchset will fix that. And it's ready now. Tarball: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/?view=tar This patch takes debugging a leap beyond where it is now NOTE: Only for the zfsboot script -- there is another separate effort to improve the debugging in bsdinstall in-general; those patches are here: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_debug/?view=tar -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: BE Loader Menu (was Re: rcs)
On Oct 21, 2013, at 6:32 AM, Kris Moore wrote: On 10/20/2013 20:58, Julian Elischer wrote: Kris. exactly what features need to be added to the boot process to allow what you want to do, but without using grub? Here's a list of the features we are using from grub: For install medium: * Hybrid DVD ISO/USB image in same file I've been using ISOLINUX for that functionality since 2006. My DRUID installer is shipped as a hybrid ISO using ISOLINUX. * BIOS / UEFI loaders on same image Haven't played with UEFI yet. * Graphics support ISOLINUX has that. Can we compare screens? I really like the way ISOLINUX's vesamenu.c32 module allows you to even go so far as to customize the scrollbar coloring (e.g., in the below image, notice how I've given the scrollbar a blue color so it completes the theme). http://twitpic.com/17z5yi Compare that to the same menu, but previously themed in blue: http://twitpic.com/17xas6 I wonder if grub lets you do that (scrollbar color, hilite color, and color of header/footer text as well). For installed system: * Script-able ZFS BE menus This can be achieved in loader(8) with some basic work, building upon what we already have. Around this time last year, I actually added code to allow scripting of the loader(8) menu, but said functionality is still unused (waiting for some userland tool or some C code integrated directly into loader(8) to actually start scripting the menu-system. Until that happens... it's just my little secret that the Forth menus are actually scriptable from a higher language. (been trying for years it seems to find someone whom cares to sit down and show them how it works -- maybe at vBSDcon). * Options for setting default timeouts / default BEs to load, etc I don't see why this too couldn't be done with loader(8). We already have a timeout system for booting with defaults... and I gather that whatever scripting there is to change the loaded defaults would take place prior to reboot (some tool munges loader.conf and then reboots; current defaults are displayed and after the configured timeout, it boots with the displayed configuration). * Script-able menu to boot other OS's, such as Windows / Linux Well, I use SYSLINUX for that. SYSLINUX is the same product as the aforementioned ISOLINUX, except that it's focused on system versus install media. Of course... in your case, grub loads FreeBSD directly. When I use syslinux as a bootloader, I get the FreeBSD Forth beastie menu after selecting FreeBSD from the syslinux vesamenu. * Loads before kernel Yeah, I got patches for our menu to follow suit. http://druidbsd.cvs.sf.net/viewvc/druidbsd/forth_zfs/ (thinking abut committing it already) * Graphics support (I.E. countdown indicator, wallpapers, arrow-key support, etc) Well, I don't think the countdown indicator is going to get anyone all too excited (I believe we've had that for years). The wallpapers of course... can't do those as a default, but I think there's a graphical loader project -- but of course that won't service serial boot. I've thought about using arrow keys to navigate before... It involves implementing a whole new layer on top of the menu. One that would use some static character (e.g., ) to show you which item you are over (or in the case of color... we could *maybe* use ANSI reverse). Definitely not easy. New state machine would be required. New rendering code would be required as well. The good news though is that it would get rid of the number prefixes). If the native loader is able to handle the ZFS BE stuff, then I don't see a reason why I can't make it an option in the installer, even without graphics and the other niceties that grub brings. Is this working in the stable/10 branch now? No... just the patchset I pointed to... http://druidbsd.cvs.sf.net/viewvc/druidbsd/forth_zfs/ Which produces the following: http://twitpic.com/dhv2b6 Which achieves: a. Don't load kernel before menu b. Scripting of selections can be done via $kernels in loader.conf What we are lacking at the moment is... Root Device selection menu. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[CFT] Improve bsdinstall debugging
Hi all, I've got a set of patches to apply to bsdinstall to improve debugging. Would like some testing and/or review feedback. Patches are here (tarball form; apply in any order using svn patch): http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_debug/?view=tar http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/?view=tar The net effect of the patches is to: 1. Increase debugging to include commands and their output 2. Add descriptions in the debug output to show what we're doing 3. Provide better error reporting through dialog(1) (when interactive) 4. Implement debugging for things we may not expect to go wrong 5. Show function names in debug messages 6. Log values of important environment variables 7. Type-check arguments that need to be a specific type (e.g., integer) 8. Double-quote values that we can't guarantee to not contain $IFS 9. In the zfsboot script, catch any error from all 48+ fork-execs 10. Abort immediately on error if nonInteractive is set (zfsboot) 11. Implement `-D file' for bsdinstall to change BSDINSTALL_LOG 12. If the log file can't be written, print a single warning and don't log 13. Fix dangling participle Begun in log message to Began 14. Rewrite docsinstall module to not depend on the log file for error NB: Also brings in bsdconfig package management API to docsinstall Right now, it should be noted that I went to town on the zfsboot script but I was a lot more sedate when patching the rest of bsdinstall. The initial motivation for the debugging additions for the rest of bsdinstall (besides zfsboot) was from flo@ whom was unable to deploy to his Soekris boxen via PXE because /tmp was read-only when bsdinstall started (and if it couldn't write to its log file, it just exited -- without perhaps giving you a chance to run a script to bootstrap a writable /tmp). These patches fixed flo@'s problem with bsdinstall not launching. The motivation for the zfsboot debugging additions was a thread wherein someone had some ad hoc graid metadata on a disk and we were unable to (from the user interface *or* the debug logfile) diagnose the situation. These patches fixed Johan's problems with that. Then... I decided to take it one step further and centralize the debugging API -- currently stashed in zfsboot for local usage... but with feedback from this thread, I'd like to then uplift those into the library files and then pepper them throughout the rest of bsdinstall to make its debugging as strong as that in zfsboot. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 19, 2013, at 8:34 AM, Allan Jude wrote: On 2013-10-19 11:31, Johan Broman wrote: On 19/10/13 17:23, Allan Jude wrote: On 2013-10-19 10:56, Johan Broman wrote: Hi! Just tested the root-on-ZFS install option using FreeBSD 10 beta 1. I have 4 SATA drives in my server. I select all four of them in a RAIDZ1 setup. I hit enter to continue the installation and the zpool is created, but I'm then returned to the zpool selection screen again. It turned out that two of the drives had previously been used in a (Linux) software mirror setup and because of this they got activated in /dev/raid/r0. Because of this I ended up in an endless bsdinstall loop. Removing the raid device using the graid command resolved the situation. Now maybe this is working as designed, but there was no warning/alert to the fact that the devices couldn't be used. Perhaps a warning should be rasied in this situation? Thanks for all the great work on the new installer, really looking forward to FreeBSD 10! Cheers Johan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org Errors like that normally generate a msgbox dialog with the error output from whichever command failed. I'll have to dig into it and see where that problem is. I've seen other people have problems creating ZFS arrays after graid, but in that case it was an incomplete graid label causing a device to be locked but not appear in the graid status output. Ah ok. A msgbox did appear but the drives that had the problem (ada2 and ada3) wasn't visible in the output. (not sure if the box itself has a size limit or maybe I was just unable to scroll down and see the errors?). The only visible output was that it was able to create labels on ada0 and ada1. /Johan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org Ahh yes, you have to press 'page-down' to scroll the msgbox. I tried to add a scrollbar but turns out that is not possible. The only indication that there is more message to read, is a small 'xx%' in the bottom right. We might have to look at breaking that output up or something. The only reason for a msgbox widget to scroll is if it is displayed at maximum height or width of the screen and it *still* has more data to display than can be presented at-once. If... however... the msgbox widget is *not* full-height or full-width yet... it is requiring you to scroll -- then we've found a bug. Can we get a screen shot? -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 19, 2013, at 8:43 AM, Teske, Devin wrote: On Oct 19, 2013, at 8:34 AM, Allan Jude wrote: On 2013-10-19 11:31, Johan Broman wrote: On 19/10/13 17:23, Allan Jude wrote: On 2013-10-19 10:56, Johan Broman wrote: Hi! Just tested the root-on-ZFS install option using FreeBSD 10 beta 1. I have 4 SATA drives in my server. I select all four of them in a RAIDZ1 setup. I hit enter to continue the installation and the zpool is created, but I'm then returned to the zpool selection screen again. It turned out that two of the drives had previously been used in a (Linux) software mirror setup and because of this they got activated in /dev/raid/r0. Because of this I ended up in an endless bsdinstall loop. Removing the raid device using the graid command resolved the situation. Now maybe this is working as designed, but there was no warning/alert to the fact that the devices couldn't be used. Perhaps a warning should be rasied in this situation? Thanks for all the great work on the new installer, really looking forward to FreeBSD 10! Cheers Johan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org Errors like that normally generate a msgbox dialog with the error output from whichever command failed. I'll have to dig into it and see where that problem is. I've seen other people have problems creating ZFS arrays after graid, but in that case it was an incomplete graid label causing a device to be locked but not appear in the graid status output. Ah ok. A msgbox did appear but the drives that had the problem (ada2 and ada3) wasn't visible in the output. (not sure if the box itself has a size limit or maybe I was just unable to scroll down and see the errors?). The only visible output was that it was able to create labels on ada0 and ada1. /Johan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org Ahh yes, you have to press 'page-down' to scroll the msgbox. I tried to add a scrollbar but turns out that is not possible. The only indication that there is more message to read, is a small 'xx%' in the bottom right. We might have to look at breaking that output up or something. The only reason for a msgbox widget to scroll is if it is displayed at maximum height or width of the screen and it *still* has more data to display than can be presented at-once. I should clarify... The zfsboot script doesn't use dialog(1) directly. It uses the bsdconfig API. That being said, msgbox widgets automatically scale their size to fit the content being displayed. So whenever a msgbox is thrown up using this API... the widget will never scroll unless the box can't be made big enough to hold the entire content (either the screen resolution or terminal size is too small; we maxed out the size of the widget; and there's still hidden content). But... While all of bsdconfig uses this API, hardly any of bsdinstall uses this API. If... however... the msgbox widget is *not* full-height or full-width yet... it is requiring you to scroll -- then we've found a bug. Can we get a screen shot? So we really need to nail down precisely which error box this is so that we can address whether the issue is in-fact an instance of using the old error-box handling instead of the auto-sizing API. So... With this described API, you should never have to scroll a box unless it can't fit all the data *and* you should be able to immediately identify when that becomes the case... 1. The widget spans the entire width of the screen. 2. The widget spans the entire height of the screen. 3. Both 1 and 2. It's in *those* cases that you should then *EXPECT* to find that the region can scroll with cursor keys and page up/down (look for the scroll percentage in the widget as Allan suggested. I don't want to see the scroll percentage doohickey *unless* the widget is auto-sized to full-width or full-height. Meaning, there's either a bug in the API or someone fell into a trap (there are a couple). -- Devin -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 19, 2013, at 9:27 AM, Allan Jude wrote: On 2013-10-19 11:55, Teske, Devin wrote: On Oct 19, 2013, at 8:43 AM, Teske, Devin wrote: On Oct 19, 2013, at 8:34 AM, Allan Jude wrote: On 2013-10-19 11:31, Johan Broman wrote: On 19/10/13 17:23, Allan Jude wrote: On 2013-10-19 10:56, Johan Broman wrote: Hi! Just tested the root-on-ZFS install option using FreeBSD 10 beta 1. I have 4 SATA drives in my server. I select all four of them in a RAIDZ1 setup. I hit enter to continue the installation and the zpool is created, but I'm then returned to the zpool selection screen again. It turned out that two of the drives had previously been used in a (Linux) software mirror setup and because of this they got activated in /dev/raid/r0. Because of this I ended up in an endless bsdinstall loop. Removing the raid device using the graid command resolved the situation. Now maybe this is working as designed, but there was no warning/alert to the fact that the devices couldn't be used. Perhaps a warning should be rasied in this situation? Thanks for all the great work on the new installer, really looking forward to FreeBSD 10! Cheers Johan ___ freebsd-current@freebsd.org mailing list https://urldefense.proofpoint.com/v1/url?u=http://lists.freebsd.org/mailman/listinfo/freebsd-currentk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=AnourZYCpeybP9aMxTFH%2B4GBOf0xikQ2jpEi%2BPFWcKo%3D%0As=fec708d91fa0976a0e297ef3628851168f36e9cb34727dc7935e26ae190e90da To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org Errors like that normally generate a msgbox dialog with the error output from whichever command failed. I'll have to dig into it and see where that problem is. I've seen other people have problems creating ZFS arrays after graid, but in that case it was an incomplete graid label causing a device to be locked but not appear in the graid status output. Ah ok. A msgbox did appear but the drives that had the problem (ada2 and ada3) wasn't visible in the output. (not sure if the box itself has a size limit or maybe I was just unable to scroll down and see the errors?). The only visible output was that it was able to create labels on ada0 and ada1. /Johan ___ freebsd-current@freebsd.org mailing list https://urldefense.proofpoint.com/v1/url?u=http://lists.freebsd.org/mailman/listinfo/freebsd-currentk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=AnourZYCpeybP9aMxTFH%2B4GBOf0xikQ2jpEi%2BPFWcKo%3D%0As=fec708d91fa0976a0e297ef3628851168f36e9cb34727dc7935e26ae190e90da To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org Ahh yes, you have to press 'page-down' to scroll the msgbox. I tried to add a scrollbar but turns out that is not possible. The only indication that there is more message to read, is a small 'xx%' in the bottom right. We might have to look at breaking that output up or something. The only reason for a msgbox widget to scroll is if it is displayed at maximum height or width of the screen and it *still* has more data to display than can be presented at-once. I should clarify... The zfsboot script doesn't use dialog(1) directly. It uses the bsdconfig API. That being said, msgbox widgets automatically scale their size to fit the content being displayed. So whenever a msgbox is thrown up using this API... the widget will never scroll unless the box can't be made big enough to hold the entire content (either the screen resolution or terminal size is too small; we maxed out the size of the widget; and there's still hidden content). But... While all of bsdconfig uses this API, hardly any of bsdinstall uses this API. If... however... the msgbox widget is *not* full-height or full-width yet... it is requiring you to scroll -- then we've found a bug. Can we get a screen shot? So we really need to nail down precisely which error box this is so that we can address whether the issue is in-fact an instance of using the old error-box handling instead of the auto-sizing API. So... With this described API, you should never have to scroll a box unless it can't fit all the data *and* you should be able to immediately identify when that becomes the case... 1. The widget spans the entire width of the screen. 2. The widget spans the entire height of the screen. 3. Both 1 and 2. It's in *those* cases that you should then *EXPECT* to find that the region can scroll with cursor keys and page up/down (look for the scroll percentage in the widget as Allan suggested. I don't want to see the scroll percentage doohickey *unless* the widget is auto-sized to full-width or full-height. Meaning, there's either a bug in the API or someone fell into a trap (there are a couple). the error output msgbox is huge
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 19, 2013, at 10:07 AM, Johan Broman wrote: On 19/10/13 18:27, Allan Jude wrote: On 2013-10-19 11:55, Teske, Devin wrote: On Oct 19, 2013, at 8:43 AM, Teske, Devin wrote: On Oct 19, 2013, at 8:34 AM, Allan Jude wrote: On 2013-10-19 11:31, Johan Broman wrote: On 19/10/13 17:23, Allan Jude wrote: On 2013-10-19 10:56, Johan Broman wrote: Hi! Just tested the root-on-ZFS install option using FreeBSD 10 beta 1. I have 4 SATA drives in my server. I select all four of them in a RAIDZ1 setup. I hit enter to continue the installation and the zpool is created, but I'm then returned to the zpool selection screen again. It turned out that two of the drives had previously been used in a (Linux) software mirror setup and because of this they got activated in /dev/raid/r0. Because of this I ended up in an endless bsdinstall loop. Removing the raid device using the graid command resolved the situation. Now maybe this is working as designed, but there was no warning/alert to the fact that the devices couldn't be used. Perhaps a warning should be rasied in this situation? Thanks for all the great work on the new installer, really looking forward to FreeBSD 10! Cheers Johan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org Errors like that normally generate a msgbox dialog with the error output from whichever command failed. I'll have to dig into it and see where that problem is. I've seen other people have problems creating ZFS arrays after graid, but in that case it was an incomplete graid label causing a device to be locked but not appear in the graid status output. Ah ok. A msgbox did appear but the drives that had the problem (ada2 and ada3) wasn't visible in the output. (not sure if the box itself has a size limit or maybe I was just unable to scroll down and see the errors?). The only visible output was that it was able to create labels on ada0 and ada1. /Johan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org Ahh yes, you have to press 'page-down' to scroll the msgbox. I tried to add a scrollbar but turns out that is not possible. The only indication that there is more message to read, is a small 'xx%' in the bottom right. We might have to look at breaking that output up or something. The only reason for a msgbox widget to scroll is if it is displayed at maximum height or width of the screen and it *still* has more data to display than can be presented at-once. I should clarify... The zfsboot script doesn't use dialog(1) directly. It uses the bsdconfig API. That being said, msgbox widgets automatically scale their size to fit the content being displayed. So whenever a msgbox is thrown up using this API... the widget will never scroll unless the box can't be made big enough to hold the entire content (either the screen resolution or terminal size is too small; we maxed out the size of the widget; and there's still hidden content). But... While all of bsdconfig uses this API, hardly any of bsdinstall uses this API. If... however... the msgbox widget is *not* full-height or full-width yet... it is requiring you to scroll -- then we've found a bug. Can we get a screen shot? So we really need to nail down precisely which error box this is so that we can address whether the issue is in-fact an instance of using the old error-box handling instead of the auto-sizing API. So... With this described API, you should never have to scroll a box unless it can't fit all the data *and* you should be able to immediately identify when that becomes the case... 1. The widget spans the entire width of the screen. 2. The widget spans the entire height of the screen. 3. Both 1 and 2. It's in *those* cases that you should then *EXPECT* to find that the region can scroll with cursor keys and page up/down (look for the scroll percentage in the widget as Allan suggested. I don't want to see the scroll percentage doohickey *unless* the widget is auto-sized to full-width or full-height. Meaning, there's either a bug in the API or someone fell into a trap (there are a couple). the error output msgbox is huge, probably 100+ lines (the screen is what, 24 lines high, and with the ok button, top and bottom reserved space etc, can display maybe 18 lines at once) It contains all the shell output from everything we do, creating the gparts, setting up gnop, all of the redundant destroys etc. I don't think the TINY little % in the bottom right is really enough of an indicator to the user that they CAN scroll, let alone HOW to scroll (IIRC the arrow keys do
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 19, 2013, at 10:07 AM, Johan Broman wrote: I recreated the graid mirror on ada2 and ada3 and reran the installation. I'm unable to scroll the msgbox using PgDn or arrow keys. There is no indication that the action failed and I'm returned to the ZFS setup screen if I hit OK. I have screen shots (taken with my phone) of the msgbox and ps auxwww output. Let me know what kind of debug info you would like. I've put the screen shots here: http://212.181.212.146/bsdinstall I've added a patch to fix debugging in the zfsboot script... http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/ Feedback welcome. Johan,... Can you see if the patch sheds some better light as to what's failing? The patch won't fix the problem, but it should give us an accurate error message so that we can learn what precisely is returning an error status. Thanks in advance. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: BE Loader Menu (was Re: rcs)
On Oct 10, 2013, at 9:42 AM, Julian Elischer wrote: On 10/11/13 12:39 AM, Teske, Devin wrote: On Oct 10, 2013, at 9:06 AM, Julian Elischer wrote: On 10/10/13 1:05 AM, Teske, Devin wrote: I'm late to the party again ;D (didn't realize the rcs thread had turned BE) Both problems can be solved. The loading of the kernel *after* choosing your boot device is trivial. We've been doing it at $work for *years* (almost a decade?) I can put that in, whenever. Probably at the same time as implementing the live/dynamic BE menus for selecting the root device. yeah it always pisses me of when the menu comes up after the kernel is loaded because 99% of the time, I'm in the menu because I want to boot a DIFFERENT kernel.. Same thought I had about 7 years ago. After hearing that others (especially you, Julian) think the same thoughts... I'm happily ready to merge a patch from VICOR to achieve this. PLEASE!.. put it up for review somewhere... Done... http://druidbsd.cvs.sf.net/viewvc/druidbsd/forth_zfs/ All new code. Had to make it programmable: http://twitpic.com/dhv2b6 See that the patch adds documentation for loader.conf(5). It also clarifies a horrible description of start versus initialize in loader.4th(8). I wonder if we can we get a reviewboard instance for the project some time? Do let me know what you think. I went all-out on this one. -- Devin P.S. I probably hadn't have put so much thought into this one if it hadn't been specifically you asking ;D but I think you know that. _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 10, 2013, at 12:00 AM, Nathan Whitehorn wrote: On 10/09/13 18:55, Teske, Devin wrote: On Oct 8, 2013, at 11:19 PM, Nathan Whitehorn wrote: On 10/09/13 01:13, Allan Jude wrote: On 2013-10-08 16:17, Nathan Whitehorn wrote: On 10/07/13 21:59, Allan Jude wrote: Devin Teske and I have been working on a big patch to bsdinstall to implement installing on a ZFS pool. It supports both GPT and MBR, the 4k sector gnop trick, and optional GELI encryption. We would like to commit this in time for 10.0-BETA1 so it needs some testing to work out any obvious bugs before we send it off to re@ to get it committed. It includes a single configuration menu that allows you to select all of the required details, including which drives to use (gets details from camcontrol, also includes an inspection utility that presents the detailed output of camcontrol inquiry/identify, and gpart show), what ZFS RAID level to use (taking in to consideration the selected number of drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc. Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1 3. Remove the dialog asking if you wish to enable crash dumps, this feature has been combined into the regular 'services to enable' dialog and enabled by default You can browse the patches here: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/ I've built a bootonly.iso (10.0-ALPHA4) to make testing easier, available compressed (48 MB) or uncompressed (211 MB): http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso We look forward to your feedback Thanks for doing this! I had a few comments: 1. ZFS is not bootable on all architectures. Could you adjust that menu item to only display for i386, amd64, and (I think?) sparc64. Use uname -m, not -p, for this. I had not considered that, I'll make that change 1a. The script is broken on sparc64 in any case, which uses VTOC8 instead of GPT. I'll disable sparc64 as well 2. Why are you using camcontrol? That is guaranteed not to work on non-CAM systems. You should use the GEOM ident string if you need an ID. The GEOM ident string doesn't do enough to help the user identify which drive is which. More data is not exposed anywhere that I could find What we really need, is dev.ada.0.desc% like we have for network interfaces and a slew of other devices. GEOM data is great, but it is not exposed in a shell friendly way any place that I could find, other than the sysctl with DOT and XML data. This is one of the reasons the partition editor is written in C. There are a few other odd corner-cases where C is much more powerful than the command-line for the GEOM operations that partedit needs to do. I'm not sure how to usefully get it just from the shell. You can see how to do it in C in the boot_disk() routine of partedit/auto_part.c. 3. Any plans to integrate this into the regular partition editor? ZFS support is important enough that I will definitely not get in the way, even as a bolt-on, but it would be a shame for it to stay that way. The editor is also designed for ZFS to be added. I am a sysadmin, not a programmer. I can't write C. Most people deploying servers can't write C. I agree with Devin Teske, if everything was in shell it would be a lot more usable for non-developers, who probably make up the majority of people who deploy FreeBSD. There are some cases the other way too. Devin is probably the most shell-proficient FreeBSD committer. Well, there's jilles ;D (he writes/maintains the sh(1) implementation itself). I certainly can't write shell scripts at that level, which means that for me bsdconfig, for example, is effectively read-only (and quite hard to read as well). In the past few days of working on the bsdinstall_zfs patch-set with Allan Jude, I learned something new actually. It seems that sh(1) doesn't suffer so much from this read only concept. Imagine you're starting a new C project on an operating system that has all new syscalls and all new APIs that you've never seen. Would be pretty hard, no doubt. Now imagine that you're thrown a life-line called POSIX and hey, now you're slinging code in MinGW on Windows when you don't know a lick of M$ syscalls or APIs. In a similar manner, I've witnessed functionality be added that is truly functional *without* using any of the API calls. Then I come in and do a round of optimizations to leverage the existing API. Shell is kinda like that... I'm noticing Allan Jude doesn't know all the API calls yet (who could? other than me of course -- and even I have trouble remembering them _all_) yet this doesn't phase him (or others) from jumping in and lending a hand. No different than C, but the read-only aspect
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 10, 2013, at 4:55 AM, Nathan Whitehorn wrote: On 10/10/13 09:20, Allan Jude wrote: On 2013-10-10 03:00, Nathan Whitehorn wrote: On 10/09/13 18:55, Teske, Devin wrote: On Oct 8, 2013, at 11:19 PM, Nathan Whitehorn wrote: On 10/09/13 01:13, Allan Jude wrote: On 2013-10-08 16:17, Nathan Whitehorn wrote: On 10/07/13 21:59, Allan Jude wrote: Devin Teske and I have been working on a big patch to bsdinstall to implement installing on a ZFS pool. It supports both GPT and MBR, the 4k sector gnop trick, and optional GELI encryption. We would like to commit this in time for 10.0-BETA1 so it needs some testing to work out any obvious bugs before we send it off to re@ to get it committed. It includes a single configuration menu that allows you to select all of the required details, including which drives to use (gets details from camcontrol, also includes an inspection utility that presents the detailed output of camcontrol inquiry/identify, and gpart show), what ZFS RAID level to use (taking in to consideration the selected number of drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc. Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1 3. Remove the dialog asking if you wish to enable crash dumps, this feature has been combined into the regular 'services to enable' dialog and enabled by default You can browse the patches here: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/ I've built a bootonly.iso (10.0-ALPHA4) to make testing easier, available compressed (48 MB) or uncompressed (211 MB): http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso We look forward to your feedback Thanks for doing this! I had a few comments: 1. ZFS is not bootable on all architectures. Could you adjust that menu item to only display for i386, amd64, and (I think?) sparc64. Use uname -m, not -p, for this. I had not considered that, I'll make that change 1a. The script is broken on sparc64 in any case, which uses VTOC8 instead of GPT. I'll disable sparc64 as well 2. Why are you using camcontrol? That is guaranteed not to work on non-CAM systems. You should use the GEOM ident string if you need an ID. The GEOM ident string doesn't do enough to help the user identify which drive is which. More data is not exposed anywhere that I could find What we really need, is dev.ada.0.desc% like we have for network interfaces and a slew of other devices. GEOM data is great, but it is not exposed in a shell friendly way any place that I could find, other than the sysctl with DOT and XML data. This is one of the reasons the partition editor is written in C. There are a few other odd corner-cases where C is much more powerful than the command-line for the GEOM operations that partedit needs to do. I'm not sure how to usefully get it just from the shell. You can see how to do it in C in the boot_disk() routine of partedit/auto_part.c. 3. Any plans to integrate this into the regular partition editor? ZFS support is important enough that I will definitely not get in the way, even as a bolt-on, but it would be a shame for it to stay that way. The editor is also designed for ZFS to be added. I am a sysadmin, not a programmer. I can't write C. Most people deploying servers can't write C. I agree with Devin Teske, if everything was in shell it would be a lot more usable for non-developers, who probably make up the majority of people who deploy FreeBSD. There are some cases the other way too. Devin is probably the most shell-proficient FreeBSD committer. Well, there's jilles ;D (he writes/maintains the sh(1) implementation itself). I certainly can't write shell scripts at that level, which means that for me bsdconfig, for example, is effectively read-only (and quite hard to read as well). In the past few days of working on the bsdinstall_zfs patch-set with Allan Jude, I learned something new actually. It seems that sh(1) doesn't suffer so much from this read only concept. Imagine you're starting a new C project on an operating system that has all new syscalls and all new APIs that you've never seen. Would be pretty hard, no doubt. Now imagine that you're thrown a life-line called POSIX and hey, now you're slinging code in MinGW on Windows when you don't know a lick of M$ syscalls or APIs. In a similar manner, I've witnessed functionality be added that is truly functional *without* using any of the API calls. Then I come in and do a round of optimizations to leverage the existing API. Shell is kinda like that... I'm noticing Allan Jude doesn't know all the API calls yet (who could? other than me of course -- and even I have trouble remembering them _all_) yet this doesn't phase him
Re: BE Loader Menu (was Re: rcs)
On Oct 10, 2013, at 9:06 AM, Julian Elischer wrote: On 10/10/13 1:05 AM, Teske, Devin wrote: I'm late to the party again ;D (didn't realize the rcs thread had turned BE) Both problems can be solved. The loading of the kernel *after* choosing your boot device is trivial. We've been doing it at $work for *years* (almost a decade?) I can put that in, whenever. Probably at the same time as implementing the live/dynamic BE menus for selecting the root device. yeah it always pisses me of when the menu comes up after the kernel is loaded because 99% of the time, I'm in the menu because I want to boot a DIFFERENT kernel.. Same thought I had about 7 years ago. After hearing that others (especially you, Julian) think the same thoughts... I'm happily ready to merge a patch from VICOR to achieve this. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 10, 2013, at 10:18 AM, Allan Jude wrote: On 2013-10-10 11:30, Teske, Devin wrote: On Oct 10, 2013, at 4:55 AM, Nathan Whitehorn wrote: On 10/10/13 09:20, Allan Jude wrote: On 2013-10-10 03:00, Nathan Whitehorn wrote: On 10/09/13 18:55, Teske, Devin wrote: On Oct 8, 2013, at 11:19 PM, Nathan Whitehorn wrote: On 10/09/13 01:13, Allan Jude wrote: On 2013-10-08 16:17, Nathan Whitehorn wrote: On 10/07/13 21:59, Allan Jude wrote: Devin Teske and I have been working on a big patch to bsdinstall to implement installing on a ZFS pool. It supports both GPT and MBR, the 4k sector gnop trick, and optional GELI encryption. We would like to commit this in time for 10.0-BETA1 so it needs some testing to work out any obvious bugs before we send it off to re@ to get it committed. It includes a single configuration menu that allows you to select all of the required details, including which drives to use (gets details from camcontrol, also includes an inspection utility that presents the detailed output of camcontrol inquiry/identify, and gpart show), what ZFS RAID level to use (taking in to consideration the selected number of drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc. Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1 3. Remove the dialog asking if you wish to enable crash dumps, this feature has been combined into the regular 'services to enable' dialog and enabled by default You can browse the patches here: https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=ckXs2H5TEcKdiN%2F9l%2FClInsMys9LmTwEbElzzrBchMU%3D%0As=6361ad5ccc04a6d8401158a3ee1b3d8df9fcd2ef47ff6ad6365a57b907cceaec I've built a bootonly.iso (10.0-ALPHA4) to make testing easier, available compressed (48 MB) or uncompressed (211 MB): https://urldefense.proofpoint.com/v1/url?u=http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xzk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=ckXs2H5TEcKdiN%2F9l%2FClInsMys9LmTwEbElzzrBchMU%3D%0As=570daea25d9e629788c76c2b5a6eae19d32050363986a1004a6434c3466965c2 https://urldefense.proofpoint.com/v1/url?u=http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.isok=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=ckXs2H5TEcKdiN%2F9l%2FClInsMys9LmTwEbElzzrBchMU%3D%0As=f52d604a2503a48b409f92f8376bb773de4dbc469d1e74b16d051a20ad894820 We look forward to your feedback Thanks for doing this! I had a few comments: 1. ZFS is not bootable on all architectures. Could you adjust that menu item to only display for i386, amd64, and (I think?) sparc64. Use uname -m, not -p, for this. I had not considered that, I'll make that change 1a. The script is broken on sparc64 in any case, which uses VTOC8 instead of GPT. I'll disable sparc64 as well 2. Why are you using camcontrol? That is guaranteed not to work on non-CAM systems. You should use the GEOM ident string if you need an ID. The GEOM ident string doesn't do enough to help the user identify which drive is which. More data is not exposed anywhere that I could find What we really need, is dev.ada.0.desc% like we have for network interfaces and a slew of other devices. GEOM data is great, but it is not exposed in a shell friendly way any place that I could find, other than the sysctl with DOT and XML data. This is one of the reasons the partition editor is written in C. There are a few other odd corner-cases where C is much more powerful than the command-line for the GEOM operations that partedit needs to do. I'm not sure how to usefully get it just from the shell. You can see how to do it in C in the boot_disk() routine of partedit/auto_part.c. 3. Any plans to integrate this into the regular partition editor? ZFS support is important enough that I will definitely not get in the way, even as a bolt-on, but it would be a shame for it to stay that way. The editor is also designed for ZFS to be added. I am a sysadmin, not a programmer. I can't write C. Most people deploying servers can't write C. I agree with Devin Teske, if everything was in shell it would be a lot more usable for non-developers, who probably make up the majority of people who deploy FreeBSD. There are some cases the other way too. Devin is probably the most shell-proficient FreeBSD committer. Well, there's jilles ;D (he writes/maintains the sh(1) implementation itself). I certainly can't write shell scripts at that level, which means that for me bsdconfig, for example, is effectively read-only (and quite hard to read as well). In the past few days of working on the bsdinstall_zfs patch-set
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 10, 2013, at 10:54 AM, Allan Jude wrote: On 2013-10-10 13:21, Warren Block wrote: [off-list reply] I have not tested the new version yet. Is there any chance it supports setting up a gmirror? Something I've been meaning to discuss with Devin is the concept of presets. The user would be able to choose from a list of presets: Filesystem Layout - Merged / filesystem Split /, /var, /tmp, /usr filesystems then Filesystem Type --- UFS ZFS then Disk Layout --- Plain disk Mirror RAID Those choices might change depending on earlier ones, like if they had chosen ZFS earlier they would get a choice of Mirror, RAID-Z1, RAID-Z2, and so on. All this would build up a configuration, and at the end the user can edit it (filesystem sizes, for instance), then pick Go and walk away. All of the UFS stuff is in C, so is read-only for me. It is something I'd like to do, but it is really iffy if that can be done in time for 10.0 In general, gmirror support could be added to the ZFS section (renamed to something else). So in our current 'zfs' menu, would become the 'new partition manager' menu. Add an extra option for ZFS or UFS, and then the vdev menu would context switch between zfs vdevs and ufs options like gmirror and gstripe or something Adding an extra option for / or / /var /tmp /usr is not a bad idea either, maybe even for ZFS, offering a simpler set of datasets that might be some people's preference (especially if I can't do the following in time:) In a future revision of the zfsboot stuff, I'd like to offer a dialog where users can actually adjust the ZFS datasets. provide a menu listing the created dataset, with option to add more, and when you select a dataset, you can adjust its options or delete it (some really scary validation needs to happen here so you can't delete /usr if you have /usr/local) If the UFS stuff took the form of the current ZFS stuff (give me the entire disk, and i'll run with it), it would be fairly easy to implement, and could probably be done between 10.0-BETA1 and 10.0-BETA2 The more powerful features of the current partedit code, where it actually shows what is on your disk already, and allows you to just adjust it, and in general manage dual booting etc, that is a lot more complicated. But the API exists (and we get a taste of the API in zfsboot). Specifically, a script would mangle a disk and then call f_device_rescan to get the menus to populate new entries. That aside, one of my plans for diskmgmt was to essentially let the user perform free-wheeling adhoc configurations. Here's an extreme example of what I mean**: 1. Boot a box with naked disks 2. Create some gmultipath labels 3. Put a GPT scheme on the disk with one BSD partition 4. Put some UFS labels in the partition 5. Build a zpool out of a secondary label of each multipath'd disk ** Because afterall, FreeBSD shouldn't prevent you from doing something like that. I think that if we try to tackle scenarios like that, then we'll really come up with a winning diskmgmt module (because right now I can't go do some UFS setup and then shove the products into a zpool). _When_ the ZFS management stuff is all integrated (which I plan on integrating bits from pjd's zfsconf too), I think it should be agnostic about what you give it as a device *and* allow you to configure many types of things without being forced into an either-or situation (because often it can be a I want both situation). (tuppence) -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 9, 2013, at 8:29 AM, Warren Block wrote: On Tue, 8 Oct 2013, Teske, Devin wrote: But shell is nasty; slow; and not as powerful as C (it depends in what context; the first is rhetoric, the second is only true for poor implement- ations, and the third may be true in some contexts, but I consider the answer to how maintainable is it to be a factor in the power of a language, so don't necessarily consider C to be more powerful than shell as the latter is as-or-more maintainable with fewer LoC and a higher return on investment; see previous [above] arguments). My question would be: why are sh and C the only choices? If the answer is because that's all we have in base, is that a valid concern? As far as sh, it lacks many high- or even mid-level constructs and has real problems with quoting, Enter helper functions... dte...@scribe9.vicor.com bsdconfig $ grep '^# f_' /usr/share/bsdconfig/strings.subr # f_substr $string $start [ $length ] # f_snprintf $var_to_set $size $format ... # f_vsnprintf $var_to_set $size $format $format_args # f_longest_line_length # f_number_of_lines # f_isinteger $arg # f_uriencode [$text] # f_uridecode [$text] # f_replaceall $string $find $replace [$var_to_set] # f_str2varname $string [$var_to_set] # f_shell_escape $string [$var_to_set] # f_shell_unescape $string [$var_to_set] # f_expand_number $string [$var_to_set] The ones dealing specifically with quoting are f_shell_escape() and f_shell_unescape(). With those two helper functions, I consider quoting in sh(1) to be complete and maleable to handle all situations. parsing, Surely, a strength. It can take a while to engrain or impart all the expansions and at- what stage (and in what context) they are performed, but when you understand the mechanics of (for example) in what order each element (escaped-sequences, params, newlines, and sub-shells) is expanded... the parsing becomes an asset, not a hindrance (imho). Not to mention the flexibility afforded by compound strings. Take for example, the following syntax (leveraging ordered expansion and compound strings): f_dialog_input bar \ 'Enter *any* text; including globs [*] and shell special chars [$`#'']... f_dialog_input baz \ Enter some *more* non-conforming text please... f_shell_escape $bar bar f_shell_escape $baz baz items='$bar' '$baz' show=You entered:\n1=%s\n2=%s\n eval printf \\$show\ $items I'd be surprised if you could get anything out but the values you enter. and output (21 3, for example). I think it helps to highlight a technique for solving the general-purpose issue of how you launch a program and have it talk to a parent's maintained file-descriptor. In C, you'd have to pass special args to posix_spawn(3) to get this functionality. Talking about succinctness of syntax... that sequence is likely much shorter than any syntax I've ever seen in C for setting up file descriptors. I've interfaced with dialog(1) from within C and getting the data back from it is non-trivial. ASIDE: You may be asking yourself... why on Earth would you fork-exec an instance of dialog(1) from within C instead of using dialog(3)? There are some rare cases when this is advantageous. Usually when you want to support a drop- in replacement (like Xdialog(1)) for a long-running stateful widget (e.g. gauge). Usually (=99% of the time) a C program _should_ use dialog(3) but in those rare cases ( see http://druidbsd.cvs.sf.net/viewvc/druidbsd/fdpv/ ) thank goodness I haven't had to mangle the file descriptors to get back any data (I call out to gauge and just pump it massive amounts of data via stdin through a pipe passed to the posix_spawn(3) call -- no data to read back. So I don't see the above syntax as anything to point out -- C programs have more complicated setup for doing the same thing (they just have to do it less often). Oh... and in bsdconfig, you'll notice I don't do that. Upon initialization I clone the *real* terminal fd's _once_ and then use variables to reference the dup'd fd. These make it harder to do things (aka, more code to accomplish a task, more code to be maintained, more difficult to modify) than the higher level Perubython languages. Only because those languages have APIs that handle those things. Which I might add that those APIs have to be maintained. In that spirit, bsdconfig(8) provides the API for shell -- and similarly, said API (like most APIs) will be need much less maintenance than the programs using said API (how often do we make changes to posix_spawn(3)? or how often does Perubython change an API call?). It's in that light that the bsdconfig(8) shell API is also treated just like any other API with respect to changes (I treat all the *.subr includes as though they are an ABI -- changes to function arguments must be well thought out). In any case, thanks for working on this. A functioning program in any language is better than a non-existent better one. Indeed
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 9, 2013, at 9:05 AM, Allan Jude wrote: On 2013-10-09 03:22, Matthias Gamsjager wrote: Hi, tried 10-8 iso in Virtualbox but after reboot I was looking at a bootloader with a nice '-' but nothing more. post setup 'gpart list' showed no entries at all. On Wed, Oct 9, 2013 at 5:49 AM, Allan Jude free...@allanjude.com mailto:free...@allanjude.com wrote: On 2013-10-07 15:59, Allan Jude wrote: Devin Teske and I have been working on a big patch to bsdinstall to implement installing on a ZFS pool. It supports both GPT and MBR, the 4k sector gnop trick, and optional GELI encryption. We would like to commit this in time for 10.0-BETA1 so it needs some testing to work out any obvious bugs before we send it off to re@ to get it committed. It includes a single configuration menu that allows you to select all of the required details, including which drives to use (gets details from camcontrol, also includes an inspection utility that presents the detailed output of camcontrol inquiry/identify, and gpart show), what ZFS RAID level to use (taking in to consideration the selected number of drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc. Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1 3. Remove the dialog asking if you wish to enable crash dumps, this feature has been combined into the regular 'services to enable' dialog and enabled by default You can browse the patches here: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/ I've built a bootonly.iso (10.0-ALPHA4) to make testing easier, available compressed (48 MB) or uncompressed (211 MB): http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso We look forward to your feedback We've made more improvements, including corporating most all of the feedback we've gotten so far Outstanding items: 1. Apply the changes to ipv6 config the way we did ipv4 2. improve disk identification (model info and serial # instead of one or the other) 3. Include a helpful message before the GELI step where you have to enter your password many times, the user will be less confused if it is explained why they have to enter their password 3 * number of disks times 4. Validate vdev type choice inside the vdev type menu, and warn the user if they have made an invalid selection, so they can add more disks or chance their selection, without having to try to start the installation first 5. Whatever else you guys find wrong tonight I generated new test images, and attached the patch (which got REALLY big when Devin Teske decided to fix all of the things: http://www.allanjude.com/bsd/zfsbootonly.2013-10-08.iso.xz http://www.allanjude.com/bsd/zfsbootonly.2013-10-08.iso -- Allan Jude ___ freebsd-current@freebsd.org mailto:freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org mailto:freebsd-current-unsubscr...@freebsd.org This was a variable scoping error made by Devin when he refactored some of the code into a subroutine. I have fixed it Ouch and apologies. It was too hard to see at 2AM after starting-in on a deeper pathos to the keymap patches (which I'm still working on). Allan alerted me to the issue but when I started digging-in, I realized much deeper changes were required. And then I passed out around 2:30AM (keyboard-faceplant). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 8, 2013, at 11:19 PM, Nathan Whitehorn wrote: On 10/09/13 01:13, Allan Jude wrote: On 2013-10-08 16:17, Nathan Whitehorn wrote: On 10/07/13 21:59, Allan Jude wrote: Devin Teske and I have been working on a big patch to bsdinstall to implement installing on a ZFS pool. It supports both GPT and MBR, the 4k sector gnop trick, and optional GELI encryption. We would like to commit this in time for 10.0-BETA1 so it needs some testing to work out any obvious bugs before we send it off to re@ to get it committed. It includes a single configuration menu that allows you to select all of the required details, including which drives to use (gets details from camcontrol, also includes an inspection utility that presents the detailed output of camcontrol inquiry/identify, and gpart show), what ZFS RAID level to use (taking in to consideration the selected number of drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc. Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1 3. Remove the dialog asking if you wish to enable crash dumps, this feature has been combined into the regular 'services to enable' dialog and enabled by default You can browse the patches here: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/ I've built a bootonly.iso (10.0-ALPHA4) to make testing easier, available compressed (48 MB) or uncompressed (211 MB): http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso We look forward to your feedback Thanks for doing this! I had a few comments: 1. ZFS is not bootable on all architectures. Could you adjust that menu item to only display for i386, amd64, and (I think?) sparc64. Use uname -m, not -p, for this. I had not considered that, I'll make that change 1a. The script is broken on sparc64 in any case, which uses VTOC8 instead of GPT. I'll disable sparc64 as well 2. Why are you using camcontrol? That is guaranteed not to work on non-CAM systems. You should use the GEOM ident string if you need an ID. The GEOM ident string doesn't do enough to help the user identify which drive is which. More data is not exposed anywhere that I could find What we really need, is dev.ada.0.desc% like we have for network interfaces and a slew of other devices. GEOM data is great, but it is not exposed in a shell friendly way any place that I could find, other than the sysctl with DOT and XML data. This is one of the reasons the partition editor is written in C. There are a few other odd corner-cases where C is much more powerful than the command-line for the GEOM operations that partedit needs to do. I'm not sure how to usefully get it just from the shell. You can see how to do it in C in the boot_disk() routine of partedit/auto_part.c. 3. Any plans to integrate this into the regular partition editor? ZFS support is important enough that I will definitely not get in the way, even as a bolt-on, but it would be a shame for it to stay that way. The editor is also designed for ZFS to be added. I am a sysadmin, not a programmer. I can't write C. Most people deploying servers can't write C. I agree with Devin Teske, if everything was in shell it would be a lot more usable for non-developers, who probably make up the majority of people who deploy FreeBSD. There are some cases the other way too. Devin is probably the most shell-proficient FreeBSD committer. Well, there's jilles ;D (he writes/maintains the sh(1) implementation itself). I certainly can't write shell scripts at that level, which means that for me bsdconfig, for example, is effectively read-only (and quite hard to read as well). In the past few days of working on the bsdinstall_zfs patch-set with Allan Jude, I learned something new actually. It seems that sh(1) doesn't suffer so much from this read only concept. Imagine you're starting a new C project on an operating system that has all new syscalls and all new APIs that you've never seen. Would be pretty hard, no doubt. Now imagine that you're thrown a life-line called POSIX and hey, now you're slinging code in MinGW on Windows when you don't know a lick of M$ syscalls or APIs. In a similar manner, I've witnessed functionality be added that is truly functional *without* using any of the API calls. Then I come in and do a round of optimizations to leverage the existing API. Shell is kinda like that... I'm noticing Allan Jude doesn't know all the API calls yet (who could? other than me of course -- and even I have trouble remembering them _all_) yet this doesn't phase him (or others) from jumping in and lending a hand. No different than C, but the read-only aspect is lessened significantly I believe because there are so many people out there that know enough good shell syntax and are
BE Loader Menu (was Re: rcs)
On Oct 9, 2013, at 9:42 AM, Kris Moore wrote: On 10/09/2013 12:33, Allan Jude wrote: On 2013-10-09 12:23, Freddie Cash wrote: On Wed, Oct 9, 2013 at 9:12 AM, Freddie Cash fjwc...@gmail.com wrote: On Wed, Oct 9, 2013 at 6:41 AM, Julian Elischer jul...@freebsd.orgwrote: On 10/9/13 2:35 AM, Freddie Cash wrote: On Tue, Oct 8, 2013 at 9:09 AM, Alfred Perlstein bri...@mu.org wrote: You're right on the money, to be honest this is one of the reasons why I've switched to using OSX as my desktop OS. zsh, vim, screen by default. and upgrades work. At the end of the day I'm spending time doing work, not mucking about my workspace to make it usable for development. I think this was brought up at BSDCan in the discussion about making FreeBSD a more featured development platform. Speaking of... has anyone tried PCBSD? PC-BSD isn't much different from FreeBSD. The installer is GUI and support ZFS, there are some GUI setup tools on first boot for X, there are some GUI tools to select binary drivers for X, and there ​​are working pkgng repos available. I had a lot of issues with PC-BSD 9.0 and 9.1 as I was trying to do things the FreeBSD way which broke a lot of things that were done the PC-BSD way (aka don't manually edit config files used for booting). ​Switching to the rolling-release (aka PC-BSD 9-STABLE) and moving all my config file edits into filename.conf.local fixed my issues. Things have been running smooth, and I finally understand the beauty and simplicity of freebsd-update + pkg. OS gets updated once per month, packages get updated twice per month, no more compiling things from source. It's like using Ubuntu/Debian but with the power and features of FreeBSD. :) ​ When they went to a ZFS-only system, using GRUB, with no alternative, then I'm afraid they lost me. I want a root filesystem on UFS for reliabailty and simpleness. I can debug it's media if needed. Before then I really liked it (though ther eis not enough information on how it works interneally if you want to use it. hopefully that will come.. and I LIKE PBIs FreeBSD should adopt PBIs for sure. With PBIs you could make even quite base items separately installable. versioning problems go away. There's no GRUB in a default install of PC-BSD 9.0, 9.1, or 9.2. Even on a ZFS-only setup (which is what I run). It's using the FreeBSD loader, with custom artwork and menus. ​Hrm, it seems they've changed things with the 9.2 installer. It does use GRUB2 (e!) for the boot loader, and integrates support for ZFS boot environments (via beadm) into it. :( Shame they didn't use the BE support in the FreeBSD loader for this. Wonder if my 9-STABLE-based PC-BSD install will get upgraded to GRUB? The reason they went to grub2, is that the way the freebsd loader menus work, it loads the kernel before it draws the menu. This means if there is a problem with your kernel (probably the most valuable time to have boot environments) then the menu never comes up, and you cannot select which BE to boot from. Grub doesn't rely on a FreeBSD kernel until after you select which BE to boot from. Kris and I discussed it at length with Devin Teske, and while he has demonstrated being able to populate a lower menu with the ZFS datasets, I am not sure if the other issue can be resolved. I'm late to the party again ;D (didn't realize the rcs thread had turned BE) Both problems can be solved. The loading of the kernel *after* choosing your boot device is trivial. We've been doing it at $work for *years* (almost a decade?) I can put that in, whenever. Probably at the same time as implementing the live/dynamic BE menus for selecting the root device. Yea, GRUB is not my first choice, but ATM this is the cleanest way we can do ZFS BE's. However if you don't like ZFS / GRUB, you can always use regular FreeBSD and just grab our toolchain from sysutils/pcbsd-utils* in ports or use our PKGNG repo: http://wiki.pcbsd.org/index.php/Turn_FreeBSD_into_PC-BSD%C2%AE I'm planning on using GRUB to do UEFI booting for 10.0 as well. But when the FreeBSD loader matures to the point of having support for all these features, I'll gladly move us back. I had not been pushing hard on finishing the BE Forth because I knew that it would be a greater value add if we actually had Boot from ZFS finished first. Would be easier to test, that is, if I actually had a system booting from ZFS -- which is what Allan, jmg, and myself are hopefully working toward for FreeBSD 10.0-R. (smiles) -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 8, 2013, at 8:49 PM, Allan Jude wrote: On 2013-10-07 15:59, Allan Jude wrote: Devin Teske and I have been working on a big patch to bsdinstall to implement installing on a ZFS pool. It supports both GPT and MBR, the 4k sector gnop trick, and optional GELI encryption. We would like to commit this in time for 10.0-BETA1 so it needs some testing to work out any obvious bugs before we send it off to re@ to get it committed. It includes a single configuration menu that allows you to select all of the required details, including which drives to use (gets details from camcontrol, also includes an inspection utility that presents the detailed output of camcontrol inquiry/identify, and gpart show), what ZFS RAID level to use (taking in to consideration the selected number of drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc. Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1 3. Remove the dialog asking if you wish to enable crash dumps, this feature has been combined into the regular 'services to enable' dialog and enabled by default You can browse the patches here: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/ I've built a bootonly.iso (10.0-ALPHA4) to make testing easier, available compressed (48 MB) or uncompressed (211 MB): http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso We look forward to your feedback We've made more improvements, including corporating most all of the feedback we've gotten so far Outstanding items: 1. Apply the changes to ipv6 config the way we did ipv4 2. improve disk identification (model info and serial # instead of one or the other) 3. Include a helpful message before the GELI step where you have to enter your password many times, the user will be less confused if it is explained why they have to enter their password 3 * number of disks times I'm hopeful that we can script the application of a password that we first prompt for. What tool is prompting for a password? Can we not just provide an answer on stdin? (e.g., echo $pass | tool_that_needs_pass) 4. Validate vdev type choice inside the vdev type menu, and warn the user if they have made an invalid selection, so they can add more disks or chance their selection, without having to try to start the installation first This will be done with fanciness ;D (read: ... --and-widget --infobox ... and sundry smartness; retaining as much as possible the ability to do things out of order but never arise at a point of astonishment). 5. Whatever else you guys find wrong tonight I generated new test images, and attached the patch (which got REALLY big when Devin Teske decided to fix all of the things: And then I merged all of the things into HEAD, so the patch-set shrunk back to its normal size. Now we have global exit codes which will make merging of code that is based off of Thomas Dickey's samples easier. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 9, 2013, at 10:47 AM, Allan Jude wrote: On 2013-10-09 13:21, Teske, Devin wrote: On Oct 8, 2013, at 8:49 PM, Allan Jude wrote: On 2013-10-07 15:59, Allan Jude wrote: Devin Teske and I have been working on a big patch to bsdinstall to implement installing on a ZFS pool. It supports both GPT and MBR, the 4k sector gnop trick, and optional GELI encryption. We would like to commit this in time for 10.0-BETA1 so it needs some testing to work out any obvious bugs before we send it off to re@ to get it committed. It includes a single configuration menu that allows you to select all of the required details, including which drives to use (gets details from camcontrol, also includes an inspection utility that presents the detailed output of camcontrol inquiry/identify, and gpart show), what ZFS RAID level to use (taking in to consideration the selected number of drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc. Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1 3. Remove the dialog asking if you wish to enable crash dumps, this feature has been combined into the regular 'services to enable' dialog and enabled by default You can browse the patches here: https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=oPEdrpFUFqEutq%2B7DHGyBIA0gTt5%2BZ6FWvbN6q4bjm4%3D%0As=32f69a33d9c98c63ecf3962b89eea0af631f5a2fd6e82b55afb6ec139f3f803d I've built a bootonly.iso (10.0-ALPHA4) to make testing easier, available compressed (48 MB) or uncompressed (211 MB): https://urldefense.proofpoint.com/v1/url?u=http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xzk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=oPEdrpFUFqEutq%2B7DHGyBIA0gTt5%2BZ6FWvbN6q4bjm4%3D%0As=fd0f3ed623f12c8cb5eee607cf82122523a0800ab9f5624cb3f77ae2646e1b1b https://urldefense.proofpoint.com/v1/url?u=http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.isok=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0Am=oPEdrpFUFqEutq%2B7DHGyBIA0gTt5%2BZ6FWvbN6q4bjm4%3D%0As=fa49aaa528513a591ac40a2f0c5666abae2d7269bd66ac70d7100475a9574300 We look forward to your feedback We've made more improvements, including corporating most all of the feedback we've gotten so far Outstanding items: 1. Apply the changes to ipv6 config the way we did ipv4 2. improve disk identification (model info and serial # instead of one or the other) 3. Include a helpful message before the GELI step where you have to enter your password many times, the user will be less confused if it is explained why they have to enter their password 3 * number of disks times I'm hopeful that we can script the application of a password that we first prompt for. What tool is prompting for a password? Can we not just provide an answer on stdin? (e.g., echo $pass | tool_that_needs_pass) It is 'geli create' and 'geli attach'. I am not sure if we want to have the password show up in the process list (obviously in the installer this is less of an issue, but) It won't. echo is a shell built-in. If we're uber paranoid... we prefix the word builtin; e.g.,... builtin echo $pass | tool_that_needs_pass You'll see the tool in ps, but you won't see the echo (that's part of the shell invocation -- all you see in ps is an instance of sh). 4. Validate vdev type choice inside the vdev type menu, and warn the user if they have made an invalid selection, so they can add more disks or chance their selection, without having to try to start the installation first This will be done with fanciness ;D (read: ... --and-widget --infobox ... and sundry smartness; retaining as much as possible the ability to do things out of order but never arise at a point of astonishment). I don't think we need --and-widget, just in the function where we apply the results of the menu selection, The purpose of --and-widget with an --infobox is to let the user know that validation is occurring each/every time they make a selection. Seeing the infobox before being returned to the previous menu (in the case of selecting a valid vdev_type) is to cement in the mind of the user that their selection was validated. Of course, in the case of an invalid selection, they get a message box. What the message box says depends on: 1. Are they trying to select a vdev_type for which they don't have enough devices? Tell them that they don't have enough devices, and bring them back to the vdev_type menu (to select a different [valid] vdev type *or* cancel and go back to the ZFS menu where they can optionally Rescan for more devices -- allowing that vdev_type to be selected without issue). In the prompt
Re: rcs
On Oct 9, 2013, at 2:48 PM, Adrian Chadd wrote: On 9 October 2013 11:28, Bruce Cran br...@cran.org.uk wrote: On 10/9/2013 3:17 PM, Alfred Perlstein wrote: Only a few years ago you could take a dvd or memstick of FreeBSD and have 1000s of packages to choose from during your install. That is broken now? At some point it was decided that the installer should be as simple as possible and package installation was a post-install task, for once the system was up and running. I think that might be changing with bsdinstall getting pkgng support, but it did seem like a fairly major regression at the time. That wasn't the rationale. sysinstall was broken and more people wanted a replacement than wanted to see it fixed. bsdinstall is simply coming of age still (sysinstall had a 15 year run). I think it was more a this is a stepping stone to a bigger change.. .. and that didn't happen soon after. Adrian is right. It was a first step (nobody intended to see functionality get dropped -- it's a matter of finding the time to replace a 15-year code line -- sysinstall). Only a very small handful of us are actively working on it. It's further along than you think. You can always help out by testing. The other thing here is the introduction of the uhm, livefs. It's big and it took up the space once occupied by the first set of packages. Honestly, I'm kinda surprised how big base is these days, but I have other fights to fight right now. The livefs is a bee in my bonnet too. But as you say... more pressing things to worry about right now than break something that is working currently. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 9, 2013, at 8:11 PM, Allan Jude wrote: On 2013-10-09 14:14, Teske, Devin wrote: On Oct 9, 2013, at 10:47 AM, Allan Jude wrote: On 2013-10-09 13:21, Teske, Devin wrote: On Oct 8, 2013, at 8:49 PM, Allan Jude wrote: On 2013-10-07 15:59, Allan Jude wrote: Devin Teske and I have been working on a big patch to bsdinstall to implement installing on a ZFS pool. It supports both GPT and MBR, the 4k sector gnop trick, and optional GELI encryption. We would like to commit this in time for 10.0-BETA1 so it needs some testing to work out any obvious bugs before we send it off to re@ to get it committed. It includes a single configuration menu that allows you to select all of the required details, including which drives to use (gets details from camcontrol, also includes an inspection utility that presents the detailed output of camcontrol inquiry/identify, and gpart show), what ZFS RAID level to use (taking in to consideration the selected number of drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc. Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1 3. Remove the dialog asking if you wish to enable crash dumps, this feature has been combined into the regular 'services to enable' dialog and enabled by default You can browse the patches here: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/ I've built a bootonly.iso (10.0-ALPHA4) to make testing easier, available compressed (48 MB) or uncompressed (211 MB): http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso We look forward to your feedback We've made more improvements, including corporating most all of the feedback we've gotten so far Outstanding items: 1. Apply the changes to ipv6 config the way we did ipv4 2. improve disk identification (model info and serial # instead of one or the other) 3. Include a helpful message before the GELI step where you have to enter your password many times, the user will be less confused if it is explained why they have to enter their password 3 * number of disks times I'm hopeful that we can script the application of a password that we first prompt for. What tool is prompting for a password? Can we not just provide an answer on stdin? (e.g., echo $pass | tool_that_needs_pass) It is 'geli create' and 'geli attach'. I am not sure if we want to have the password show up in the process list (obviously in the installer this is less of an issue, but) It won't. echo is a shell built-in. If we're uber paranoid... we prefix the word builtin; e.g.,... builtin echo $pass | tool_that_needs_pass You'll see the tool in ps, but you won't see the echo (that's part of the shell invocation -- all you see in ps is an instance of sh). I have implemented this, 'geli init' takes the -J file flag, which can be set to - to mean stdin. 'geli attach' is the same except the flag is -j I first prompt for the password with a dialog --passwordbox, so the user only has to enter it once Do we think it makes sense to prompt the user twice, and confirm the two entered passphrases are the same? I think so. I'm afraid someone could make a typo and then not be able to log into their system because the password that was actually used is different than what the user thinks it is. 4. Validate vdev type choice inside the vdev type menu, and warn the user if they have made an invalid selection, so they can add more disks or chance their selection, without having to try to start the installation first This will be done with fanciness ;D (read: ... --and-widget --infobox ... and sundry smartness; retaining as much as possible the ability to do things out of order but never arise at a point of astonishment). I don't think we need --and-widget, just in the function where we apply the results of the menu selection, The purpose of --and-widget with an --infobox is to let the user know that validation is occurring each/every time they make a selection. Seeing the infobox before being returned to the previous menu (in the case of selecting a valid vdev_type) is to cement in the mind of the user that their selection was validated. Of course, in the case of an invalid selection, they get a message box. What the message box says depends on: 1. Are they trying to select a vdev_type for which they don't have enough devices? Tell them that they don't have enough devices, and bring them back to the vdev_type menu (to select a different [valid] vdev type *or* cancel and go back to the ZFS menu where they can optionally Rescan for more devices -- allowing that vdev_type to be selected without issue). In the prompt that tells them that they don't have enough disks in their system to select that vdev_type
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 7, 2013, at 10:48 PM, Allan Jude wrote: On 2013-10-08 01:11, Teske, Devin wrote: On Oct 7, 2013, at 10:07 PM, Allan Jude wrote: On 2013-10-07 15:59, Allan Jude wrote: Devin Teske and I have been working on a big patch to bsdinstall to implement installing on a ZFS pool. It supports both GPT and MBR, the 4k sector gnop trick, and optional GELI encryption. We would like to commit this in time for 10.0-BETA1 so it needs some testing to work out any obvious bugs before we send it off to re@ to get it committed. It includes a single configuration menu that allows you to select all of the required details, including which drives to use (gets details from camcontrol, also includes an inspection utility that presents the detailed output of camcontrol inquiry/identify, and gpart show), what ZFS RAID level to use (taking in to consideration the selected number of drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc. Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1 3. Remove the dialog asking if you wish to enable crash dumps, this feature has been combined into the regular 'services to enable' dialog and enabled by default You can browse the patches here: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/ I've built a bootonly.iso (10.0-ALPHA4) to make testing easier, available compressed (48 MB) or uncompressed (211 MB): http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso We look forward to your feedback I've generated a new version of the ISO incorporating all of todays changes and moving up to 10.0-ALPHA5 It includes 2 patches from Warren Block, improving the keymap menu and indicating which network interfaces are wireless Excellent. However, I have to say... you opened a can of worms by touching keymap. Eventually I plan to rewrite all of the scripts to the same format that zfsboot is using. I've already started rewriting keymap to the new format. Of course, this opened up another can of worms... the simple things like: DIALOG_OK DIALOG_CANCEL DIALOG_ESC Yeah, after having them, I felt a bit naked not having all of those 'constants' defined. The constants are now where they belong... `dialog.subr' I also went through bsdconfig(8) with a fine-tooth comb and made use of the new constants everywhere. I think it's a definite improvement. Originally, all I had done was add --defaultno to the dialog command, but Warren's patch makes a lot of sense, allow the user to 'try' the new keymap before trying to do the rest of the install based on it. I rewrote warren's code into the stateful design and cleaned it up. I feel would be much better off in the `dialog.subr' module. So,... I've started peppering their usage everywhere in bsdconfig to make them proper. That means they will just be transparent from including `dialog.subr'. This is definately something that I think is a good idea, but my focus is on improving the usability and functionality of the installer in time for 10.0. Unifying everything to the bsdconfig style is slightly lower priority. Admittedly, the `bsdconfig networking` stuff is quite nice *nods* I took a look at the bsdinstall networking stuff, and I wanted to run away. Let's push that into the future. It wasn't too hard to get those global exit codes deployed, so I did that, but like you say... I agree we should minimize extra work that will be eventually slated for a future release. -- Devin http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso Excellent, thanks! -- Allan Jude ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 8, 2013, at 1:17 PM, Nathan Whitehorn wrote: On 10/07/13 21:59, Allan Jude wrote: Devin Teske and I have been working on a big patch to bsdinstall to implement installing on a ZFS pool. It supports both GPT and MBR, the 4k sector gnop trick, and optional GELI encryption. We would like to commit this in time for 10.0-BETA1 so it needs some testing to work out any obvious bugs before we send it off to re@ to get it committed. It includes a single configuration menu that allows you to select all of the required details, including which drives to use (gets details from camcontrol, also includes an inspection utility that presents the detailed output of camcontrol inquiry/identify, and gpart show), what ZFS RAID level to use (taking in to consideration the selected number of drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc. Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1 3. Remove the dialog asking if you wish to enable crash dumps, this feature has been combined into the regular 'services to enable' dialog and enabled by default You can browse the patches here: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/ I've built a bootonly.iso (10.0-ALPHA4) to make testing easier, available compressed (48 MB) or uncompressed (211 MB): http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso We look forward to your feedback Thanks for doing this! I had a few comments: 1. ZFS is not bootable on all architectures. Could you adjust that menu item to only display for i386, amd64, and (I think?) sparc64. Use uname -m, not -p, for this. I think we can do that no problem. 1a. The script is broken on sparc64 in any case, which uses VTOC8 instead of GPT. *nods* we'll have to purloin your VTOC8 code before we offer it to sparc64. 2. Why are you using camcontrol? That is guaranteed not to work on non-CAM systems. You should use the GEOM ident string if you need an ID. I imagine we could get the info from many places, and to be honest, I never knew about diskinfo until last week (but apparently it's been around since 5.x days). The one place where I suspect camcontrol(8) usage is appropriate is the mini disk inspector dialog. The camcontrol inquiry output is specifically helpful if/when you're doing multipathing (and you need to identify which da# devices are duplicates of the same device -- given Serial#). But I gather you mean for disk description in the device menu (for picking text to go along with the device name). I'm open to using a different tool... or multiple tools. Do you think it would be better to just stick to one tool? or to query a few? (which ones win-out?) 3. Any plans to integrate this into the regular partition editor? ZFS support is important enough that I will definitely not get in the way, even as a bolt-on, but it would be a shame for it to stay that way. The editor is also designed for ZFS to be added. Yes and yes. (and didn't know the editor was designed for ZFS too) 4. What is this gnop stuff for? Thanks goes to Freddie Cash for his excellent explanation (I had to admit, I didn't understand it at that level) 5. I think some substantial part of the MBR code will blow up if you are reinitalizing a previously formatted disk (the bsdlabel will be retasted and come back from the dead). Will not some combination of gpart destroy -F and (insert suggestion) work? -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 8, 2013, at 2:28 PM, Nathan Whitehorn wrote: On 10/08/13 23:21, Kurt Lidl wrote: On 10/8/2013, Nathan Whitehorn wrote: On 10/07/13 21:59, Allan Jude wrote: Devin Teske and I have been working on a big patch to bsdinstall to implement installing on a ZFS pool. It supports both GPT and MBR, the 4k sector gnop trick, and optional GELI encryption. We would like to commit this in time for 10.0-BETA1 so it needs some testing to work out any obvious bugs before we send it off to re@ to get it committed. It includes a single configuration menu that allows you to select all of the required details, including which drives to use (gets details from camcontrol, also includes an inspection utility that presents the detailed output of camcontrol inquiry/identify, and gpart show), what ZFS RAID level to use (taking in to consideration the selected number of drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc. Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1 3. Remove the dialog asking if you wish to enable crash dumps, this feature has been combined into the regular 'services to enable' dialog and enabled by default You can browse the patches here: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/ I've built a bootonly.iso (10.0-ALPHA4) to make testing easier, available compressed (48 MB) or uncompressed (211 MB): http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso We look forward to your feedback Thanks for doing this! I had a few comments: 1. ZFS is not bootable on all architectures. Could you adjust that menu item to only display for i386, amd64, and (I think?) sparc64. Use uname -m, not -p, for this. 1a. The script is broken on sparc64 in any case, which uses VTOC8 instead of GPT. 2. Why are you using camcontrol? That is guaranteed not to work on non-CAM systems. You should use the GEOM ident string if you need an ID. 3. Any plans to integrate this into the regular partition editor? ZFS support is important enough that I will definitely not get in the way, even as a bolt-on, but it would be a shame for it to stay that way. The editor is also designed for ZFS to be added. 4. What is this gnop stuff for? 5. I think some substantial part of the MBR code will blow up if you are reinitalizing a previously formatted disk (the bsdlabel will be retasted and come back from the dead). -Nathan I wrote some support for adding ZFS to the partition editor a couple of months ago, around the time that 9.2 was branched. One of the biggest things that I have not done is integrate setting up a zfs mirror between any disks before creating the zpool. Nor did I do the gnop hack to create 4K disk blocks before creating the pool. I more or less changed the partedit program to take an argument when invoked with ufs (same as current behavior) or zfs, which knows about zfs setup stuff. The hookup to the actual scripts was, shall we say, much less intrusive than what has been published here. I guess it's time to dig out the patches and make them available to others. That would be great! The original idea was to have a zfs_ops.c to go with gpart_ops.c and have some fields in the disk item struct that said which to use. Not sure if that's ultimately workable, though. I can hack on it once the patches are there in any case. One of the options presented was to integrate ZFS knowledge into the partedit module. However, another approach that is not being spoken about is in the other direction: rewriting partedit to be in sh(1). What I'd like to see is the --treeview widget of dialog(1) fixed before such a transition is made, so the former option may be the best bet for now. For anyone that hasn't run the treeview or treeview2 samples in head/contrib/dialog/samples ... it's not anything we'd want to use to replace dialog(3) with. I've seen much nicer treeview implementations than what is in [c]dialog (currently we're using dialog-1.2-20130925 and the treeview for some reason incorporates a radio-button column which we do _not_ want for a disk partition layout menu). Of course, right now you're all looking at me like I'm crazy. Like a fox. There are real tangible benefits to rewriting the C code of partedit into shell syntax. The value-add is not just perceived. Some of the values obtained are: + Conflated namespace with singular API rather than having to learn how to script each individual blocking-utility which implements a unique method for scripting its input. + Top-down integration of new functionality. We're integrating ZFS right now and that's great that we have a fully-functional libzfs et. al. for doing this stuff in C, but usually the command-line utility functionality precedes programatic
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 7, 2013, at 2:47 PM, Allan Jude wrote: On 2013-10-07 17:00, Outback Dingo wrote: On Mon, Oct 7, 2013 at 4:54 PM, Allan Jude free...@allanjude.com wrote: On 2013-10-07 16:43, Warren Block wrote: On Mon, 7 Oct 2013, Allan Jude wrote: Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no Please see http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/162175 . ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org That is a good idea, I'll add that I've also hit this: http://lists.freebsd.org/pipermail/freebsd-sysinstall/2013-September/000949.html and plan to throw in a fix for that. Which makes more sense, blindly 'killall dhclient' before we try to acquire a new lease, or detect that dhclient is running and try to use the IP that has already been assigned? jeez after i just updated and patched my source tree to do a build, you go and fix things :P anyway you can just generate a master diff against the tree that we can apply, seems not ive to revert the patches and reapply all the newest ones. -- Allan Jude ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org Attached find the patch from last night against a fresh HEAD I am working on incorporating Warren's patch now ... and I'm working to review and validate what Allan is writing ;D A couple of big patches to incorporate within the hour (next 15 minutes). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 7, 2013, at 3:21 PM, Outback Dingo wrote: On Mon, Oct 7, 2013 at 5:49 PM, Teske, Devin devin.te...@fisglobal.com wrote: On Oct 7, 2013, at 2:47 PM, Allan Jude wrote: On 2013-10-07 17:00, Outback Dingo wrote: On Mon, Oct 7, 2013 at 4:54 PM, Allan Jude free...@allanjude.com wrote: On 2013-10-07 16:43, Warren Block wrote: On Mon, 7 Oct 2013, Allan Jude wrote: Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no Please see http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/162175 . ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org That is a good idea, I'll add that I've also hit this: http://lists.freebsd.org/pipermail/freebsd-sysinstall/2013-September/000949.html and plan to throw in a fix for that. Which makes more sense, blindly 'killall dhclient' before we try to acquire a new lease, or detect that dhclient is running and try to use the IP that has already been assigned? jeez after i just updated and patched my source tree to do a build, you go and fix things :P anyway you can just generate a master diff against the tree that we can apply, seems not ive to revert the patches and reapply all the newest ones. -- Allan Jude ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org Attached find the patch from last night against a fresh HEAD I am working on incorporating Warren's patch now ... and I'm working to review and validate what Allan is writing ;D A couple of big patches to incorporate within the hour (next 15 minutes). okies dokies guess ill revert it all and wait till you guys give the green light to try again Thanks... you know... Allan is giving me a work-out. He's running like the wind and I'm trying to catch my breath as he I trail behind him. Go Allan Go! ;D -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 7, 2013, at 5:18 PM, Warren Block wrote: On Mon, 7 Oct 2013, Allan Jude wrote: On 2013-10-07 16:43, Warren Block wrote: On Mon, 7 Oct 2013, Allan Jude wrote: Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no Please see http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/162175 . That is a good idea, I'll add that I've also hit this: http://lists.freebsd.org/pipermail/freebsd-sysinstall/2013-September/000949.html and plan to throw in a fix for that. Which makes more sense, blindly 'killall dhclient' before we try to acquire a new lease, or detect that dhclient is running and try to use the IP that has already been assigned? killall seems all right. In fact, internally, it's going to do that check. If nothing named dhclient is running, it will return immediately. Although on 10.0, dhclient does not want to die sometimes. Here's another PR for usability: http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/161547 Hope you guys aren't rewriting bsdconfig's network.subr -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 7, 2013, at 5:31 PM, Allan Jude wrote: On 2013-10-07 20:21, Teske, Devin wrote: On Oct 7, 2013, at 5:18 PM, Warren Block wrote: On Mon, 7 Oct 2013, Allan Jude wrote: On 2013-10-07 16:43, Warren Block wrote: On Mon, 7 Oct 2013, Allan Jude wrote: Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no Please see http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/162175 . That is a good idea, I'll add that I've also hit this: http://lists.freebsd.org/pipermail/freebsd-sysinstall/2013-September/000949.html and plan to throw in a fix for that. Which makes more sense, blindly 'killall dhclient' before we try to acquire a new lease, or detect that dhclient is running and try to use the IP that has already been assigned? killall seems all right. In fact, internally, it's going to do that check. If nothing named dhclient is running, it will return immediately. Although on 10.0, dhclient does not want to die sometimes. Here's another PR for usability: http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/161547 Hope you guys aren't rewriting bsdconfig's network.subr These changes are in usr.sbin/bsdinstall/scripts/network and usr.sbin/bsdinstall/scripts/network_ipv4 Correct, but I meant reinvent actually. There's an enormous API for dealing with the network. /usr/share/bsdconfig/media/tcpip.subr Is for acquiring/maintaining active TCP connections Meanwhile,... /usr/share/bsdconfig/network/*.subr Is for configuring permanent network settings that stick after a reboot. I've not touched bsdconfig's network stuff Probably worth a look. It can probably replace bsdinstall's netconfig. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI
On Oct 7, 2013, at 10:07 PM, Allan Jude wrote: On 2013-10-07 15:59, Allan Jude wrote: Devin Teske and I have been working on a big patch to bsdinstall to implement installing on a ZFS pool. It supports both GPT and MBR, the 4k sector gnop trick, and optional GELI encryption. We would like to commit this in time for 10.0-BETA1 so it needs some testing to work out any obvious bugs before we send it off to re@ to get it committed. It includes a single configuration menu that allows you to select all of the required details, including which drives to use (gets details from camcontrol, also includes an inspection utility that presents the detailed output of camcontrol inquiry/identify, and gpart show), what ZFS RAID level to use (taking in to consideration the selected number of drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc. Additional, it includes some other changes to bsdinstall: 1. Change the default to the 'non-standard keyboard mapping' prompt to no 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1 3. Remove the dialog asking if you wish to enable crash dumps, this feature has been combined into the regular 'services to enable' dialog and enabled by default You can browse the patches here: http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/ I've built a bootonly.iso (10.0-ALPHA4) to make testing easier, available compressed (48 MB) or uncompressed (211 MB): http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso We look forward to your feedback I've generated a new version of the ISO incorporating all of todays changes and moving up to 10.0-ALPHA5 It includes 2 patches from Warren Block, improving the keymap menu and indicating which network interfaces are wireless Excellent. However, I have to say... you opened a can of worms by touching keymap. Eventually I plan to rewrite all of the scripts to the same format that zfsboot is using. I've already started rewriting keymap to the new format. Of course, this opened up another can of worms... the simple things like: DIALOG_OK DIALOG_CANCEL DIALOG_ESC I feel would be much better off in the `dialog.subr' module. So,... I've started peppering their usage everywhere in bsdconfig to make them proper. That means they will just be transparent from including `dialog.subr'. http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso Excellent, thanks! -- Devin P.S. I have to get apache installed on one of my jails to server you the video ;) I'll have a URL for you tomorrow morning. Trying to work through all the changes in keymap to get it upgraded. _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: r253070 and disappearing zpool
On Jul 22, 2013, at 12:29 AM, Andriy Gapon wrote: [turning this into a public discussion with Glen's permission] on 10/07/2013 21:05 Glen Barber said the following: Hi, My setup is like this: root@nucleus:/usr/src # zpool list NAME SIZE ALLOC FREECAP DEDUP HEALTH ALTROOT zboot0 9.94G 379M 9.57G 3% 1.00x ONLINE - zroot0 159G 113G 46.2G70% 1.00x ONLINE - root@nucleus:/usr/src # zpool get bootfs NAMEPROPERTY VALUE SOURCE zboot0 bootfs- default zroot0 bootfs- default root@nucleus:/usr/src # zfs list zboot0 NAME USED AVAIL REFER MOUNTPOINT zboot0 379M 9.41G 281M /bootdir root@nucleus:/usr/src # zfs list zroot0 NAME USED AVAIL REFER MOUNTPOINT zroot0 113G 43.7G 147M / 'zroot0' is a GELI-backed pool, so I have this to fix the boot process: root@nucleus:/usr/src # ll /boot lrwxr-xr-x 1 root wheel 12 Aug 25 2012 /boot@ - bootdir/boot I upgraded from head/ on July 1 to r253159, and when I rebooted the system, I could correctly boot from the /bootdir/boot. Once I enter the GELI passphrase, / (from zroot0) is mounted. Normally, everything would be okay at that point, but since the upgrade, /bootdir/boot disappears because the zboot0 pool is not imported as it was before. Any thoughts? I think that this setup (on ZFS level) is quite untypical, although not impossible on FreeBSD (and perhaps only FreeBSD). It's untypical because you have separate boot pool (where loader, loader.conf and kernel are taken from) and root pool (where / is mounted from). There is this magic zpool.cache file that essentially tells what pools should be automatically imported. On FreeBSD this file lives in /boot/zfs directory while originally (in Solaris and its descendants) its home is /etc/zfs. Until recently FreeBSD could use only zpool.cache from a boot pool and, in fact, if a root pool was different from a boot pool the root pool had to be in zpool.cache. I changed things a little bit and now a root pool does not have to be in zpool.cache. Also, now zpool.cache is taken from the root pool, or to be more precise from a root filesystem (whatever happens to be /boot/zfs/zpool.cache after / is mounted). I am considering if perhaps now we should move zpool.cache back to /etc/zfs/. So, I see three ways of resolving the problem that my changes caused for your configuration. 1. [the easiest] Put zpool.cache loading instructions that used to be in defaults/loader.conf into your loader.conf. This way everything should work as before -- zpool.cache would be loaded from your boot pool. 2. Somehow (I don't want to go into any technical details here) arrange that your root pool has /boot/zfs/zpool.cache that describes your boot pool. This is probably hard given that your /boot is a symlink at the moment. This probably would be easier to achieve if zpool.cache lived in /etc/zfs. 3. [my favorite] Remove an artificial difference between your boot and root pools, so that they are a single root+boot pool (as zfs gods intended). As far as I understand your setup, you use GELI to protect some sensitive data. Apparently your kernel is not sensitive data, so I wonder if your /bin/sh or /sbin/init are really sensitive either. So perhaps you can arrange your unencrypted pool to hold all of the base system (boot + root) and put all your truly sensitive filesystems (like e.g. /home or /var/data or /opt/xyz) onto your encrypted pool. ZFS is really flexible, you can use mountpoint and canmount properties to place your filesystems from same or different pools into whatever file namespace hierarchy you desire. Remember that your filesystem hierarchy in the mount namespace does not always have to be the same as your ZFS dataset hierarchy. I hope that this makes sense to you. If you have any additional questions, please do not hesitate. P.S. ZFS/FreeBSD boot process is extremely flexible. For example zfsboot can take zfsloader from pool1/fsA, zfsloader can boot kernel from pool2/fsB and kernel can mount / from pool3/fsC. Of these 3 filesystems from where should zpool.cache be taken? My firm opinion is that it should be taken from / (pool3/fsC in the example above). Because it is the root filesystem that defines what a system is going to do ultimately: what daemons are started, with what configurations, etc. And thus it should also determine what pools to auto-import. We can say that zpool.cache is analogous to /etc/fstab in this respect. So I understand that my change causes a problem for a setup like yours, but I believe that the change is correct. +1 for zpool.cache on / (pool3/fsC in last example) -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 14, 2013, at 11:23 PM, Baptiste Daroussin wrote: On Sun, Jul 14, 2013 at 10:12:19AM +, Teske, Devin wrote: On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote: On 14/07/2013 06:48, Teske, Devin wrote: Question: Where can I learn more about the actual format of what's in the new tarballs? This is going to be important not for bsdconfig, but $work (we have our own build platform; I'm going to have to rewrite it from mastering PLIST files to mastering YAML MANIFEST files and I want to know all the gritty details). We do need a pkg-manifest(5) man page, which can double as a pkg-tarball(5) page since the manifest file is where most of the interesting bits are. A pkg tarball is a compressed tar archive like so: lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz -rw-r--r-- 0 root wheel 530 Jan 1 1970 +COMPACT_MANIFEST -rw-r--r-- 0 root wheel6385 Jan 1 1970 +MANIFEST -rw-r--r-- 0 root wheel 17567 Jan 1 1970 +MTREE_DIRS -r--r--r-- 0 root wheel 19453 Jul 7 12:26 /usr/local/etc/bash_completion.d/_pkg.bash -r-xr-xr-x 0 root wheel 629 Jul 7 12:26 /usr/local/etc/periodic/daily/400.status-pkg -r-xr-xr-x 0 root wheel 823 Jul 7 12:26 /usr/local/etc/periodic/daily/411.pkg-backup -r-xr-xr-x 0 root wheel 678 Jul 7 12:26 /usr/local/etc/periodic/daily/490.status-pkg-changes -r-xr-xr-x 0 root wheel2558 Jul 7 12:26 /usr/local/etc/periodic/security/410.pkg-audit -r-xr-xr-x 0 root wheel 611 Jul 7 12:26 /usr/local/etc/periodic/security/460.pkg-checksum -r--r--r-- 0 root wheel 839 Jul 7 12:26 /usr/local/etc/pkg.conf.sample -r--r--r-- 0 root wheel 43432 Jul 7 12:26 /usr/local/include/pkg.h -r--r--r-- 0 root wheel 727558 Jul 7 12:26 /usr/local/lib/libpkg.a lrwxr-xr-x 0 root wheel 0 Jul 7 12:26 /usr/local/lib/libpkg.so - libpkg.so.1 -r--r--r-- 0 root wheel 1227064 Jul 7 12:26 /usr/local/lib/libpkg.so.1 -rw-r--r-- 0 root wheel 187 Jul 7 12:26 /usr/local/libdata/pkgconfig/pkg.pc [... etc ...] Interesting. I notice that (while looking ahead to see a prefix: of /usr/local in the +MANIFEST), the tarball itself has files that include /usr/local in their path. Does pkg handle packages where the prefix (in +MANIFEST) is /usr/local _but_ tar tf of the unxz'd tarball is _not_ /usr/local? (e.g. pkg_install style) or is this the new way going forward? There must at least be a +MANIFEST -- other meta data files are optional. +COMPACT_MANIFEST is a subset of the full +MANIFEST. They're both YAML documents. +COMPACT_MANIFEST looks like this, for example: --- name: pkg version: 1.1.4 origin: ports-mgmt/pkg comment: New generation package manager arch: freebsd:9:x86:64 www: https://urldefense.proofpoint.com/v1/url?u=http://wiki.freebsd.org/pkgngk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=LPoff3ddLDoHE30AU6R5vgBklqxsjVAosXGjJop6uO4%3D%0As=30b27cf33d7e594144fc38f20bc75d84464788e715f7fc82b2495f813fd4be2d maintainer: port...@freebsd.orgmailto:port...@freebsd.org prefix: /usr/local licenselogic: single licenses: - BSD flatsize: 6311507 desc: | New Generation package management tool for FreeBSD WWW: https://urldefense.proofpoint.com/v1/url?u=http://wiki.freebsd.org/pkgngk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=cLKrok7Cl%2BjTLEB024lmmMXfxb1fKAOZvvxLV8RjU%2FU%3D%0As=cef93a5fe98a3fa0fbc9df3e0d11c9bbeaee44290306834ddef30f2ba925446c categories: - ports-mgmt shlibs_required: - libpkg.so.1 shlibs_provided: - libpkg.so.1 message: | If you are upgrading from the old package format, first run: # pkg2ng Nice. Very nice. +MTREE_DIRS is a compatibility thing with the old pkg_tools. It's not needed in general as +MANIFEST can provide all that meta data itself. It isn't going to be deprecated for at least as long as the ports tree continues to support pkg_tools though. Beyond that, the rest of the pkg tarball just contains a tar archive of all the files, directories, sym-links etc to be installed by the package. Note that pkg(8) has no problem with creating an empty directory for a package, unlike pkg_tools. Excellent! Now, while you can grovel through the details of pkg tarballs and internal data formats like this, be aware: the format of the manifest and the details of the meta-data included in the pkg-tarballs is subject to change without warning as we develop pkg(8) further. We only promise API stability through the pkg(8) commands or for the libpkg.so library functions; reports of breakage from usage outside those APIs will receive little sympathy. Understood. So just to give you a better idea of how we manage packages here. We've realized that if you want to package for FreeBSD (in 9 and older), you could get by alright if you knew how to create a +CONTENTS file from scratch. For RedHat, it's the SPECFILE. And now for FreeBSD 10, it looks an awful lot like a SPECFILE (they are both in-fact YAML). The only thing you need if you
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 14, 2013, at 11:27 PM, Baptiste Daroussin wrote: On Sun, Jul 14, 2013 at 07:29:50AM +, Teske, Devin wrote: Or you can maintain the old cruft for your business -- just don't expect anyone else to use it, or even want to. I have no intention of making old-world packages... but I also have no intention of using pkg create. So you are taking as an exemple rpm but with RPM you do want to use what they do provide, aka rpmbuild but with pkgng you want to manually handwrite the packages because we haven't hidden the format behind opaque names and format. Close... I use rpmbuild because I haven't figured out how to subvert it to create cpio-balls for rpm(8). I know how to extract the contents of binary rpm's using rpm2cpio and I know how to extract the meta data and supplant it into a template SPECFILE using rpm -q -p file.rpm --qf queryformat ... but I haven't spent enough time with rpm's to know how to master them without rpmbuild. The best I've done is learn how to use .rpmmacros to make rpmbuild behave in a sane manner that is conducive to (a) building the archive out of the current directory (including all temporary files, etc.) and (b) working on source-controlled checkouts. However, even if I did figure out the cpio-ball/metadata format, it is quite convenient to have rpmbuild do the work -- which unfortunately introduces the constraint that you have to generate the archive on the system for which it will be installed (something you eluded that is inconclusive -- I'll cover that in a reply to that-reply directly). The convenience however is not because the tool does everything I need it to do, but the convenience comes from the translation of YAML into metadata. It doesn't appear that this will be necessary with pkgng packages, just as it wasn't necessary for pkg_install packages. To be more clear,... because tools like pkg_create and pkg create take a file-in, add data to it, and then (the most important part) put a modified form into the archive (which quite-closely resembles the original), it is then possible to imagine that: (1) there's value in learning both the input format and the output format so that you can validate packages, fix broken packages, etc. ASIDE: For example... a binary release package ends up with incorrect dependencies encoded within its +CONTENTS file -- I can slurp the package into our pkgbase, adjust the PLIST (generated from +CONTENTS), and produce a fixed package in mere moments (and I didn't have to wake anybody in the middle of the night, ask that FTP mirrors be updated, balk at the core team -- none of that... the problem was solved and in the next release we hope for the best that the dependency chain isn't busted for the same binary package within the set of X.Y-RELEASE packages). (2) there's value in writing a shell-script method to translate the input format into the output format from a system of makefiles calling said shell scripts out of a revision-controlled checkout ASIDE: This one should be obvious. If you can assemble a package using nothing but shell scripts (and mostly awk, ls, find, cp, mkdir, ln, cvs, make, md5/sha256/sha512/openssl, and etc.), then you don't have to port the port system (our port system being called pkgbase -- and working to keep you stable rather than keep you up-to-date). You know in that case you should probably create your own RPM without RPM just write by hand the header and happened the cpio to it, in that case you will end up having the same at what you are trying to do with pkg. It's a good idea... except one thing... the YAML SPECFILE (minus the elements that are specific to the `-bb' process of rpmbuild) get completely changed in the process. Doing this would require a YAML parser + metadata encoder process-pair. Whereas for FreeBSD it's YAML-in, YAML-out. So I don't need a YAML-parser, just a line-parser and light YAML-encoder -- which could be fixed-strings, doesn't have to be complex... e.g., awk). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 14, 2013, at 11:33 PM, Baptiste Daroussin wrote:
On Sun, Jul 14, 2013 at 04:52:26PM +, Teske, Devin wrote:
On Jul 14, 2013, at 8:01 AM, Chris Rees wrote:
On 14 Jul 2013, at 08:29, Teske, Devin wrote:
Simple, really.
Let's take RPM for example. The RPM package format has been ported to other
platforms.
So does pkgng ported on Linux, OS X, dragonfly, NetBSD...
Sweet! (bright future!)
But, I can't take archivers/rpm4 and build on RPM on FreeBSD and install it on
RedHat.
Yes you can, I do it at work all the time, on FreeBSD I do create AIX rpms and
RedHat rpms.
What version of RedHat?
Worked on RedHat EL4 using rpm3, but in-practice -- attempts to recreate that
workflow on RedHat EL6 using rpm4 have failed.
Please see a copy/paste of the output of when we build an RPM on FreeBSD-8.1
and try to install it on RedHat Enterprise Linux 6:
http://pastebin.com/zpzjxP2T
Spoiler: package {X} is intended for a freebsd operating system (not
installed)
This is because the RPM format records the platform that you build your RPM
on (not the binaries, just the RPM) *into* said RPM.
So does pkgng.
Good to know.
This actually adds a requirement to the RPM production that the RPMs be
produced on the platform that they will be installed-to.
No.
Yes. See pastebin link above.
Currently, no such restriction exists for the building of FreeBSD packages
(within our system). This would have been true if we had ported pkg_create (and
may continue to be true if we ported pkg and its ilk), but let's say for the
sake of argument that the future of pkg looks bright and it gets ported to
all sorts of systems (ported in a fashion similar to RPM) *and* we find one day
that the +MANIFEST starts containing a target-platform (resulting in refusal to
install a *.txz package because it was rolled on a different platform.
Thank for describing the exact situation pkg is right now.
Glad to help ;D
(not so happy about the target platform being recorded -- is there an override?
setting UNAME_{a,r,etc.}?)
In that case, we'd then prefer to by-pass the tools and use our own method of
creating the tar-ball to lift such a restriction.
The restriction you are speaking about does not exists.
See pastebin link above.
ASIDE: If I knew how to force rpmbuild into creating androgynous packages for
other architectures, I'd be doing that to life the restriction there too, but I
haven't figured out that.
Basically... within our pkgbase tree, we like the branch within the tree to
dictate how a package is built... not what platform you're on. The goal being
that we can run a single package-build host that builds all of our packages
from a single platform.
You can do it with pkgng just easily, as well as you can do it with rpm.
W/respect to RPM, see pastebin link above.
--
Devin
_
The information contained in this message is proprietary and/or confidential.
If you are not the intended recipient, please: (i) delete the message and all
copies; (ii) do not disclose, distribute or use the message in any manner; and
(iii) notify the sender immediately. In addition, please be aware that any
message addressed to our domain is subject to archiving and review by persons
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 14, 2013, at 11:38 PM, Baptiste Daroussin wrote: On Sat, Jul 13, 2013 at 06:51:00PM -0500, Mark Felder wrote: On Sat, Jul 13, 2013, at 13:54, Teske, Devin wrote: If FTP access (or any of the other remote access methods) are going away for HEAD pkg access, I'll need to know so I can make the appropriate changes in the HEAD branch of bsdconfig. It's simpler than you think. The new pkg uses libfetch. You can have your PACAKGESITE be file:// ftp:// http:// https:// ... I do suspect that HTTP_PROXY support is probably not available as I recall seeing a post where someone was asking about that getting implemented for fetch. I'll have to research that again, though. Yes pkgng do support proxies, because libfetch do support them. I think Mark was saying that libfetch doesn't yet support http proxy. However, I've often done: env http_proxy=user:pass@host:port fetch this-thing or that-thing So I'm not sure if Mark was implying that this doesn't yet work with PACKAGESITE because you need a way of passing the additional env? (talk of a PKG_ENV iirc)? pkgng also support ssh:// That's cool ;D -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 14, 2013, at 11:42 PM, Baptiste Daroussin wrote: On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote: On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote: Hi, I have just committed (r253305) a change the make pkg_install not being built and installed by default on HEAD. If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your src.conf(5) I think while a good move, we need to allow a window of development to re-work other HEAD components. It might also be worth developing a lint-tool to make sure we get every last instance (both from C code and sh code) within our base before we motion to cut a release with this change. I for one am effected and will have to change things. Can you point us at a guide (or even better, a Wiki) that can smooth the process? I have already pointed you to such resources a couple of time before you started working on the package management part of bsdconfig. Respectfully... look at the reply-text. I didn't mention bsdconfig. I was asking in-general for others. I think we're all in agreement that a transition Wiki does not exist. I'm not upset that the conversation was turned into a bsdconfig-specific one (on the contrary, elated)... but for the very reason that you had already given me pointers in the past, I did not come right out and say hey, could you re-tell me everything you told me before because I've forgotten most of it? but instead I pointed out that we don't have a transition Wiki. So I do supremely apologize if I ask a question that you already answered in the past. My excuse is that you pointed me to such resources almost 10 months ago. And for that entire 10 months, I'd been working on the package management part of bsdconfig. I *just* got my head above water and then it was announced that pkg_install is going away, and what perfect timing -- having just finished the pkg_install-based framework for which I plan to base the pkgng integration. May I remember you that at this time I have recommanded you to directly work on the pkgng version? This comment doesn't seem to be related to any of the reply-text above it. I'll assume it's tainted by the other messages in this thread. For bsdconfig, let me just re-iterate... + bsdconfig is the tool to replace sysinstall configPackages. + It will not do anything weird or strange or out of place ** + It will try to be an example as well as a re-usable and extendable tool (and on the back of those last two comments, again, bsdconfig pkgng integration is not associated with any funky handling of package MANIFEST files or any black-magic voodoo that I do at $work with pkgbase). ** Unless a UI enhancement can be made by subverting a decidedly non-UI tool to provide more feedback than simply blocking on an opaque process. I have also explained you 2 or 3 times at least what is a catalog in pkgng how it works what pkg query and pkg rquery will give you as a script writter. Again, apologies if I ask a question that *seems* like it's been asked before. And it would be a lie for me to say I've forgotten *everything* from our prior conversations. However, I may repeat some questions as it was almost 10 months ago that I was last looking at pkgng (getting bsdconfig packages done for 9.2-R was a *lot* of work and it depleted my brain of pretty much everything else for that entire time -- except for $work of course, but every night for 10 months, 7 days a week, I ate, drank, slept, and smoked bsdconfig packages). If I could recount something from our conversations from 10 months ago... I would say that you and I agreed that I needed to get the framework done and that I could best do that unencumbered if it were pkg_install based. Well that's been done (it was a hell of a lot of work, but I got it done). I'm saying that one sole request (to get the framework ready for pkgng) is now done and we need to reconvene to take the next step. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 13, 2013, at 11:13 PM, Garrett Wollman wrote: In article 20130714054840$7...@grapevine.csail.mit.edu, dte...@freebsd.org writes: How about rquery? What protocol does that use? and what does it talk to? It accesses the sqlite database in /var/db/pkg that was previously retrieved from the remote repository. This previously retrieved has me worried. One of the things that you could do with sysinstall was: 1. Browse packages (nothing else) 2. Exit sysinstall And your system is completely unmodified and the mirror your chose is forgotten. Now from what you explained of pkg, I'm worried that for bsdconfig: 1. Browse packages (nothing else) 2. Exit bsdconfig and now because the pkg rquery has staged a db, future pkg commands are now influenced. In which case, I think I would best off downloading the sqlite db to a temporary location and pointing pkg at it (if that's possible). Because I'm not sure that I want adhoc browsing in bsdconfig to change the default mirror (but maybe that's desired? I'm not sure). Question: Where can I learn more about the actual format of what's in the new tarballs? This is going to be important not for bsdconfig, but $work (we have our own build platform; I'm going to have to rewrite it from mastering PLIST files to mastering YAML MANIFEST files and I want to know all the gritty details). You really shouldn't know about the actual format of the tarballs; your time would be better spent learning the new pkg create, pkg register, and pkg repo commands. Depending on your needs, you might want to write to the libpkg API instead. That won't work for us. You're not going to like the answer, but it does mean that things like pkg create, pkg register and pkg repo won't work for us. Because FreeBSD packages are just tar-balls, I created a build-system that allows us to master and re-master them on *any* platform with nothing more than make and a few standard UNIX utilities such as awk and etc. This makes it extremely convenient for developers and integrators. You can feel free to check out the build tree on Mac OS X, make some changes to a file, check them in, build the package, throw it in the repos, and all from the comfort of your own OS. Yes, it flies in the face of dog-fooding, but the value-add that it brings to the table is quite tangible w/respect to our workflow. So the format is still something I want to know quite well. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 14, 2013, at 12:06 AM, Garrett Wollman wrote: In article 20130714064601$3...@grapevine.csail.mit.edu, dte...@freebsd.org writes: [I wrote:] It accesses the sqlite database in /var/db/pkg that was previously retrieved from the remote repository. Now from what you explained of pkg, I'm worried that for bsdconfig: 1. Browse packages (nothing else) 2. Exit bsdconfig and now because the pkg rquery has staged a db, future pkg commands are now influenced. Only if you update /usr/local/etc/pkg.conf to set a permanent PACKAGESITE. Otherwise, it will notice that the remote repo catalog you have isn't for your currently selected remote repo, and use that instead. You really shouldn't know about the actual format of the tarballs; your time would be better spent learning the new pkg create, pkg register, and pkg repo commands. Depending on your needs, you might want to write to the libpkg API instead. That won't work for us. You're not going to like the answer, but it does mean that things like pkg create, pkg register and pkg repo won't work for us. Congratulations for building your entire workflow around a horrible kluge straight out of 1993. Prejudice much? FreeBSD, however, is moving on. Moving from tarballs to tarballs, yep... moving along nicely. (And it's long past time!) I love your enthusiastic embrace. Your developers will just have to deal. You're making the wrong argument. Bringing in a new package system doesn't help developers or integrators. It doesn't matter if you switch pkg_create with pkg_create. Developers like being able to go into a build tree and say make and end-up with a package. How that build tree does it should be sufficiently advanced enough to be indistinguishable from magic. Your answer implies that this infrastructure is unnecessary, when I can indeed tell you that even if it is not divulged to me what goes into a package, I'll still end up ripping it open and creating my own build platform that doesn't depend on platform-specific tools. And when the day comes that FreeBSD uses something other than tarballs, then I'll then force the developers to build the packages on the platform using said tools; but until then, why pigeon-hole the process of building a package into one OS when developers could care less whether their make uses pkg_create, pkg, rpm, dpkg, or what else? To give you an idea as to just how helpful this is... Imagine the following hierarchy: src/pkgbase/depend/mystuff/script1 src/pkgbase/depend/mystuff/textfile1 src/pkgbase/depend/mystuff/sourcefile.c src/pkgbase/depend/mystuff/Makefile You are a developer. You want to ship a package that contains script1, textfile1, and binary1 (which is compiled by saying make to turn sourcefile.c into binary1) You want to ship 8 types of packages: FreeBSD-4.11 FreeBSD-8.1 (i386) FreeBSD-8.1 (amd64) RedHat EL 4 RedHat EL 6 (i386) RedHat EL 6 (x86_64) Debian Wheezy Debian Wheezy 64-bit This is where my framework comes in-handy... cd ~/src/pkgbase/freebsd/RELENG_4/category/mystuff make # it pulled the necessary bits from src/pkgbase/depend/mystuff and built the .tgz cd ~/src/pkgbase/freebsd/RELENG_8/category/mystuff make # it pulled the necessary bits from the depend dir and built .tbz cd ~/src/pkgbase/redhat/rhel4/category/sub-category/mystuff make # pulled in depend and made .rpm cd ~/src/pkgbase/redhat/rhel6/category/sub-category/mystuff make # pulled in depend and made .rpm etc. Of course, *any* time the depend tree has binaries in it... you have to first do a make in there on the platform you want to ship the binary for, and then do make depend in the platform-specific tree to pull in the binaries. Once you've done that, you don't have to muck with the depend tree again unless there are changes there. So, I assume that your prejudice remarks are because you haven't either seen (a) such a platform or (b) such a need for said platform. Yeah, I could rewrite the freebsd-specific logic to use pkg create, but let me tell you... When you have to touch a file that needs to get shipped out to multiple platforms... It's damned nice to be able to build the FreeBSD packages under RedHat *BECAUSE* the redhat RPMs can't be built under anything else (building an RPM on FreeBSD and attempting to install it on RedHat results in an error message similar to this is an rpm for FreeBSD; go away). Whereas FreeBSD will never balk about a package built on another platform. It's a huge time-saving measure... not having to jump over to each/every unique platform to package things up *IF/WHEN* you know that there are no binaries in the package *or* you've already checked the pre-compiled binaries into the arch-specific hierarchy. Or you can maintain the old cruft for your business -- just don't expect anyone else to use it, or even want to. I have no intention of making old-world packages... but I also have no intention of using
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 13, 2013, at 9:24 PM, Peter Wemm wrote: On Sat, Jul 13, 2013 at 11:54 AM, Teske, Devin devin.te...@fisglobal.com wrote: So yes... I'm asking... in a HEAD world, what is the officially supported method of acquisition? * pkgng will be configured to pull via http from project infrastructure by default. So is this the right _first step_ patch for HEAD bsdconfig? NOTE: Based on feedback, this will be the list of what official pkg infrastructure we want to support from the base. The selection would be communicated to pkg via $PACKAGESITE. Index: head/usr.sbin/bsdconfig/share/media/http.subr === --- head/usr.sbin/bsdconfig/share/media/http.subr (revision 25) +++ head/usr.sbin/bsdconfig/share/media/http.subr (working copy) @@ -77,7 +77,9 @@ f_dialog_menu_media_http() f_dialog_title_restore local prompt=$msg_please_select_the_site_closest_to_you_or_other local menu_list= - 'URL' '$msg_specify_some_other_http_site' + '$msg_main_site' 'http.freebsd.org' + 'URL' '$msg_specify_some_other_http_site' + 'IPv6 $msg_main_site' 'http.freebsd.org' # END-QUOTE local hline=$msg_select_a_site_thats_close Index: head/usr.sbin/bsdconfig/share/media/ftp.subr === --- head/usr.sbin/bsdconfig/share/media/ftp.subr(revision 25) +++ head/usr.sbin/bsdconfig/share/media/ftp.subr(working copy) @@ -73,157 +73,7 @@ f_dialog_menu_media_ftp() f_dialog_title_restore local prompt=$msg_please_select_the_site_closest_to_you_or_other local menu_list= - '$msg_main_site' 'ftp.freebsd.org' - 'URL' '$msg_specify_some_other_ftp_site' - '$msg_snapshots_server_japan' - 'snapshots.jp.freebsd.org' - '$msg_snapshots_server_sweden' - 'snapshots.se.freebsd.org' - 'IPv6 $msg_main_site' 'ftp.freebsd.org' - ' IPv6 $msg_ireland' 'ftp3.ie.freebsd.org' - ' IPv6 $msg_israel' 'ftp.il.freebsd.org' - ' IPv6 $msg_japan''ftp2.jp.freebsd.org' - ' IPv6 $msg_usa' 'ftp4.us.freebsd.org' - ' IPv6 $msg_turkey' 'ftp2.tr.freebsd.org' - '$msg_primary''ftp1.freebsd.org' - ' $msg_primary #2''ftp2.freebsd.org' - ' $msg_primary #3''ftp3.freebsd.org' - ' $msg_primary #4''ftp4.freebsd.org' - ' $msg_primary #5''ftp5.freebsd.org' - ' $msg_primary #6''ftp6.freebsd.org' - ' $msg_primary #7''ftp7.freebsd.org' - ' $msg_primary #8''ftp8.freebsd.org' - ' $msg_primary #9''ftp9.freebsd.org' - ' $msg_primary #10' 'ftp10.freebsd.org' - ' $msg_primary #11' 'ftp11.freebsd.org' - ' $msg_primary #12' 'ftp12.freebsd.org' - ' $msg_primary #13' 'ftp13.freebsd.org' - ' $msg_primary #14' 'ftp14.freebsd.org' - '$msg_argentina' 'ftp.ar.freebsd.org' - '$msg_australia' 'ftp.au.freebsd.org' - ' $msg_australia #2' 'ftp2.au.freebsd.org' - ' $msg_australia #3' 'ftp3.au.freebsd.org' - '$msg_austria''ftp.at.freebsd.org' - ' $msg_austria #2''ftp2.at.freebsd.org' - '$msg_brazil' 'ftp.br.freebsd.org' - ' $msg_brazil #2' 'ftp2.br.freebsd.org' - ' $msg_brazil #3' 'ftp3.br.freebsd.org' - ' $msg_brazil #4' 'ftp4.br.freebsd.org' - ' $msg_brazil #5' 'ftp5.br.freebsd.org' - ' $msg_brazil #6' 'ftp6.br.freebsd.org' - ' $msg_brazil #7' 'ftp7.br.freebsd.org' - '$msg_canada' 'ftp.ca.freebsd.org' - '$msg_china' 'ftp.cn.freebsd.org' - ' $msg_china #2' 'ftp2.cn.freebsd.org' - '$msg_croatia''ftp.hr.freebsd.org' - '$msg_czech_republic' 'ftp.cz.freebsd.org' - '$msg_denmark''ftp.dk.freebsd.org' - ' $msg_denmark #2''ftp2.dk.freebsd.org' - '$msg_estonia''ftp.ee.freebsd.org' - '$msg_finland''ftp.fi.freebsd.org' - '$msg_france' 'ftp.fr.freebsd.org' - ' $msg_france #2' 'ftp2.fr.freebsd.org' - ' $msg_france #3' 'ftp3.fr.freebsd.org
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote: On 14/07/2013 06:48, Teske, Devin wrote: Question: Where can I learn more about the actual format of what's in the new tarballs? This is going to be important not for bsdconfig, but $work (we have our own build platform; I'm going to have to rewrite it from mastering PLIST files to mastering YAML MANIFEST files and I want to know all the gritty details). We do need a pkg-manifest(5) man page, which can double as a pkg-tarball(5) page since the manifest file is where most of the interesting bits are. A pkg tarball is a compressed tar archive like so: lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz -rw-r--r-- 0 root wheel 530 Jan 1 1970 +COMPACT_MANIFEST -rw-r--r-- 0 root wheel6385 Jan 1 1970 +MANIFEST -rw-r--r-- 0 root wheel 17567 Jan 1 1970 +MTREE_DIRS -r--r--r-- 0 root wheel 19453 Jul 7 12:26 /usr/local/etc/bash_completion.d/_pkg.bash -r-xr-xr-x 0 root wheel 629 Jul 7 12:26 /usr/local/etc/periodic/daily/400.status-pkg -r-xr-xr-x 0 root wheel 823 Jul 7 12:26 /usr/local/etc/periodic/daily/411.pkg-backup -r-xr-xr-x 0 root wheel 678 Jul 7 12:26 /usr/local/etc/periodic/daily/490.status-pkg-changes -r-xr-xr-x 0 root wheel2558 Jul 7 12:26 /usr/local/etc/periodic/security/410.pkg-audit -r-xr-xr-x 0 root wheel 611 Jul 7 12:26 /usr/local/etc/periodic/security/460.pkg-checksum -r--r--r-- 0 root wheel 839 Jul 7 12:26 /usr/local/etc/pkg.conf.sample -r--r--r-- 0 root wheel 43432 Jul 7 12:26 /usr/local/include/pkg.h -r--r--r-- 0 root wheel 727558 Jul 7 12:26 /usr/local/lib/libpkg.a lrwxr-xr-x 0 root wheel 0 Jul 7 12:26 /usr/local/lib/libpkg.so - libpkg.so.1 -r--r--r-- 0 root wheel 1227064 Jul 7 12:26 /usr/local/lib/libpkg.so.1 -rw-r--r-- 0 root wheel 187 Jul 7 12:26 /usr/local/libdata/pkgconfig/pkg.pc [... etc ...] Interesting. I notice that (while looking ahead to see a prefix: of /usr/local in the +MANIFEST), the tarball itself has files that include /usr/local in their path. Does pkg handle packages where the prefix (in +MANIFEST) is /usr/local _but_ tar tf of the unxz'd tarball is _not_ /usr/local? (e.g. pkg_install style) or is this the new way going forward? There must at least be a +MANIFEST -- other meta data files are optional. +COMPACT_MANIFEST is a subset of the full +MANIFEST. They're both YAML documents. +COMPACT_MANIFEST looks like this, for example: --- name: pkg version: 1.1.4 origin: ports-mgmt/pkg comment: New generation package manager arch: freebsd:9:x86:64 www: https://urldefense.proofpoint.com/v1/url?u=http://wiki.freebsd.org/pkgngk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=LPoff3ddLDoHE30AU6R5vgBklqxsjVAosXGjJop6uO4%3D%0As=30b27cf33d7e594144fc38f20bc75d84464788e715f7fc82b2495f813fd4be2d maintainer: port...@freebsd.org prefix: /usr/local licenselogic: single licenses: - BSD flatsize: 6311507 desc: | New Generation package management tool for FreeBSD WWW: http://wiki.freebsd.org/pkgng categories: - ports-mgmt shlibs_required: - libpkg.so.1 shlibs_provided: - libpkg.so.1 message: | If you are upgrading from the old package format, first run: # pkg2ng Nice. Very nice. +MTREE_DIRS is a compatibility thing with the old pkg_tools. It's not needed in general as +MANIFEST can provide all that meta data itself. It isn't going to be deprecated for at least as long as the ports tree continues to support pkg_tools though. Beyond that, the rest of the pkg tarball just contains a tar archive of all the files, directories, sym-links etc to be installed by the package. Note that pkg(8) has no problem with creating an empty directory for a package, unlike pkg_tools. Excellent! Now, while you can grovel through the details of pkg tarballs and internal data formats like this, be aware: the format of the manifest and the details of the meta-data included in the pkg-tarballs is subject to change without warning as we develop pkg(8) further. We only promise API stability through the pkg(8) commands or for the libpkg.so library functions; reports of breakage from usage outside those APIs will receive little sympathy. Understood. So just to give you a better idea of how we manage packages here. We've realized that if you want to package for FreeBSD (in 9 and older), you could get by alright if you knew how to create a +CONTENTS file from scratch. For RedHat, it's the SPECFILE. And now for FreeBSD 10, it looks an awful lot like a SPECFILE (they are both in-fact YAML). So rather than teach the people here how to use tools, I taught them what I think is more important... how to manage +CONTENTS, SPECFILE, and now up-and-coming, +MANIFEST. However, I'd be lying if I said I taught them *all* the gory details. In reality, for +CONTENTS they edit a PLIST which is essentially +CONTENTS
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 14, 2013, at 8:01 AM, Chris Rees wrote: On 14 Jul 2013, at 08:29, Teske, Devin wrote: To give you an idea as to just how helpful this is... Imagine the following hierarchy: src/pkgbase/depend/mystuff/script1 src/pkgbase/depend/mystuff/textfile1 src/pkgbase/depend/mystuff/sourcefile.c src/pkgbase/depend/mystuff/Makefile You are a developer. You want to ship a package that contains script1, textfile1, and binary1 (which is compiled by saying make to turn sourcefile.c into binary1) You want to ship 8 types of packages: FreeBSD-4.11 FreeBSD-8.1 (i386) FreeBSD-8.1 (amd64) RedHat EL 4 RedHat EL 6 (i386) RedHat EL 6 (x86_64) Debian Wheezy Debian Wheezy 64-bit This is where my framework comes in-handy... cd ~/src/pkgbase/freebsd/RELENG_4/category/mystuff make # it pulled the necessary bits from src/pkgbase/depend/mystuff and built the .tgz cd ~/src/pkgbase/freebsd/RELENG_8/category/mystuff make # it pulled the necessary bits from the depend dir and built .tbz cd ~/src/pkgbase/redhat/rhel4/category/sub-category/mystuff make # pulled in depend and made .rpm cd ~/src/pkgbase/redhat/rhel6/category/sub-category/mystuff make # pulled in depend and made .rpm etc. Of course, *any* time the depend tree has binaries in it... you have to first do a make in there on the platform you want to ship the binary for, and then do make depend in the platform-specific tree to pull in the binaries. Once you've done that, you don't have to muck with the depend tree again unless there are changes there. So, I assume that your prejudice remarks are because you haven't either seen (a) such a platform or (b) such a need for said platform. Yeah, I could rewrite the freebsd-specific logic to use pkg create, but let me tell you... When you have to touch a file that needs to get shipped out to multiple platforms... It's damned nice to be able to build the FreeBSD packages under RedHat *BECAUSE* the redhat RPMs can't be built under anything else (building an RPM on FreeBSD and attempting to install it on RedHat results in an error message similar to this is an rpm for FreeBSD; go away). Whereas FreeBSD will never balk about a package built on another platform. It's a huge time-saving measure... not having to jump over to each/every unique platform to package things up *IF/WHEN* you know that there are no binaries in the package *or* you've already checked the pre-compiled binaries into the arch-specific hierarchy. Or you can maintain the old cruft for your business -- just don't expect anyone else to use it, or even want to. I have no intention of making old-world packages... but I also have no intention of using pkg create. You still haven't really explained at all why you can't use libpkg. If it doesn't run on Debian (not tried), it's got to be easier to port it than rewrite a hacked version, hasn't it??? At least then you'll also be contributing back. Simple, really. Let's take RPM for example. The RPM package format has been ported to other platforms. But, I can't take archivers/rpm4 and build on RPM on FreeBSD and install it on RedHat. This is because the RPM format records the platform that you build your RPM on (not the binaries, just the RPM) *into* said RPM. This actually adds a requirement to the RPM production that the RPMs be produced on the platform that they will be installed-to. Currently, no such restriction exists for the building of FreeBSD packages (within our system). This would have been true if we had ported pkg_create (and may continue to be true if we ported pkg and its ilk), but let's say for the sake of argument that the future of pkg looks bright and it gets ported to all sorts of systems (ported in a fashion similar to RPM) *and* we find one day that the +MANIFEST starts containing a target-platform (resulting in refusal to install a *.txz package because it was rolled on a different platform. In that case, we'd then prefer to by-pass the tools and use our own method of creating the tar-ball to lift such a restriction. ASIDE: If I knew how to force rpmbuild into creating androgynous packages for other architectures, I'd be doing that to life the restriction there too, but I haven't figured out that. Basically... within our pkgbase tree, we like the branch within the tree to dictate how a package is built... not what platform you're on. The goal being that we can run a single package-build host that builds all of our packages from a single platform. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 14, 2013, at 11:50 AM, Adrian Chadd wrote: ... I bet you could do that. I bet you could build the rpm inside a linux jail and have the relevant uname bits overridden in the right way. There's an idea. -- Devin On 14 July 2013 09:52, Teske, Devin devin.te...@fisglobal.com wrote: On Jul 14, 2013, at 8:01 AM, Chris Rees wrote: On 14 Jul 2013, at 08:29, Teske, Devin wrote: To give you an idea as to just how helpful this is... Imagine the following hierarchy: src/pkgbase/depend/mystuff/script1 src/pkgbase/depend/mystuff/textfile1 src/pkgbase/depend/mystuff/sourcefile.c src/pkgbase/depend/mystuff/Makefile You are a developer. You want to ship a package that contains script1, textfile1, and binary1 (which is compiled by saying make to turn sourcefile.c into binary1) You want to ship 8 types of packages: FreeBSD-4.11 FreeBSD-8.1 (i386) FreeBSD-8.1 (amd64) RedHat EL 4 RedHat EL 6 (i386) RedHat EL 6 (x86_64) Debian Wheezy Debian Wheezy 64-bit This is where my framework comes in-handy... cd ~/src/pkgbase/freebsd/RELENG_4/category/mystuff make # it pulled the necessary bits from src/pkgbase/depend/mystuff and built the .tgz cd ~/src/pkgbase/freebsd/RELENG_8/category/mystuff make # it pulled the necessary bits from the depend dir and built .tbz cd ~/src/pkgbase/redhat/rhel4/category/sub-category/mystuff make # pulled in depend and made .rpm cd ~/src/pkgbase/redhat/rhel6/category/sub-category/mystuff make # pulled in depend and made .rpm etc. Of course, *any* time the depend tree has binaries in it... you have to first do a make in there on the platform you want to ship the binary for, and then do make depend in the platform-specific tree to pull in the binaries. Once you've done that, you don't have to muck with the depend tree again unless there are changes there. So, I assume that your prejudice remarks are because you haven't either seen (a) such a platform or (b) such a need for said platform. Yeah, I could rewrite the freebsd-specific logic to use pkg create, but let me tell you... When you have to touch a file that needs to get shipped out to multiple platforms... It's damned nice to be able to build the FreeBSD packages under RedHat *BECAUSE* the redhat RPMs can't be built under anything else (building an RPM on FreeBSD and attempting to install it on RedHat results in an error message similar to this is an rpm for FreeBSD; go away). Whereas FreeBSD will never balk about a package built on another platform. It's a huge time-saving measure... not having to jump over to each/every unique platform to package things up *IF/WHEN* you know that there are no binaries in the package *or* you've already checked the pre-compiled binaries into the arch-specific hierarchy. Or you can maintain the old cruft for your business -- just don't expect anyone else to use it, or even want to. I have no intention of making old-world packages... but I also have no intention of using pkg create. You still haven't really explained at all why you can't use libpkg. If it doesn't run on Debian (not tried), it's got to be easier to port it than rewrite a hacked version, hasn't it??? At least then you'll also be contributing back. Simple, really. Let's take RPM for example. The RPM package format has been ported to other platforms. But, I can't take archivers/rpm4 and build on RPM on FreeBSD and install it on RedHat. This is because the RPM format records the platform that you build your RPM on (not the binaries, just the RPM) *into* said RPM. This actually adds a requirement to the RPM production that the RPMs be produced on the platform that they will be installed-to. Currently, no such restriction exists for the building of FreeBSD packages (within our system). This would have been true if we had ported pkg_create (and may continue to be true if we ported pkg and its ilk), but let's say for the sake of argument that the future of pkg looks bright and it gets ported to all sorts of systems (ported in a fashion similar to RPM) *and* we find one day that the +MANIFEST starts containing a target-platform (resulting in refusal to install a *.txz package because it was rolled on a different platform. In that case, we'd then prefer to by-pass the tools and use our own method of creating the tar-ball to lift such a restriction. ASIDE: If I knew how to force rpmbuild into creating androgynous packages for other architectures, I'd be doing that to life the restriction there too, but I haven't figured out that. Basically... within our pkgbase tree, we like the branch within the tree to dictate how a package is built... not what platform you're on. The goal being that we can run a single package-build host that builds all of our packages from a single platform. -- Devin _ The information contained
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 14, 2013, at 3:43 PM, Craig Rodrigues wrote:
On Sat, Jul 13, 2013 at 10:54 PM, Teske, Devin
devin.te...@fisglobal.commailto:devin.te...@fisglobal.com wrote:
I assume that poudiere builds packages from ports.
Yes.
That's not how we build package repositories here (and would expect that there
are many more like us).
How do you build packages if you are not using FreeBSD ports?
Two spectrums...
When developer A says he needs xerces, I go to:
(for an active release, such as 8.4 or 9.1)
ftp://ftp.freebsd.org/pub/FreeBSD/releases/{arch}/X.Y-RELEASE/packages/Allftp://ftp.freebsd.org/pub/FreeBSD/releases/%7Barch%7D/X.Y-RELEASE/packages/All
(for an in-active release, such as 8.1 or 9.0)
ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/{arch}/X.Y-RELEASE/packages/Allftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/%7Barch%7D/X.Y-RELEASE/packages/All
Meanwhile, when either developer B or integrator A says I have software X that
I want to package, we use pkgbase to build the FreeBSD package, the RedHat
RPM, etc.
Do you have your own Makefiles which resemble FreeBSD ports,
or are you doing something completely different?
That's essentially what pkgbase is. Except instead of building makefiles that
build the 3rd-party software, we rely on the 3rd-party compilation process to
produce the binaries and have a configuration file (pkgbase.conf) which
describes the vendor layout.
The vendor layout can be anything and do anything and prepare anything. But the
pkgbase.conf says fileA in the vendor layout goes to src/fileA (where
src/ looks exactly like the un-packed tar-ball, *minus* +CONTENTS -- which is
generated from PLIST when you say make).
Since we're using CVS, people that are familiar with CVS would think it's quite
cute that it does a cvs admin -ko on files that are pulled from the vendor
branch (resulting in keyword expansion to retain its vendor location -- so if a
problem happens in the field, we know that the file that should be edited --
based on its keyword expansion -- is over in the vendor depend tree which,
upon modification, will cause a cascading rebuild of packages for releases we
actively maintain [through a sliding window]).
So it's very similar to ports, but we need the mechanism of being able to (for
the benefit of developer A/B and integrator A above) freeze the software
distribution of software X.
The ports tree manages that through a collection of centrally located distfiles
(which acts as a fall-back for when vendor distfiles dry up). But that method
doesn't work so well for us because when we find a bug we often fix it
ourselves in our own local branch and then communicate the fix back the vendor
(leaving it up to them to incorporate it). The next time around on our
migration to a new release, *if* we have made a local patch, producing a custom
version, *and* the vendor has not taken a patch, we'll _first_ re-evaluate the
new version with a documented test-case or (if that fails) we'll keep the patch
in-place.
The ports tree moves too fast for us. And many times we find a bug, we end up
performing cost/benefit analysis only to find that we can live with the bug
*if* the problem has been fixed already. In rare cases, we'll merge a single
patch from a future revision to get things going. But we *won't* just blindly
take a new version simply because it fixes a bug (as it may introduce new bugs
-- unwinding many man-hours of many testers and developers attempting to lead
up to a release of a wholly contained full-working software entity).
I hope this helps to explain a little more about how we take FreeBSD releases,
vet them, and them blast them out to thousands of machines over many customers
after many months of testing.
Now to say that we don't use ports would be a lie. Sometimes a developer asks
for a package that is not available on the release we're shipping. This
happened recently. So I went into the ports tree, built the package on 8.1
(which had 99% binary packages from 8.1-RELEASE -- see above URLs) and then I
said make package and threw it in the repository.
ASIDE: For diligence, make describe iirc generates my INDEX entry to append.
We expect to start with an empty directory, go grab packages that we want (one
by one) from FTP. Then put those tarballs into a directory. There's no
additional step because we download the INDEX file to the repository too. Local
repository is built.
What FTP site are you grabbing packages from, the FreeBSD ftp site (or mirrors)
or one of your own?
We grab from either ftp.f.o or ftp-archive.f.o and plop into an NFS-based repo
(which has a structure mirroring that of the ftp{,-archive}.f.o structure).
How are you generating the INDEX file?
Well, since we prefer to ship 100% with the packages that were cut with
X.Y-RELEASE, the INDEX file that's up on the ftp.f.o site does fine (except
when we have to inject things from ports -- for example, when
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 13, 2013, at 1:07 AM, Baptiste Daroussin wrote: On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote: On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote: Hi, I have just committed (r253305) a change the make pkg_install not being built and installed by default on HEAD. If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your src.conf(5) I think while a good move, we need to allow a window of development to re-work other HEAD components. It might also be worth developing a lint-tool to make sure we get every last instance (both from C code and sh code) within our base before we motion to cut a release with this change. I for one am effected and will have to change things. If you are referring to bsdconfig's package management, If you choose to read into it that way, sure. Yes. that's what I'm talking about. NOTE: I can't be the only person in HEAD that's using pkg_install still. it is not working anyway HEAD as we do not and will not provides any pkg_install for HEAD via any of the usual distribution process: http, ftp, CD. I'm quite aware that changes will be needed. I'm still interested in a transition wiki if one exists. Secondly if I'm properly reading src.conf.5 If there's a reference to WITH_BSDCONFIG in src.conf.5, that needs to be removed. I wasn't aware that it was referenced in there. bsdconfig is not installed by default, Wrong, please see... http://svnweb.freebsd.org/base?view=revisionrevision=252862 which means we have a non default code depending on a now non default code, Wrong. I don't see a problem here, should I? Yes. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 13, 2013, at 8:17 AM, Teske, Devin wrote: On Jul 13, 2013, at 1:07 AM, Baptiste Daroussin wrote: On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote: On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote: Hi, I have just committed (r253305) a change the make pkg_install not being built and installed by default on HEAD. If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your src.conf(5) [snip] I for one am effected and will have to change things. If you are referring to bsdconfig's package management, [snip] Yes. that's what I'm talking about. [snip] it is not working anyway HEAD as we do not and will not provides any pkg_install for HEAD via any of the usual distribution process: http, ftp, CD. The FTP mirror of packages is going away? (if you said yes and pointed at a prior conversation about leading up to this, I would not be surprised -- I'm just questioning it because I don't see the value in pairing-down methods of acquisition) If this is the case, what's the surviving acquisition method? A custom TCP protocol perhaps? There may be those that wish to use pkg in the pkg_add manner and download things and then inspect them manually before adding them. For example, I often go the freshports.orghttp://freshports.org to find a package that fills a need... download it from the official FTP server(s), inspect all of them, and choose the one that best fits me need (and only then installing from the local file; tossing the rest). If I go through the pkg tool, I have to inspect things *after* they've been installed which is not to my satisfaction. [snip bsdconfig is not installed by default, Wrong, please see... http://svnweb.freebsd.org/base?view=revisionrevision=252862 [snip] The first thing that comes to mind in reprogramming bsdconfig's package management for pkgng... We have a *very* large list of FTP mirrors. This will presumably be replaced with a list of pkg mirrors. Do we have such a list that we want to program into the base configuration of bsdconfig? -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 13, 2013, at 11:16 AM, Kimmo Paasiala wrote: On Sat, Jul 13, 2013 at 7:46 PM, Teske, Devin devin.te...@fisglobal.com wrote: On Jul 13, 2013, at 8:17 AM, Teske, Devin wrote: On Jul 13, 2013, at 1:07 AM, Baptiste Daroussin wrote: On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote: On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote: Hi, I have just committed (r253305) a change the make pkg_install not being built and installed by default on HEAD. If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your src.conf(5) [snip] I for one am effected and will have to change things. If you are referring to bsdconfig's package management, [snip] Yes. that's what I'm talking about. [snip] it is not working anyway HEAD as we do not and will not provides any pkg_install for HEAD via any of the usual distribution process: http, ftp, CD. The FTP mirror of packages is going away? (if you said yes and pointed at a prior conversation about leading up to this, I would not be surprised -- I'm just questioning it because I don't see the value in pairing-down methods of acquisition) If this is the case, what's the surviving acquisition method? A custom TCP protocol perhaps? There may be those that wish to use pkg in the pkg_add manner and download things and then inspect them manually before adding them. For example, I often go the freshports.orghttps://urldefense.proofpoint.com/v1/url?u=http://freshports.org/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=Ft5J2W3Nm8xze74zARHsiglLFTGOYrxAzaubbP7wvcM%3D%0As=cb1df85e237d1a6be19f981e8b352419dd056ea0f4e782b330cb9c7cfd15fda5 to find a package that fills a need... download it from the official FTP server(s), inspect all of them, and choose the one that best fits me need (and only then installing from the local file; tossing the rest). If I go through the pkg tool, I have to inspect things *after* they've been installed which is not to my satisfaction. [snip bsdconfig is not installed by default, Wrong, please see... http://svnweb.freebsd.org/base?view=revisionrevision=252862 [snip] The first thing that comes to mind in reprogramming bsdconfig's package management for pkgng... We have a *very* large list of FTP mirrors. This will presumably be replaced with a list of pkg mirrors. Do we have such a list that we want to program into the base configuration of bsdconfig? -- Devin Come on, this only concerns 10-CURRENT. Right; which is why -current is CC'd. Where is it stated that the FTP mirrors for FreeBSD 8/9 packages are going away? I don't know where you got the impression that I was concerned with 8/9 packages. I'm strictly concerned with HEAD and strictly concerned with planning for the future. So yes... I'm asking... in a HEAD world, what is the officially supported method of acquisition? I saw mention on a page that rsync access will provided for those who want to mirror but my I'm not really interested in mirroring while I would like to continue to be able to grab packages myself without installing them. Maybe the answer is that pkg has a method of acquiring a single package (without dependencies) without downloading it. That would solve my personal workflow issue (again, I like to download tarballs, inspect them, and then install them from the locally downloaded file -- then if it passes validation tests, that downloaded file is migrated to our own distribution servers; it's a work-flow for validating packages -- it has less to do with how pkgng works and more just whether I can get packages in a fashion such as FTP or HTTP or whatever. Now that aside, bsdconfig currently is a different story. To rewrite the packages module to work in a HEAD world for HEAD, using pkgng servers, I'm going to have to change the way things are done. Currently there is an abstraction layer for fetching packages from media (which can be FTP, HTTP, HTTP Proxy, NFS, Local Directory, CDROM, a DOS partition, a UFS partition, and [god forbid] Floppy -- with perhaps SMB/CIFS down the road). When it wants to add a package, it calls the routine to get the package data by name on standard-out and pumps it into pkg_add (after of course initializing the media). This abstraction layer is useful to pkgng as it prepares the source. What I *suspect* will have to stop happening is the direct-fetching of the package data and piping that into a program (however, I *could* continue to do that .. pkg supports adding of local packages and iirc supports reading from standard-in). Instead, perhaps just call pkg in a small-handful of different ways (pointed at an ftp server; pointed at a local file; pointed at http; etc.). But the preparation of an NFS mount would be handled by the abstraction layer leading up to the calling of pkg. That issue aside (whether to have pkg do the bidding or to use
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 13, 2013, at 4:51 PM, Mark Felder wrote: On Sat, Jul 13, 2013, at 13:54, Teske, Devin wrote: If FTP access (or any of the other remote access methods) are going away for HEAD pkg access, I'll need to know so I can make the appropriate changes in the HEAD branch of bsdconfig. It's simpler than you think. The new pkg uses libfetch. You can have your PACAKGESITE be file:// ftp:// http:// https:// ... In bsdconfig (as one might expect), before a user installs a package, he or she is presented with a list of packages with descriptions (and perhaps other limited information). ASIDE: For efficiency, I will actually need three things: (1) a list of all packages (2) their descriptions and (3) their run-time dependencies. NOTE: I'll need run-time dependencies so that as they're checking boxes in the UI, I can update accordingly (has nothing to do with how the dependencies are handled when the packages get installed; I'll let pkg handle that when it comes time, but for the UI and for the review screen, I want the user to be informed and I can do that more efficiently if I have a master-list and don't have to make continuous queries). That being said, I was planning on doing a pkg rquery to get that master list of [minimally] 3-pieces of information. What protocol does that rquery run on? What would one have to do to set up their own server that can accept an rquery? (our customers don't have Internet access) Last but not least... Can you do an rquery on a local repository? (say, one that has been mounted via NFS or some other media, local or otherwise but looking like a local filesystem once-mounted). What would be required to get a local repository like that going? I do suspect that HTTP_PROXY support is probably not available as I recall seeing a post where someone was asking about that getting implemented for fetch. I'll have to research that again, though. Thanks. Keep me up to date on that. Neither sysinstall nor bsdinstall really give HTTP_PROXY access much thought (they support it, but only minimally). They just construct a raw HTTP header with the FTP URL and send that directly to the proxy. One cute thing it does (when initializing the connection) is test for Squid and if-so, appends ;type=i to the URL (to force binary download versus ascii). No support for proxy-server authentication (however user/pass authentication for the FTP server is passed through to the proxy-server). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 13, 2013, at 9:24 PM, Peter Wemm wrote: On Sat, Jul 13, 2013 at 11:54 AM, Teske, Devin devin.te...@fisglobal.com wrote: So yes... I'm asking... in a HEAD world, what is the officially supported method of acquisition? This has been answered elsewhere, but to be absolutely clear: * 10.0 will ship with pkgng format packages. Ok. Respectfully not what I was inquiring about. * pkgng will be configured to pull via http from project infrastructure by default. Thanks. Good to know. So when you don't define PACKAGESITE, it defaults to HTTP. How about rquery? What protocol does that use? and what does it talk to? * The project will not be providing pkg_install format packages via any method for 10.x. Yes, I know this from previous announcements (not the topic at-hand; respectfully). This doesn't stop third parties building and providing them unofficially but my understanding is that they won't be on any project operated freebsd.org sites. Question: Where can I learn more about the actual format of what's in the new tarballs? This is going to be important not for bsdconfig, but $work (we have our own build platform; I'm going to have to rewrite it from mastering PLIST files to mastering YAML MANIFEST files and I want to know all the gritty details). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 13, 2013, at 8:33 PM, Craig Rodrigues wrote: On Sat, Jul 13, 2013 at 8:18 PM, Teske, Devin devin.te...@fisglobal.commailto:devin.te...@fisglobal.com wrote: Can you do an rquery on a local repository? (say, one that has been mounted via NFS or some other media, local or otherwise but looking like a local filesystem once-mounted). What would be required to get a local repository like that going? (1) Install the port: ports-mgmgt/poudriere , and follow the instructions at: https://fossil.etoilebsd.net/poudriere/https://urldefense.proofpoint.com/v1/url?u=https://fossil.etoilebsd.net/poudriere/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=ITD4dgEgVeGBJaxdFd9siyBYqr8XS3YLpKM1XB2BIlo%3D%0As=3f2fd1a60ccad617509daca0f40579fc7aa17cfc3a6ee176d38a4f2de3c913a0 (2) Build your own repository of packages I assume that poudiere builds packages from ports. That's not how we build package repositories here (and would expect that there are many more like us). We expect to start with an empty directory, go grab packages that we want (one by one) from FTP. Then put those tarballs into a directory. There's no additional step because we download the INDEX file to the repository too. Local repository is built. Is it reasonable to assume that this is still possible? We don't want to have to set up poudriere or build our own packages for packages that were already built at the time of release. (we don't track anything except security releng branches). (3) Make that repository available in a local file system or NFS That's what we want. I believe that pkg and poudriere address pretty much all the questions you have been asking, so you just need to set up the environment so that you can do experiments to integrate it with bsdconfig. But I want bsdconfig to work with local repositories without having poudriere. PC-BSD and DragonFlyBSD have switched to using pkg, so it is quite well tested. Cool. Baptiste and the rest of the pkgng and poudriere team have put a lot of effort and thought into this, and I believe you will be pleased with the results once you migrate over to these tools. I think so too. But right now a *lot* of unanswered questions. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] No more pkg_install on HEAD by default
On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote: Hi, I have just committed (r253305) a change the make pkg_install not being built and installed by default on HEAD. If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your src.conf(5) I think while a good move, we need to allow a window of development to re-work other HEAD components. It might also be worth developing a lint-tool to make sure we get every last instance (both from C code and sh code) within our base before we motion to cut a release with this change. I for one am effected and will have to change things. Can you point us at a guide (or even better, a Wiki) that can smooth the process? -- Devin NOTE: I cut ports off the CC as they don't need to worry about this (this being heading toward cut of a RELEASE with untold uses of pkg_tools in base) _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[CFT] sysutils/bsdconfg (0.9.0) and sysutils/sysrc (5.2)
Hey all, With SVN r252862, bsdconfig(8) and sysrc(8) are alive in HEAD. They are for the most-part vestigial organs that will be enmeshed further as they breathe fresh-air, but for now they certainly won't cause the build the break (100% shell; the only thing that gets compiled are the man-pages) and won't disturb anything else. What's been unfurled into HEAD is exactly the same as what's just been released to ports: sysutils/bsdconfig (latest version 0.9.0) sysutils/sysrc (latest version 5.2) My plan is to merge these to stable/9 in 3 days. Reasons for merge are stated in above-revision log-message (mail will be sent to releng and anybody else that cares to receive it or redirect it). This e-mail is about testing. I have been testing on 9.0-R all this time and there should be no problems with 9.1. But if some brave souls will install the port onto 9.1-R and give it a go, that would be great (this is, afterall, a Call For Testing). If you test bsdconfig on anything higher than 9.1-R (say, 9.1-STABLE), package management may not be kind to you (but there's a work-around, go into the options dialog and set your release as 9.1-RELEASE). This has to do with the way binary packages are stored on the FTP server (of course, if you have your own local repository, success depends on your ability to make the repository look official -- just as you had to with sysinstall). No bother testing bsdconfig on anything older than 9.0-R (while sysrc requires only 4.x or higher). Thanks all so much in-advance. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: FreeBSD-10.0 code freeze
On May 27, 2013, at 6:33 AM, Oded Shanoon wrote: Hi, I heard a rumor that the 10.0 code freeze will be in the end of July. Is that true? No. The release engineering team will shift focus to 9.2-R after the current focus of 8.4-R. Can anyone tell me when does it plan to be? 10 will be the focal point sometime after 9.2-R. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[UPDATE] sysutils/bsdconfig snapshot
Hello, I've taken a new snapshot of HEAD usr.sbin/bsdconfig and made it available through the ports tree (as sysutils/bsdconfig). The last snapshot was quite some time months ago, so a lot has changed since then. Most notably, we now have the beginnings of the package management module, edging ever-closer to a 1.0 release status. I'd like to see if there are any interested folks out there that could give my updated version a go and provide some feedback. Any/all feedback would be greatly appreciated. Just an FYI however… this code is only expected to work on 9.0-R or higher. -- Cheers, Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [UPDATE] sysutils/bsdconfig snapshot
(replying to my own request with some additional details on what to try in this update) On May 7, 2013, at 1:36 PM, Teske, Devin wrote: Hello, I've taken a new snapshot of HEAD usr.sbin/bsdconfig and made it available through the ports tree (as sysutils/bsdconfig). The last snapshot was quite some time months ago, so a lot has changed since then. Files have moved, there's now a /usr/share/bsdconfig directory for shared include files, and the fake language files have been removed. That's what you'll notice from what the new port installs. Most notably, we now have the beginnings of the package management module, edging ever-closer to a 1.0 release status. Of which it's important to note that this initial work is based on the old pkg_tools and not pkgng. We'll convert it to pkgng later, but first we need to make sure the framework for interacting with huge package data is usable. One of the challenges in testing this feature is that on 9.1, there aren't any packages available (yet; that should change eventually -- being restored retroactively). However, if you don't use the FTP media types, you'll get a lot further. What I've been using as a test is to download the 9.0-RELEASE packages/INDEX file and throw that into a local packages directory and then point the Directory media type at the parent directory where the packages directory was created. For example, … cd /tmp mkdir packages fetch ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/9.0-RELEASE/packages/INDEX Then use the Directory media type and enter a value of /tmp. The packages/INDEX file will be found in the media (/tmp directory) and the the packages interface will then be presented. If you're already running 9.0-RELEASE, then you can choose the FTP Passive media type, then choose the second menu item title URL and enter the URL ftp://ftp-archive.freebsd.org;. ASIDE: /me notes that he really ought to change the FTP routine to use a default $FTPMODE of auto instead of active (currently, FTP means explicitly active, and FTP Passive means explicitly passive). Once you're in the package interface, you will see: 1. List of package categories. NOTE: For a package category to be listed, it has to have at least one package. This list also includes a dummy category named All. 2. For each category (including the All dummy category), the number of: 2.a. packages available in that category 2.b. packages that are installed from that category 2.c. selected packages (a selected package is one that has been been marked for install, re-install, un-install, or marked as a dependency of a package to install/re-install) are displayed. 3. Above the category list is a Review menu item. Use this to review the selections you've made. Choosing a category intuitively displays the packages available from that category. In the list of packages, packages that are already installed are marked with an X; those marked for install have an I; re-install have an R; un-install, U; and dependencies are marked with a D. Selecting a package that is not marked will turn the mark into an I. Selecting a package that is marked with an X will bring up a menu asking if you would like to mark it for Uninstall, Re-install, or return it back to just being installed. Selecting a package that is marked as a dependency turns it into a package requested for installation (D mark becomes I mark). If without leaving the package selection screen you then re-select the same package (which was a dependency turned into an I), the mark goes away, but if you exit and return to the package selection menu, the mark is restored as a D. This is because package dependencies are not calculated until you return to the category list (when counts must be updated). When you're navigating the menus, you may notice an asterisk (*) on some menu items. This indicates that a page has been cached. Unlike sysinstall which would spend ~10 minutes (on my hardware) processing the INDEX file, building the in-memory structures for all category submenus, bsdconfig only generates the menus you request, and does them on-the-fly. The first time you are presented with the list of package categories, no category has an asterisk next to it. The first time you enter a package category, this is a small 1-5 second delay (depending on hardware) to generate the menu(s) for that category. When you return to the category list, that category will now have an asterisk displayed next to it to let you know: a. this category has been visited and b. the menus have been cached in-memory so re-entering this category will be fast (less than one second). Another notable feature once you get to the package menus is that not all packages are displayed on one page. There is a global PACKAGES_MENU_SIZE which defaults to 2000 and sets the maximum number of packages that can appear on one page (in contrast, sysinstall would display all 21,000
