Re: ZFS Boot patch

[Teske, Devin]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512


On Dec 18, 2013, at 9:34 AM, Allan Jude wrote:

On 2013-12-18 12:27, Allan Jude wrote:
An issue we thought we had fixed, was not actually fixed.

When doing a GELI based Root-on-ZFS install, the 'bootpool' is not
always properly mounted in the installed system.

The lines added to loader.conf to make it use the zpool.cache file and
learn of the existence of the 2nd pool are required, and have the
desired effect.

However, it seems that the 2nd pool is not always listed in the cache file.

The attached patch should fix this issue.

Hopefully this can get MFCd in time for the next RC


A note for the release notes. If you have an existing install, it can be
'repaired' easily:

zpool import -f bootpool

Yes, I too had noticed this. Figured it was a minor annoyance.



zpool set cachefile=/boot/zfs/zpool.cache bootpool


I'll have to re-test, but I didn't find this to be necessary. After importing
the bootpool and rebooting, I noticed it always came back.

The explanation I told myself was:

1. You boot into the new system
2. You import the bootpool
3. That adds an entry to /boot/zfs/zpool.cache
4. Next time you reboot, that entry is still in there

But what I was trying to figure out was how exactly to get the bootpool
to auto-import into the newline installed system -- I think the above line
you shared shows me precisely how we can accomplish that (I see it
in the patch you submitted).


This will add the bootpool to the existing zpool.cache (which contains
the data for your main pool)

This only applies to users who opted to encrypt their zpool.


Minor nit... not only for geli; now all MBR layouts use a bootpool. I found
this to be required for the edge-case of MBR+NOSWAP (which I fiddled
with for almost a week and got nowhere without using a bootpool).
- -- 
Devin
-BEGIN PGP SIGNATURE-
Comment: GPGTools - https://gpgtools.org

iQEcBAEBCgAGBQJSsgTUAAoJEKrMn5R9npq5SwEH/2iCbpkLaYcKOpn8ibMAxY7k
CUvpP8A5r3LEIxrf6lSIuiF0UpavTBQDZLS0n4mjAsQDQh2nyvXp8URpbjFUoHtS
CKCP8MlaNit5xEnnIx3nCOuU7yelBkTUUqzSGfKFZc0GHO1uhV9OTvuwNvyrNce2
y4/6l5ieUH4deoLTNQDIS4p6BoJr94JIopWLx/A+jF3SOlt34Z/LxeVEvQGeUaZ7
7YeAfg/NxeAGSnmRBNPox+HloGJvrHXzAhZLmXZI0cUUwvue/icmYH/hth9bmNYZ
NGMJckMyAZm600Vi9OoDqJf8y+sHjrYmLAuyXu1hfDb6CBfZD9S49CsTWV40W54=
=rP4m
-END PGP SIGNATURE-

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Defaults in 10.0 ZFS through bsdinstall

[Teske, Devin]

On Nov 14, 2013, at 12:34 PM, Marcus Reid wrote:

> On Thu, Nov 14, 2013 at 06:35:43PM +0000, Teske, Devin wrote:
>> On Nov 14, 2013, at 10:21 AM, Allan Jude wrote:
>>> On 2013-11-14 9:34, Reid, Marcus wrote:
>>> 
>>> Hi,
>>> 
>>> I noticed a couple of things with the ZFS defaults that result from
>>> using the new installer in 10.0-BETA3.
>>> 
>>> One, atime is turned off everywhere by default.  There was a thread
>>> on this list on June 8 with a subject of 'Changing the default for
>>> ZFS atime to off?', and from what I can tell the idea of turning off
>>> atime by default was not a popular one.
>>> 
>>> Two, and probably less controversial, is that fletcher4 is specified
>>> exlicitly on the root pool, even though it is default (wouldn't you
>>> just want to go with the default, in case it changes?)
>>> 
>>> Marcus
>> 
>> I have never heard a good argument for having atime on.
> 
> Can you address some of the issues that people brought up in the thread
> I mentioned earlier?  I'll summarize some:
> 
>  - breaks some software (MTAs were mentioned), and the admin should
>know when to turn atime on in those cases.
> 
>  - "any mail program using mbox mail folders uses them to correctly
>report which mailboxes have not been read yet"
> 

I'm looking at HEAD and I don't see "atime=off" for the /var dataset.
Knowing that most folks (accepting the defaults) will store their mail
in /var/mail ... does the concern over atime=off still exist?

However, I did notice that before we go creating the /var dataset, we
do set "atime=off" for the boot pool/dataset.

Is inheritance at-play here? and /var is turning up with atime=off even
though it's not specified?

In that case, I certainly agree we should remove atime=off from the last
place it is used -- the boot pool (nowhere else).




>  - "Of course it can't be turned off by default.  It is specified by
>POSIX."
> 
>  - "If the overhead were a real problem then file systems would use a
>special atime cache"
> 
>> The performance penalty on ZFS is quite large
> 
> This is subjective, and if it were that bad it seems like other
> ZFS-based systems would have turned it off as well, or done something
> like caching/logging or relatime it to mitigate the performance hit.
> 

I do recall discussions about bringing over relatime and making it the
default.

Allan was the one that brought up the performance hit...
Similarly where I $work, we turn it off where need performance (large
storage arrays and such).

I gather it's only a matter of time until relatime is default.



>> and it also makes your snapshots grow constant.
> 
> This does not seem to be the case.  For example, if I snapshot
> zroot/usr/src and then tar it up, I generate a bunch of writes and the
> snapshot grows to 70.5MB.  However, when I tar it up a second time,
> there are writes but the snapshot remains 70.5MB.  It seems to use the
> same blocks for subsequent atime updates.
> 
> If you made another snapshot, you would see more space used for that
> one, of course.
> 

I'll defer to Allan to expand (was his recollection on snapshots)


>> If you have a use for it, you can turn it on I guess. This would be
>> solved by having the dataset editor we're planning for 10.1
>> 
>> Seems easy enough to turn on once installed. And like Allan says, the
>> editor for 10.1 would allow changing of the default.
> 
> The same case could be made for turning it off, make that an option in
> the installer instead.
> 

Fair enough. Good idea.



> Some people don't seem to use atime on a regular basis, but I use it a
> lot when troubleshooting things, and I think others do to when they get
> to the point where they realize how useful it is.
> 
>> I'll add that if scripting it, you can (in current state for 10.0)
>> change the dataset options at-will.
>> 
>> The fletcher4 thing is a leftover, from older versions of ZFS where
>> the default was fletcher2, I suppose we can remove it
> 
> While I'm picking nits:
> 
> There are some other locally set things that are default (exec set to on
> on zroot/var/tmp is one), that might just inherit that from zroot/var.
> 

Need a +1 from Allan on that.


> Also, zroot doesn't appear to be mounted on /zroot, as it is on other
> ZFS-based systems.  This has the side-effect of newly created
> datasets not having a mountpoint by default, instead of /zroot/dataset.
> I don't know of the reason behind this change either.
> 

Hmm, wonder if that's

Re: Defaults in 10.0 ZFS through bsdinstall

[Teske, Devin]

On Nov 14, 2013, at 12:34 PM, Marcus Reid wrote:

> On Thu, Nov 14, 2013 at 06:35:43PM +0000, Teske, Devin wrote:
>> On Nov 14, 2013, at 10:21 AM, Allan Jude wrote:
>>> On 2013-11-14 9:34, Reid, Marcus wrote:
>>> 
>>> Hi,
>>> 
>>> I noticed a couple of things with the ZFS defaults that result from
>>> using the new installer in 10.0-BETA3.
>>> 
>>> One, atime is turned off everywhere by default.  There was a thread
>>> on this list on June 8 with a subject of 'Changing the default for
>>> ZFS atime to off?', and from what I can tell the idea of turning off
>>> atime by default was not a popular one.
>>> 
>>> Two, and probably less controversial, is that fletcher4 is specified
>>> exlicitly on the root pool, even though it is default (wouldn't you
>>> just want to go with the default, in case it changes?)
>>> 
>>> Marcus
>> 
>> I have never heard a good argument for having atime on.
> 
> Can you address some of the issues that people brought up in the thread
> I mentioned earlier?  I'll summarize some:
> 
>  - breaks some software (MTAs were mentioned), and the admin should
>know when to turn atime on in those cases.
> 
>  - "any mail program using mbox mail folders uses them to correctly
>report which mailboxes have not been read yet"
> 

I'm looking at HEAD and I don't see "atime=off" for the /var dataset.
Knowing that most folks (accepting the defaults) will store their mail
in /var/mail ... does the concern over atime=off still exist?

However, I did notice that before we go creating the /var dataset, we
do set "atime=off" for the boot pool/dataset.

Is inheritance at-play here? and /var is turning up with atime=off even
though it's not specified?

In that case, I certainly agree we should remove atime=off from the last
place it is used -- the boot pool (nowhere else).




>  - "Of course it can't be turned off by default.  It is specified by
>POSIX."
> 
>  - "If the overhead were a real problem then file systems would use a
>special atime cache"
> 
>> The performance penalty on ZFS is quite large
> 
> This is subjective, and if it were that bad it seems like other
> ZFS-based systems would have turned it off as well, or done something
> like caching/logging or relatime it to mitigate the performance hit.
> 

I do recall discussions about bringing over relatime and making it the
default.

Allan was the one that brought up the performance hit...
Similarly where I $work, we turn it off where need performance (large
storage arrays and such).

I gather it's only a matter of time until relatime is default.



>> and it also makes your snapshots grow constant.
> 
> This does not seem to be the case.  For example, if I snapshot
> zroot/usr/src and then tar it up, I generate a bunch of writes and the
> snapshot grows to 70.5MB.  However, when I tar it up a second time,
> there are writes but the snapshot remains 70.5MB.  It seems to use the
> same blocks for subsequent atime updates.
> 
> If you made another snapshot, you would see more space used for that
> one, of course.
> 

I'll defer to Allan to expand (was his recollection on snapshots)


>> If you have a use for it, you can turn it on I guess. This would be
>> solved by having the dataset editor we're planning for 10.1
>> 
>> Seems easy enough to turn on once installed. And like Allan says, the
>> editor for 10.1 would allow changing of the default.
> 
> The same case could be made for turning it off, make that an option in
> the installer instead.
> 

Fair enough. Good idea.



> Some people don't seem to use atime on a regular basis, but I use it a
> lot when troubleshooting things, and I think others do to when they get
> to the point where they realize how useful it is.
> 
>> I'll add that if scripting it, you can (in current state for 10.0)
>> change the dataset options at-will.
>> 
>> The fletcher4 thing is a leftover, from older versions of ZFS where
>> the default was fletcher2, I suppose we can remove it
> 
> While I'm picking nits:
> 
> There are some other locally set things that are default (exec set to on
> on zroot/var/tmp is one), that might just inherit that from zroot/var.
> 

Need a +1 from Allan on that.


> Also, zroot doesn't appear to be mounted on /zroot, as it is on other
> ZFS-based systems.  This has the side-effect of newly created
> datasets not having a mountpoint by default, instead of /zroot/dataset.
> I don't know of the reason behind this change either.
> 

Hmm, wonder if that's

Re: [CFT] bsdinstall and zfsboot enhancements

[Teske, Devin]

On Nov 12, 2013, at 4:02 AM, Johan Hendriks wrote:

> Teske, Devin schreef:
>> Hi all,
>> 
>> Another Call For Testing...
>> This one is for bsdinstall.
>> 
>> Two patchsets are required for this CFT:
>> 
>> http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_debug/
>> http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/
>> 
>> The enhancements are:
>> 
>> + Add a `-D FILE" command-line option for overriding the
>> path to the bsdinstall log file (BSDINSTALL_LOG env var).
>> 
>> + Document new `-D FILE' in the man page for bsdinstall.
>> 
>> + If FILE in `-D FILE' begins with a +, debug output goes to
>> stdout (interleaved between dialog(1) invocations/output) as
>> well as to FILE (minus the leading + of course).
>> 
>> + If BSDINSTALL_LOG cannot be written, then debugging is
>> disabled (except in the case of a leading + in the pathname,
>> wherein debug will still be printed to stdout).
>> 
>> + Update source code format to approximate a future shstyle(9)
>> 
>> + Fix a dangling participle ("Begun ..." -> "Began ...")
>> 
>> + Rewrite the docsinstall script (was necessary to abate direct
>> dependency on BSDINSTALL_LOG (instead, use fault-tolerant
>> bsdconfig framework which displays appropriate errors for
>> package management).
>> 
>> + Add additional debug output for dhclient/rtsol/wpa_cliscan
>> 
>> + Display script errors in a textbox rather than just on stdout
>> 
>> + Update many coments.
>> 
>> + Add new f_show_err() API call (like f_show_msg but changes
>> the dialog title to "Error")(see bsdconfig's `common.subr').
>> 
>> + Add new f_eval_catch() API call for executing a command via
>> eval but not before logging the command to debug. Several
>> example cases documented in API header for function in
>> bsdconfig's `common.subr'.
>> 
>> + Fix dialog auto-sizing when launched as an rvalue to a pipe
>> for indirected scripts (previously would default to 80x24 sizing
>> in this case, now it can autosize to full size even when in a pipe
>> chain).
>> 
>> + Fix a bug in f_snprintf wherein if the $format argument began
>> with a "-" or "--" it would be misinterpreted as a flag to printf. (this
>> is in bsdcofig's strings.subr)
>> 
>> + Add accompanying f_sprintf() and f_vsprintf() to go along with
>> already existing f_snprintf() and f_vsnprintf() (see bsdconfig's
>> strings.subr).
>> 
>> + Update bsdinstall's "config" script to adjust ttyu* entries in
>> /etc/ttys when it is determined that we are in-fact doing an install
>> over serial (e.g. bhyve).
>> 
>> + Remove some unnecessary default ZFS datasets from the
>> automatic "zfsboot" script. Such as: /usr/ports/distfiles
>> /usr/ports/packages /usr/obj /var/db /var/empty /var/mail and
>> /var/run (these can all be created as-needed once the system is
>> installed).
>> 
>> + Remove setuid=off for /usr/home (as discussed with others from
>> last round of CFT).
>> 
>> + Fix some i18n string violations in "zfsboot"
>> 
>> + Bolster debugging output in "zfsboot"
>> 
>> + Fix some string quoting issues in "zfsboot"
>> 
>> + Fix some variable scope issues in "zfsboot"
>> 
>> + Only display the ZFS vdev type selection menu when running
>> interactively (duh).
>> 
>> + Change "Create" to "Install" in "zfsboot" main menu
>> 
>> + Increase pedantic error checking in "zfsboot" (type-check
>> arguments and such).
>> 
>> + Add a call to "graid destroy" to kill automatic metadata (part of
>> the series of pedantic destructions we do when bootstrapping
>> a new/naked disk).
>> 
>> + Make judicious use of new f_eval_catch() in "zfsboot"
>> 
>> + More comments (zfsboot).
>> 
>> + Fixup some variable names for consistency (zfsboot).
>> 
> I did not try this installer myself, but one question, can i seperate the 
> swap space from the pool?
> I have seen to many strange hangs on a server where i use swap on zfs!
> 

If I understand correctly, it does this by default.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Default MBR boot "manager"

[Teske, Devin]

On Nov 11, 2013, at 2:57 PM, Nathan Whitehorn wrote:

> On 11/11/13 15:51, Teske, Devin wrote:
>> On Nov 11, 2013, at 1:43 PM, Nathan Whitehorn wrote:
>> 
>>> On 11/11/13 15:39, Teske, Devin wrote:
>>>> On Nov 11, 2013, at 1:32 PM, Nathan Whitehorn wrote:
>>>> 
>>>>> On 11/11/13 15:19, Teske, Devin wrote:
>>>>>> Topic: Lenovo Laptops and bsdinstall zfsboot with MBR layout...
>>>>>> 
>>>>>> Should we do the quick patch to change the default
>>>>>> from /boot/boot0 to /boot/mbr:
>>>>>> 
>>>>>> Index: zfsboot
>>>>>> ===
>>>>>> --- zfsboot (revision 258016)
>>>>>> +++ zfsboot (working copy)
>>>>>> @@ -764,7 +764,7 @@ zfs_create_diskpart()
>>>>>> #
>>>>>> f_eval_catch $funcname gpart "$GPART_CREATE" mbr \$disk 
>>>>>> ||
>>>>>>  return $FAILURE
>>>>>> -   f_eval_catch $funcname gpart "$GPART_BOOTCODE" 
>>>>>> /boot/boot0 \
>>>>>> +   f_eval_catch $funcname gpart "$GPART_BOOTCODE" /boot/mbr 
>>>>>> \
>>>>>>  \$disk || return $FAILURE
>>>>>> 
>>>>>> #
>>>>>> 
>>>>>> That would fix things for Lenovo laptops for the next
>>>>>> release until I finish up the bootcode selection menu.
>>>>>> I'd like to take my time in making sure Allan and I design
>>>>>> a worthy bootcode selection menu.
>>>>> This patch looks good (I don't remember why it was boot0 in the first 
>>>>> place). I think gpart automatically installs something like /boot/mbr by 
>>>>> default, so I'd be interested to know if making the diff purely negative 
>>>>> still works.
>>>>> 
>>>>> On another note, I think we should move away from a selector. Right now, 
>>>>> we have three kinds of boot code:
>>>>> 1. ZFS boot code
>>>>> 2. UFS boot code
>>>>> 3. boot0
>>>>> 
>>>>> Unifying 1 and 2 would help a lot -- I don't know of any reason we need 
>>>>> both except for tradition. #3 is probably best done as a post-install 
>>>>> config step ("Install FreeBSD boot manager" or something), which also 
>>>>> means it works for UFS systems.
>>>> Well, I'm sensitive to the fact that sysinstall offered "none" and
>>>> even explained why in an F1 dialog that brought up "drives.hlp"
>>>> to explain that you might want to keep whatever (alternate) boot
>>>> manager you may be using already.
>>>> 
>>>> In a proposed selector, the full breadth of options that I was
>>>> envisioning was:
>>>> 
>>>> GPT + gptboot
>>>> GPT + none (use your existing boot manager... syslinux?)
>>>> MBR + mbr
>>>> MBR + boot0
>>>> MBR + none (again, BYOBM)
>>>> 
>>>> Hadn't got around to zfsboot yet. Where would that go? at the top?
>>>> 
>>>> GPT + zfsboot ?
>>>> 
>>>> (and of course, this is x86 specific... I was gleaning from sysinstall
>>>> that for systems like pc98, they call it an IPL and there's only two
>>>> options... a standard IPL or bring your own boot manager, aka "none").
>>>> 
>>>> I imagine that there would be architectures that are like the ol' pc98,
>>>> wherein they don't have all these options (is, for example? sparc64
>>>> GPT only?)
>>> This would be super-unportable. SPARC uses VTOC8, for example, and doesn't 
>>> support GPT at all. PC98 has the differences you mentioned. PowerPC uses 
>>> MBR sometimes, sometimes APM, sometimes GPT. You never have a choice. No 
>>> platforms except x86 have any analog to boot0. Etc, etc. This is why I'd 
>>> like to pull ZFS into partedit, which already knows how to set up 
>>> everything and does the right thing everywhere. For the only system (x86) 
>>> where there is a real choice (do you want to replace whatever you have 
>>> already with boot0?), it makes sense to do this as a post-install conf

Re: Default MBR boot "manager"

[Teske, Devin]

On Nov 11, 2013, at 1:43 PM, Nathan Whitehorn wrote:

> On 11/11/13 15:39, Teske, Devin wrote:
>> On Nov 11, 2013, at 1:32 PM, Nathan Whitehorn wrote:
>> 
>>> On 11/11/13 15:19, Teske, Devin wrote:
>>>> Topic: Lenovo Laptops and bsdinstall zfsboot with MBR layout...
>>>> 
>>>> Should we do the quick patch to change the default
>>>> from /boot/boot0 to /boot/mbr:
>>>> 
>>>> Index: zfsboot
>>>> ===
>>>> --- zfsboot (revision 258016)
>>>> +++ zfsboot (working copy)
>>>> @@ -764,7 +764,7 @@ zfs_create_diskpart()
>>>> #
>>>> f_eval_catch $funcname gpart "$GPART_CREATE" mbr \$disk ||
>>>>  return $FAILURE
>>>> -   f_eval_catch $funcname gpart "$GPART_BOOTCODE" /boot/boot0 
>>>> \
>>>> +   f_eval_catch $funcname gpart "$GPART_BOOTCODE" /boot/mbr \
>>>>  \$disk || return $FAILURE
>>>> 
>>>> #
>>>> 
>>>> That would fix things for Lenovo laptops for the next
>>>> release until I finish up the bootcode selection menu.
>>>> I'd like to take my time in making sure Allan and I design
>>>> a worthy bootcode selection menu.
>>> This patch looks good (I don't remember why it was boot0 in the first 
>>> place). I think gpart automatically installs something like /boot/mbr by 
>>> default, so I'd be interested to know if making the diff purely negative 
>>> still works.
>>> 
>>> On another note, I think we should move away from a selector. Right now, we 
>>> have three kinds of boot code:
>>> 1. ZFS boot code
>>> 2. UFS boot code
>>> 3. boot0
>>> 
>>> Unifying 1 and 2 would help a lot -- I don't know of any reason we need 
>>> both except for tradition. #3 is probably best done as a post-install 
>>> config step ("Install FreeBSD boot manager" or something), which also means 
>>> it works for UFS systems.
>> Well, I'm sensitive to the fact that sysinstall offered "none" and
>> even explained why in an F1 dialog that brought up "drives.hlp"
>> to explain that you might want to keep whatever (alternate) boot
>> manager you may be using already.
>> 
>> In a proposed selector, the full breadth of options that I was
>> envisioning was:
>> 
>> GPT + gptboot
>> GPT + none (use your existing boot manager... syslinux?)
>> MBR + mbr
>> MBR + boot0
>> MBR + none (again, BYOBM)
>> 
>> Hadn't got around to zfsboot yet. Where would that go? at the top?
>> 
>> GPT + zfsboot ?
>> 
>> (and of course, this is x86 specific... I was gleaning from sysinstall
>> that for systems like pc98, they call it an IPL and there's only two
>> options... a standard IPL or bring your own boot manager, aka "none").
>> 
>> I imagine that there would be architectures that are like the ol' pc98,
>> wherein they don't have all these options (is, for example? sparc64
>> GPT only?)
> 
> This would be super-unportable. SPARC uses VTOC8, for example, and doesn't 
> support GPT at all. PC98 has the differences you mentioned. PowerPC uses MBR 
> sometimes, sometimes APM, sometimes GPT. You never have a choice. No 
> platforms except x86 have any analog to boot0. Etc, etc. This is why I'd like 
> to pull ZFS into partedit, which already knows how to set up everything and 
> does the right thing everywhere. For the only system (x86) where there is a 
> real choice (do you want to replace whatever you have already with boot0?), 
> it makes sense to do this as a post-install config.

Two migration paths before us, and I do rather like the idea of
benefiting from all your work there.

My biggest concern is how to maximize functionality in the
migration of the ZFS stuff to partedit.

You know the code better there better than I, ... have you given
much thought to how you might integrate what we've done?

It's sad that we would be giving up i18n, X11, and discrete
scripting (surely there are more parts to ZFS than what partedit
supports now -- e.g., datasets, etc.).

Naturally, the scripting can be solved. i18n is a bit harder to
solve as it's a "start from the bottom" venture. And I fear X11 is
a lost cause in its current state for partedit.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Default MBR boot "manager"

[Teske, Devin]

On Nov 11, 2013, at 1:32 PM, Nathan Whitehorn wrote:

> On 11/11/13 15:19, Teske, Devin wrote:
>> Topic: Lenovo Laptops and bsdinstall zfsboot with MBR layout...
>> 
>> Should we do the quick patch to change the default
>> from /boot/boot0 to /boot/mbr:
>> 
>> Index: zfsboot
>> ===
>> --- zfsboot (revision 258016)
>> +++ zfsboot (working copy)
>> @@ -764,7 +764,7 @@ zfs_create_diskpart()
>> #
>> f_eval_catch $funcname gpart "$GPART_CREATE" mbr \$disk ||
>>  return $FAILURE
>> -   f_eval_catch $funcname gpart "$GPART_BOOTCODE" /boot/boot0 \
>> +   f_eval_catch $funcname gpart "$GPART_BOOTCODE" /boot/mbr \
>>  \$disk || return $FAILURE
>> 
>> #
>> 
>> That would fix things for Lenovo laptops for the next
>> release until I finish up the bootcode selection menu.
>> I'd like to take my time in making sure Allan and I design
>> a worthy bootcode selection menu.
> 
> This patch looks good (I don't remember why it was boot0 in the first place). 
> I think gpart automatically installs something like /boot/mbr by default, so 
> I'd be interested to know if making the diff purely negative still works.
> 
> On another note, I think we should move away from a selector. Right now, we 
> have three kinds of boot code:
> 1. ZFS boot code
> 2. UFS boot code
> 3. boot0
> 
> Unifying 1 and 2 would help a lot -- I don't know of any reason we need both 
> except for tradition. #3 is probably best done as a post-install config step 
> ("Install FreeBSD boot manager" or something), which also means it works for 
> UFS systems.

Well, I'm sensitive to the fact that sysinstall offered "none" and
even explained why in an F1 dialog that brought up "drives.hlp"
to explain that you might want to keep whatever (alternate) boot
manager you may be using already.

In a proposed selector, the full breadth of options that I was
envisioning was:

GPT + gptboot
GPT + none (use your existing boot manager... syslinux?)
MBR + mbr
MBR + boot0
MBR + none (again, BYOBM)

Hadn't got around to zfsboot yet. Where would that go? at the top?

GPT + zfsboot ?

(and of course, this is x86 specific... I was gleaning from sysinstall
that for systems like pc98, they call it an IPL and there's only two
options... a standard IPL or bring your own boot manager, aka "none").

I imagine that there would be architectures that are like the ol' pc98,
wherein they don't have all these options (is, for example? sparc64
GPT only?)
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Default MBR boot "manager"

[Teske, Devin]

Topic: Lenovo Laptops and bsdinstall zfsboot with MBR layout...

Should we do the quick patch to change the default
from /boot/boot0 to /boot/mbr:

Index: zfsboot
===
--- zfsboot (revision 258016)
+++ zfsboot (working copy)
@@ -764,7 +764,7 @@ zfs_create_diskpart()
#
f_eval_catch $funcname gpart "$GPART_CREATE" mbr \$disk ||
 return $FAILURE
-   f_eval_catch $funcname gpart "$GPART_BOOTCODE" /boot/boot0 \
+   f_eval_catch $funcname gpart "$GPART_BOOTCODE" /boot/mbr \
 \$disk || return $FAILURE

#

That would fix things for Lenovo laptops for the next
release until I finish up the bootcode selection menu.
I'd like to take my time in making sure Allan and I design
a worthy bootcode selection menu.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] bsdinstall and zfsboot enhancements

[Teske, Devin]

On Nov 11, 2013, at 12:59 PM, Peter Grehan wrote:

> Hi Devin,
> 
>> When boot is executed, I know I can see "kenv console", but hadn't realized
>> that there were/are a host of others that are slurped into the kernel for 
>> later
>> (very purposeful) fetching.
>> 
>> So when you say that bhyve requests a serial console... I assume now it's
>> setting variables... but via raw Forth? C code? loader.conf(5)? I've seen my
>> menu come up under bhyve, and I noticed that it only has a 5-second count-
>> down instead of the usual 9 -- but I'm curious how you're exporting the 
>> variables.
> 
> bhyve uses the loader facility to set env variables directly from loader 
> code. This is no different than other arch ports of the loader that set env 
> variables. The forced setting of boot_serial is just a convenience for users 
> since that is the only supported console type.
> 

Thank you much for the explanation. I see it from end-to-end now.


> It doesn't touch the timeout code - should be whatever FreeBSD loader forth 
> scripts tell it to do.
> 

Ah, must have been something Michael did. I noticed this whilst getting
demos from him on his laptop.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] bsdinstall and zfsboot enhancements

[Teske, Devin]

On Nov 11, 2013, at 12:56 PM, Nathan Whitehorn wrote:

> On 11/11/13 14:52, Teske, Devin wrote:
>> On Nov 11, 2013, at 12:44 PM, Peter Grehan wrote:
>> 
>>> Hi Devin,
>>> 
>>>> Question:
>>>> Does bhyve set kern.console irrespective of loader.conf values?
>>> The kernel sets it based on what it determines the console to be. Bhyve 
>>> influences that by requesting a serial console. This is no different than 
>>> booting on a headless machine with a serial console.
>>> 
>> Well, for a headless meachine, I would set console=comconsole,vidconsole
>> in loader.conf(5), then our Forth code slurps it in via loader.4th + 
>> support.4th
>> routines...
>> 
>> When boot is executed, I know I can see "kenv console", but hadn't realized
>> that there were/are a host of others that are slurped into the kernel for 
>> later
>> (very purposeful) fetching.
>> 
>> So when you say that bhyve requests a serial console... I assume now it's
>> setting variables... but via raw Forth? C code? loader.conf(5)? I've seen my
>> menu come up under bhyve, and I noticed that it only has a 5-second count-
>> down instead of the usual 9 -- but I'm curious how you're exporting the 
>> variables.
> 
> I think you've misunderstood. kern.console isn't set by loader. It reflects 
> the state of the kernel, which decides what to do autonomously based on a 
> number of driver and platform-dependent things including, but not limited to, 
> kenv (loader variables, for instance).

I was thinking it, but didn't express it properly... yes... kern.console is 
influenced
by loader variables.

"slurped" is definitely not the technical explanation I should be striving for 
;D (that
is, if I'm trying to be clear).
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] bsdinstall and zfsboot enhancements

[Teske, Devin]

On Nov 11, 2013, at 12:54 PM, Nathan Whitehorn wrote:

> On 11/11/13 14:30, Nathan Whitehorn wrote:
>> On 11/11/13 14:18, Teske, Devin wrote:
>>> -BEGIN PGP SIGNED MESSAGE-
>>> Hash: SHA512
>>> 
>>> 
>>> On Nov 11, 2013, at 11:46 AM, Michael Dexter wrote:
>>> 
>>> 
>>> Hello all,
>>> 
>>> I have been experimenting with various BSD and GNU/Linux boot media
>>> under bhyve and noticed that we may want to accommodate the "LiveCD"
>>> mode of the installer, which in turn requires the correct console.
>>> 
>>> Currently, one is prompted for VT100 for installation but this does not
>>> appear to work/stick for LiveCD mode.
>>> 
>>> Can anyone verify this?
>>> 
>>> 
>>> While I developed this patch...
>>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/usr.sbin%253A%253Absdinstall%253A%253Ascripts%253A%253Aconfig.patch?revision%3D1.10%26view%3Dmarkup&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=aZg%2BxwqLTX6Zcpf3Rc44iPtAQhFrpqoS4FC5B8ykxJQ%3D%0A&s=6d54d337ea5472f5713ddbe7788d3248c45feefd4b291eab0a976c39e9d40428
>>>  
>>> 
>>> Reasons exist to search for a better solution, see here:
>>> https://urldefense.proofpoint.com/v1/url?u=http://lists.freebsd.org/pipermail/freebsd-current/2013-November/046148.html&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=aZg%2BxwqLTX6Zcpf3Rc44iPtAQhFrpqoS4FC5B8ykxJQ%3D%0A&s=5f8128901747346f937ffc4478eadb4bc39059504258dfb9b36f0fb9e7a61b79
>>>  
>>> (and messages that follow it)
>>> 
>>> Is modifying init(8) still the way to go? What modification do we want to 
>>> make?
>>> I'll do the work if we can come to consensus.
>>> 
>>> Or should we touch up the patch in some way to address the original 
>>> concerns?
>>> 
>> 
>> I think modifying init is the way to go -- it keeps the install system from 
>> interfering with the installed one, as well as fixing this kind of issue 
>> with moved hard drives or PXE booting or what have you. If we can provide a 
>> guarantee that any system that displays text has a working console (unless 
>> explicitly configured not to), useability is improved.
>> 
>> I would propose one of the following (and volunteer to write the code):
>> 
>> Option A
>> 
>> 
>> 1. init checks if there is an entry in /etc/ttys for the terminal[s] 
>> corresponding to the value[s] in kern.console
>> 2. If an entry for each console terminal exists in /etc/ttys, enable it
>> 3. If not, invent one with a terminal type of "ansi"
>> 
>> The one issue here is that someone may want to force a particular entry to 
>> off and still have it be the kernel console. This is tricky. We could invent 
>> a new "status" field that is not "on" or "off" ("auto", maybe, or 
>> "ifconsole"?). Which brings us to:
> 
> One easy way to accomplish this is just to only implement (1) and (3), then 
> comment out the ttyu0 entry in /etc/ttys on x86 instead of marking it "off". 
> Then the behavior is just that a tty marked "off" stays off, one marked "on" 
> stays on, and one not present spawns login with a terminal type corresponding 
> to "console" (by default "unknown") if it happens to be the console. I will 
> implement this over the next few days and then send patches unless anyone has 
> an objection.

I love it. (smiles)

Excellent idea and that's the winner I think.

+1
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] bsdinstall and zfsboot enhancements

[Teske, Devin]

On Nov 11, 2013, at 12:44 PM, Peter Grehan wrote:

> Hi Devin,
> 
>> Question:
>> Does bhyve set kern.console irrespective of loader.conf values?
> 
> The kernel sets it based on what it determines the console to be. Bhyve 
> influences that by requesting a serial console. This is no different than 
> booting on a headless machine with a serial console.
> 

Well, for a headless meachine, I would set console=comconsole,vidconsole
in loader.conf(5), then our Forth code slurps it in via loader.4th + support.4th
routines...

When boot is executed, I know I can see "kenv console", but hadn't realized
that there were/are a host of others that are slurped into the kernel for later
(very purposeful) fetching.

So when you say that bhyve requests a serial console... I assume now it's
setting variables... but via raw Forth? C code? loader.conf(5)? I've seen my
menu come up under bhyve, and I noticed that it only has a 5-second count-
down instead of the usual 9 -- but I'm curious how you're exporting the 
variables.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] bsdinstall and zfsboot enhancements

[Teske, Devin]

On Nov 11, 2013, at 12:30 PM, Nathan Whitehorn wrote:

> On 11/11/13 14:18, Teske, Devin wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA512
>> 
>> 
>> On Nov 11, 2013, at 11:46 AM, Michael Dexter wrote:
>> 
>> 
>> Hello all,
>> 
>> I have been experimenting with various BSD and GNU/Linux boot media
>> under bhyve and noticed that we may want to accommodate the "LiveCD"
>> mode of the installer, which in turn requires the correct console.
>> 
>> Currently, one is prompted for VT100 for installation but this does not
>> appear to work/stick for LiveCD mode.
>> 
>> Can anyone verify this?
>> 
>> 
>> While I developed this patch...
>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/usr.sbin%253A%253Absdinstall%253A%253Ascripts%253A%253Aconfig.patch?revision%3D1.10%26view%3Dmarkup&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=CRY%2BFNOe6Q01s5oKeCFeGOhE0bW%2BvEmw6GTjyy5eHFo%3D%0A&s=3e5345ea6b13f84e719068bb993b66978f1971b648b430edfde9165677c279de
>> 
>> Reasons exist to search for a better solution, see here:
>> https://urldefense.proofpoint.com/v1/url?u=http://lists.freebsd.org/pipermail/freebsd-current/2013-November/046148.html&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=CRY%2BFNOe6Q01s5oKeCFeGOhE0bW%2BvEmw6GTjyy5eHFo%3D%0A&s=9ebff0354adb2634006e52cb4c0048287b7518bd7f4defe420ce5f34675ce5cd
>> (and messages that follow it)
>> 
>> Is modifying init(8) still the way to go? What modification do we want to 
>> make?
>> I'll do the work if we can come to consensus.
>> 
>> Or should we touch up the patch in some way to address the original concerns?
>> 
> 
> I think modifying init is the way to go -- it keeps the install system from 
> interfering with the installed one, as well as fixing this kind of issue with 
> moved hard drives or PXE booting or what have you. If we can provide a 
> guarantee that any system that displays text has a working console (unless 
> explicitly configured not to), useability is improved.
> 
> I would propose one of the following (and volunteer to write the code):
> 
> Option A
> 
> 
> 1. init checks if there is an entry in /etc/ttys for the terminal[s] 
> corresponding to the value[s] in kern.console

Is kern.console set by bhyve while NULL or unset when booting from
a physical PC (bare metal)?


> 2. If an entry for each console terminal exists in /etc/ttys, enable it
> 3. If not, invent one with a terminal type of "ansi"
> 
> The one issue here is that someone may want to force a particular entry to 
> off and still have it be the kernel console. This is tricky. We could invent 
> a new "status" field that is not "on" or "off" ("auto", maybe, or 
> "ifconsole"?). Which brings us to:
> 

Trying to think of an edge-case where they would want to force it to off.
If the kern.console setting is only expected to be set via ... help me out 
here...

+ bhyve ?
+ pxe ?
+ what else ?

Then maybe the secret sauce shouldn't be in /etc/ttys, but in the value that
is passed in via kern.console (that is, ... of the above ruminated technologies
which allow you to customize the value?)



> Option B
> ---
> 
> Very similar to Option A, except only provide an automatic console using (2) 
> and (3) if the "console" terminal is marked "on". This would increase the 
> magic attached to "console" in /etc/ttys, but fix the problem with (A).
> 
> It's possible another approach would work as well. Does anyone have thoughts 
> on this?

If I'm not wrong, there's a third Option C that skirts the issue by bolstering 
the ability
to change things for different needs.

That prior change that I had made to bsdinstall/scripts/config (iirc) was a big 
agressive
because it tried to automatically figure things out.

If we simply made it a choice (a scriptable one), then that would solve the 
problem for
the installation.

But for the LiveCD issue (Michael wants to bring up the installer's LiveCD via 
serial),
then perhaps we could solve it by using unionfs in the installer to make the 
install
environment writable (and again, change on-the-fly for different needs).

Just a thought. Tell me if it's not an option ;D I won't mind.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] bsdinstall and zfsboot enhancements

[Teske, Devin]

On Nov 11, 2013, at 12:35 PM, Nathan Whitehorn wrote:

> On 11/11/13 14:30, Teske, Devin wrote:
>> (disabling default gpg-signing until they fix a bug with the quoting)
>> 
>> On Nov 11, 2013, at 11:46 AM, Michael Dexter wrote:
>> 
>>> Hello all,
>>> 
>>> I have been experimenting with various BSD and GNU/Linux boot media
>>> under bhyve and noticed that we may want to accommodate the "LiveCD"
>>> mode of the installer, which in turn requires the correct console.
>>> 
>>> Currently, one is prompted for VT100 for installation but this does not
>>> appear to work/stick for LiveCD mode.
>>> 
>>> Can anyone verify this?
>>> 
>> Sorry, I mistook your issue in the previous e-mail to be on-going with the
>> thread it was inline with.
>> 
>> "LiveCD" changes things a bit.
>> 
>> (thinks)
>> 
>> I would expect that a prompt could do:
>> 
>>  1. modify /etc/ttys on the boot media
>>  2. run "init q"
>> 
>> But there's a couple of assumptions...
>> 
>>  3. Can we even write to /etc/ttys?
>>  NB: We can write to /tmp because it's an md0 swap device
>> 
>> So do we have to get fancy with slipping a unionfs layer backed by another
>> md swap device above the root?
>> 
>> That would make every file on the boot media writable (writes would go to
>> the swap-backed md device and if you execute "rm -fW file" you can get back
>> files that have been previously unlinked -- unlinks are stored as whiteouts 
>> in
>> the swap backed md device). For all intents and purposes, the read-only file-
>> system becomes writable and we could then munge /etc/ttys to enable serial
>> only when a menu item is chosen.
>> 
>> If I'm off-base, let me know... sounds like a lot of trouble.
>> 
>> The alternative being that you enable serial by default but then I have to 
>> tell
>> field engineers to unplug barcode readers before they do an install??? 
>> (that's
>> a question, it may be entirely safe, but I've never tried, seems unsafe)
>> 
>> Question is, how would you disable it? Goes back to writable filesystem.
> 
> This is why I don't think we want to modify /etc/ttys. Instead we want to 
> have init do the right thing (follow the boot console) with a single 
> unchanged /etc/ttys.

Let me see if I've got it right...

Already-knowns:
+ The boot variables are influenced by loader.conf which is read-only media.
+ But Michael is loading the media from bhyve.

Question:
Does bhyve set kern.console irrespective of loader.conf values?

If so, then yeah... I'm all for supporting your stated Option-A in the other 
thread.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] bsdinstall and zfsboot enhancements

[Teske, Devin]

(disabling default gpg-signing until they fix a bug with the quoting)

On Nov 11, 2013, at 11:46 AM, Michael Dexter wrote:

> 
> Hello all,
> 
> I have been experimenting with various BSD and GNU/Linux boot media
> under bhyve and noticed that we may want to accommodate the "LiveCD"
> mode of the installer, which in turn requires the correct console.
> 
> Currently, one is prompted for VT100 for installation but this does not
> appear to work/stick for LiveCD mode.
> 
> Can anyone verify this?
> 

Sorry, I mistook your issue in the previous e-mail to be on-going with the
thread it was inline with.

"LiveCD" changes things a bit.

(thinks)

I would expect that a prompt could do:

1. modify /etc/ttys on the boot media
2. run "init q"

But there's a couple of assumptions...

3. Can we even write to /etc/ttys?
NB: We can write to /tmp because it's an md0 swap device

So do we have to get fancy with slipping a unionfs layer backed by another
md swap device above the root?

That would make every file on the boot media writable (writes would go to
the swap-backed md device and if you execute "rm -fW file" you can get back
files that have been previously unlinked -- unlinks are stored as whiteouts in
the swap backed md device). For all intents and purposes, the read-only file-
system becomes writable and we could then munge /etc/ttys to enable serial
only when a menu item is chosen.

If I'm off-base, let me know... sounds like a lot of trouble.

The alternative being that you enable serial by default but then I have to tell
field engineers to unplug barcode readers before they do an install??? (that's
a question, it may be entirely safe, but I've never tried, seems unsafe)

Question is, how would you disable it? Goes back to writable filesystem.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] bsdinstall and zfsboot enhancements

[Teske, Devin]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512


On Nov 11, 2013, at 11:46 AM, Michael Dexter wrote:


Hello all,

I have been experimenting with various BSD and GNU/Linux boot media
under bhyve and noticed that we may want to accommodate the "LiveCD"
mode of the installer, which in turn requires the correct console.

Currently, one is prompted for VT100 for installation but this does not
appear to work/stick for LiveCD mode.

Can anyone verify this?


While I developed this patch...
http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/usr.sbin%3A%3Absdinstall%3A%3Ascripts%3A%3Aconfig.patch?revision=1.10&view=markup

Reasons exist to search for a better solution, see here:
http://lists.freebsd.org/pipermail/freebsd-current/2013-November/046148.html
(and messages that follow it)

Is modifying init(8) still the way to go? What modification do we want to make?
I'll do the work if we can come to consensus.

Or should we touch up the patch in some way to address the original concerns?
- -- 
Devin
-BEGIN PGP SIGNATURE-
Comment: GPGTools - https://gpgtools.org

iQEcBAEBCgAGBQJSgTuCAAoJEKrMn5R9npq5M90H/RrIhOqhCAPj9lyG9ElM6NlC
G2EDyajzIqntzOQY6O9AZXl+CrKndMkOAthOAYBfWTXCIKV0J5tlV167Gmfm25tZ
GQpJkneOiHwMuhj7Fe0WWAKaXWwLj8ASTI6bj2bFz3Xls/5p0Q6YA0TMBraUp9K3
NVJGbscp4S+BG6wYQjoAEzwek2Kz5yc4jHSJhfraLTEWmUwHczPV7Levp37rLeiW
n/OQT1p3w5OJVgnRFy9ptQCST/MaYU9KJWP360bXnqGYg6NrrrlFdIejnIIsSkxX
u21Mu+sF76S3rg0D+L+ByZgbP957azcJ9NeDTooi3Gb+/TqgWL9j4Qyp7DkR2no=
=T2FX
-END PGP SIGNATURE-

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [Review] bsdconfig pkgng integration

[Teske, Devin]

On Nov 6, 2013, at 5:13 PM, Teske, Devin wrote:

> 
> On Nov 6, 2013, at 5:02 PM, Glen Barber wrote:
> 
>> On Thu, Nov 07, 2013 at 12:48:02AM +0000, Teske, Devin wrote:
>>> I think that you're thinking that:
>>> 
>>> pkg.eu.freebsd.org
>>> pkg.us-east.freebsd.org
>>> pkg.us-west.freebsd.org
>>> 
>>> Will somehow change?
>> 
>> I think you're thinking they *won't* change.  Don't assume.
>> 
> 
> Well, what *kind* of change?
> 
> Changes to what they serve should be no problem...
> 
> but what I was assuming that wouldn't change was
> their actual names. (e.g. "pkg.eu" stays "pkg.eu").
> 
> Couldn't care about any other change, because that
> would percolate at runtime off the DNS servers.

We concluded that despite the fact that pkg.* are officially
supported names, a POLA issue has been brought to the
top of the discussion.

That is to say, that the concern is that someone may choose
pkg.eu or pkg.us-* and then become astonished if/when
that name mirror is down (precisely because we do not yet
have geodns and because the pkg.{eu,us-*} names are
currently one-to-one mappings).

So in that spirit, we move forward with only the pkg.f.o entry.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [Review] bsdconfig pkgng integration

[Teske, Devin]

On Nov 6, 2013, at 5:02 PM, Glen Barber wrote:

> On Thu, Nov 07, 2013 at 12:48:02AM +0000, Teske, Devin wrote:
>> I think that you're thinking that:
>> 
>> pkg.eu.freebsd.org
>> pkg.us-east.freebsd.org
>> pkg.us-west.freebsd.org
>> 
>> Will somehow change?
> 
> I think you're thinking they *won't* change.  Don't assume.
> 

Well, what *kind* of change?

Changes to what they serve should be no problem...

but what I was assuming that wouldn't change was
their actual names. (e.g. "pkg.eu" stays "pkg.eu").

Couldn't care about any other change, because that
would percolate at runtime off the DNS servers.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [Review] bsdconfig pkgng integration

[Teske, Devin]

On Nov 6, 2013, at 4:22 PM, Glen Barber wrote:

> On Thu, Nov 07, 2013 at 12:13:49AM +0000, Teske, Devin wrote:
>> So what hard-coding are you talking about?
>> 
> 
> You are trying to hard-code hostnames for a service in the FreeBSD
> src/ tree, when it is *absolutely* unnecessary.
> 

Not all hostnames are equal.

As you know, pkg.freebsd.org is one that you would never deny.

So just as pkg.freebsd.org is "different" from a normal hostname,
pkg.eu.freebsd.org and pkg.us-east.freebsd.org and
pkg.us-west.freebsd.org are similarly "different" (in the same way).

I'm not trying to hard-code any ol' hostnames...

I'm trying to hardcode the hostnames which can be administered
to point to one or more grouped servers.

You talk about how pkg.freebsd.org is unique because we can
simply change where it points...

I'm talking about doing the same thing for the other 3 names we
have which were designed for the same thing.

I think that you're thinking that:

pkg.eu.freebsd.org
pkg.us-east.freebsd.org
pkg.us-west.freebsd.org

Will somehow change?

The only way they would change is in the same way that you
would change pkg.freebsd.org -- I don't know how much clearer
I can make this.



>> You talk about how "if a node goes down we take it out of DNS"
>> but that has absolutely nothing to do with me because I'm not
>> putting A/-resolving names in the menu.
>> 
> 
> I did not once say anything about A or  records.
> 

You talked about a node going down.
A node has a A or quad-A record.
The name for that node is served via the SRV record.

Read the above paragraph again carefully.

I said "A/-resolving names"
That is equivalent to "node name"
and you were talking about "what if a node goes down."

And my answer to that was...
I don't care if a node goes down, because I'm not using
node names in the menu.




>> You do realize don't you that pkg.eu.f.o is a locale-specific name
>> that will eventually hold potentially many-more European server
>> names, right?
>> 
> 
> So?
> 

So, the name "pkg.eu" is never going to change. It, like pkg.freebsd.org
will just have modified SRV records to track the European nodes.



>> You do realize that the actual European server is *NOT* pkg.eu,
>> right?
>> 
> 
> *Sigh*...
> 

Your sign leads me to be concerned that there is a false
assumption that the DNS name that serves the SRV records should
be a one-to-one mapping.

There's absolutely no reason whatsoever to dedicate an entire locale
specific domain to one SRV record. That's silly. As we grow, this domain
-- which is decidedly locale-specific by way of its actual name -- should
be used for what it was intended... a locale abstraction of many-to-one.


>> You do realize that while the name pkg.f.o may wholly encompass
>> all the mirrors, that this will not always be true, right?
>> 
> 
> No, you are wrong.
> 

I surely hope I am not.

You're saying that:

+ We'll never have more than one server in Europe.
+ We'll never have more than one server in the West Coast of the USA.
+ We'll never have more than one server in the East Coast of the USA.

That makes me very sad. Very sad indeed.

I envisaged a growth that would have had dozens upon dozens of
servers all throughout, and the locale-specific domains would then
return geographically based SRV records (of which we administer on
the back-end).



>> I have no idea what you're talking about with the updating of config
>> files.
>> 
> 
> Clearly.  If you hard-code anything other than pkg.FreeBSD.org in
> bsdconfig, now clusteradm has to become aware of it, and make sure that
> record *always* exists, no matter what the endpoint is.
> 

http://lists.freebsd.org/pipermail/freebsd-pkg/2013-October/000107.html

According to that mail, clusteradm *should* already be away of the
three I mentioned... pkg.eu, pkg.us-east, and pkg.us-west




> This nonsense happened with sysinstall, and anything else that used the
> FTP mirror list.  And when a node disappears, for whatever reason, it is
> an absolute nightmare to sort out.
> 

I have to continue to beat the drum...

You had to do that in sysinstall, because the names that were used there
were A/quad-A resolving names (node names).

Just as pkg.freebsd.org is not one of those names... pkg.eu and pkg.us-*
are not like the names used in sysinstall.

So please... tell me again, how or why you would *ever* have to modify
pkg.eu or pkg.us-* ???


> For the last time, you do not need to have *any* host entries other than
> 'pkg.FreeBSD.org'.  Period.
> 

Until you acknowledge that pkg.eu and pkg.us-* are of the same pedigree
as pkg.f.o, I assume that you are still confu

Re: [Review] bsdconfig pkgng integration

[Teske, Devin]

On Nov 6, 2013, at 4:01 PM, Glen Barber wrote:

> On Wed, Nov 06, 2013 at 11:53:28PM +0000, Teske, Devin wrote:
>>> And if we need to change something with the infrastructure, we now have
>>> to tell people that this was done.  You are putting hard-coded
>>> limitations on us.  Don't.
>>> 
>> 
>> I disagree.
>> 
>> You're making the argument for "what if a CDN goes away".
>> 
> 
> No, you are thinking about it backwards.
> 
> If a node goes down (there is no "CDN"), we take it out of DNS.
> A single entry, removed from the pkg.FreeBSD.org SRV record.  We do not
> have to redirect anything, we do not have to update various
> configuration files in various places in the tree.
> 

Please help me to understand why on Earth you think:

"[I am] putting hard-coded limitations on [you]"

The facts:

1. The DNS names I am talking about:
1.a. pkg.f.o
1.b. pkg.eu.f.o
2. Do not resolve to an A or  record

So what hard-coding are you talking about?

You talk about how "if a node goes down we take it out of DNS"
but that has absolutely nothing to do with me because I'm not
putting A/-resolving names in the menu.

You do realize don't you that pkg.eu.f.o is a locale-specific name
that will eventually hold potentially many-more European server
names, right?

You do realize that the actual European server is *NOT* pkg.eu,
right?

You do realize that while the name pkg.f.o may wholly encompass
all the mirrors, that this will not always be true, right?

I have no idea what you're talking about with the updating of config
files.

The only reason you would ever have to update a config file is if
pkg.eu.f.o went away, but ...

Now I will repeat...

pkg.eu.f.o is NOT A REAL SERVER

It is a name, that resolves to the SRV record of the European server.

That being said... that name should not go away unless we no longer
have even one single server in Europe.

Oh, and by the way...

The name "pkg.f.o" does *NOT* resolve to "pkg.eu"... pkg.eu is a sibling
name that is disassociated -- it was actually *designed* to be used for
this.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [Review] bsdconfig pkgng integration

[Teske, Devin]

On Nov 6, 2013, at 3:53 PM, Teske, Devin wrote:

> 
> On Nov 6, 2013, at 3:44 PM, Glen Barber wrote:
> 
>> On Wed, Nov 06, 2013 at 11:37:43PM +0000, Teske, Devin wrote:
>>>> You do not need to show any servers except pkg.FreeBSD.org.
>>> 
>>> Bapt *just* got finished telling me in IRC that we should also list:
>>> 
>>> pkg.eu
>>> pkg.us-*
>>> 
>> 
>> Why?
>> 
>>> Which I assume to be the same make/model of pkg.f.o (does not
>>> return an A or  but only SVR).
>>> 
>> 
>> And if we need to change something with the infrastructure, we now have
>> to tell people that this was done.  You are putting hard-coded
>> limitations on us.  Don't.
>> 
> 
> I disagree.
> 
> You're making the argument for "what if a CDN goes away".
> 
> Well... people stop using that entry because it's broken and in the next 
> release
> you remove it.
> 
> What sounds like a maintenance issue will actually save you support events.
> And for whatever hours that this proposed maintenance actually cost you, 
> you'll
> more than make up that in prevented support events.
> 
> I'm talking about saving time (because we're all volunteers, right?)
> 
> Let me be very explicit...
> 
> 
> 
>> With a single entry (pkg.FreeBSD.org), we do not have to worry about
>> what is broken if (when) we need to change something.
>> 
> 
> Let's say that the one entry doesn't do what you want (which requires trusting
> the CDN maintenance on the back-end)...
> 
> You're telling me that it's more efficient to immediately defer every single 
> user
> to the support forums the moment there's a single hiccup in the ONE and ONLY
> CDN name that we're providing?
> 
> Ok... but let's look at one alternative...
> 
> If we listed more than one CDN, and one breaks... the wise and tenacious user
> simply ... tries another?
> 
> But if there are no other choices... then you'll be headed strait to the 
> forums or
> mailing lists and generating what I call a "support event".
> 
> I'm sorry, but I don't agree that listing only one single entry is going to 
> be beneficial
> to the end-user; only to the person that thinks having a single CDN to 
> maintain is
> going to actually save them work (which may only be true if they are not the 
> ones
> that have to manage support events).

I think there's a lot of misunderstanding going on because of the way that 
clusteradm
has set up our CDN.

It is true that right now, pkg.f.o returns SRV records for all known mirrors.

It is also true that previously mentioned pkg.eu.f.o returns the one and only 
SRV record
for the one mirror in Europe.

However, I am planning for a day when our CDN is administered in a more 
traditional
fashion. That is...

I envisage pkg.f.o not returning all known mirrors (because there will be too 
many as
the CDN grows), but instead pkg.f.o would return only the 3 closest mirrors.

Meanwhile, pkg.eu.f.o would grow to return records for the top-3 closest 
servers in Europe.

Just because the amount of machines that we have available for our CDN is 
small, doesn't
mean that it will be small forever.

Eventually (as is the plan), the list of servers should far exceed the number 
of SRV records
you will want to return. And when we get to that point, I highly recommend that 
we allow the
user to choose the CDN specific to their country (at the very least).
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [Review] bsdconfig pkgng integration

[Teske, Devin]

On Nov 6, 2013, at 3:44 PM, Glen Barber wrote:

> On Wed, Nov 06, 2013 at 11:37:43PM +0000, Teske, Devin wrote:
>>> You do not need to show any servers except pkg.FreeBSD.org.
>> 
>> Bapt *just* got finished telling me in IRC that we should also list:
>> 
>> pkg.eu
>> pkg.us-*
>> 
> 
> Why?
> 
>> Which I assume to be the same make/model of pkg.f.o (does not
>> return an A or  but only SVR).
>> 
> 
> And if we need to change something with the infrastructure, we now have
> to tell people that this was done.  You are putting hard-coded
> limitations on us.  Don't.
> 

I disagree.

You're making the argument for "what if a CDN goes away".

Well... people stop using that entry because it's broken and in the next release
you remove it.

What sounds like a maintenance issue will actually save you support events.
And for whatever hours that this proposed maintenance actually cost you, you'll
more than make up that in prevented support events.

I'm talking about saving time (because we're all volunteers, right?)

Let me be very explicit...



> With a single entry (pkg.FreeBSD.org), we do not have to worry about
> what is broken if (when) we need to change something.
> 

Let's say that the one entry doesn't do what you want (which requires trusting
the CDN maintenance on the back-end)...

You're telling me that it's more efficient to immediately defer every single 
user
to the support forums the moment there's a single hiccup in the ONE and ONLY
CDN name that we're providing?

Ok... but let's look at one alternative...

If we listed more than one CDN, and one breaks... the wise and tenacious user
simply ... tries another?

But if there are no other choices... then you'll be headed strait to the forums 
or
mailing lists and generating what I call a "support event".

I'm sorry, but I don't agree that listing only one single entry is going to be 
beneficial
to the end-user; only to the person that thinks having a single CDN to maintain 
is
going to actually save them work (which may only be true if they are not the 
ones
that have to manage support events).
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [Review] bsdconfig pkgng integration

[Teske, Devin]

On Nov 6, 2013, at 3:32 PM, Glen Barber wrote:

> On Wed, Nov 06, 2013 at 11:28:25PM +0000, Teske, Devin wrote:
>> We do not ever want to display the contents of the SVR record in the list.
>> There are multiple pkg*.freebsd.org SVR zones.
>> 
> 
> You do not need to show any servers except pkg.FreeBSD.org.

Bapt *just* got finished telling me in IRC that we should also list:

pkg.eu
pkg.us-*

Which I assume to be the same make/model of pkg.f.o (does not
return an A or  but only SVR).

If that is the case (that pkg.eu.f.o is just like pkg.f.o -- and it just
happens to be a name that returns SVR names for servers local
to Europe), then I don't see why we would want to omit this.

I'm not the only one you have to convince...

bapt was the one that brought up the existence of the other names.



>  In fact,
> you can use that by default, and not show any servers at all.  Do not
> add more complexity to this than there needs to be.  By adding explicit
> listing of servers here, you are putting a dependency on clusteradm that
> is not necessary at all.
> 

I want to get some confusion out of the way...

I do not want to add even one single name that resolves to an A or .

I want *only* names that resolve to SVR.

I've just heard from bapt that pkg.f.o is not the only such name, that there
are other such names that similarly do not return A or .

I don't see the increase in complexity or maintenance.

I'm asking for locale-specific CDN names.

The user ought to be able to choose the master CDN or a locale-specific
CDN. This is not rocket science.
-- 
Devin




> Glen
> 

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [Review] bsdconfig pkgng integration

[Teske, Devin]

On Nov 6, 2013, at 3:19 PM, Allan Jude wrote:

> On 2013-11-06 18:15, Teske, Devin wrote:
>> On Nov 6, 2013, at 3:02 PM, Allan Jude wrote:
>> 
>>> On 2013-11-06 13:32, Teske, Devin wrote:
>>>> Subject says it all.
>>>> 
>>>> I need a review on the attached patch.
>>>> 
>>>> Come on, you know you want to...
>>>> 
>>> I think you missed something, you attached a 1 line diff
>>> 
>> Oh I see... too much to review.
>> 
>> Hey, don't all review it at once now...
>> 
>> Seriously... come on...
>> 
>> I'm asking for some feedback on the most important part of the patchset...
>> 
>> The user-facing element.
>> 
>> The least everyone can do is chime in with their favorite mirrors.
>> 
>> So... please...
>> 
>> Tell me if the mirror list is incomplete (I *know* there are more mirrors
>> than that, and I *know* we want pkg.eu and pkg.us-* ... but I don't actually
>> know what people want to see ... so please... feedback!)
> pkg.freebsd.org is a SVR record

This matters not... see SVN r257755


> that lists the mirrors,

We do not ever want to display the contents of the SVR record in the list.
There are multiple pkg*.freebsd.org SVR zones.


> and pkg
> automatically picks the correct one,

pkg does no such thing. It picks the first one that works.

The decisioning for which order and what content gets returned from the SVR 
query
is on the back-end, not on the pkg-end; pkg just takes the first one that works.

For example, "host -t SVR _http._tcp.pkg.freebsd.org" (for me at least)
returns 3 entries. pkg goes after the first one.

I'd be shocked if you told me that pkg was doing geo-mapping with the 3 names it
returns and determining which was was closest.

Rather, pkg just does a DNS query and goes through the list of hosts.



> so it is best to list only that 1
> mirror.
> 

No. You're thinking that I'm talking about the contents of the SVR query.
I'm not. I'm instead talking about...

*actual* and *real* multiple names that you can query SVR data from.

pkg.freebsd.org is only one

There are multiples, and I believe that we should add them.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [CFT] bsdinstall and zfsboot enhancements

[Teske, Devin]

On Nov 5, 2013, at 7:54 PM, Craig Rodrigues wrote:

> 
> 
> 
> On Sat, Nov 2, 2013 at 10:59 AM, Teske, Devin  
> wrote:
> 
> + Update bsdinstall's "config" script to adjust ttyu* entries in
> /etc/ttys when it is determined that we are in-fact doing an install
> over serial (e.g. bhyve).
> 
> I think this is useful.  In earlier versions of BHyve, the default was to use 
> console over a PCI serial device. 
> With this commit from Neel Natu: 
> http://lists.freebsd.org/pipermail/svn-src-all/2013-October/076027.html
> it is now possible to specify a different device for the console, such as 
> COM1.
> 

Unfortunately, that's getting the axe.

I agree with Nathan... the installer shouldn't make this assumption.

I think init should be fixed instead.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [Review] bsdconfig pkgng integration

[Teske, Devin]

On Nov 6, 2013, at 3:02 PM, Allan Jude wrote:

> On 2013-11-06 13:32, Teske, Devin wrote:
>> Subject says it all.
>> 
>> I need a review on the attached patch.
>> 
>> Come on, you know you want to...
>> 
> I think you missed something, you attached a 1 line diff
> 

Oh I see... too much to review.

Hey, don't all review it at once now...

Seriously... come on...

I'm asking for some feedback on the most important part of the patchset...

The user-facing element.

The least everyone can do is chime in with their favorite mirrors.

So... please...

Tell me if the mirror list is incomplete (I *know* there are more mirrors
than that, and I *know* we want pkg.eu and pkg.us-* ... but I don't actually
know what people want to see ... so please... feedback!)
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

[Review] bsdconfig pkgng integration

[Teske, Devin]

Subject says it all.

I need a review on the attached patch.

Come on, you know you want to...
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
Index: http.subr
===
--- http.subr   (revision 256550)
+++ http.subr   (working copy)
@@ -79,6 +79,7 @@ f_dialog_menu_media_http()
local menu_list="
'$msg_main_site'  'ftp.freebsd.org'
'URL' '$msg_specify_some_other_http_site'
+   'pkg $msg_main_site'  'pkg.freebsd.org'
" # END-QUOTE
local hline="$msg_select_a_site_thats_close"
 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu

[Teske, Devin]

On Nov 6, 2013, at 1:18 AM, Lars Engels wrote:

> Am 05.11.2013 18:06, schrieb Kurt Lidl:
> 
>> Well, I'd probably be in support of this change - it sure beats having
>> to interrupt the normal boot sequence and typing:
>>   unload
>>   load /boot/kernel.old/kernel
>>   load /boot/kernel.old/opensolaris.ko
>>   load /boot/kernel.old/zfs.ko
>>   boot
> 
> To load an older kernel I always just type
> 
> boot kernel.old
> 
> 
> Doesn't that unload the currently loaded kernel automatically?

Actually... it does.

Thanks for pointing that out (forgot about that).

I think we still want the loader_delay feature added by the last
patch I shared on-list.

Yeah? I think it's still a good (optional) value-add.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu

[Teske, Devin]

On Nov 5, 2013, at 9:53 AM, Teske, Devin wrote:

> 
> On Nov 5, 2013, at 9:28 AM, Nathan Whitehorn wrote:
> 
>> On 11/05/13 11:06, Kurt Lidl wrote:
>>> 
>>>> You can try enabling the beastie menu on sparc64 by editing
>>>> /boot/loader.rc:
>>>> 
>>>> === Change #1 in /boot/loader.rc to enable beastie menu ===
>>>> 
>>>> Find:
>>>>   \ Reads and processes loader.conf variables
>>>>   \ NOTE: Change to `initialize' if you enable the below boot menu
>>>>   start
>>>> 
>>>> Change "start" to "initialize" as shown below:
>>>>   \ Reads and processes loader.conf variables
>>>>   \ NOTE: Change to `initialize' if you enable the below boot menu
>>>>   initialize
>>>> 
>>>> === Change #2 in [same file] to enable beastie menu ===
>>>> 
>>>> Find:
>>>>   \ Uncomment to enable boot menu
>>>>   \ include /boot/beastie.4th
>>>>   \ beastie-start
>>>> 
>>>> Uncomment "beastie-start" as shown below:
>>>>   \ Uncomment to enable boot menu
>>>>   \ include /boot/beastie.4th
>>>>   beastie-start
>>>> 
>>>> ==
>>>> 
>>>> If you find that making those two trivial changes, that you are able
>>>> to load
>>>> the menu... then maybe it's time for us to start thinking about
>>>> enabling the
>>>> beastie menu by-default for the sparc64 architecture.
>>> 
>>> Seems to work just fine.  I tested by booting, toggling through the
>>> different kernel choices (/boot/kernel/kernel /boot/kernel.old/kernel)
>>> and both worked correctly.
>>> 
>>> (Although I uncommented the "include /boot/beastie.4th" line too.)
>>> 
>>>> Does anybody else have any thoughts on enabling it for sparc64?
>>> 
>>> Well, I'd probably be in support of this change - it sure beats having
>>> to interrupt the normal boot sequence and typing:
>>>  unload
>>>  load /boot/kernel.old/kernel
>>>  load /boot/kernel.old/opensolaris.ko
>>>  load /boot/kernel.old/zfs.ko
>>>  boot
>>> When I need to get back to the prior version of the kernel.
>> 
>> Is there a way to make this work even without the beastie menu? A way to
>> interrupt the boot before kernel load (even holding down a key) would be
>> really valuable, even on systems that do not support fancy terminals
>> with colors and such.
> 
> Nathan,
> 
> Can you drop into your /boot/loader.conf:
> 
>   loader_delay=3
> 
> And reboot?
> 
> I'm trying to think how we could use that to our advantage. I'm interested
> in creative applications thereof.
> 
> For more skinny on what that does... "man delay.4th", it spills the beans
> on a "delayed command execution" module that I added a few years ago.
> 
> For example... Here's my idea of making things easier on the user that
> wants to load a different kernel, but *without* using a menu...
> 
> 1. We have loader_delay default to 3
> 2. Dots are displayed for 3 seconds before we do anything (like load a
> kernel)
> 3. User presses ENTER during those 3 seconds, and the delay is truncated
> 4. User presses Ctrl-C during those 3 seconds (or ESC) and they get a
> prompt (before the kernel has loaded).
> 
> Then here's what I would do (as to not have to load separate modules)...
> 
> set kernel=kernel.old
> boot
> 
> + The forth code figures out that the kernel is in /boot
> + If loader.conf had opensolaris_load=YES and
> + zfs_load=YES then the forth code also loads those from kernel.old
> 
> But let's say loader.conf was bare, you could do:
> 
> set kernel=kernel.old
> set opensolaris_load=YES
> set zfs_load=YES
> boot
> 
> NB: The reason this works is because you would not call "start" in your
> loader.rc (which actually loads things), but instead "initialize" (which only
> loads loader.conf et. al.) followed by dc_execute

Err, dc_execute is what I called it nearly a decade ago when I first wrote it
at $work. I forgot that I renamed it to delay_execute (used to be "dc_execute"
for "delayed-command execute").



> to fire off "autoboot" in 3
> seconds (with the aforementioned allowed interrupt ability).
> 
> Is that what you were thinking? Will that work?
> 
> Bonus: It would be extremely trivial to implement.

I wipped up a patch... (and tested it about 12 dif

Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu

[Teske, Devin]

On Nov 5, 2013, at 9:28 AM, Nathan Whitehorn wrote:

> On 11/05/13 11:06, Kurt Lidl wrote:
>> 
>>> You can try enabling the beastie menu on sparc64 by editing
>>> /boot/loader.rc:
>>> 
>>> === Change #1 in /boot/loader.rc to enable beastie menu ===
>>> 
>>> Find:
>>>\ Reads and processes loader.conf variables
>>>\ NOTE: Change to `initialize' if you enable the below boot menu
>>>start
>>> 
>>> Change "start" to "initialize" as shown below:
>>>\ Reads and processes loader.conf variables
>>>\ NOTE: Change to `initialize' if you enable the below boot menu
>>>initialize
>>> 
>>> === Change #2 in [same file] to enable beastie menu ===
>>> 
>>> Find:
>>>\ Uncomment to enable boot menu
>>>\ include /boot/beastie.4th
>>>\ beastie-start
>>> 
>>> Uncomment "beastie-start" as shown below:
>>>\ Uncomment to enable boot menu
>>>\ include /boot/beastie.4th
>>>beastie-start
>>> 
>>> ==
>>> 
>>> If you find that making those two trivial changes, that you are able
>>> to load
>>> the menu... then maybe it's time for us to start thinking about
>>> enabling the
>>> beastie menu by-default for the sparc64 architecture.
>> 
>> Seems to work just fine.  I tested by booting, toggling through the
>> different kernel choices (/boot/kernel/kernel /boot/kernel.old/kernel)
>> and both worked correctly.
>> 
>> (Although I uncommented the "include /boot/beastie.4th" line too.)
>> 
>>> Does anybody else have any thoughts on enabling it for sparc64?
>> 
>> Well, I'd probably be in support of this change - it sure beats having
>> to interrupt the normal boot sequence and typing:
>>   unload
>>   load /boot/kernel.old/kernel
>>   load /boot/kernel.old/opensolaris.ko
>>   load /boot/kernel.old/zfs.ko
>>   boot
>> When I need to get back to the prior version of the kernel.
> 
> Is there a way to make this work even without the beastie menu? A way to
> interrupt the boot before kernel load (even holding down a key) would be
> really valuable, even on systems that do not support fancy terminals
> with colors and such.

Nathan,

Can you drop into your /boot/loader.conf:

loader_delay=3

And reboot?

I'm trying to think how we could use that to our advantage. I'm interested
in creative applications thereof.

For more skinny on what that does... "man delay.4th", it spills the beans
on a "delayed command execution" module that I added a few years ago.

For example... Here's my idea of making things easier on the user that
wants to load a different kernel, but *without* using a menu...

1. We have loader_delay default to 3
2. Dots are displayed for 3 seconds before we do anything (like load a
kernel)
3. User presses ENTER during those 3 seconds, and the delay is truncated
4. User presses Ctrl-C during those 3 seconds (or ESC) and they get a
prompt (before the kernel has loaded).

Then here's what I would do (as to not have to load separate modules)...

set kernel=kernel.old
boot

+ The forth code figures out that the kernel is in /boot
+ If loader.conf had opensolaris_load=YES and
+ zfs_load=YES then the forth code also loads those from kernel.old

But let's say loader.conf was bare, you could do:

set kernel=kernel.old
set opensolaris_load=YES
set zfs_load=YES
boot

NB: The reason this works is because you would not call "start" in your
loader.rc (which actually loads things), but instead "initialize" (which only
loads loader.conf et. al.) followed by dc_execute to fire off "autoboot" in 3
seconds (with the aforementioned allowed interrupt ability).

Is that what you were thinking? Will that work?

Bonus: It would be extremely trivial to implement.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu

[Teske, Devin]

On Nov 5, 2013, at 9:06 AM, Kurt Lidl wrote:

> 
>> You can try enabling the beastie menu on sparc64 by editing
>> /boot/loader.rc:
>> 
>> === Change #1 in /boot/loader.rc to enable beastie menu ===
>> 
>> Find:
>>  \ Reads and processes loader.conf variables
>>  \ NOTE: Change to `initialize' if you enable the below boot menu
>>  start
>> 
>> Change "start" to "initialize" as shown below:
>>  \ Reads and processes loader.conf variables
>>  \ NOTE: Change to `initialize' if you enable the below boot menu
>>  initialize
>> 
>> === Change #2 in [same file] to enable beastie menu ===
>> 
>> Find:
>>  \ Uncomment to enable boot menu
>>  \ include /boot/beastie.4th
>>  \ beastie-start
>> 
>> Uncomment "beastie-start" as shown below:
>>  \ Uncomment to enable boot menu
>>  \ include /boot/beastie.4th
>>  beastie-start
>> 
>> ==
>> 
>> If you find that making those two trivial changes, that you are able to load
>> the menu... then maybe it's time for us to start thinking about enabling the
>> beastie menu by-default for the sparc64 architecture.
> 
> Seems to work just fine.  I tested by booting, toggling through the
> different kernel choices (/boot/kernel/kernel /boot/kernel.old/kernel)
> and both worked correctly.
> 
> (Although I uncommented the "include /boot/beastie.4th" line too.)
> 

Oops, heh, good eye ;D and sorry if that caused any pain for you.



>> Does anybody else have any thoughts on enabling it for sparc64?
> 
> Well, I'd probably be in support of this change - it sure beats having
> to interrupt the normal boot sequence and typing:
>   unload
>   load /boot/kernel.old/kernel
>   load /boot/kernel.old/opensolaris.ko
>   load /boot/kernel.old/zfs.ko
>   boot
> When I need to get back to the prior version of the kernel.
> 

We'll have to draw up a clean patch to draw this in for sparc64.
For i386 and amd64, they have a dedicated separate loader.rc
stashed in sys/boot/i386/loader/

Not sure of the cleanest way to do this for sparc64 (dup the
loader.rc and make sparc64 drop it's own? or perhaps make
sparc64 drop the i386 loader.rc? I think amd64 does the latter).
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu

[Teske, Devin]

On Nov 4, 2013, at 2:43 PM, Kurt Lidl wrote:

>> On Nov 2, 2013, at 1:45 AM, Sam Fourman Jr. wrote:
>>> On Sat, Nov 2, 2013 at 7:09 AM, Teske, Devin  
>>> wrote:
>>> Hi all,
>>> 
>>> Here's a chance to test out the kernel selection menu enhancements
>>> to the boot loader menu before they go into HEAD.
>>> 
>>> Discussion welcome, feedback desired.
>>> 
>>> No recompile needed, just drop the new forth files onto a HEAD or
>>> stable/9 box and reboot.
>>> --
>>> Cheers,
>>> Devin
>>> 
>>> where are the forth files in question?
>>> 
>> 
>> D'Oh!
>> 
>> Here they are:
>> 
>> http://druidbsd.cvs.sf.net/viewvc/druidbsd/forth_zfs/
>> 
>> Supplied as patches to either stable/9 or head.
>> --
>> Devin
> 
> Hmmm.  I saw no appreciable changes to behavior when I patched all
> the files in /boot with these versions.  This was on a sparc64 host
> running 10-BETA3 (compile this morning).
> 

Excellent!

Thank you for testing.

NB: That's what *should* happen on sparc64 since that architecture
doesn't actually enable the beastie menu (sad, I know... I wish that the
beastie menu was active on all platforms by default).



> Notably, the kernel and modules still loaded before presenting me
> with the option to tell it which kernel to load:
> 
> Executing last command: boot /pci@1f,0/pci@1/scsi@8/disk@0,0:a
> Boot device: /pci@1f,0/pci@1/scsi@8/disk@0,0:a  File and args:
> 
> >> FreeBSD/sparc64 ZFS boot block
>   Boot path:   /pci@1f,0/pci@1/scsi@8/disk@0,0:a
> Consoles: Open Firmware console
> \
> FreeBSD/sparc64 ZFS enabled bootstrap loader, Revision 1.0
> (r...@snap.freebsd.org, Sun Oct 27 07:20:42 UTC 2013)
> bootpath="zfs:sys/ROOT/default:"
> Loading /boot/defaults/loader.conf
> /boot/kernel/kernel data=0x8d54c0+0x66180 syms=[0x8+0x954f0+0x8+0x8d7ef]
> /boot/kernel/zfs.ko text=0x223328 data=0xa4e0+0x17fc0 
> syms=[0x8+0x197b8+0x8+0x143f8]
> loading required module 'opensolaris'
> /boot/kernel/opensolaris.ko text=0x3130 data=0x2c8+0x2030 
> syms=[0x8+0xd98+0x8+0x929]
> /boot/kernel/geom_mirror.ko text=0x38430 data=0x5b0+0x20 
> syms=[0x8+0x16b0+0x8+0x119e]
> 
> Hit [Enter] to boot immediately, or any other key for command prompt.
> Booting [/boot/kernel/kernel] in 9 seconds...
> 

You can try enabling the beastie menu on sparc64 by editing
/boot/loader.rc:

=== Change #1 in /boot/loader.rc to enable beastie menu ===

Find:
\ Reads and processes loader.conf variables
\ NOTE: Change to `initialize' if you enable the below boot menu
start

Change "start" to "initialize" as shown below:
\ Reads and processes loader.conf variables
\ NOTE: Change to `initialize' if you enable the below boot menu
initialize

=== Change #2 in [same file] to enable beastie menu ===

Find:
\ Uncomment to enable boot menu
\ include /boot/beastie.4th
\ beastie-start

Uncomment "beastie-start" as shown below:
\ Uncomment to enable boot menu
\ include /boot/beastie.4th
beastie-start

==

If you find that making those two trivial changes, that you are able to load
the menu... then maybe it's time for us to start thinking about enabling the
beastie menu by-default for the sparc64 architecture.

Does anybody else have any thoughts on enabling it for sparc64?

-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu

[Teske, Devin]

On Nov 2, 2013, at 8:27 PM, Artem Belevich wrote:

> On Sat, Nov 2, 2013 at 9:47 AM, Teske, Devin  
> wrote:
>>> where are the forth files in question?
>>> 
>> 
>> D'Oh!
>> 
>> Here they are:
>> 
>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/forth_zfs/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=LikaP7fhAuWFwcCB34idLlWshVBNdWkfLMLvZ5MEnX8%3D%0A&s=3d2d4cfda3daa8bd25f165bb1622cf30975b89ae1bbcaea301da0b8e2f5b00f8
>> 
>> Supplied as patches to either stable/9 or head.
>> --
>> Devin
> 
> Thanks for the patches. Tried on stable/9 and they worked great for me.
> Do you have anything similar to select boot environment as well?
> 

You're right to assume that these patches are actually an incremental step
in that direction (that is to say, that it's silly to load a kernel before the 
menu
if the menu will allow you to select an ad hoc root device).

So indeed, the next step is to allow selecting your root device (be it a ZFS
snapshot or alternate root for whatever purpose).

I've been giving some thought to this one... and I'm actually thinking about
writing this one as an FICL call-out (that is to say, written in the C code of
the loader which is then invoked by the Forth code).

I have already done mockups for this, but it will take some time before you
see a worthy patchset (think 10.1).
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [CFT] bsdinstall and zfsboot enhancements

[Teske, Devin]

On Nov 2, 2013, at 8:40 PM, Nathan Whitehorn wrote:

> On 11/02/13 12:59, Teske, Devin wrote:
>> Hi all,
>> 
>> Another Call For Testing...
>> This one is for bsdinstall.
> Will look at the rest later...
>> 
>> + Update bsdinstall's "config" script to adjust ttyu* entries in
>> /etc/ttys when it is determined that we are in-fact doing an install
>> over serial (e.g. bhyve).
>> 
> 
> I think this is the wrong solution.

Thank you for your feedback. Sincerely.


> The installer is run in a lot of circumstances, and tying it to the boot 
> environment is a mistake. If we want serial consoles to default to on for 
> x86, they should default to on (they do for most other architectures). The 
> magic should never ever be in the installer.
> 

Well, I would say we *don't* want serial to be on by default...
That would actually hurt me at $work where we actually use
the serial ports for barcode readers.


> Setting the terminal type to vt100 unconditionally is also questionable. 
> Using kbdcontrol also doesn't do the right thing, since it will turn on 
> serial consoles if you install to, say, a disk image from an xterm or if you 
> use newcons.

Kk.

I'm CC'ing Peter Grehan, because we only arrived at this solution
based on a quick discussion with Michael Dexter at vBSDcon with
respect to bhyve.

Peter, can you restate the problem for Nathan so that we can
maybe find a better home for this change? or perhaps more clearly
define (than I) how we arrived at the code for the bhyve work?
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

[CFT] bsdinstall and zfsboot enhancements

[Teske, Devin]

Hi all,

Another Call For Testing...
This one is for bsdinstall.

Two patchsets are required for this CFT:

http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_debug/
http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/

The enhancements are:

+ Add a `-D FILE" command-line option for overriding the
path to the bsdinstall log file (BSDINSTALL_LOG env var).

+ Document new `-D FILE' in the man page for bsdinstall.

+ If FILE in `-D FILE' begins with a +, debug output goes to
stdout (interleaved between dialog(1) invocations/output) as
well as to FILE (minus the leading + of course).

+ If BSDINSTALL_LOG cannot be written, then debugging is
disabled (except in the case of a leading + in the pathname,
wherein debug will still be printed to stdout).

+ Update source code format to approximate a future shstyle(9)

+ Fix a dangling participle ("Begun ..." -> "Began ...")

+ Rewrite the docsinstall script (was necessary to abate direct
dependency on BSDINSTALL_LOG (instead, use fault-tolerant
bsdconfig framework which displays appropriate errors for
package management).

+ Add additional debug output for dhclient/rtsol/wpa_cliscan

+ Display script errors in a textbox rather than just on stdout

+ Update many coments.

+ Add new f_show_err() API call (like f_show_msg but changes
the dialog title to "Error")(see bsdconfig's `common.subr').

+ Add new f_eval_catch() API call for executing a command via
eval but not before logging the command to debug. Several
example cases documented in API header for function in
bsdconfig's `common.subr'.

+ Fix dialog auto-sizing when launched as an rvalue to a pipe
for indirected scripts (previously would default to 80x24 sizing
in this case, now it can autosize to full size even when in a pipe
chain).

+ Fix a bug in f_snprintf wherein if the $format argument began
with a "-" or "--" it would be misinterpreted as a flag to printf. (this
is in bsdcofig's strings.subr)

+ Add accompanying f_sprintf() and f_vsprintf() to go along with
already existing f_snprintf() and f_vsnprintf() (see bsdconfig's
strings.subr).

+ Update bsdinstall's "config" script to adjust ttyu* entries in
/etc/ttys when it is determined that we are in-fact doing an install
over serial (e.g. bhyve).

+ Remove some unnecessary default ZFS datasets from the
automatic "zfsboot" script. Such as: /usr/ports/distfiles
/usr/ports/packages /usr/obj /var/db /var/empty /var/mail and
/var/run (these can all be created as-needed once the system is
installed).

+ Remove setuid=off for /usr/home (as discussed with others from
last round of CFT).

+ Fix some i18n string violations in "zfsboot"

+ Bolster debugging output in "zfsboot"

+ Fix some string quoting issues in "zfsboot"

+ Fix some variable scope issues in "zfsboot"

+ Only display the ZFS vdev type selection menu when running
interactively (duh).

+ Change "Create" to "Install" in "zfsboot" main menu

+ Increase pedantic error checking in "zfsboot" (type-check
arguments and such).

+ Add a call to "graid destroy" to kill automatic metadata (part of
the series of pedantic destructions we do when bootstrapping
a new/naked disk).

+ Make judicious use of new f_eval_catch() in "zfsboot"

+ More comments (zfsboot).

+ Fixup some variable names for consistency (zfsboot).

-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [CFT] Kernel-Selection Enhancemnt to Boot Menu

[Teske, Devin]

On Nov 2, 2013, at 1:45 AM, Sam Fourman Jr. wrote:

> 
> 
> 
> On Sat, Nov 2, 2013 at 7:09 AM, Teske, Devin  
> wrote:
> Hi all,
> 
> Here's a chance to test out the kernel selection menu enhancements
> to the boot loader menu before they go into HEAD.
> 
> Discussion welcome, feedback desired.
> 
> No recompile needed, just drop the new forth files onto a HEAD or
> stable/9 box and reboot.
> --
> Cheers,
> Devin
> 
> where are the forth files in question? 
> 

D'Oh!

Here they are:

http://druidbsd.cvs.sf.net/viewvc/druidbsd/forth_zfs/

Supplied as patches to either stable/9 or head.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

[CFT] Kernel-Selection Enhancemnt to Boot Menu

[Teske, Devin]

Hi all,

Here's a chance to test out the kernel selection menu enhancements
to the boot loader menu before they go into HEAD.

Discussion welcome, feedback desired.

No recompile needed, just drop the new forth files onto a HEAD or
stable/9 box and reboot.
-- 
Cheers,
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 21, 2013, at 10:18 AM, Allan Jude wrote:

> On 2013-10-21 13:14, Teske, Devin wrote:
>> On Oct 21, 2013, at 10:12 AM, Allan Jude wrote:
>> 
>>> On 2013-10-21 12:19, Teske, Devin wrote:
>>>> On Oct 21, 2013, at 9:06 AM, Allan Jude wrote:
>>>> 
>>>>> On 2013-10-21 12:04, Teske, Devin wrote:
>>>>>> On Oct 21, 2013, at 8:50 AM, Allan Jude wrote:
>>>>>> 
>>>>>>> On 2013-10-21 11:45, Johan Broman wrote:
>>>>>>>> Hi!
>>>>>>>> 
>>>>>>>> Sorry for the delayed answer. I've patched zfsboot and rebuilt the
>>>>>>>> release. I now get an error message that ada2 can't be used, which is
>>>>>>>> correct. Good stuff! :)
>>>>>>>> 
>>>>>>>> ( I've recreated the test environment using a KVM guest with four SATA
>>>>>>>> drives instead of the server I was using. I makes it easier to test
>>>>>>>> stuff. )
>>>>>>>> 
>>>>>>>> Here's the screenshot:
>>>>>>>> 
>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.png&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=f6fc4ec7c9d8b9486e897156dac5af7c3f64e3fae746dcf6c66ad91564a8ce99
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Maybe one should be unable to select drives that are part of a graid
>>>>>>>> in the first place? Or is that out-of-scope for bsdinstall at this
>>>>>>>> point? (As I guess that requires too many changes/new lines)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Cheers
>>>>>>>> Johan
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 19/10/13 22:20, Teske, Devin wrote:
>>>>>>>>> On Oct 19, 2013, at 10:07 AM, Johan Broman wrote:
>>>>>>>>> 
>>>>>>>>>> I recreated the graid mirror on ada2 and ada3 and reran the
>>>>>>>>>> installation. I'm unable to scroll the msgbox using PgDn or arrow
>>>>>>>>>> keys. There is no indication that the action failed and I'm returned
>>>>>>>>>> to the ZFS setup screen if I hit OK.
>>>>>>>>>> 
>>>>>>>>>> I have screen shots (taken with my phone) of the msgbox and "ps
>>>>>>>>>> auxwww" output. Let me know what kind of debug info you would like.
>>>>>>>>>> I've put the screen shots here:
>>>>>>>>>> 
>>>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=6322000e13ed155bda748698c4a0d54c9de7c29f5566affe202d7c5a29917cd1
>>>>>>>>> I've added a patch to fix debugging in the zfsboot script...
>>>>>>>>> 
>>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=1dc96a8b5450d27fe8210b361c9ed736ccd448f78df6aabe170bb80e31bca6d9
>>>>>>>>> 
>>>>>>>>> Feedback welcome.
>>>>>>>>> 
>>>>>>>>> Johan,...
>>>>>>>>> 
>>>>>>>>> Can you see if the patch sheds some better light as to what's failing?
>>>>>>>>> 
>>>>>>>>> The patch won't fix the problem, but it should give us an accurate 
>>>>>>>>> error
>>>>>>>>> message so that we can learn what precisely is returning an error
>>>>>>>>> status.
>>>>>>>>> 
>>>>>>>>> Thanks in advance.
>>>>>>>>> 
>>>>>>> I do notice that Devin's manually prefixing the error message with the
>>>>>>> tool name, is partially redundancy w

Re: [RFC] libdispatch (aka Grand Central Dispatch) in base

[Teske, Devin]

On Oct 29, 2013, at 9:01 PM, Nathan Whitehorn wrote:

> On 10/29/13 21:04, Teske, Devin wrote:
>> Hi all,
>> 
>> I'd like to bring up the discussion for topic..
>> 
>> Importing libdispatch (aka Apple's Grand Central Dispatch) into base 
>> (contrib?).
>> 
>> Specifically into HEAD then MFC'd only as far back as stable/10.
>> 
>> Here's the reason why:
>> http://devinteske.com/freebsd-installer-enhancements
>> 
>> Summary:
>> For the purpose of providing a concurrency model better than pthreads for the
>> expressed desire to bring about concurrent data processing (applicable 
>> directly
>> to distributions, packages, signing and more).
>> 
>> Multiple people have confirmed with me with respect to the above blog article
>> that the concurrency model would be most efficient with libdispatch.
>> 
>> Since the tool mentioned in the blog is
>> a. Compiling with clang
>> b. Requires newest dialog(3) that is only in stable/10 or higher
>> 
>> I'd say that it looks like a match made in heaven.
>> 
>> But of course, there's that one hang-up... dispatch is not available in base 
>> yet.
>> 
>> Is anyone working on getting dispatch into base?
> 
> I have no opinion on GCD in base -- probably a good idea -- but I was hoping 
> you could explain further what you are trying to do here. Parallelism in 
> these steps is usually of very limited utility. For checksum evaluation, it 
> could speed things up on multicore systems. For fetch and extract, however, 
> it either has no effect (if you are bandwidth limited during fetch) or slows 
> things down significantly by causing extra seeks. This problem is especially 
> bad for CD installs. For extract, it can actually cause lasting problems on 
> the installed system by increasing disk fragmentation.

The parallel concurrency was actually an after-thought.

First... I started with this...
http://pastebin.com/LvDtJNGh

A straight-forward attempt to see if I could add X11 support
by finding a feature that dialog(1) and Xdialog(1) agree upon.

They agree that when they are reading pipe data for the gauge
widget, they will update their prompt text when the input looks like:

XXX
New prompt text
XXX

NB: Literally "XXX" on a line by itself, sandwiching the new prompt.

I then quickly rewrote that into C and added dozens of features to
get where the fdpv prototype is at now.

I'm glad we're having this discussion, because I acknowledge that
I very well can push ahead without concurrency (and we may want
to do that, because the utility brings other benefits that are foremost
on my list behind parallelism).

Trying to achieve:

1. X11 support
2. Get rid of the need for temporary files (without sacrificing signing)
3. Internationalization

So would you recommend just moving forward without the parallelism
so we could add that as a value-add down the line?

That would actually speed up the development of fdpv as a new base
utility that can take either stdin or a list of fifo's with size and signature
data, and verify the data before forking off a tar command, or pkg etc.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [RFC] libdispatch (aka Grand Central Dispatch) in base

[Teske, Devin]

On Oct 29, 2013, at 7:17 PM, Freddie Cash wrote:

> 
> On Oct 29, 2013 7:07 PM, "Teske, Devin"  wrote:
> >
> > Hi all,
> >
> > I'd like to bring up the discussion for topic..
> >
> > Importing libdispatch (aka Apple's Grand Central Dispatch) into base 
> > (contrib?).
> 
> Hasn't this been done already? There's mention of it in the archives from 
> 2009/2010. There was a port by Robert Watson et al to 8.1 mentioned in the 
> quarterly status reports, and indications it would be imported after 
> 8.1-release.
> 

I knew about the port, and I had heard about previous efforts, but when I go
scrounging around HEAD, I can't find it. Let alone older branches.

Do you know where we should be looking? I already checked contrib, and lib.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

[RFC] libdispatch (aka Grand Central Dispatch) in base

[Teske, Devin]

Hi all,

I'd like to bring up the discussion for topic..

Importing libdispatch (aka Apple's Grand Central Dispatch) into base (contrib?).

Specifically into HEAD then MFC'd only as far back as stable/10.

Here's the reason why:
http://devinteske.com/freebsd-installer-enhancements

Summary:
For the purpose of providing a concurrency model better than pthreads for the
expressed desire to bring about concurrent data processing (applicable directly
to distributions, packages, signing and more).

Multiple people have confirmed with me with respect to the above blog article
that the concurrency model would be most efficient with libdispatch.

Since the tool mentioned in the blog is
a. Compiling with clang
b. Requires newest dialog(3) that is only in stable/10 or higher

I'd say that it looks like a match made in heaven.

But of course, there's that one hang-up... dispatch is not available in base 
yet.

Is anyone working on getting dispatch into base?
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

[CFT] Improve bsdinstall debugging

[Teske, Devin]

Hi all,

I've got a set of patches to apply to bsdinstall to improve
debugging.

Would like some testing and/or review feedback.

Patches are here (tarball form; apply in any order using
"svn patch"):

http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_debug/?view=tar
http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/?view=tar

The net effect of the patches is to:

1. Increase debugging to include commands and their output
2. Add descriptions in the debug output to show what we're doing
3. Provide better error reporting through dialog(1) (when interactive)
4. Implement debugging for things we may not expect to go wrong
5. Show function names in debug messages
6. Log values of important environment variables
7. Type-check arguments that need to be a specific type (e.g., integer)
8. Double-quote values that we can't guarantee to not contain $IFS
9. In the zfsboot script, catch any error from all 48+ fork-execs
10. Abort immediately on error if nonInteractive is set (zfsboot)
11. Implement `-D file' for bsdinstall to change BSDINSTALL_LOG
12. If the log file can't be written, print a single warning and don't log
13. Fix dangling participle "Begun" in log message to "Began"
14. Rewrite "docsinstall" module to not depend on the log file for error
NB: Also brings in bsdconfig package management API to docsinstall

Right now, it should be noted that I went to town on the zfsboot script
but I was a lot more sedate when patching the rest of bsdinstall.

The initial motivation for the debugging additions for the rest of bsdinstall
(besides zfsboot) was from flo@ whom was unable to deploy to his
Soekris boxen via PXE because /tmp was read-only when bsdinstall
started (and if it couldn't write to its log file, it just exited -- without 
perhaps
giving you a chance to run a script to bootstrap a writable /tmp). These
patches fixed flo@'s problem with bsdinstall not launching.

The motivation for the zfsboot debugging additions was a thread wherein
someone had some ad hoc graid metadata on a disk and we were unable
to (from the user interface *or* the debug logfile) diagnose the situation.
These patches fixed Johan's problems with that. Then... I decided to take
it one step further and centralize the debugging API -- currently stashed in
"zfsboot" for local usage... but with feedback from this thread, I'd like to 
then
uplift those into the library files and then pepper them throughout the rest
of bsdinstall to make its debugging as strong as that in "zfsboot".
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: BE Loader Menu (was Re: rcs)

[Teske, Devin]

On Oct 21, 2013, at 6:32 AM, Kris Moore wrote:

> On 10/20/2013 20:58, Julian Elischer wrote:
>> Kris.  exactly what features need to be added to the boot process to
>> allow
>> what you want to do, but without using grub?
>> 
> 
> Here's a list of the features we are using from grub:
> 
> For install medium:
> 
> * Hybrid DVD ISO/USB image in same file

I've been using ISOLINUX for that functionality since 2006.
My DRUID installer is shipped as a hybrid ISO using ISOLINUX.


> * BIOS / UEFI loaders on same image

Haven't played with UEFI yet.


> * Graphics support

ISOLINUX has that. Can we compare screens?

I really like the way ISOLINUX's vesamenu.c32 module allows
you to even go so far as to customize the scrollbar coloring (e.g.,
in the below image, notice how I've given the scrollbar a blue
color so it completes the theme).

http://twitpic.com/17z5yi

Compare that to the same menu, but previously themed in blue:
http://twitpic.com/17xas6

I wonder if grub lets you do that (scrollbar color, hilite color, and
color of header/footer text as well).


> For installed system:
> 
> * Script-able ZFS BE menus

This can be achieved in loader(8) with some basic work, building
upon what we already have. Around this time last year, I actually
added code to allow scripting of the loader(8) menu, but said
functionality is still unused (waiting for some userland tool or some
C code integrated directly into loader(8) to actually start scripting
the menu-system.

Until that happens... it's just my little secret that the Forth menus
are actually scriptable from a higher language. (been trying for
years it seems to find someone whom cares to sit down and show
them how it works -- maybe at vBSDcon).


> * Options for setting default timeouts / default BEs to load, etc

I don't see why this too couldn't be done with loader(8). We already
have a timeout system for booting with defaults... and I gather that
whatever scripting there is to change the loaded defaults would take
place prior to reboot (some tool munges loader.conf and then
reboots; current defaults are displayed and after the configured
timeout, it boots with the displayed configuration).



> * Script-able menu to boot other OS's, such as Windows / Linux

Well, I use SYSLINUX for that. SYSLINUX is the same product as the
aforementioned ISOLINUX, except that it's focused on system versus
install media.

Of course... in your case, grub loads FreeBSD directly. When I use
syslinux as a bootloader, I get the FreeBSD Forth beastie menu after
selecting FreeBSD from the syslinux vesamenu.


> * Loads before kernel

Yeah, I got patches for our menu to follow suit.
http://druidbsd.cvs.sf.net/viewvc/druidbsd/forth_zfs/

(thinking abut committing it already)




> * Graphics support (I.E. countdown indicator, wallpapers, arrow-key
> support, etc)
> 

Well, I don't think the countdown indicator is going to get anyone all
too excited (I believe we've had that for years).

The wallpapers of course... can't do those as a default, but I think there's
a graphical loader project -- but of course that won't service serial boot.

I've thought about using arrow keys to navigate before...

It involves implementing a whole new layer on top of the menu. One that
would use some static character (e.g., ">") to show you which item you
are "over" (or in the case of color... we could *maybe* use ANSI reverse).
Definitely not easy. New state machine would be required. New rendering
code would be required as well. The good news though is that it would get
rid of the number prefixes).



> If the native loader is able to handle the ZFS BE stuff, then I don't
> see a reason why I can't make it an option in the installer, even
> without graphics and the other niceties that grub brings. Is this
> working in the stable/10 branch now?
> 

No... just the patchset I pointed to...

http://druidbsd.cvs.sf.net/viewvc/druidbsd/forth_zfs/

Which produces the following:

http://twitpic.com/dhv2b6

Which achieves:

a. Don't load kernel before menu
b. Scripting of selections can be done via $kernels in loader.conf

What we are lacking at the moment is...

Root Device selection menu.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 21, 2013, at 9:04 AM, Teske, Devin wrote:

> 
> On Oct 21, 2013, at 8:50 AM, Allan Jude wrote:
> 
>> On 2013-10-21 11:45, Johan Broman wrote:
>>> Hi!
>>> 
>>> Sorry for the delayed answer. I've patched zfsboot and rebuilt the
>>> release. I now get an error message that ada2 can't be used, which is
>>> correct. Good stuff! :)
>>> 
>>> ( I've recreated the test environment using a KVM guest with four SATA
>>> drives instead of the server I was using. I makes it easier to test
>>> stuff. )
>>> 
>>> Here's the screenshot:
>>> 
>>> https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.png&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=f6fc4ec7c9d8b9486e897156dac5af7c3f64e3fae746dcf6c66ad91564a8ce99
>>> 
>>> 
>>> Maybe one should be unable to select drives that are part of a graid
>>> in the first place? Or is that out-of-scope for bsdinstall at this
>>> point? (As I guess that requires too many changes/new lines)
>>> 
>>> 
>>> Cheers
>>> Johan
>>> 
>>> 
>>> On 19/10/13 22:20, Teske, Devin wrote:
>>>> 
>>>> On Oct 19, 2013, at 10:07 AM, Johan Broman wrote:
>>>> 
>>>>> I recreated the graid mirror on ada2 and ada3 and reran the
>>>>> installation. I'm unable to scroll the msgbox using PgDn or arrow
>>>>> keys. There is no indication that the action failed and I'm returned
>>>>> to the ZFS setup screen if I hit OK.
>>>>> 
>>>>> I have screen shots (taken with my phone) of the msgbox and "ps
>>>>> auxwww" output. Let me know what kind of debug info you would like.
>>>>> I've put the screen shots here:
>>>>> 
>>>>> https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=6322000e13ed155bda748698c4a0d54c9de7c29f5566affe202d7c5a29917cd1
>>>> 
>>>> I've added a patch to fix debugging in the zfsboot script...
>>>> 
>>>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=1dc96a8b5450d27fe8210b361c9ed736ccd448f78df6aabe170bb80e31bca6d9
>>>> 
>>>> Feedback welcome.
>>>> 
>>>> Johan,...
>>>> 
>>>> Can you see if the patch sheds some better light as to what's failing?
>>>> 
>>>> The patch won't fix the problem, but it should give us an accurate error
>>>> message so that we can learn what precisely is returning an error
>>>> status.
>>>> 
>>>> Thanks in advance.
>>>> 
>> I do notice that Devin's manually prefixing the error message with the
>> tool name, is partially redundancy when the tool does it it self, but we
>> can't always be sure it will do that.
>> 
> 
> The next patchset will fix that.
> 

And it's ready now.

Tarball:
http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/?view=tar

This patch takes debugging a leap beyond where it is now

NOTE: Only for the zfsboot script -- there is another separate effort to
improve the debugging in bsdinstall in-general; those patches are here:
http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_debug/?view=tar
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 21, 2013, at 10:12 AM, Allan Jude wrote:

> On 2013-10-21 12:19, Teske, Devin wrote:
>> On Oct 21, 2013, at 9:06 AM, Allan Jude wrote:
>> 
>>> On 2013-10-21 12:04, Teske, Devin wrote:
>>>> On Oct 21, 2013, at 8:50 AM, Allan Jude wrote:
>>>> 
>>>>> On 2013-10-21 11:45, Johan Broman wrote:
>>>>>> Hi!
>>>>>> 
>>>>>> Sorry for the delayed answer. I've patched zfsboot and rebuilt the
>>>>>> release. I now get an error message that ada2 can't be used, which is
>>>>>> correct. Good stuff! :)
>>>>>> 
>>>>>> ( I've recreated the test environment using a KVM guest with four SATA
>>>>>> drives instead of the server I was using. I makes it easier to test
>>>>>> stuff. )
>>>>>> 
>>>>>> Here's the screenshot:
>>>>>> 
>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.png&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=f6fc4ec7c9d8b9486e897156dac5af7c3f64e3fae746dcf6c66ad91564a8ce99
>>>>>> 
>>>>>> 
>>>>>> Maybe one should be unable to select drives that are part of a graid
>>>>>> in the first place? Or is that out-of-scope for bsdinstall at this
>>>>>> point? (As I guess that requires too many changes/new lines)
>>>>>> 
>>>>>> 
>>>>>> Cheers
>>>>>> Johan
>>>>>> 
>>>>>> 
>>>>>> On 19/10/13 22:20, Teske, Devin wrote:
>>>>>>> On Oct 19, 2013, at 10:07 AM, Johan Broman wrote:
>>>>>>> 
>>>>>>>> I recreated the graid mirror on ada2 and ada3 and reran the
>>>>>>>> installation. I'm unable to scroll the msgbox using PgDn or arrow
>>>>>>>> keys. There is no indication that the action failed and I'm returned
>>>>>>>> to the ZFS setup screen if I hit OK.
>>>>>>>> 
>>>>>>>> I have screen shots (taken with my phone) of the msgbox and "ps
>>>>>>>> auxwww" output. Let me know what kind of debug info you would like.
>>>>>>>> I've put the screen shots here:
>>>>>>>> 
>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=6322000e13ed155bda748698c4a0d54c9de7c29f5566affe202d7c5a29917cd1
>>>>>>> I've added a patch to fix debugging in the zfsboot script...
>>>>>>> 
>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=1dc96a8b5450d27fe8210b361c9ed736ccd448f78df6aabe170bb80e31bca6d9
>>>>>>> 
>>>>>>> Feedback welcome.
>>>>>>> 
>>>>>>> Johan,...
>>>>>>> 
>>>>>>> Can you see if the patch sheds some better light as to what's failing?
>>>>>>> 
>>>>>>> The patch won't fix the problem, but it should give us an accurate error
>>>>>>> message so that we can learn what precisely is returning an error
>>>>>>> status.
>>>>>>> 
>>>>>>> Thanks in advance.
>>>>>>> 
>>>>> I do notice that Devin's manually prefixing the error message with the
>>>>> tool name, is partially redundancy when the tool does it it self, but we
>>>>> can't always be sure it will do that.
>>>>> 
>>>> The next patchset will fix that.
>>>> 
>>>> I'm dropping the tool name from the msgbox contents and putting it in
>>>> the title (e.g., '"Error: gpart") that way... even if the tool spits out 
>>>> its own
>>>> name (or not), we'll know what exactly what was going on by looking
>>>> at the title.
>>>> 
>>>> 
>>>>> the graid thing is rather hard to detect, especially whe

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 21, 2013, at 10:18 AM, Allan Jude wrote:

> On 2013-10-21 13:14, Teske, Devin wrote:
>> On Oct 21, 2013, at 10:12 AM, Allan Jude wrote:
>> 
>>> On 2013-10-21 12:19, Teske, Devin wrote:
>>>> On Oct 21, 2013, at 9:06 AM, Allan Jude wrote:
>>>> 
>>>>> On 2013-10-21 12:04, Teske, Devin wrote:
>>>>>> On Oct 21, 2013, at 8:50 AM, Allan Jude wrote:
>>>>>> 
>>>>>>> On 2013-10-21 11:45, Johan Broman wrote:
>>>>>>>> Hi!
>>>>>>>> 
>>>>>>>> Sorry for the delayed answer. I've patched zfsboot and rebuilt the
>>>>>>>> release. I now get an error message that ada2 can't be used, which is
>>>>>>>> correct. Good stuff! :)
>>>>>>>> 
>>>>>>>> ( I've recreated the test environment using a KVM guest with four SATA
>>>>>>>> drives instead of the server I was using. I makes it easier to test
>>>>>>>> stuff. )
>>>>>>>> 
>>>>>>>> Here's the screenshot:
>>>>>>>> 
>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.png&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=f6fc4ec7c9d8b9486e897156dac5af7c3f64e3fae746dcf6c66ad91564a8ce99
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Maybe one should be unable to select drives that are part of a graid
>>>>>>>> in the first place? Or is that out-of-scope for bsdinstall at this
>>>>>>>> point? (As I guess that requires too many changes/new lines)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Cheers
>>>>>>>> Johan
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 19/10/13 22:20, Teske, Devin wrote:
>>>>>>>>> On Oct 19, 2013, at 10:07 AM, Johan Broman wrote:
>>>>>>>>> 
>>>>>>>>>> I recreated the graid mirror on ada2 and ada3 and reran the
>>>>>>>>>> installation. I'm unable to scroll the msgbox using PgDn or arrow
>>>>>>>>>> keys. There is no indication that the action failed and I'm returned
>>>>>>>>>> to the ZFS setup screen if I hit OK.
>>>>>>>>>> 
>>>>>>>>>> I have screen shots (taken with my phone) of the msgbox and "ps
>>>>>>>>>> auxwww" output. Let me know what kind of debug info you would like.
>>>>>>>>>> I've put the screen shots here:
>>>>>>>>>> 
>>>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=6322000e13ed155bda748698c4a0d54c9de7c29f5566affe202d7c5a29917cd1
>>>>>>>>> I've added a patch to fix debugging in the zfsboot script...
>>>>>>>>> 
>>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=1dc96a8b5450d27fe8210b361c9ed736ccd448f78df6aabe170bb80e31bca6d9
>>>>>>>>> 
>>>>>>>>> Feedback welcome.
>>>>>>>>> 
>>>>>>>>> Johan,...
>>>>>>>>> 
>>>>>>>>> Can you see if the patch sheds some better light as to what's failing?
>>>>>>>>> 
>>>>>>>>> The patch won't fix the problem, but it should give us an accurate 
>>>>>>>>> error
>>>>>>>>> message so that we can learn what precisely is returning an error
>>>>>>>>> status.
>>>>>>>>> 
>>>>>>>>> Thanks in advance.
>>>>>>>>> 
>>>>>>> I do notice that Devin's manually prefixing the error message with the
>>>>>>> tool name, is partially redundancy w

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 21, 2013, at 9:06 AM, Allan Jude wrote:

> On 2013-10-21 12:04, Teske, Devin wrote:
>> On Oct 21, 2013, at 8:50 AM, Allan Jude wrote:
>> 
>>> On 2013-10-21 11:45, Johan Broman wrote:
>>>> Hi!
>>>> 
>>>> Sorry for the delayed answer. I've patched zfsboot and rebuilt the
>>>> release. I now get an error message that ada2 can't be used, which is
>>>> correct. Good stuff! :)
>>>> 
>>>> ( I've recreated the test environment using a KVM guest with four SATA
>>>> drives instead of the server I was using. I makes it easier to test
>>>> stuff. )
>>>> 
>>>> Here's the screenshot:
>>>> 
>>>> https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.png&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=f6fc4ec7c9d8b9486e897156dac5af7c3f64e3fae746dcf6c66ad91564a8ce99
>>>> 
>>>> 
>>>> Maybe one should be unable to select drives that are part of a graid
>>>> in the first place? Or is that out-of-scope for bsdinstall at this
>>>> point? (As I guess that requires too many changes/new lines)
>>>> 
>>>> 
>>>> Cheers
>>>> Johan
>>>> 
>>>> 
>>>> On 19/10/13 22:20, Teske, Devin wrote:
>>>>> On Oct 19, 2013, at 10:07 AM, Johan Broman wrote:
>>>>> 
>>>>>> I recreated the graid mirror on ada2 and ada3 and reran the
>>>>>> installation. I'm unable to scroll the msgbox using PgDn or arrow
>>>>>> keys. There is no indication that the action failed and I'm returned
>>>>>> to the ZFS setup screen if I hit OK.
>>>>>> 
>>>>>> I have screen shots (taken with my phone) of the msgbox and "ps
>>>>>> auxwww" output. Let me know what kind of debug info you would like.
>>>>>> I've put the screen shots here:
>>>>>> 
>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=6322000e13ed155bda748698c4a0d54c9de7c29f5566affe202d7c5a29917cd1
>>>>> I've added a patch to fix debugging in the zfsboot script...
>>>>> 
>>>>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=1dc96a8b5450d27fe8210b361c9ed736ccd448f78df6aabe170bb80e31bca6d9
>>>>> 
>>>>> Feedback welcome.
>>>>> 
>>>>> Johan,...
>>>>> 
>>>>> Can you see if the patch sheds some better light as to what's failing?
>>>>> 
>>>>> The patch won't fix the problem, but it should give us an accurate error
>>>>> message so that we can learn what precisely is returning an error
>>>>> status.
>>>>> 
>>>>> Thanks in advance.
>>>>> 
>>> I do notice that Devin's manually prefixing the error message with the
>>> tool name, is partially redundancy when the tool does it it self, but we
>>> can't always be sure it will do that.
>>> 
>> The next patchset will fix that.
>> 
>> I'm dropping the tool name from the msgbox contents and putting it in
>> the title (e.g., '"Error: gpart") that way... even if the tool spits out its 
>> own
>> name (or not), we'll know what exactly what was going on by looking
>> at the title.
>> 
>> 
>>> the graid thing is rather hard to detect, especially when it is a
>>> faulted array that doesn't even appear in graid status etc.
>>> 
>> I believe the idea behind the script is that whatever you tell it to use will
>> be destroyed.
>> 
>> Allan, maybe perhaps we could add some code that attempts to dis-
>> assemble a graid to make the disk usable?
>> 
>> Johan, what would you be more apt to expect? That it killed your graid
>> or that it gave an error? (/me thinks what the recourse to the error might
>> entail -- going to partedit?)
> Your recourse would be switching to the shell (control+alt+f4) and
> destroying the graid.
> 
> I am a little hesitant to go destroying graids unprompted. If we had the
> geom.confxml parsing, we might be able to detect it and ask the user
> what to do
> 

Well, my concern with going and asking about each/every configuration
before we clear it...

Hasn't the user already answered a "Last Chance!" dialog giving the go-
ahead to nuke any/all data *and* configurations on a drive?
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 21, 2013, at 8:45 AM, Johan Broman wrote:

> Hi!
> 
> Sorry for the delayed answer. I've patched zfsboot and rebuilt the release. I 
> now get an error message that ada2 can't be used, which is correct. Good 
> stuff! :)
> 
> ( I've recreated the test environment using a KVM guest with four SATA drives 
> instead of the server I was using. I makes it easier to test stuff. )
> 
> Here's the screenshot:
> 
> http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.png
> 

Thank you so much, definitely says we're headed in the right direction.

It was bothering me that the debugging wasn't "up to-par."



> 
> Maybe one should be unable to select drives that are part of a graid in the 
> first place? Or is that out-of-scope for bsdinstall at this point? (As I 
> guess that requires too many changes/new lines)
> 

Perhaps we just need to do a better job of clearing previous
configurations from the disks that are to be used in the pool.
-- 
Devin

> On 19/10/13 22:20, Teske, Devin wrote:
>> 
>> On Oct 19, 2013, at 10:07 AM, Johan Broman wrote:
>> 
>>> I recreated the graid mirror on ada2 and ada3 and reran the installation. 
>>> I'm unable to scroll the msgbox using PgDn or arrow keys. There is no 
>>> indication that the action failed and I'm returned to the ZFS setup screen 
>>> if I hit OK.
>>> 
>>> I have screen shots (taken with my phone) of the msgbox and "ps auxwww" 
>>> output. Let me know what kind of debug info you would like. I've put the 
>>> screen shots here:
>>> 
>>> https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=HwEpYSg1%2Bqv6o4vps1DECabNoHOTpZj5VxD1rbHI5jU%3D%0A&s=a27573c7279ac5bc926ca2dc872a18550ee8244421b91d85b9f51b202b23e168
>> 
>> I've added a patch to fix debugging in the zfsboot script...
>> 
>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=HwEpYSg1%2Bqv6o4vps1DECabNoHOTpZj5VxD1rbHI5jU%3D%0A&s=195830666030a264348cc1f37355789cd9ada04e0851b8c5769ce6426403fa28
>> 
>> Feedback welcome.
>> 
>> Johan,...
>> 
>> Can you see if the patch sheds some better light as to what's failing?
>> 
>> The patch won't fix the problem, but it should give us an accurate error
>> message so that we can learn what precisely is returning an error status.
>> 
>> Thanks in advance.
>> 

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 21, 2013, at 8:50 AM, Allan Jude wrote:

> On 2013-10-21 11:45, Johan Broman wrote:
>> Hi!
>> 
>> Sorry for the delayed answer. I've patched zfsboot and rebuilt the
>> release. I now get an error message that ada2 can't be used, which is
>> correct. Good stuff! :)
>> 
>> ( I've recreated the test environment using a KVM guest with four SATA
>> drives instead of the server I was using. I makes it easier to test
>> stuff. )
>> 
>> Here's the screenshot:
>> 
>> https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall/Screenshot_2013-10-21.png&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=f6fc4ec7c9d8b9486e897156dac5af7c3f64e3fae746dcf6c66ad91564a8ce99
>> 
>> 
>> Maybe one should be unable to select drives that are part of a graid
>> in the first place? Or is that out-of-scope for bsdinstall at this
>> point? (As I guess that requires too many changes/new lines)
>> 
>> 
>> Cheers
>> Johan
>> 
>> 
>> On 19/10/13 22:20, Teske, Devin wrote:
>>> 
>>> On Oct 19, 2013, at 10:07 AM, Johan Broman wrote:
>>> 
>>>> I recreated the graid mirror on ada2 and ada3 and reran the
>>>> installation. I'm unable to scroll the msgbox using PgDn or arrow
>>>> keys. There is no indication that the action failed and I'm returned
>>>> to the ZFS setup screen if I hit OK.
>>>> 
>>>> I have screen shots (taken with my phone) of the msgbox and "ps
>>>> auxwww" output. Let me know what kind of debug info you would like.
>>>> I've put the screen shots here:
>>>> 
>>>> https://urldefense.proofpoint.com/v1/url?u=http://212.181.212.146/bsdinstall&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=6322000e13ed155bda748698c4a0d54c9de7c29f5566affe202d7c5a29917cd1
>>> 
>>> I've added a patch to fix debugging in the zfsboot script...
>>> 
>>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=B6jLX5vSxIfvyks3HH55lYWtfhRBpGZ3nVA65M%2FgmXM%3D%0A&s=1dc96a8b5450d27fe8210b361c9ed736ccd448f78df6aabe170bb80e31bca6d9
>>> 
>>> Feedback welcome.
>>> 
>>> Johan,...
>>> 
>>> Can you see if the patch sheds some better light as to what's failing?
>>> 
>>> The patch won't fix the problem, but it should give us an accurate error
>>> message so that we can learn what precisely is returning an error
>>> status.
>>> 
>>> Thanks in advance.
>>> 
> I do notice that Devin's manually prefixing the error message with the
> tool name, is partially redundancy when the tool does it it self, but we
> can't always be sure it will do that.
> 

The next patchset will fix that.

I'm dropping the tool name from the msgbox contents and putting it in
the title (e.g., '"Error: gpart") that way... even if the tool spits out its own
name (or not), we'll know what exactly what was going on by looking
at the title.


> the graid thing is rather hard to detect, especially when it is a
> faulted array that doesn't even appear in graid status etc.
> 

I believe the idea behind the script is that whatever you tell it to use will
be destroyed.

Allan, maybe perhaps we could add some code that attempts to dis-
assemble a graid to make the disk usable?

Johan, what would you be more apt to expect? That it killed your graid
or that it gave an error? (/me thinks what the recourse to the error might
entail -- going to partedit?)
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 19, 2013, at 10:07 AM, Johan Broman wrote:

> I recreated the graid mirror on ada2 and ada3 and reran the installation. I'm 
> unable to scroll the msgbox using PgDn or arrow keys. There is no indication 
> that the action failed and I'm returned to the ZFS setup screen if I hit OK.
> 
> I have screen shots (taken with my phone) of the msgbox and "ps auxwww" 
> output. Let me know what kind of debug info you would like. I've put the 
> screen shots here:
> 
> http://212.181.212.146/bsdinstall

I've added a patch to fix debugging in the zfsboot script...

http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/

Feedback welcome.

Johan,...

Can you see if the patch sheds some better light as to what's failing?

The patch won't fix the problem, but it should give us an accurate error
message so that we can learn what precisely is returning an error status.

Thanks in advance.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 19, 2013, at 10:07 AM, Johan Broman wrote:

> 
> 
> On 19/10/13 18:27, Allan Jude wrote:
>> On 2013-10-19 11:55, Teske, Devin wrote:
>>> On Oct 19, 2013, at 8:43 AM, Teske, Devin wrote:
>>> 
>>>> On Oct 19, 2013, at 8:34 AM, Allan Jude wrote:
>>>> 
>>>>> On 2013-10-19 11:31, Johan Broman wrote:
>>>>>> 
>>>>>> On 19/10/13 17:23, Allan Jude wrote:
>>>>>>> On 2013-10-19 10:56, Johan Broman wrote:
>>>>>>>> Hi!
>>>>>>>> 
>>>>>>>> Just tested the root-on-ZFS install option using FreeBSD 10 beta 1. I
>>>>>>>> have 4 SATA drives in my server. I select all four of them in a RAIDZ1
>>>>>>>> setup. I hit enter to continue the installation and the zpool is
>>>>>>>> created, but I'm then returned to the zpool selection screen again. It
>>>>>>>> turned out that two of the drives had previously been used in a
>>>>>>>> (Linux) software mirror setup and because of this they got activated
>>>>>>>> in /dev/raid/r0. Because of this I ended up in an endless bsdinstall
>>>>>>>> loop.
>>>>>>>> 
>>>>>>>> Removing the raid device using the graid command resolved the
>>>>>>>> situation.
>>>>>>>> 
>>>>>>>> Now maybe this is working as designed, but there was no warning/alert
>>>>>>>> to the fact that the devices couldn't be used. Perhaps a warning
>>>>>>>> should be rasied in this situation?
>>>>>>>> 
>>>>>>>> Thanks for all the great work on the new installer, really looking
>>>>>>>> forward to FreeBSD 10!
>>>>>>>> 
>>>>>>>> Cheers
>>>>>>>> Johan
>>>>>>>> ___
>>>>>>>> freebsd-current@freebsd.org mailing list
>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>>>>>>>> To unsubscribe, send any mail to
>>>>>>>> "freebsd-current-unsubscr...@freebsd.org"
>>>>>>> Errors like that normally generate a msgbox dialog with the error output
>>>>>>> from whichever command failed. I'll have to dig into it and see where
>>>>>>> that problem is. I've seen other people have problems creating ZFS
>>>>>>> arrays after graid, but in that case it was an incomplete graid label
>>>>>>> causing a device to be locked but not appear in the graid status output.
>>>>>>> 
>>>>>> Ah ok. A msgbox did appear but the drives that had the problem (ada2
>>>>>> and ada3) wasn't visible in the output. (not sure if the box itself
>>>>>> has a size limit or maybe I was just unable to scroll down and see the
>>>>>> errors?). The only visible output was that it was able to create
>>>>>> labels on ada0 and ada1.
>>>>>> 
>>>>>> /Johan
>>>>>> ___
>>>>>> freebsd-current@freebsd.org mailing list
>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>>>>>> To unsubscribe, send any mail to
>>>>>> "freebsd-current-unsubscr...@freebsd.org"
>>>>> Ahh yes, you have to press 'page-down' to scroll the msgbox. I tried to
>>>>> add a scrollbar but turns out that is not possible.
>>>>> 
>>>>> The only indication that there is more message to read, is a small 'xx%'
>>>>> in the bottom right. We might have to look at breaking that output up or
>>>>> something.
>>>>> 
>>>> 
>>>> The only reason for a msgbox widget to scroll is if it is displayed at
>>>> maximum height or width of the screen and it *still* has more data
>>>> to display than can be presented at-once.
>>>> 
>>> I should clarify...
>>> 
>>> The zfsboot script doesn't use dialog(1) directly. It uses the bsdconfig 
>>> API.
>>> That being said, msgbox widgets automatically scale their size to fit the
>>> content being displayed. So whenever a msgbox is thrown up using this
>>> API... the widge

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 19, 2013, at 9:27 AM, Allan Jude wrote:

> On 2013-10-19 11:55, Teske, Devin wrote:
>> On Oct 19, 2013, at 8:43 AM, Teske, Devin wrote:
>> 
>>> On Oct 19, 2013, at 8:34 AM, Allan Jude wrote:
>>> 
>>>> On 2013-10-19 11:31, Johan Broman wrote:
>>>>> 
>>>>> On 19/10/13 17:23, Allan Jude wrote:
>>>>>> On 2013-10-19 10:56, Johan Broman wrote:
>>>>>>> Hi!
>>>>>>> 
>>>>>>> Just tested the root-on-ZFS install option using FreeBSD 10 beta 1. I
>>>>>>> have 4 SATA drives in my server. I select all four of them in a RAIDZ1
>>>>>>> setup. I hit enter to continue the installation and the zpool is
>>>>>>> created, but I'm then returned to the zpool selection screen again. It
>>>>>>> turned out that two of the drives had previously been used in a
>>>>>>> (Linux) software mirror setup and because of this they got activated
>>>>>>> in /dev/raid/r0. Because of this I ended up in an endless bsdinstall
>>>>>>> loop.
>>>>>>> 
>>>>>>> Removing the raid device using the graid command resolved the
>>>>>>> situation.
>>>>>>> 
>>>>>>> Now maybe this is working as designed, but there was no warning/alert
>>>>>>> to the fact that the devices couldn't be used. Perhaps a warning
>>>>>>> should be rasied in this situation?
>>>>>>> 
>>>>>>> Thanks for all the great work on the new installer, really looking
>>>>>>> forward to FreeBSD 10!
>>>>>>> 
>>>>>>> Cheers
>>>>>>> Johan
>>>>>>> ___
>>>>>>> freebsd-current@freebsd.org mailing list
>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://lists.freebsd.org/mailman/listinfo/freebsd-current&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=AnourZYCpeybP9aMxTFH%2B4GBOf0xikQ2jpEi%2BPFWcKo%3D%0A&s=fec708d91fa0976a0e297ef3628851168f36e9cb34727dc7935e26ae190e90da
>>>>>>> To unsubscribe, send any mail to
>>>>>>> "freebsd-current-unsubscr...@freebsd.org"
>>>>>> Errors like that normally generate a msgbox dialog with the error output
>>>>>> from whichever command failed. I'll have to dig into it and see where
>>>>>> that problem is. I've seen other people have problems creating ZFS
>>>>>> arrays after graid, but in that case it was an incomplete graid label
>>>>>> causing a device to be locked but not appear in the graid status output.
>>>>>> 
>>>>> Ah ok. A msgbox did appear but the drives that had the problem (ada2
>>>>> and ada3) wasn't visible in the output. (not sure if the box itself
>>>>> has a size limit or maybe I was just unable to scroll down and see the
>>>>> errors?). The only visible output was that it was able to create
>>>>> labels on ada0 and ada1.
>>>>> 
>>>>> /Johan
>>>>> ___
>>>>> freebsd-current@freebsd.org mailing list
>>>>> https://urldefense.proofpoint.com/v1/url?u=http://lists.freebsd.org/mailman/listinfo/freebsd-current&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=AnourZYCpeybP9aMxTFH%2B4GBOf0xikQ2jpEi%2BPFWcKo%3D%0A&s=fec708d91fa0976a0e297ef3628851168f36e9cb34727dc7935e26ae190e90da
>>>>> To unsubscribe, send any mail to
>>>>> "freebsd-current-unsubscr...@freebsd.org"
>>>> Ahh yes, you have to press 'page-down' to scroll the msgbox. I tried to
>>>> add a scrollbar but turns out that is not possible.
>>>> 
>>>> The only indication that there is more message to read, is a small 'xx%'
>>>> in the bottom right. We might have to look at breaking that output up or
>>>> something.
>>>> 
>>> 
>>> The only reason for a msgbox widget to scroll is if it is displayed at
>>> maximum height or width of the screen and it *still* has more data
>>> to display than can be presented at-once.
>>> 
>> I should clarify...
>> 
>> The zfsboot script doesn't use dialog(1) directly. It uses the bsdconfig API.
&g

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 19, 2013, at 8:43 AM, Teske, Devin wrote:

> 
> On Oct 19, 2013, at 8:34 AM, Allan Jude wrote:
> 
>> On 2013-10-19 11:31, Johan Broman wrote:
>>> 
>>> 
>>> On 19/10/13 17:23, Allan Jude wrote:
>>>> On 2013-10-19 10:56, Johan Broman wrote:
>>>>> Hi!
>>>>> 
>>>>> Just tested the root-on-ZFS install option using FreeBSD 10 beta 1. I
>>>>> have 4 SATA drives in my server. I select all four of them in a RAIDZ1
>>>>> setup. I hit enter to continue the installation and the zpool is
>>>>> created, but I'm then returned to the zpool selection screen again. It
>>>>> turned out that two of the drives had previously been used in a
>>>>> (Linux) software mirror setup and because of this they got activated
>>>>> in /dev/raid/r0. Because of this I ended up in an endless bsdinstall
>>>>> loop.
>>>>> 
>>>>> Removing the raid device using the graid command resolved the
>>>>> situation.
>>>>> 
>>>>> Now maybe this is working as designed, but there was no warning/alert
>>>>> to the fact that the devices couldn't be used. Perhaps a warning
>>>>> should be rasied in this situation?
>>>>> 
>>>>> Thanks for all the great work on the new installer, really looking
>>>>> forward to FreeBSD 10!
>>>>> 
>>>>> Cheers
>>>>> Johan
>>>>> ___
>>>>> freebsd-current@freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>>>>> To unsubscribe, send any mail to
>>>>> "freebsd-current-unsubscr...@freebsd.org"
>>>> Errors like that normally generate a msgbox dialog with the error output
>>>> from whichever command failed. I'll have to dig into it and see where
>>>> that problem is. I've seen other people have problems creating ZFS
>>>> arrays after graid, but in that case it was an incomplete graid label
>>>> causing a device to be locked but not appear in the graid status output.
>>>> 
>>> 
>>> Ah ok. A msgbox did appear but the drives that had the problem (ada2
>>> and ada3) wasn't visible in the output. (not sure if the box itself
>>> has a size limit or maybe I was just unable to scroll down and see the
>>> errors?). The only visible output was that it was able to create
>>> labels on ada0 and ada1.
>>> 
>>> /Johan
>>> ___
>>> freebsd-current@freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>>> To unsubscribe, send any mail to
>>> "freebsd-current-unsubscr...@freebsd.org"
>> Ahh yes, you have to press 'page-down' to scroll the msgbox. I tried to
>> add a scrollbar but turns out that is not possible.
>> 
>> The only indication that there is more message to read, is a small 'xx%'
>> in the bottom right. We might have to look at breaking that output up or
>> something.
>> 
> 
> 
> The only reason for a msgbox widget to scroll is if it is displayed at
> maximum height or width of the screen and it *still* has more data
> to display than can be presented at-once.
> 

I should clarify...

The zfsboot script doesn't use dialog(1) directly. It uses the bsdconfig API.
That being said, msgbox widgets automatically scale their size to fit the
content being displayed. So whenever a msgbox is thrown up using this
API... the widget will never scroll unless the box can't be made big enough
to hold the entire content (either the screen resolution or terminal size is
too small; we maxed out the size of the widget; and there's still hidden
content).

But...

While all of bsdconfig uses this API, hardly any of bsdinstall uses this API.



> If... however... the msgbox widget is *not* full-height or full-width
> yet... it is requiring you to scroll -- then we've found a bug.
> 
> Can we get a screen shot?

So we really need to nail down precisely which error box this is so that
we can address whether the issue is in-fact an instance of using the old
error-box handling instead of the auto-sizing API.

So...

With this described API, you should never have to scroll a box unless it
can't fit all the data *and* you should be able to immediately identify when
that becomes the case...

1. The widget spans the entire width of the screen.

2. The widget spans the entire height of the screen.

3. 

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 19, 2013, at 8:34 AM, Allan Jude wrote:

> On 2013-10-19 11:31, Johan Broman wrote:
>> 
>> 
>> On 19/10/13 17:23, Allan Jude wrote:
>>> On 2013-10-19 10:56, Johan Broman wrote:
 Hi!
 
 Just tested the root-on-ZFS install option using FreeBSD 10 beta 1. I
 have 4 SATA drives in my server. I select all four of them in a RAIDZ1
 setup. I hit enter to continue the installation and the zpool is
 created, but I'm then returned to the zpool selection screen again. It
 turned out that two of the drives had previously been used in a
 (Linux) software mirror setup and because of this they got activated
 in /dev/raid/r0. Because of this I ended up in an endless bsdinstall
 loop.
 
 Removing the raid device using the graid command resolved the
 situation.
 
 Now maybe this is working as designed, but there was no warning/alert
 to the fact that the devices couldn't be used. Perhaps a warning
 should be rasied in this situation?
 
 Thanks for all the great work on the new installer, really looking
 forward to FreeBSD 10!
 
 Cheers
 Johan
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to
 "freebsd-current-unsubscr...@freebsd.org"
>>> Errors like that normally generate a msgbox dialog with the error output
>>> from whichever command failed. I'll have to dig into it and see where
>>> that problem is. I've seen other people have problems creating ZFS
>>> arrays after graid, but in that case it was an incomplete graid label
>>> causing a device to be locked but not appear in the graid status output.
>>> 
>> 
>> Ah ok. A msgbox did appear but the drives that had the problem (ada2
>> and ada3) wasn't visible in the output. (not sure if the box itself
>> has a size limit or maybe I was just unable to scroll down and see the
>> errors?). The only visible output was that it was able to create
>> labels on ada0 and ada1.
>> 
>> /Johan
>> ___
>> freebsd-current@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to
>> "freebsd-current-unsubscr...@freebsd.org"
> Ahh yes, you have to press 'page-down' to scroll the msgbox. I tried to
> add a scrollbar but turns out that is not possible.
> 
> The only indication that there is more message to read, is a small 'xx%'
> in the bottom right. We might have to look at breaking that output up or
> something.
> 


The only reason for a msgbox widget to scroll is if it is displayed at
maximum height or width of the screen and it *still* has more data
to display than can be presented at-once.

If... however... the msgbox widget is *not* full-height or full-width
yet... it is requiring you to scroll -- then we've found a bug.

Can we get a screen shot?
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: BE Loader Menu (was Re: rcs)

[Teske, Devin]

On Oct 10, 2013, at 9:42 AM, Julian Elischer wrote:

> On 10/11/13 12:39 AM, Teske, Devin wrote:
>> On Oct 10, 2013, at 9:06 AM, Julian Elischer wrote:
>> 
>>> On 10/10/13 1:05 AM, Teske, Devin wrote:
>>>> I'm late to the party again ;D (didn't realize the rcs thread had turned 
>>>> BE)
>>>> 
>>>> Both problems can be solved.
>>>> The loading of the kernel *after* choosing your boot device is trivial.
>>>> We've been doing it at $work for *years* (almost a decade?)
>>>> 
>>>> I can put that in, whenever. Probably at the same time as implementing
>>>> the live/dynamic BE menus for selecting the root device.
>>> yeah it always pisses me of when the menu comes up after the kernel is 
>>> loaded because 99% of the time, I'm in the menu because I want to boot a 
>>> DIFFERENT kernel..
>>> 
>> Same thought I had about 7 years ago. After hearing that others
>> (especially you, Julian) think the same thoughts...
>> 
>> I'm happily ready to merge a patch from VICOR to achieve this.
> PLEASE!..   put it up for review somewhere...
> 

Done...

http://druidbsd.cvs.sf.net/viewvc/druidbsd/forth_zfs/

All new code. Had to make it programmable:

http://twitpic.com/dhv2b6

See that the patch adds documentation for loader.conf(5).
It also clarifies a horrible description of "start" versus "initialize" in 
loader.4th(8).


> I wonder if we can we get a reviewboard instance for the project some time?
> 

Do let me know what you think. I went all-out on this one.
-- 
Devin

P.S. I probably hadn't have put so much thought into this one if it hadn't been
specifically you asking ;D but I think you know that.

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 10, 2013, at 10:54 AM, Allan Jude wrote:

> On 2013-10-10 13:21, Warren Block wrote:
>> [off-list reply]
>> 
>> I have not tested the new version yet.  Is there any chance it
>> supports setting up a gmirror?
>> 
>> Something I've been meaning to discuss with Devin is the concept of
>> presets.  The user would be able to choose from a list of presets:
>> 
>>  Filesystem Layout
>>  -
>>  Merged / filesystem
>>  Split /, /var, /tmp, /usr filesystems
>> 
>> then
>> 
>>  Filesystem Type
>>  ---
>>  UFS
>>  ZFS
>> 
>> then
>> 
>>  Disk Layout
>>  ---
>>  Plain disk
>>  Mirror
>>  RAID
>> 
>> Those choices might change depending on earlier ones, like if they had
>> chosen ZFS earlier they would get a choice of Mirror, RAID-Z1,
>> RAID-Z2, and so on.
>> 
>> All this would build up a configuration, and at the end the user can
>> edit it (filesystem sizes, for instance), then pick Go and walk away.
> 
> All of the UFS stuff is in C, so is read-only for me.
> 
> It is something I'd like to do, but it is really iffy if that can be
> done in time for 10.0
> 
> In general, gmirror support could be added to the ZFS section (renamed
> to something else). So in our current 'zfs' menu, would become the 'new
> partition manager' menu. Add an extra option for ZFS or UFS, and then
> the vdev menu would context switch between zfs vdevs and ufs options
> like gmirror and gstripe or something
> 
> Adding an extra option for / or / /var /tmp /usr is not a bad idea
> either, maybe even for ZFS, offering a simpler set of datasets that
> might be some people's preference (especially if I can't do the
> following in time:)
> 
> In a future revision of the zfsboot stuff, I'd like to offer a dialog
> where users can actually adjust the ZFS datasets. provide a menu listing
> the created dataset, with option to add more, and when you select a
> dataset, you can adjust its options or delete it (some really scary
> validation needs to happen here so you can't delete /usr if you have
> /usr/local)
> 
> If the UFS stuff took the form of the current ZFS stuff (give me the
> entire disk, and i'll run with it), it would be fairly easy to
> implement, and could probably be done between 10.0-BETA1 and 10.0-BETA2
> 
> The more powerful features of the current partedit code, where it
> actually shows what is on your disk already, and allows you to just
> adjust it, and in general manage dual booting etc, that is a lot more
> complicated.
> 

But the API exists (and we get a taste of the API in zfsboot). Specifically, a 
script
would mangle a disk and then call f_device_rescan to get the menus to populate
new entries.

That aside, one of my plans for diskmgmt was to essentially let the user perform
free-wheeling adhoc configurations.

Here's an extreme example of what I mean**:

1. Boot a box with naked disks
2. Create some gmultipath labels
3. Put a GPT scheme on the disk with one BSD partition
4. Put some UFS labels in the partition
5. Build a zpool out of a secondary label of each multipath'd disk

** Because afterall, FreeBSD shouldn't prevent you from doing something like 
that.

I think that if we try to tackle scenarios like that, then we'll really come up 
with a
winning diskmgmt module (because right now I can't go do some UFS setup and
then shove the products into a zpool).

_When_ the ZFS management stuff is all integrated (which I plan on integrating
bits from pjd's zfsconf too), I think it should be agnostic about what you give 
it as
a device *and* allow you to configure many types of things without being forced
into an "either-or" situation (because often it can be a "I want both" 
situation).

(tuppence)
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 10, 2013, at 10:18 AM, Allan Jude wrote:

> On 2013-10-10 11:30, Teske, Devin wrote:
>> On Oct 10, 2013, at 4:55 AM, Nathan Whitehorn wrote:
>> 
>>> On 10/10/13 09:20, Allan Jude wrote:
>>>> On 2013-10-10 03:00, Nathan Whitehorn wrote:
>>>>> On 10/09/13 18:55, Teske, Devin wrote:
>>>>>> On Oct 8, 2013, at 11:19 PM, Nathan Whitehorn wrote:
>>>>>> 
>>>>>>> On 10/09/13 01:13, Allan Jude wrote:
>>>>>>>> On 2013-10-08 16:17, Nathan Whitehorn wrote:
>>>>>>>>> On 10/07/13 21:59, Allan Jude wrote:
>>>>>>>>>> Devin Teske and I have been working on a big patch to bsdinstall to
>>>>>>>>>> implement installing on a ZFS pool. It supports both GPT and MBR, 
>>>>>>>>>> the 4k
>>>>>>>>>> sector gnop trick, and optional GELI encryption. We would like to 
>>>>>>>>>> commit
>>>>>>>>>> this in time for 10.0-BETA1 so it needs some testing to work out any
>>>>>>>>>> obvious bugs before we send it off to re@ to get it committed.
>>>>>>>>>> 
>>>>>>>>>> It includes a single configuration menu that allows you to select 
>>>>>>>>>> all of
>>>>>>>>>> the required details, including which drives to use (gets details 
>>>>>>>>>> from
>>>>>>>>>> camcontrol, also includes an inspection utility that presents the
>>>>>>>>>> detailed output of camcontrol inquiry/identify, and gpart show), what
>>>>>>>>>> ZFS RAID level to use (taking in to consideration the selected 
>>>>>>>>>> number of
>>>>>>>>>> drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Additional, it includes some other changes to bsdinstall:
>>>>>>>>>> 1. Change the default to the 'non-standard keyboard mapping' prompt 
>>>>>>>>>> to no
>>>>>>>>>> 2. Replace the 3 separate dialogs to configure an ipv4 address with 
>>>>>>>>>> just 1
>>>>>>>>>> 3. Remove the dialog asking if you wish to enable crash dumps, this
>>>>>>>>>> feature has been combined into the regular 'services to enable' 
>>>>>>>>>> dialog
>>>>>>>>>> and enabled by default
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> You can browse the patches here:
>>>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=ckXs2H5TEcKdiN%2F9l%2FClInsMys9LmTwEbElzzrBchMU%3D%0A&s=6361ad5ccc04a6d8401158a3ee1b3d8df9fcd2ef47ff6ad6365a57b907cceaec
>>>>>>>>>> 
>>>>>>>>>> I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
>>>>>>>>>> available compressed (48 MB) or uncompressed (211 MB):
>>>>>>>>>> 
>>>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=ckXs2H5TEcKdiN%2F9l%2FClInsMys9LmTwEbElzzrBchMU%3D%0A&s=570daea25d9e629788c76c2b5a6eae19d32050363986a1004a6434c3466965c2
>>>>>>>>>> 
>>>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=ckXs2H5TEcKdiN%2F9l%2FClInsMys9LmTwEbElzzrBchMU%3D%0A&s=f52d604a2503a48b409f92f8376bb773de4dbc469d1e74b16d051a20ad894820
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> We look forward to your feedback
>>>>>>>>>> 
>>>>>>>>> Thanks for doing this! I had a few comments:
>>>>>>>>> 1. ZFS is not bootable on all architectures. Could you adjust that 
>>>>>>>>&g

Re: BE Loader Menu (was Re: rcs)

[Teske, Devin]

On Oct 10, 2013, at 9:06 AM, Julian Elischer wrote:

> On 10/10/13 1:05 AM, Teske, Devin wrote:
>> 
>> I'm late to the party again ;D (didn't realize the rcs thread had turned BE)
>> 
>> Both problems can be solved.
>> The loading of the kernel *after* choosing your boot device is trivial.
>> We've been doing it at $work for *years* (almost a decade?)
>> 
>> I can put that in, whenever. Probably at the same time as implementing
>> the live/dynamic BE menus for selecting the root device.
> 
> yeah it always pisses me of when the menu comes up after the kernel is loaded 
> because 99% of the time, I'm in the menu because I want to boot a DIFFERENT 
> kernel..
> 

Same thought I had about 7 years ago. After hearing that others
(especially you, Julian) think the same thoughts...

I'm happily ready to merge a patch from VICOR to achieve this.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 10, 2013, at 4:55 AM, Nathan Whitehorn wrote:

> On 10/10/13 09:20, Allan Jude wrote:
>> On 2013-10-10 03:00, Nathan Whitehorn wrote:
>>> On 10/09/13 18:55, Teske, Devin wrote:
>>>> On Oct 8, 2013, at 11:19 PM, Nathan Whitehorn wrote:
>>>> 
>>>>> On 10/09/13 01:13, Allan Jude wrote:
>>>>>> On 2013-10-08 16:17, Nathan Whitehorn wrote:
>>>>>>> On 10/07/13 21:59, Allan Jude wrote:
>>>>>>>> Devin Teske and I have been working on a big patch to bsdinstall to
>>>>>>>> implement installing on a ZFS pool. It supports both GPT and MBR, the 
>>>>>>>> 4k
>>>>>>>> sector gnop trick, and optional GELI encryption. We would like to 
>>>>>>>> commit
>>>>>>>> this in time for 10.0-BETA1 so it needs some testing to work out any
>>>>>>>> obvious bugs before we send it off to re@ to get it committed.
>>>>>>>> 
>>>>>>>> It includes a single configuration menu that allows you to select all 
>>>>>>>> of
>>>>>>>> the required details, including which drives to use (gets details from
>>>>>>>> camcontrol, also includes an inspection utility that presents the
>>>>>>>> detailed output of camcontrol inquiry/identify, and gpart show), what
>>>>>>>> ZFS RAID level to use (taking in to consideration the selected number 
>>>>>>>> of
>>>>>>>> drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Additional, it includes some other changes to bsdinstall:
>>>>>>>> 1. Change the default to the 'non-standard keyboard mapping' prompt to 
>>>>>>>> no
>>>>>>>> 2. Replace the 3 separate dialogs to configure an ipv4 address with 
>>>>>>>> just 1
>>>>>>>> 3. Remove the dialog asking if you wish to enable crash dumps, this
>>>>>>>> feature has been combined into the regular 'services to enable' dialog
>>>>>>>> and enabled by default
>>>>>>>> 
>>>>>>>> 
>>>>>>>> You can browse the patches here:
>>>>>>>> http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/
>>>>>>>> 
>>>>>>>> I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
>>>>>>>> available compressed (48 MB) or uncompressed (211 MB):
>>>>>>>> 
>>>>>>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz
>>>>>>>> 
>>>>>>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso
>>>>>>>> 
>>>>>>>> 
>>>>>>>> We look forward to your feedback
>>>>>>>> 
>>>>>>> Thanks for doing this! I had a few comments:
>>>>>>> 1. ZFS is not bootable on all architectures. Could you adjust that menu
>>>>>>> item to only display for i386, amd64, and (I think?) sparc64. Use uname
>>>>>>> -m, not -p, for this.
>>>>>> I had not considered that, I'll make that change
>>>>>> 
>>>>>>> 1a. The script is broken on sparc64 in any case, which uses VTOC8
>>>>>>> instead of GPT.
>>>>>> I'll disable sparc64 as well
>>>>>> 
>>>>>>> 2. Why are you using camcontrol? That is guaranteed not to work on
>>>>>>> non-CAM systems. You should use the GEOM ident string if you need an ID.
>>>>>> The GEOM ident string doesn't do enough to help the user identify which
>>>>>> drive is which.
>>>>>> More data is not exposed anywhere that I could find
>>>>>> 
>>>>>> What we really need, is dev.ada.0.desc% like we have for network
>>>>>> interfaces and a slew of other devices. GEOM data is great, but it is
>>>>>> not exposed in a shell friendly way any place that I could find, other
>>>>>> than the sysctl with DOT and XML data.
>>>>> This is one of the reasons the partition editor is written in C. There
>>>>> are a few other odd corner-cases where C is much more powerful than the
&g

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 10, 2013, at 12:00 AM, Nathan Whitehorn wrote:

> On 10/09/13 18:55, Teske, Devin wrote:
>> On Oct 8, 2013, at 11:19 PM, Nathan Whitehorn wrote:
>> 
>>> On 10/09/13 01:13, Allan Jude wrote:
>>>> On 2013-10-08 16:17, Nathan Whitehorn wrote:
>>>>> On 10/07/13 21:59, Allan Jude wrote:
>>>>>> Devin Teske and I have been working on a big patch to bsdinstall to
>>>>>> implement installing on a ZFS pool. It supports both GPT and MBR, the 4k
>>>>>> sector gnop trick, and optional GELI encryption. We would like to commit
>>>>>> this in time for 10.0-BETA1 so it needs some testing to work out any
>>>>>> obvious bugs before we send it off to re@ to get it committed.
>>>>>> 
>>>>>> It includes a single configuration menu that allows you to select all of
>>>>>> the required details, including which drives to use (gets details from
>>>>>> camcontrol, also includes an inspection utility that presents the
>>>>>> detailed output of camcontrol inquiry/identify, and gpart show), what
>>>>>> ZFS RAID level to use (taking in to consideration the selected number of
>>>>>> drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
>>>>>> 
>>>>>> 
>>>>>> Additional, it includes some other changes to bsdinstall:
>>>>>> 1. Change the default to the 'non-standard keyboard mapping' prompt to no
>>>>>> 2. Replace the 3 separate dialogs to configure an ipv4 address with just 
>>>>>> 1
>>>>>> 3. Remove the dialog asking if you wish to enable crash dumps, this
>>>>>> feature has been combined into the regular 'services to enable' dialog
>>>>>> and enabled by default
>>>>>> 
>>>>>> 
>>>>>> You can browse the patches here:
>>>>>> http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/
>>>>>> 
>>>>>> I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
>>>>>> available compressed (48 MB) or uncompressed (211 MB):
>>>>>> 
>>>>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz
>>>>>> 
>>>>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso
>>>>>> 
>>>>>> 
>>>>>> We look forward to your feedback
>>>>>> 
>>>>> Thanks for doing this! I had a few comments:
>>>>> 1. ZFS is not bootable on all architectures. Could you adjust that menu
>>>>> item to only display for i386, amd64, and (I think?) sparc64. Use uname
>>>>> -m, not -p, for this.
>>>> I had not considered that, I'll make that change
>>>> 
>>>>> 1a. The script is broken on sparc64 in any case, which uses VTOC8
>>>>> instead of GPT.
>>>> I'll disable sparc64 as well
>>>> 
>>>>> 2. Why are you using camcontrol? That is guaranteed not to work on
>>>>> non-CAM systems. You should use the GEOM ident string if you need an ID.
>>>> The GEOM ident string doesn't do enough to help the user identify which
>>>> drive is which.
>>>> More data is not exposed anywhere that I could find
>>>> 
>>>> What we really need, is dev.ada.0.desc% like we have for network
>>>> interfaces and a slew of other devices. GEOM data is great, but it is
>>>> not exposed in a shell friendly way any place that I could find, other
>>>> than the sysctl with DOT and XML data.
>>> This is one of the reasons the partition editor is written in C. There
>>> are a few other odd corner-cases where C is much more powerful than the
>>> command-line for the GEOM operations that partedit needs to do. I'm not
>>> sure how to usefully get it just from the shell. You can see how to do
>>> it in C in the boot_disk() routine of partedit/auto_part.c.
>>> 
>>>>> 3. Any plans to integrate this into the regular partition editor? ZFS
>>>>> support is important enough that I will definitely not get in the way,
>>>>> even as a bolt-on, but it would be a shame for it to stay that way. The
>>>>> editor is also designed for ZFS to be added.
>>>> I am a sysadmin, not a programmer. I can't write C. Most people
>>>> deploying servers can't write C. I agree with Devin Teske, if everyt

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 9, 2013, at 8:11 PM, Allan Jude wrote:

> On 2013-10-09 14:14, Teske, Devin wrote:
>> On Oct 9, 2013, at 10:47 AM, Allan Jude wrote:
>> 
>>> On 2013-10-09 13:21, Teske, Devin wrote:
>>>> On Oct 8, 2013, at 8:49 PM, Allan Jude wrote:
>>>> 
>>>>> On 2013-10-07 15:59, Allan Jude wrote:
>>>>>> Devin Teske and I have been working on a big patch to bsdinstall to
>>>>>> implement installing on a ZFS pool. It supports both GPT and MBR, the 4k
>>>>>> sector gnop trick, and optional GELI encryption. We would like to commit
>>>>>> this in time for 10.0-BETA1 so it needs some testing to work out any
>>>>>> obvious bugs before we send it off to re@ to get it committed.
>>>>>> 
>>>>>> It includes a single configuration menu that allows you to select all of
>>>>>> the required details, including which drives to use (gets details from
>>>>>> camcontrol, also includes an inspection utility that presents the
>>>>>> detailed output of camcontrol inquiry/identify, and gpart show), what
>>>>>> ZFS RAID level to use (taking in to consideration the selected number of
>>>>>> drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
>>>>>> 
>>>>>> 
>>>>>> Additional, it includes some other changes to bsdinstall:
>>>>>> 1. Change the default to the 'non-standard keyboard mapping' prompt to no
>>>>>> 2. Replace the 3 separate dialogs to configure an ipv4 address with just 
>>>>>> 1
>>>>>> 3. Remove the dialog asking if you wish to enable crash dumps, this
>>>>>> feature has been combined into the regular 'services to enable' dialog
>>>>>> and enabled by default
>>>>>> 
>>>>>> 
>>>>>> You can browse the patches here:
>>>>>> http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/
>>>>>> 
>>>>>> I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
>>>>>> available compressed (48 MB) or uncompressed (211 MB):
>>>>>> 
>>>>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz
>>>>>> 
>>>>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso
>>>>>> 
>>>>>> 
>>>>>> We look forward to your feedback
>>>>>> 
>>>>> We've made more improvements, including corporating most all of the
>>>>> feedback we've gotten so far
>>>>> 
>>>>> 
>>>>> Outstanding items:
>>>>> 1. Apply the changes to ipv6 config the way we did ipv4
>>>>> 2. improve disk identification (model info and serial # instead of one
>>>>> or the other)
>>>>> 3. Include a helpful message before the GELI step where you have to
>>>>> enter your password many times, the user will be less confused if it is
>>>>> explained why they have to enter their password 3 * number of disks times
>>>> I'm hopeful that we can script the application of a password that we
>>>> first prompt for.
>>>> 
>>>> What tool is prompting for a password? Can we not just provide an answer
>>>> on stdin? (e.g., echo "$pass" | tool_that_needs_pass)
>>>> 
>>> It is 'geli create' and 'geli attach'. I am not sure if we want to have
>>> the password show up in the process list (obviously in the installer
>>> this is less of an issue, but)
>>> 
>> It won't.
>> 
>> echo is a shell built-in.
>> 
>> If we're uber paranoid... we prefix the word "builtin"; e.g.,...
>> 
>> builtin echo "$pass" | tool_that_needs_pass
>> 
>> You'll see the tool in ps, but you won't see the echo (that's part of the
>> shell invocation -- all you see in ps is an instance of sh).
>> 
> I have implemented this, 'geli init' takes the -J  flag, which can
> be set to - to mean stdin. 'geli attach' is the same except the flag is -j
> 
> I first prompt for the password with a dialog --passwordbox, so the user
> only has to enter it once
> 
> Do we think it makes sense to prompt the user twice, and confirm the two
> entered passphrases are the same?
> 

I think so. I'm afraid someone could make a typo and then not be able to
log 

Re: rcs

[Teske, Devin]

On Oct 9, 2013, at 2:48 PM, Adrian Chadd wrote:

> On 9 October 2013 11:28, Bruce Cran  wrote:
> 
>> On 10/9/2013 3:17 PM, Alfred Perlstein wrote:
>> 
>>> Only a few years ago you could take a dvd or memstick of FreeBSD and have
>>> 1000s of packages to choose from during your install.  That is broken now?
>>> 
>> 
>> At some point it was decided that the installer should be as simple as
>> possible and package installation was a post-install task, for once the
>> system was up and running. I think that might be changing with bsdinstall
>> getting pkgng support, but it did seem like a fairly major regression at
>> the time.
> 

That wasn't the rationale. sysinstall was broken and more people wanted a
replacement than wanted to see it fixed.

bsdinstall is simply coming of age still (sysinstall had a 15 year run).


> 
> I think it was more a "this is a stepping stone to a bigger change.." ..
> and that didn't happen soon after.
> 

Adrian is right. It was a "first step" (nobody intended to see functionality
get dropped -- it's a matter of finding the time to replace a 15-year code
line -- sysinstall).

Only a very small handful of us are actively working on it. It's further along
than you think. You can always help out by testing.



> The other thing here is the introduction of the uhm, "livefs". It's big and
> it took up the space once occupied by the first set of packages. Honestly,
> I'm kinda surprised how big base is these days, but I have other fights to
> fight right now.
> 

The livefs is a bee in my bonnet too. But as you say... more pressing things
to worry about right now than break something that is working currently.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 9, 2013, at 10:47 AM, Allan Jude wrote:

> On 2013-10-09 13:21, Teske, Devin wrote:
>> On Oct 8, 2013, at 8:49 PM, Allan Jude wrote:
>> 
>>> On 2013-10-07 15:59, Allan Jude wrote:
>>>> Devin Teske and I have been working on a big patch to bsdinstall to
>>>> implement installing on a ZFS pool. It supports both GPT and MBR, the 4k
>>>> sector gnop trick, and optional GELI encryption. We would like to commit
>>>> this in time for 10.0-BETA1 so it needs some testing to work out any
>>>> obvious bugs before we send it off to re@ to get it committed.
>>>> 
>>>> It includes a single configuration menu that allows you to select all of
>>>> the required details, including which drives to use (gets details from
>>>> camcontrol, also includes an inspection utility that presents the
>>>> detailed output of camcontrol inquiry/identify, and gpart show), what
>>>> ZFS RAID level to use (taking in to consideration the selected number of
>>>> drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
>>>> 
>>>> 
>>>> Additional, it includes some other changes to bsdinstall:
>>>> 1. Change the default to the 'non-standard keyboard mapping' prompt to no
>>>> 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1
>>>> 3. Remove the dialog asking if you wish to enable crash dumps, this
>>>> feature has been combined into the regular 'services to enable' dialog
>>>> and enabled by default
>>>> 
>>>> 
>>>> You can browse the patches here:
>>>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=oPEdrpFUFqEutq%2B7DHGyBIA0gTt5%2BZ6FWvbN6q4bjm4%3D%0A&s=32f69a33d9c98c63ecf3962b89eea0af631f5a2fd6e82b55afb6ec139f3f803d
>>>> 
>>>> I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
>>>> available compressed (48 MB) or uncompressed (211 MB):
>>>> 
>>>> https://urldefense.proofpoint.com/v1/url?u=http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=oPEdrpFUFqEutq%2B7DHGyBIA0gTt5%2BZ6FWvbN6q4bjm4%3D%0A&s=fd0f3ed623f12c8cb5eee607cf82122523a0800ab9f5624cb3f77ae2646e1b1b
>>>> 
>>>> https://urldefense.proofpoint.com/v1/url?u=http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=oPEdrpFUFqEutq%2B7DHGyBIA0gTt5%2BZ6FWvbN6q4bjm4%3D%0A&s=fa49aaa528513a591ac40a2f0c5666abae2d7269bd66ac70d7100475a9574300
>>>> 
>>>> 
>>>> We look forward to your feedback
>>>> 
>>> We've made more improvements, including corporating most all of the
>>> feedback we've gotten so far
>>> 
>>> 
>>> Outstanding items:
>>> 1. Apply the changes to ipv6 config the way we did ipv4
>>> 2. improve disk identification (model info and serial # instead of one
>>> or the other)
>>> 3. Include a helpful message before the GELI step where you have to
>>> enter your password many times, the user will be less confused if it is
>>> explained why they have to enter their password 3 * number of disks times
>> I'm hopeful that we can script the application of a password that we
>> first prompt for.
>> 
>> What tool is prompting for a password? Can we not just provide an answer
>> on stdin? (e.g., echo "$pass" | tool_that_needs_pass)
>> 
> It is 'geli create' and 'geli attach'. I am not sure if we want to have
> the password show up in the process list (obviously in the installer
> this is less of an issue, but)
> 

It won't.

echo is a shell built-in.

If we're uber paranoid... we prefix the word "builtin"; e.g.,...

builtin echo "$pass" | tool_that_needs_pass

You'll see the tool in ps, but you won't see the echo (that's part of the
shell invocation -- all you see in ps is an instance of sh).



>> 
>>> 4. Validate vdev type choice inside the vdev type menu, and warn the
>>> user if they have made an invalid selection, so they can add more disks
>>> or chance their selection, without having to try to start the
>>> installation first
>> This will be done with fanciness ;D (read: ... --and-widget --infobox ... and
>> sundry s

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 8, 2013, at 8:49 PM, Allan Jude wrote:

> On 2013-10-07 15:59, Allan Jude wrote:
>> Devin Teske and I have been working on a big patch to bsdinstall to
>> implement installing on a ZFS pool. It supports both GPT and MBR, the 4k
>> sector gnop trick, and optional GELI encryption. We would like to commit
>> this in time for 10.0-BETA1 so it needs some testing to work out any
>> obvious bugs before we send it off to re@ to get it committed.
>> 
>> It includes a single configuration menu that allows you to select all of
>> the required details, including which drives to use (gets details from
>> camcontrol, also includes an inspection utility that presents the
>> detailed output of camcontrol inquiry/identify, and gpart show), what
>> ZFS RAID level to use (taking in to consideration the selected number of
>> drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
>> 
>> 
>> Additional, it includes some other changes to bsdinstall:
>> 1. Change the default to the 'non-standard keyboard mapping' prompt to no
>> 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1
>> 3. Remove the dialog asking if you wish to enable crash dumps, this
>> feature has been combined into the regular 'services to enable' dialog
>> and enabled by default
>> 
>> 
>> You can browse the patches here:
>> http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/
>> 
>> I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
>> available compressed (48 MB) or uncompressed (211 MB):
>> 
>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz
>> 
>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso
>> 
>> 
>> We look forward to your feedback
>> 
> 
> We've made more improvements, including corporating most all of the
> feedback we've gotten so far
> 
> 
> Outstanding items:
> 1. Apply the changes to ipv6 config the way we did ipv4
> 2. improve disk identification (model info and serial # instead of one
> or the other)
> 3. Include a helpful message before the GELI step where you have to
> enter your password many times, the user will be less confused if it is
> explained why they have to enter their password 3 * number of disks times

I'm hopeful that we can script the application of a password that we
first prompt for.

What tool is prompting for a password? Can we not just provide an answer
on stdin? (e.g., echo "$pass" | tool_that_needs_pass)



> 4. Validate vdev type choice inside the vdev type menu, and warn the
> user if they have made an invalid selection, so they can add more disks
> or chance their selection, without having to try to start the
> installation first

This will be done with fanciness ;D (read: ... --and-widget --infobox ... and
sundry smartness; retaining as much as possible the ability to do things
out of order but never arise at a point of astonishment).


> 5. Whatever else you guys find wrong tonight
> 
> I generated new test images, and attached the patch (which got REALLY
> big when Devin Teske decided to fix "all of the things":
> 

And then I merged "all of the things" into HEAD, so the patch-set shrunk
back to its normal size. Now we have global exit codes which will make
merging of code that is based off of Thomas Dickey's samples easier.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

BE Loader Menu (was Re: rcs)

[Teske, Devin]

On Oct 9, 2013, at 9:42 AM, Kris Moore wrote:

> On 10/09/2013 12:33, Allan Jude wrote:
>> On 2013-10-09 12:23, Freddie Cash wrote:
>>> On Wed, Oct 9, 2013 at 9:12 AM, Freddie Cash  wrote:
>>> 
 On Wed, Oct 9, 2013 at 6:41 AM, Julian Elischer wrote:
 
> On 10/9/13 2:35 AM, Freddie Cash wrote:
> 
>> On Tue, Oct 8, 2013 at 9:09 AM, Alfred Perlstein  wrote:
>> 
>> You're right on the money, to be honest this is one of the reasons why
>>> I've switched to using OSX as my desktop OS.
>>> 
>>> zsh, vim, screen by default.  and upgrades work.  At the end of the day
>>> I'm spending time doing work, not mucking about my workspace to make it
>>> usable for development.
>>> 
>>> I think this was brought up at BSDCan in the discussion about making
>>> FreeBSD a more featured development platform.
>>> 
>>> Speaking of... has anyone tried PCBSD?
>>> 
>> PC-BSD isn't much different from FreeBSD.  The installer is GUI and
>> support
>> ZFS, there are some GUI setup tools on first boot for X, there are some
>> GUI
>> tools to select binary drivers for X, and there ​​are working pkgng repos
>> available.
>> 
>> I had a lot of issues with PC-BSD 9.0 and 9.1 as I was trying to do
>> things
>> "the FreeBSD way" which broke a lot of things that were done "the PC-BSD
>> way" (aka don't manually edit config files used for booting).
>> 
>> ​Switching to the "rolling-release" (aka PC-BSD 9-STABLE) and moving all
>> my
>> config file edits into .conf.local fixed my issues.  Things
>> have
>> been running smooth, and I finally understand the beauty and simplicity
>> of
>> freebsd-update + pkg.  OS gets updated once per month, packages get
>> updated
>> twice per month, no more compiling things from source.  It's like using
>> Ubuntu/Debian but with the power and features of FreeBSD.  :)
>> ​
>> 
> When they went to a ZFS-only system, using GRUB, with no alternative,
> then I'm afraid they lost me.
> I want a root filesystem on UFS for reliabailty and simpleness.  I can
> debug it's media if needed.
> Before then I really liked it (though ther eis not enough information on
> how it works interneally if you want to use it.
> hopefully that will come.. and I LIKE PBIs  FreeBSD should adopt PBIs for
> sure.
> With PBIs you could make even quite base items separately installable.
> versioning problems go away.
> 
 There's no GRUB in a default install of PC-BSD 9.0, 9.1, or 9.2.  Even on
 a ZFS-only setup (which is what I run).  It's using the FreeBSD loader,
 with custom artwork and menus.
 
>>> ​Hrm, it seems they've changed things with the 9.2 installer.  It does use
>>> GRUB2 (e!) for the boot loader, and integrates support for ZFS boot
>>> environments (via beadm) into it.  :(  Shame they didn't use the BE support
>>> in the FreeBSD loader for this.  Wonder if my 9-STABLE-based PC-BSD install
>>> will get "upgraded" to GRUB?
>>> 
>> The reason they went to grub2, is that the way the freebsd loader menus
>> work, it loads the kernel before it draws the menu. This means if there
>> is a problem with your kernel (probably the most valuable time to have
>> boot environments) then the menu never comes up, and you cannot select
>> which BE to boot from. Grub doesn't rely on a FreeBSD kernel until after
>> you select which BE to boot from.
>> 
>> Kris and I discussed it at length with Devin Teske, and while he has
>> demonstrated being able to populate a lower menu with the ZFS datasets,
>> I am not sure if the other issue can be resolved.
>> 
> 

I'm late to the party again ;D (didn't realize the rcs thread had turned BE)

Both problems can be solved.
The loading of the kernel *after* choosing your boot device is trivial.
We've been doing it at $work for *years* (almost a decade?)

I can put that in, whenever. Probably at the same time as implementing
the live/dynamic BE menus for selecting the root device.



> Yea, GRUB is not my first choice, but ATM this is the "cleanest" way we
> can do ZFS BE's. However if you don't like ZFS / GRUB, you can always
> use regular FreeBSD and just grab our toolchain from
> sysutils/pcbsd-utils* in ports or use our PKGNG repo:
> 
> http://wiki.pcbsd.org/index.php/Turn_FreeBSD_into_PC-BSD%C2%AE
> 
> I'm planning on using GRUB to do UEFI booting for 10.0 as well. But when
> the FreeBSD loader matures to the point of having support for all these
> features, I'll gladly move us back.
> 

I had not been pushing hard on finishing the BE Forth because I knew that
it would be a greater value add if we actually had "Boot from ZFS" finished
first. Would be easier to test, that is, if I actually had a system booting from
ZFS -- which is what Allan, jmg, and myself are hopefully working toward
for FreeBSD 10.0-R. (smiles)
-- 
Devin

_
The information contained in this message

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 8, 2013, at 11:19 PM, Nathan Whitehorn wrote:

> On 10/09/13 01:13, Allan Jude wrote:
>> On 2013-10-08 16:17, Nathan Whitehorn wrote:
>>> On 10/07/13 21:59, Allan Jude wrote:
 Devin Teske and I have been working on a big patch to bsdinstall to
 implement installing on a ZFS pool. It supports both GPT and MBR, the 4k
 sector gnop trick, and optional GELI encryption. We would like to commit
 this in time for 10.0-BETA1 so it needs some testing to work out any
 obvious bugs before we send it off to re@ to get it committed.
 
 It includes a single configuration menu that allows you to select all of
 the required details, including which drives to use (gets details from
 camcontrol, also includes an inspection utility that presents the
 detailed output of camcontrol inquiry/identify, and gpart show), what
 ZFS RAID level to use (taking in to consideration the selected number of
 drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
 
 
 Additional, it includes some other changes to bsdinstall:
 1. Change the default to the 'non-standard keyboard mapping' prompt to no
 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1
 3. Remove the dialog asking if you wish to enable crash dumps, this
 feature has been combined into the regular 'services to enable' dialog
 and enabled by default
 
 
 You can browse the patches here:
 http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/
 
 I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
 available compressed (48 MB) or uncompressed (211 MB):
 
 http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz
 
 http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso
 
 
 We look forward to your feedback
 
>>> Thanks for doing this! I had a few comments:
>>> 1. ZFS is not bootable on all architectures. Could you adjust that menu
>>> item to only display for i386, amd64, and (I think?) sparc64. Use uname
>>> -m, not -p, for this.
>> I had not considered that, I'll make that change
>> 
>>> 1a. The script is broken on sparc64 in any case, which uses VTOC8
>>> instead of GPT.
>> I'll disable sparc64 as well
>> 
>>> 2. Why are you using camcontrol? That is guaranteed not to work on
>>> non-CAM systems. You should use the GEOM ident string if you need an ID.
>> The GEOM ident string doesn't do enough to help the user identify which
>> drive is which.
>> More data is not exposed anywhere that I could find
>> 
>> What we really need, is dev.ada.0.desc% like we have for network
>> interfaces and a slew of other devices. GEOM data is great, but it is
>> not exposed in a shell friendly way any place that I could find, other
>> than the sysctl with DOT and XML data.
> 
> This is one of the reasons the partition editor is written in C. There
> are a few other odd corner-cases where C is much more powerful than the
> command-line for the GEOM operations that partedit needs to do. I'm not
> sure how to usefully get it just from the shell. You can see how to do
> it in C in the boot_disk() routine of partedit/auto_part.c.
> 
>>> 3. Any plans to integrate this into the regular partition editor? ZFS
>>> support is important enough that I will definitely not get in the way,
>>> even as a bolt-on, but it would be a shame for it to stay that way. The
>>> editor is also designed for ZFS to be added.
>> I am a sysadmin, not a programmer. I can't write C. Most people
>> deploying servers can't write C. I agree with Devin Teske, if everything
>> was in shell it would be a lot more usable for non-developers, who
>> probably make up the majority of people who deploy FreeBSD.
> 
> There are some cases the other way too. Devin is probably the most
> shell-proficient FreeBSD committer.

Well, there's jilles ;D (he writes/maintains the sh(1) implementation itself).


> I certainly can't write shell
> scripts at that level, which means that for me bsdconfig, for example,
> is effectively read-only (and quite hard to read as well).

In the past few days of working on the bsdinstall_zfs patch-set with Allan
Jude, I learned something new actually.

It seems that sh(1) doesn't suffer so much from this "read only" concept.

Imagine you're starting a new C project on an operating system that has
all new syscalls and all new APIs that you've never seen. Would be pretty
hard, no doubt. Now imagine that you're thrown a life-line called POSIX
and hey, now you're slinging code in MinGW on Windows when you don't
know a lick of M$ syscalls or APIs.

In a similar manner, I've witnessed functionality be added that is truly
functional *without* using any of the API calls. Then I come in and do
a round of optimizations to leverage the existing API.

Shell is kinda like that...

I'm noticing Allan Jude doesn't know all the API calls yet (who could? other
than me of course -- and even I have trouble remembering them _all_) yet
this doesn't 

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 9, 2013, at 9:05 AM, Allan Jude wrote:

> On 2013-10-09 03:22, Matthias Gamsjager wrote:
>> Hi,
>> 
>> tried 10-8 iso in Virtualbox but after reboot I was looking at a
>> bootloader with a nice '-' but nothing more. 
>> post setup 'gpart list' showed no entries at all.
>> 
>> 
>> 
>> 
>> On Wed, Oct 9, 2013 at 5:49 AM, Allan Jude > > wrote:
>> 
>>On 2013-10-07 15:59, Allan Jude wrote:
>>> Devin Teske and I have been working on a big patch to bsdinstall to
>>> implement installing on a ZFS pool. It supports both GPT and
>>MBR, the 4k
>>> sector gnop trick, and optional GELI encryption. We would like
>>to commit
>>> this in time for 10.0-BETA1 so it needs some testing to work out any
>>> obvious bugs before we send it off to re@ to get it committed.
>>> 
>>> It includes a single configuration menu that allows you to
>>select all of
>>> the required details, including which drives to use (gets
>>details from
>>> camcontrol, also includes an inspection utility that presents the
>>> detailed output of camcontrol inquiry/identify, and gpart show),
>>what
>>> ZFS RAID level to use (taking in to consideration the selected
>>number of
>>> drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
>>> 
>>> 
>>> Additional, it includes some other changes to bsdinstall:
>>> 1. Change the default to the 'non-standard keyboard mapping'
>>prompt to no
>>> 2. Replace the 3 separate dialogs to configure an ipv4 address
>>with just 1
>>> 3. Remove the dialog asking if you wish to enable crash dumps, this
>>> feature has been combined into the regular 'services to enable'
>>dialog
>>> and enabled by default
>>> 
>>> 
>>> You can browse the patches here:
>>> http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/
>>> 
>>> I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
>>> available compressed (48 MB) or uncompressed (211 MB):
>>> 
>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz
>>> 
>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso
>>> 
>>> 
>>> We look forward to your feedback
>>> 
>> 
>>We've made more improvements, including corporating most all of the
>>feedback we've gotten so far
>> 
>> 
>>Outstanding items:
>>1. Apply the changes to ipv6 config the way we did ipv4
>>2. improve disk identification (model info and serial # instead of one
>>or the other)
>>3. Include a helpful message before the GELI step where you have to
>>enter your password many times, the user will be less confused if
>>it is
>>explained why they have to enter their password 3 * number of
>>disks times
>>4. Validate vdev type choice inside the vdev type menu, and warn the
>>user if they have made an invalid selection, so they can add more
>>disks
>>or chance their selection, without having to try to start the
>>installation first
>>5. Whatever else you guys find wrong tonight
>> 
>>I generated new test images, and attached the patch (which got REALLY
>>big when Devin Teske decided to fix "all of the things":
>> 
>>http://www.allanjude.com/bsd/zfsbootonly.2013-10-08.iso.xz
>> 
>>http://www.allanjude.com/bsd/zfsbootonly.2013-10-08.iso
>> 
>> 
>>--
>>Allan Jude
>> 
>> 
>>___
>>freebsd-current@freebsd.org 
>>mailing list
>>http://lists.freebsd.org/mailman/listinfo/freebsd-current
>>To unsubscribe, send any mail to
>>"freebsd-current-unsubscr...@freebsd.org
>>"
>> 
>> 
> This was a variable scoping error made by Devin when he refactored some
> of the code into a subroutine. I have fixed it
> 

Ouch and apologies.

It was too hard to see at 2AM after starting-in on a deeper pathos to the keymap
patches (which I'm still working on). Allan alerted me to the issue but when I 
started
digging-in, I realized much deeper changes were required. And then I passed out
around 2:30AM (keyboard-faceplant).
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 9, 2013, at 8:29 AM, Warren Block wrote:

> On Tue, 8 Oct 2013, Teske, Devin wrote:
> 
>> "But shell is nasty; slow; and not as powerful as C" (it depends in what
>> context; the first is rhetoric, the second is only true for poor implement-
>> ations, and the third may be true in some contexts, but I consider the
>> answer to "how maintainable is it" to be a factor in the "power" of a
>> language, so don't necessarily consider C to be more powerful than
>> shell as the latter is as-or-more maintainable with fewer LoC and a
>> higher return on investment; see previous [above] arguments).
> 
> My question would be: why are sh and C the only choices?  If the answer is 
> "because that's all we have in base", is that a valid concern?
> 
> As far as sh, it lacks many high- or even mid-level constructs and has real 
> problems with quoting,

Enter helper functions...

dte...@scribe9.vicor.com bsdconfig $ grep '^# f_' 
/usr/share/bsdconfig/strings.subr 
# f_substr "$string" $start [ $length ]
# f_snprintf $var_to_set $size $format ...
# f_vsnprintf $var_to_set $size $format $format_args
# f_longest_line_length
# f_number_of_lines
# f_isinteger $arg
# f_uriencode [$text]
# f_uridecode [$text]
# f_replaceall $string $find $replace [$var_to_set]
# f_str2varname $string [$var_to_set]
# f_shell_escape $string [$var_to_set]
# f_shell_unescape $string [$var_to_set]
# f_expand_number $string [$var_to_set]

The ones dealing specifically with quoting are f_shell_escape() and 
f_shell_unescape().
With those two helper functions, I consider quoting in sh(1) to be complete and 
maleable
to handle all situations.



> parsing,

Surely, a strength. It can take a while to engrain or impart all the expansions 
and at-
what stage (and in what context) they are performed, but when you understand the
mechanics of (for example) in what order each element (escaped-sequences, 
params,
newlines, and sub-shells) is expanded... the parsing becomes an asset, not a
hindrance (imho). Not to mention the flexibility afforded by "compound strings".

Take for example, the following syntax (leveraging ordered expansion and 
compound
strings):

f_dialog_input bar \
'Enter *any* text; including globs [*] and shell special chars 
[$`#'"']..."
f_dialog_input baz \
"Enter some *more* non-conforming text please..."
f_shell_escape "$bar" bar
f_shell_escape "$baz" baz
items="'$bar' '$baz'"
show="You entered:\n1=%s\n2=%s\n"
eval printf \"\$show\" $items

I'd be surprised if you could get anything out but the values you enter.



> and output (2>&1 >&3, for example).

I think it helps to highlight a technique for solving the general-purpose issue 
of
how you launch a program and have it talk to a parent's maintained 
file-descriptor.

In C, you'd have to pass special args to posix_spawn(3) to get this 
functionality.
Talking about succinctness of syntax... that sequence is likely much shorter 
than
any syntax I've ever seen in C for setting up file descriptors. I've interfaced 
with
dialog(1) from within C and getting the data back from it is non-trivial.

ASIDE: You may be asking yourself... why on Earth would you fork-exec an
instance of dialog(1) from within C instead of using dialog(3)? There are some
rare cases when this is advantageous. Usually when you want to support a drop-
in replacement (like Xdialog(1)) for a long-running stateful widget (e.g. 
gauge).
Usually (>=99% of the time) a C program _should_ use dialog(3) but in those
rare cases ( see http://druidbsd.cvs.sf.net/viewvc/druidbsd/fdpv/ ) thank 
goodness
I haven't had to mangle the file descriptors to get back any data (I call out 
to gauge
and just pump it massive amounts of data via stdin through a pipe passed to the
posix_spawn(3) call -- no data to read back.

So I don't see the above syntax as anything to point out -- C programs have more
complicated setup for doing the same thing (they just have to do it less often).

Oh... and in bsdconfig, you'll notice I don't do that. Upon initialization I 
clone the
*real* terminal fd's _once_ and then use variables to reference the dup'd fd.


> These make it harder to do things (aka, more code to accomplish a task, more 
> code to be maintained, more difficult to modify) than the higher level 
> Perubython languages.
> 

Only because those languages have APIs that handle those things.

Which I might add that those APIs have to be maintained. In that spirit,
bsdconfig(8) provides the API for shell -- and similarly, said API (like most
APIs) will be need much less maintenance than the programs using said
API (how often do we make changes to posix_spawn(3)? or how often
does Per

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 8, 2013, at 2:28 PM, Nathan Whitehorn wrote:

> On 10/08/13 23:21, Kurt Lidl wrote:
>> 
>> On 10/8/2013, Nathan Whitehorn wrote:
>>> On 10/07/13 21:59, Allan Jude wrote:
 Devin Teske and I have been working on a big patch to bsdinstall to
 implement installing on a ZFS pool. It supports both GPT and MBR,
 the 4k
 sector gnop trick, and optional GELI encryption. We would like to
 commit
 this in time for 10.0-BETA1 so it needs some testing to work out any
 obvious bugs before we send it off to re@ to get it committed.
 
 It includes a single configuration menu that allows you to select
 all of
 the required details, including which drives to use (gets details from
 camcontrol, also includes an inspection utility that presents the
 detailed output of camcontrol inquiry/identify, and gpart show), what
 ZFS RAID level to use (taking in to consideration the selected
 number of
 drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
 
 
 Additional, it includes some other changes to bsdinstall:
 1. Change the default to the 'non-standard keyboard mapping' prompt
 to no
 2. Replace the 3 separate dialogs to configure an ipv4 address with
 just 1
 3. Remove the dialog asking if you wish to enable crash dumps, this
 feature has been combined into the regular 'services to enable' dialog
 and enabled by default
 
 
 You can browse the patches here:
 http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/
 
 I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
 available compressed (48 MB) or uncompressed (211 MB):
 
 http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz
 
 http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso
 
 
 We look forward to your feedback
 
>>> 
>>> Thanks for doing this! I had a few comments:
>>> 1. ZFS is not bootable on all architectures. Could you adjust that menu
>>> item to only display for i386, amd64, and (I think?) sparc64. Use uname
>>> -m, not -p, for this.
>>> 1a. The script is broken on sparc64 in any case, which uses VTOC8
>>> instead of GPT.
>>> 2. Why are you using camcontrol? That is guaranteed not to work on
>>> non-CAM systems. You should use the GEOM ident string if you need an ID.
>>> 3. Any plans to integrate this into the regular partition editor? ZFS
>>> support is important enough that I will definitely not get in the way,
>>> even as a bolt-on, but it would be a shame for it to stay that way. The
>>> editor is also designed for ZFS to be added.
>>> 4. What is this gnop stuff for?
>>> 5. I think some substantial part of the MBR code will blow up if you are
>>> reinitalizing a previously formatted disk (the bsdlabel will be retasted
>>> and come back from the dead).
>>> -Nathan
>> 
>> I wrote some support for adding ZFS to the partition editor a couple of
>> months ago, around the time that 9.2 was branched.  One of the biggest
>> things that I have not done is integrate setting up a zfs mirror between
>> any disks before creating the zpool.  Nor did I do the gnop hack to
>> create 4K disk blocks before creating the pool.
>> 
>> I more or less changed the partedit program to take an argument when
>> invoked with "ufs" (same as current behavior) or "zfs", which knows
>> about zfs setup stuff.  The hookup to the actual scripts was, shall
>> we say, much less intrusive than what has been published here.
>> 
>> I guess it's time to dig out the patches and make them available to
>> others.
> 
> That would be great! The original idea was to have a "zfs_ops.c" to go
> with "gpart_ops.c" and have some fields in the disk item struct that
> said which to use. Not sure if that's ultimately workable, though. I can
> hack on it once the patches are there in any case.

One of the options presented was to integrate ZFS knowledge into the
partedit module.

However, another approach that is not being spoken about is in the
other direction: rewriting partedit to be in sh(1).

What I'd like to see is the --treeview widget of dialog(1) "fixed" before
such a transition is made, so the former option may be the best bet for
now. For anyone that hasn't run the treeview or treeview2 samples in
head/contrib/dialog/samples ... it's not anything we'd want to use to
replace dialog(3) with. I've seen much nicer treeview implementations
than what is in [c]dialog (currently we're using dialog-1.2-20130925 and
the treeview for some reason incorporates a radio-button column which
we do _not_ want for a disk partition layout menu).

Of course, right now you're all looking at me like I'm crazy.

Like a fox.

There are real tangible benefits to rewriting the C code of partedit into
shell syntax. The value-add is not just perceived. Some of the values
obtained are:

+ Conflated namespace with singular API rather than having to learn
how to script each individual blocking-utility which implements a unique
method fo

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 8, 2013, at 1:17 PM, Nathan Whitehorn wrote:

> On 10/07/13 21:59, Allan Jude wrote:
>> Devin Teske and I have been working on a big patch to bsdinstall to
>> implement installing on a ZFS pool. It supports both GPT and MBR, the 4k
>> sector gnop trick, and optional GELI encryption. We would like to commit
>> this in time for 10.0-BETA1 so it needs some testing to work out any
>> obvious bugs before we send it off to re@ to get it committed.
>> 
>> It includes a single configuration menu that allows you to select all of
>> the required details, including which drives to use (gets details from
>> camcontrol, also includes an inspection utility that presents the
>> detailed output of camcontrol inquiry/identify, and gpart show), what
>> ZFS RAID level to use (taking in to consideration the selected number of
>> drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
>> 
>> 
>> Additional, it includes some other changes to bsdinstall:
>> 1. Change the default to the 'non-standard keyboard mapping' prompt to no
>> 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1
>> 3. Remove the dialog asking if you wish to enable crash dumps, this
>> feature has been combined into the regular 'services to enable' dialog
>> and enabled by default
>> 
>> 
>> You can browse the patches here:
>> http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/
>> 
>> I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
>> available compressed (48 MB) or uncompressed (211 MB):
>> 
>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz
>> 
>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso
>> 
>> 
>> We look forward to your feedback
>> 
> 
> Thanks for doing this! I had a few comments:
> 1. ZFS is not bootable on all architectures. Could you adjust that menu
> item to only display for i386, amd64, and (I think?) sparc64. Use uname
> -m, not -p, for this.

I think we can do that no problem.


> 1a. The script is broken on sparc64 in any case, which uses VTOC8
> instead of GPT.

*nods* we'll have to purloin your VTOC8 code before we offer it to
sparc64.



> 2. Why are you using camcontrol? That is guaranteed not to work on
> non-CAM systems. You should use the GEOM ident string if you need an ID.

I imagine we could get the info from many places, and to be honest, I never
knew about "diskinfo" until last week (but apparently it's been around since
5.x days).

The one place where I suspect camcontrol(8) usage is appropriate is the
mini "disk inspector" dialog. The camcontrol inquiry output is specifically
helpful if/when you're doing multipathing (and you need to identify which
da# devices are duplicates of the same device -- given Serial#).

But I gather you mean for "disk description" in the device menu (for picking
text to go along with the device name).

I'm open to using a different tool... or multiple tools. Do you think it would 
be
better to just stick to one tool? or to query a few? (which ones win-out?)




> 3. Any plans to integrate this into the regular partition editor? ZFS
> support is important enough that I will definitely not get in the way,
> even as a bolt-on, but it would be a shame for it to stay that way. The
> editor is also designed for ZFS to be added.

Yes and yes. (and didn't know the editor was designed for ZFS too)


> 4. What is this gnop stuff for?

Thanks goes to Freddie Cash for his excellent explanation (I had to
admit, I didn't understand it at that level)


> 5. I think some substantial part of the MBR code will blow up if you are
> reinitalizing a previously formatted disk (the bsdlabel will be retasted
> and come back from the dead).

Will not some combination of "gpart destroy -F" and (insert suggestion) work?
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 7, 2013, at 10:48 PM, Allan Jude wrote:

> On 2013-10-08 01:11, Teske, Devin wrote:
>> On Oct 7, 2013, at 10:07 PM, Allan Jude wrote:
>> 
>>> On 2013-10-07 15:59, Allan Jude wrote:
>>>> Devin Teske and I have been working on a big patch to bsdinstall to
>>>> implement installing on a ZFS pool. It supports both GPT and MBR, the 4k
>>>> sector gnop trick, and optional GELI encryption. We would like to commit
>>>> this in time for 10.0-BETA1 so it needs some testing to work out any
>>>> obvious bugs before we send it off to re@ to get it committed.
>>>> 
>>>> It includes a single configuration menu that allows you to select all of
>>>> the required details, including which drives to use (gets details from
>>>> camcontrol, also includes an inspection utility that presents the
>>>> detailed output of camcontrol inquiry/identify, and gpart show), what
>>>> ZFS RAID level to use (taking in to consideration the selected number of
>>>> drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
>>>> 
>>>> 
>>>> Additional, it includes some other changes to bsdinstall:
>>>> 1. Change the default to the 'non-standard keyboard mapping' prompt to no
>>>> 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1
>>>> 3. Remove the dialog asking if you wish to enable crash dumps, this
>>>> feature has been combined into the regular 'services to enable' dialog
>>>> and enabled by default
>>>> 
>>>> 
>>>> You can browse the patches here:
>>>> http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/
>>>> 
>>>> I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
>>>> available compressed (48 MB) or uncompressed (211 MB):
>>>> 
>>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz
>>>> 
>>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso
>>>> 
>>>> 
>>>> We look forward to your feedback
>>>> 
>>> I've generated a new version of the ISO incorporating all of todays
>>> changes and moving up to 10.0-ALPHA5
>>> It includes 2 patches from Warren Block, improving the keymap menu and
>>> indicating which network interfaces are wireless
>>> 
>> Excellent.
>> 
>> However, I have to say... you opened a can of worms by touching keymap.
>> 
>> Eventually I plan to rewrite all of the scripts to the same format that 
>> zfsboot
>> is using. I've already started rewriting keymap to the new format. Of course,
>> this opened up another can of worms... the simple things like:
>> 
>> DIALOG_OK
>> DIALOG_CANCEL
>> DIALOG_ESC
> 
> Yeah, after having them, I felt a bit naked not having all of those
> 'constants' defined.
> 

The constants are now where they belong... `dialog.subr'
I also went through bsdconfig(8) with a fine-tooth comb and made use of the
new "constants" everywhere.

I think it's a definite improvement.



> Originally, all I had done was add --defaultno to the dialog command,
> but Warren's patch makes a lot of sense, allow the user to 'try' the new
> keymap before trying to do the rest of the install based on it.
> 

I rewrote warren's code into the stateful design and cleaned it up.



>> I feel would be much better off in the `dialog.subr' module. So,... I've 
>> started
>> peppering their usage everywhere in bsdconfig to make them "proper".
>> 
>> That means they will just be transparent from including `dialog.subr'.
>> 
> 
> This is definately something that I think is a good idea, but my focus
> is on improving the usability and functionality of the installer in time
> for 10.0. Unifying everything to the bsdconfig style is slightly lower
> priority. Admittedly, the `bsdconfig networking` stuff is quite nice
> 

*nods*

I took a look at the bsdinstall networking stuff, and I wanted to run away.

Let's push that into the future.

It wasn't too hard to get those global exit codes deployed, so I did that,
but like you say... I agree we should minimize "extra work" that will be
eventually slated for a future release.
-- 
Devin



>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz
>>> 
>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso
>>> 
>> Excellent, thanks!
> 
> 
> -- 
> Allan Jude
> 
> _

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 7, 2013, at 10:07 PM, Allan Jude wrote:

> On 2013-10-07 15:59, Allan Jude wrote:
>> Devin Teske and I have been working on a big patch to bsdinstall to
>> implement installing on a ZFS pool. It supports both GPT and MBR, the 4k
>> sector gnop trick, and optional GELI encryption. We would like to commit
>> this in time for 10.0-BETA1 so it needs some testing to work out any
>> obvious bugs before we send it off to re@ to get it committed.
>> 
>> It includes a single configuration menu that allows you to select all of
>> the required details, including which drives to use (gets details from
>> camcontrol, also includes an inspection utility that presents the
>> detailed output of camcontrol inquiry/identify, and gpart show), what
>> ZFS RAID level to use (taking in to consideration the selected number of
>> drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
>> 
>> 
>> Additional, it includes some other changes to bsdinstall:
>> 1. Change the default to the 'non-standard keyboard mapping' prompt to no
>> 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1
>> 3. Remove the dialog asking if you wish to enable crash dumps, this
>> feature has been combined into the regular 'services to enable' dialog
>> and enabled by default
>> 
>> 
>> You can browse the patches here:
>> http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/
>> 
>> I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
>> available compressed (48 MB) or uncompressed (211 MB):
>> 
>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz
>> 
>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso
>> 
>> 
>> We look forward to your feedback
>> 
> 
> I've generated a new version of the ISO incorporating all of todays
> changes and moving up to 10.0-ALPHA5
> It includes 2 patches from Warren Block, improving the keymap menu and
> indicating which network interfaces are wireless
> 

Excellent.

However, I have to say... you opened a can of worms by touching keymap.

Eventually I plan to rewrite all of the scripts to the same format that zfsboot
is using. I've already started rewriting keymap to the new format. Of course,
this opened up another can of worms... the simple things like:

DIALOG_OK
DIALOG_CANCEL
DIALOG_ESC

I feel would be much better off in the `dialog.subr' module. So,... I've started
peppering their usage everywhere in bsdconfig to make them "proper".

That means they will just be transparent from including `dialog.subr'.


> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz
> 
> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso
> 

Excellent, thanks!
-- 
Devin

P.S. I have to get apache installed on one of my jails to server you the video 
;)
I'll have a URL for you tomorrow morning. Trying to work through all the changes
in keymap to get it upgraded.

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 7, 2013, at 5:31 PM, Allan Jude wrote:

> On 2013-10-07 20:21, Teske, Devin wrote:
>> On Oct 7, 2013, at 5:18 PM, Warren Block wrote:
>> 
>>> On Mon, 7 Oct 2013, Allan Jude wrote:
>>> 
>>>> On 2013-10-07 16:43, Warren Block wrote:
>>>>> On Mon, 7 Oct 2013, Allan Jude wrote:
>>>>> 
>>>>>> Additional, it includes some other changes to bsdinstall:
>>>>>> 1. Change the default to the 'non-standard keyboard mapping' prompt
>>>>>> to no
>>>>> Please see http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/162175 .
>>>> That is a good idea, I'll add that
>>>> 
>>>> I've also hit this:
>>>> http://lists.freebsd.org/pipermail/freebsd-sysinstall/2013-September/000949.html
>>>> and plan to throw in a fix for that.
>>>> Which makes more sense, blindly 'killall dhclient' before we try to
>>>> acquire a new lease, or detect that dhclient is running and try to use
>>>> the IP that has already been assigned?
>>> killall seems all right.  In fact, internally, it's going to do that check. 
>>>  If nothing named dhclient is running, it will return immediately.
>>> 
>>> Although on 10.0, dhclient does not want to die sometimes.
>>> 
>>> Here's another PR for usability:
>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/161547
>> Hope you guys aren't rewriting bsdconfig's network.subr
> 
> These changes are in usr.sbin/bsdinstall/scripts/network and
> usr.sbin/bsdinstall/scripts/network_ipv4
> 

Correct, but I meant "reinvent" actually.

There's an enormous API for dealing with the network.

/usr/share/bsdconfig/media/tcpip.subr

Is for acquiring/maintaining active TCP connections
Meanwhile,...

/usr/share/bsdconfig/network/*.subr

Is for configuring permanent network settings that stick after a reboot.



> I've not touched bsdconfig's network stuff
> 

Probably worth a look. It can probably replace bsdinstall's netconfig.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 7, 2013, at 5:18 PM, Warren Block wrote:

> On Mon, 7 Oct 2013, Allan Jude wrote:
> 
>> On 2013-10-07 16:43, Warren Block wrote:
>>> On Mon, 7 Oct 2013, Allan Jude wrote:
>>> 
 Additional, it includes some other changes to bsdinstall:
 1. Change the default to the 'non-standard keyboard mapping' prompt
 to no
>>> 
>>> Please see http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/162175 .
> 
>> That is a good idea, I'll add that
>> 
>> I've also hit this:
>> http://lists.freebsd.org/pipermail/freebsd-sysinstall/2013-September/000949.html
>> and plan to throw in a fix for that.
>> Which makes more sense, blindly 'killall dhclient' before we try to
>> acquire a new lease, or detect that dhclient is running and try to use
>> the IP that has already been assigned?
> 
> killall seems all right.  In fact, internally, it's going to do that check.  
> If nothing named dhclient is running, it will return immediately.
> 
> Although on 10.0, dhclient does not want to die sometimes.
> 
> Here's another PR for usability:
> http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/161547

Hope you guys aren't rewriting bsdconfig's network.subr
-- 
Devin




_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 7, 2013, at 3:21 PM, Outback Dingo wrote:

> 
> 
> 
> On Mon, Oct 7, 2013 at 5:49 PM, Teske, Devin  
> wrote:
> 
> On Oct 7, 2013, at 2:47 PM, Allan Jude wrote:
> 
> > On 2013-10-07 17:00, Outback Dingo wrote:
> >> On Mon, Oct 7, 2013 at 4:54 PM, Allan Jude  wrote:
> >>
> >>> On 2013-10-07 16:43, Warren Block wrote:
> >>>> On Mon, 7 Oct 2013, Allan Jude wrote:
> >>>>
> >>>>> Additional, it includes some other changes to bsdinstall:
> >>>>> 1. Change the default to the 'non-standard keyboard mapping' prompt
> >>>>> to no
> >>>> Please see http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/162175 .
> >>>> ___
> >>>> freebsd-current@freebsd.org mailing list
> >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> >>>> To unsubscribe, send any mail to
> >>>> "freebsd-current-unsubscr...@freebsd.org"
> >>> That is a good idea, I'll add that
> >>>
> >>> I've also hit this:
> >>>
> >>> http://lists.freebsd.org/pipermail/freebsd-sysinstall/2013-September/000949.html
> >>> and plan to throw in a fix for that.
> >>> Which makes more sense, blindly 'killall dhclient' before we try to
> >>> acquire a new lease, or detect that dhclient is running and try to use
> >>> the IP that has already been assigned?
> >>>
> >>>
> >>>
> >> jeez after i just updated and patched my source tree to do a build, you
> >> go and fix things :P
> >> anyway you can just generate a master diff against the tree that we can
> >> apply, seems not ive to
> >> revert the patches and reapply all the newest ones.
> >>
> >>
> >>> --
> >>> Allan Jude
> >>>
> >>> ___
> >>> freebsd-current@freebsd.org mailing list
> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> >>> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
> >>>
> >> ___
> >> freebsd-current@freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> >> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
> >
> > Attached find the patch from last night against a fresh HEAD
> >
> > I am working on incorporating Warren's patch now
> >
> 
> ... and I'm working to review and validate what Allan is writing ;D
> 
> A couple of big patches to incorporate within the hour (next 15 minutes).
> 
> okies dokies guess ill revert it all and wait till you guys give the green 
> light to try again
>  

Thanks... you know... Allan is giving me a work-out. He's running like the wind 
and
I'm trying to catch my breath as he I trail behind him.

Go Allan Go!

;D
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

[Teske, Devin]

On Oct 7, 2013, at 2:47 PM, Allan Jude wrote:

> On 2013-10-07 17:00, Outback Dingo wrote:
>> On Mon, Oct 7, 2013 at 4:54 PM, Allan Jude  wrote:
>> 
>>> On 2013-10-07 16:43, Warren Block wrote:
 On Mon, 7 Oct 2013, Allan Jude wrote:
 
> Additional, it includes some other changes to bsdinstall:
> 1. Change the default to the 'non-standard keyboard mapping' prompt
> to no
 Please see http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/162175 .
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to
 "freebsd-current-unsubscr...@freebsd.org"
>>> That is a good idea, I'll add that
>>> 
>>> I've also hit this:
>>> 
>>> http://lists.freebsd.org/pipermail/freebsd-sysinstall/2013-September/000949.html
>>> and plan to throw in a fix for that.
>>> Which makes more sense, blindly 'killall dhclient' before we try to
>>> acquire a new lease, or detect that dhclient is running and try to use
>>> the IP that has already been assigned?
>>> 
>>> 
>>> 
>> jeez after i just updated and patched my source tree to do a build, you
>> go and fix things :P
>> anyway you can just generate a master diff against the tree that we can
>> apply, seems not ive to
>> revert the patches and reapply all the newest ones.
>> 
>> 
>>> --
>>> Allan Jude
>>> 
>>> ___
>>> freebsd-current@freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>>> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
>>> 
>> ___
>> freebsd-current@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
> 
> Attached find the patch from last night against a fresh HEAD
> 
> I am working on incorporating Warren's patch now
> 

... and I'm working to review and validate what Allan is writing ;D

A couple of big patches to incorporate within the hour (next 15 minutes).
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: r253070 and "disappearing" zpool

[Teske, Devin]

On Jul 22, 2013, at 12:29 AM, Andriy Gapon wrote:

> 
> [turning this into a public discussion with Glen's permission]
> 
> on 10/07/2013 21:05 Glen Barber said the following:
>> Hi,
>> 
>> My setup is like this:
>> 
>>root@nucleus:/usr/src # zpool list
>>NAME SIZE  ALLOC   FREECAP  DEDUP  HEALTH  ALTROOT
>>zboot0  9.94G   379M  9.57G 3%  1.00x  ONLINE  -
>>zroot0   159G   113G  46.2G70%  1.00x  ONLINE  -
>> 
>>root@nucleus:/usr/src # zpool get bootfs
>>NAMEPROPERTY  VALUE   SOURCE
>>zboot0  bootfs-   default
>>zroot0  bootfs-   default
>> 
>>root@nucleus:/usr/src # zfs list zboot0
>>NAME USED  AVAIL  REFER  MOUNTPOINT
>>zboot0   379M  9.41G   281M  /bootdir
>> 
>>root@nucleus:/usr/src # zfs list zroot0
>>NAME USED  AVAIL  REFER  MOUNTPOINT
>>zroot0   113G  43.7G   147M  /
>> 
>> 'zroot0' is a GELI-backed pool, so I have this to fix the boot process:
>> 
>>root@nucleus:/usr/src # ll /boot
>>lrwxr-xr-x  1 root  wheel  12 Aug 25  2012 /boot@ -> bootdir/boot
>> 
>> I upgraded from head/ on July 1 to r253159, and when I rebooted the
>> system, I could correctly boot from the "/bootdir/boot".  Once I enter
>> the GELI passphrase, "/" (from zroot0) is mounted.  Normally, everything
>> would be okay at that point, but since the upgrade, "/bootdir/boot"
>> disappears because the zboot0 pool is not imported as it was before.
>> 
>> Any thoughts?
> 
> I think that this setup (on ZFS level) is quite untypical, although not
> impossible on FreeBSD (and perhaps only FreeBSD).
> It's untypical because you have separate boot pool (where loader, loader.conf
> and kernel are taken from) and root pool (where "/" is mounted from).
> 
> There is this "magic" zpool.cache file that essentially tells what pools 
> should
> be automatically imported.  On FreeBSD this file lives in /boot/zfs directory
> while originally (in Solaris and its descendants) its home is /etc/zfs.
> 
> Until recently FreeBSD could use only zpool.cache from a boot pool and, in 
> fact,
> if a root pool was different from a boot pool the root pool had to be in
> zpool.cache.
> I changed things a little bit and now a root pool does not have to be in
> zpool.cache.  Also, now zpool.cache is taken from the root
> pool, or to be more precise from a root filesystem (whatever happens to be
> /boot/zfs/zpool.cache after "/" is mounted).  I am considering if perhaps now 
> we
> should move zpool.cache back to /etc/zfs/.
> 
> So, I see three ways of resolving the problem that my changes caused for your
> configuration.
> 
> 1.  [the easiest] Put zpool.cache loading instructions that used to be in
> defaults/loader.conf into your loader.conf.  This way everything should work 
> as
> before -- zpool.cache would be loaded from your boot pool.
> 
> 2. Somehow (I don't want to go into any technical details here) arrange that
> your root pool has /boot/zfs/zpool.cache that describes your boot pool.  This 
> is
> probably hard given that your /boot is a symlink at the moment.  This probably
> would be easier to achieve if zpool.cache lived in /etc/zfs.
> 
> 3. [my favorite]  Remove an artificial difference between your boot and root
> pools, so that they are a single root+boot pool (as zfs gods intended).  As 
> far
> as I understand your setup, you use GELI to protect some sensitive data.
> Apparently your kernel is not sensitive data, so I wonder if your /bin/sh or
> /sbin/init are really sensitive either.
> So perhaps you can arrange your unencrypted pool to hold all of the base 
> system
> (boot + root) and put all your truly sensitive filesystems (like e.g. /home or
> /var/data or /opt/xyz) onto your encrypted pool.
> ZFS is really flexible, you can use mountpoint and canmount properties to 
> place
> your filesystems from same or different pools into whatever file namespace
> hierarchy you desire.  Remember that your filesystem hierarchy in the mount
> namespace does not always have to be the same as your ZFS dataset hierarchy.
> 
> I hope that this makes sense to you.
> If you have any additional questions, please do not hesitate.
> 
> P.S.
> ZFS/FreeBSD boot process is extremely flexible.  For example zfsboot can take
> zfsloader from pool1/fsA, zfsloader can boot kernel from pool2/fsB and kernel
> can mount / from pool3/fsC.  Of these 3 filesystems from where should
> zpool.cache be taken?
> My firm opinion is that it should be taken from / (pool3/fsC in the example
> above).  Because it is the root filesystem that defines what a system is going
> to do ultimately: what daemons are started, with what configurations, etc.
> And thus it should also determine what pools to auto-import.
> We can say that zpool.cache is analogous to /etc/fstab in this respect.
> 
> So I understand that my change causes a problem for a setup like yours, but I
> believe that the change is correct.
> 

+1 for zpool.cache on / (pool3/fsC in last example)
-- 
Devin

_
Th

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 14, 2013, at 11:42 PM, Baptiste Daroussin wrote:

> On Fri, Jul 12, 2013 at 11:52:19PM +0000, Teske, Devin wrote:
>> On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote:
>> 
>>> Hi,
>>> 
>>> I have just committed (r253305) a change the make pkg_install not being 
>>> built
>>> and installed by default on HEAD.
>>> 
>>> If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your
>>> src.conf(5)
>> 
>> I think while a good move, we need to allow a window of development to 
>> re-work other HEAD components.
>> 
>> It might also be worth developing a lint-tool to make sure we get every last 
>> instance (both from C code and sh code) within our base before we motion to 
>> cut a release with this change.
>> 
>> I for one am effected and will have to change things.
>> 
>> Can you point us at a guide (or even better, a Wiki) that can smooth the 
>> process?
> 
> I have already pointed you to such resources a couple of time before you 
> started
> working on the package management part of bsdconfig.

Respectfully... look at the reply-text. I didn't mention bsdconfig. I was 
asking in-general for others. I think we're all in agreement that a transition 
Wiki does not exist. I'm not upset that the conversation was turned into a 
bsdconfig-specific one (on the contrary, elated)... but for the very reason 
that you had already given me pointers in the past, I did not come right out 
and say "hey, could you re-tell me everything you told me before because I've 
forgotten most of it?" but instead I pointed out that we don't have a 
transition Wiki.

So I do supremely apologize if I ask a question that you already answered in 
the past.

My excuse is that you pointed me to such resources almost 10 months ago. And 
for that entire 10 months, I'd been working on the package management part of 
bsdconfig. I *just* got my head above water and then it was announced that 
pkg_install is going away, and what perfect timing -- having just finished the 
pkg_install-based framework for which I plan to base the pkgng integration.


> May I remember you that at
> this time I have recommanded you to directly work on the pkgng version?

This comment doesn't seem to be related to any of the reply-text above it. I'll 
assume it's tainted by the other messages in this thread. For bsdconfig, let me 
just re-iterate...

+ bsdconfig is the tool to replace "sysinstall configPackages".
+ It will not do anything "weird" or "strange" or "out of place" **
+ It will try to be an example as well as a re-usable and extendable tool

(and on the back of those last two comments, again, bsdconfig pkgng integration 
is not associated with any funky handling of package MANIFEST files or any 
black-magic voodoo that I do at $work with "pkgbase").


** Unless a UI enhancement can be made by subverting a decidedly non-UI tool to 
provide more feedback than simply blocking on an opaque process.


> I have
> also explained you 2 or 3 times at least what is a catalog in pkgng how it 
> works
> what pkg query and pkg rquery will give you as a script writter.
> 

Again, apologies if I ask a question that *seems* like it's been asked before.

And it would be a lie for me to say I've forgotten *everything* from our prior 
conversations. However, I may repeat some questions as it was almost 10 months 
ago that I was last looking at pkgng (getting "bsdconfig packages" done for 
9.2-R was a *lot* of work and it depleted my brain of pretty much everything 
else for that entire time -- except for $work of course, but every night for 10 
months, 7 days a week, I ate, drank, slept, and smoked "bsdconfig packages").

If I could recount something from our conversations from 10 months ago... I 
would say that you and I agreed that I needed to get the framework done and 
that I could best do that unencumbered if it were pkg_install based. Well 
that's been done (it was a hell of a lot of work, but I got it done).

I'm saying that one sole request (to get the framework ready for pkgng) is now 
done and we need to reconvene to take the next step.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 14, 2013, at 11:38 PM, Baptiste Daroussin wrote:

> On Sat, Jul 13, 2013 at 06:51:00PM -0500, Mark Felder wrote:
>> On Sat, Jul 13, 2013, at 13:54, Teske, Devin wrote:
>>> 
>>> 
>>> If FTP access (or any of the other remote access methods) are going away
>>> for HEAD pkg access, I'll need to know so I can make the appropriate
>>> changes in the HEAD branch of bsdconfig.
>>> 
>> 
>> It's simpler than you think. The new pkg uses libfetch. You can have
>> your PACAKGESITE be file:// ftp:// http:// https:// ... 
>> 
>> I do suspect that HTTP_PROXY support is probably not available as I
>> recall seeing a post where someone was asking about that getting
>> implemented for fetch. I'll have to research that again, though.
> 
> Yes pkgng do support proxies, because libfetch do support them.
> 

I think Mark was saying that libfetch doesn't yet support http proxy.

However, I've often done:

env http_proxy=user:pass@host:port fetch this-thing or that-thing

So I'm not sure if Mark was implying that this doesn't yet work with 
PACKAGESITE because you need a way of passing the additional env? (talk of a 
PKG_ENV iirc)?


> pkgng also support ssh://
> 

That's cool ;D
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 14, 2013, at 11:33 PM, Baptiste Daroussin wrote:

On Sun, Jul 14, 2013 at 04:52:26PM +, Teske, Devin wrote:

On Jul 14, 2013, at 8:01 AM, Chris Rees wrote:

On 14 Jul 2013, at 08:29, Teske, Devin wrote:


Simple, really.

Let's take RPM for example. The RPM package format has been ported to other 
platforms.

So does pkgng ported on Linux, OS X, dragonfly, NetBSD...

Sweet! (bright future!)



But, I can't take archivers/rpm4 and build on RPM on FreeBSD and install it on 
RedHat.

Yes you can, I do it at work all the time, on FreeBSD I do create AIX rpms and
RedHat rpms.

What version of RedHat?

Worked on RedHat EL4 using rpm3, but in-practice -- attempts to recreate that 
workflow on RedHat EL6 using rpm4 have failed.

Please see a copy/paste of the output of when we build an RPM on FreeBSD-8.1 
and try to install it on RedHat Enterprise Linux 6:

http://pastebin.com/zpzjxP2T

Spoiler: "package {X} is intended for a freebsd operating system" (not 
installed)





This is because the RPM format records the platform that you "build" your RPM 
on (not the binaries, just the RPM) *into* said RPM.

So does pkgng.


Good to know.



This actually adds a requirement to the RPM production that the RPMs be 
produced on the platform that they will be installed-to.

No.

Yes. See pastebin link above.




Currently, no such restriction exists for the building of FreeBSD packages 
(within our system). This would have been true if we had ported pkg_create (and 
may continue to be true if we ported pkg and its ilk), but let's say for the 
sake of argument that the future of "pkg" looks bright and it gets ported to 
all sorts of systems (ported in a fashion similar to RPM) *and* we find one day 
that the +MANIFEST starts containing a target-platform (resulting in refusal to 
install a *.txz package because it was rolled on a different platform.

Thank for describing the exact situation pkg is right now.

Glad to help ;D

(not so happy about the target platform being recorded -- is there an override? 
setting UNAME_{a,r,etc.}?)




In that case, we'd then prefer to by-pass the tools and use our own method of 
creating the tar-ball to lift such a restriction.

The restriction you are speaking about does not exists.

See pastebin link above.



ASIDE: If I knew how to force rpmbuild into creating androgynous packages for 
other architectures, I'd be doing that to life the restriction there too, but I 
haven't figured out that.

Basically... within our "pkgbase" tree, we like the branch within the tree to 
dictate how a package is built... not what platform you're on. The goal being 
that we can run a single package-build host that builds all of our packages 
from a single platform.

You can do it with pkgng just easily, as well as you can do it with rpm.


W/respect to RPM, see pastebin link above.
--
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 14, 2013, at 11:27 PM, Baptiste Daroussin wrote:

> On Sun, Jul 14, 2013 at 07:29:50AM +0000, Teske, Devin wrote:
>> 
>>> Or you
>>> can maintain the old cruft for your business -- just don't expect
>>> anyone else to use it, or even want to.
>>> 
>> 
>> 
>> I have no intention of making old-world packages... but I also have no 
>> intention of using "pkg create".
> 
> So you are taking as an exemple rpm but with RPM you do want to use what they 
> do
> provide, aka rpmbuild but with pkgng you want to manually handwrite the 
> packages
> because we haven't hidden the format behind opaque names and format.
> 

Close... I use rpmbuild because I haven't figured out how to subvert it to 
create cpio-balls for rpm(8). I know how to extract the contents of binary 
rpm's using rpm2cpio and I know how to extract the meta data and supplant it 
into a template SPECFILE using rpm -q -p file.rpm --qf "queryformat" ... but I 
haven't spent enough time with rpm's to know how to master them without 
rpmbuild. The best I've done is learn how to use ".rpmmacros" to make rpmbuild 
behave in a sane manner that is conducive to (a) building the archive out of 
the current directory (including all temporary files, etc.) and (b) working on 
source-controlled checkouts.

However, even if I did figure out the cpio-ball/metadata format, it is quite 
convenient to have rpmbuild do the work -- which unfortunately introduces the 
constraint that you have to generate the archive on the system for which it 
will be installed (something you eluded that is inconclusive -- I'll cover that 
in a reply to that-reply directly).

The convenience however is not because the tool does everything I need it to 
do, but the convenience comes from the translation of YAML into metadata. It 
doesn't appear that this will be necessary with pkgng packages, just as it 
wasn't necessary for pkg_install packages.

To be more clear,... because tools like pkg_create and "pkg create" take a 
file-in, add data to it, and then (the most important part) put a modified form 
into the archive (which quite-closely resembles the original), it is then 
possible to imagine that:

(1) there's value in learning both the input format and the output format so 
that you can validate packages, fix broken packages, etc.

ASIDE: For example... a binary release package ends up with incorrect 
dependencies encoded within its +CONTENTS file -- I can slurp the package into 
our "pkgbase", adjust the PLIST (generated from +CONTENTS), and produce a fixed 
package in mere moments (and I didn't have to wake anybody in the middle of the 
night, ask that FTP mirrors be updated, balk at the core team -- none of 
that... the problem was solved and in the next release we hope for the best 
that the dependency chain isn't busted for the same binary package within the 
set of "X.Y-RELEASE" packages).

(2) there's value in writing a shell-script method to translate the input 
format into the output format from a system of makefiles calling said shell 
scripts out of a revision-controlled checkout

ASIDE: This one should be obvious. If you can assemble a package using nothing 
but shell scripts (and mostly awk, ls, find, cp, mkdir, ln, cvs, make, 
md5/sha256/sha512/openssl, and etc.), then you don't have to port the "port" 
system (our "port" system being called "pkgbase" -- and working to "keep you 
stable" rather than "keep you up-to-date").



> You know in that case you should probably create your own RPM without RPM just
> write by hand the header and happened the cpio to it, in that case you will 
> end
> up having the same at what you are trying to do with pkg.
> 

It's a good idea... except one thing... the YAML SPECFILE (minus the elements 
that are specific to the `-bb' process of rpmbuild) get completely changed in 
the process. Doing this would require a YAML parser + metadata encoder 
process-pair. Whereas for FreeBSD it's YAML-in, YAML-out. So I don't need a 
YAML-parser, just a line-parser and light YAML-encoder -- which could be 
fixed-strings, doesn't have to be complex... e.g., awk).
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 14, 2013, at 11:23 PM, Baptiste Daroussin wrote:

On Sun, Jul 14, 2013 at 10:12:19AM +, Teske, Devin wrote:

On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote:

On 14/07/2013 06:48, Teske, Devin wrote:
Question: Where can I learn more about the actual format of what's in
the new tarballs? This is going to be important not for bsdconfig,
but $work (we have our own build platform; I'm going to have to
rewrite it from mastering PLIST files to mastering YAML MANIFEST
files and I want to know all the gritty details).

We do need a pkg-manifest(5) man page, which can double as a
pkg-tarball(5) page since the manifest file is where most of the
interesting bits are.

A pkg tarball is a compressed tar archive like so:

lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz
-rw-r--r--  0 root   wheel 530 Jan  1  1970 +COMPACT_MANIFEST
-rw-r--r--  0 root   wheel6385 Jan  1  1970 +MANIFEST
-rw-r--r--  0 root   wheel   17567 Jan  1  1970 +MTREE_DIRS
-r--r--r--  0 root   wheel   19453 Jul  7 12:26
/usr/local/etc/bash_completion.d/_pkg.bash
-r-xr-xr-x  0 root   wheel 629 Jul  7 12:26
/usr/local/etc/periodic/daily/400.status-pkg
-r-xr-xr-x  0 root   wheel 823 Jul  7 12:26
/usr/local/etc/periodic/daily/411.pkg-backup
-r-xr-xr-x  0 root   wheel 678 Jul  7 12:26
/usr/local/etc/periodic/daily/490.status-pkg-changes
-r-xr-xr-x  0 root   wheel2558 Jul  7 12:26
/usr/local/etc/periodic/security/410.pkg-audit
-r-xr-xr-x  0 root   wheel 611 Jul  7 12:26
/usr/local/etc/periodic/security/460.pkg-checksum
-r--r--r--  0 root   wheel 839 Jul  7 12:26
/usr/local/etc/pkg.conf.sample
-r--r--r--  0 root   wheel   43432 Jul  7 12:26 /usr/local/include/pkg.h
-r--r--r--  0 root   wheel  727558 Jul  7 12:26 /usr/local/lib/libpkg.a
lrwxr-xr-x  0 root   wheel   0 Jul  7 12:26 /usr/local/lib/libpkg.so
-> libpkg.so.1
-r--r--r--  0 root   wheel 1227064 Jul  7 12:26 /usr/local/lib/libpkg.so.1
-rw-r--r--  0 root   wheel 187 Jul  7 12:26
/usr/local/libdata/pkgconfig/pkg.pc
[... etc ...]


Interesting. I notice that (while looking ahead to see a prefix: of /usr/local 
in the +MANIFEST), the tarball itself has files that include /usr/local in 
their path.

Does pkg handle packages where the prefix (in +MANIFEST) is "/usr/local" _but_ 
"tar tf" of the unxz'd tarball is _not_ "/usr/local"? (e.g. pkg_install style) 
or is this the new way going forward?



There must at least be a +MANIFEST -- other meta data files are
optional.  +COMPACT_MANIFEST is a subset of the full +MANIFEST.  They're
both YAML documents.  +COMPACT_MANIFEST looks like this, for example:

---
name: pkg
version: 1.1.4
origin: ports-mgmt/pkg
comment: New generation package manager
arch: freebsd:9:x86:64
www: 
https://urldefense.proofpoint.com/v1/url?u=http://wiki.freebsd.org/pkgng&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=LPoff3ddLDoHE30AU6R5vgBklqxsjVAosXGjJop6uO4%3D%0A&s=30b27cf33d7e594144fc38f20bc75d84464788e715f7fc82b2495f813fd4be2d
maintainer: port...@freebsd.org<mailto:port...@freebsd.org>
prefix: /usr/local
licenselogic: single
licenses:
- BSD
flatsize: 6311507
desc: |
New Generation package management tool for FreeBSD

WWW: 
https://urldefense.proofpoint.com/v1/url?u=http://wiki.freebsd.org/pkgng&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=cLKrok7Cl%2BjTLEB024lmmMXfxb1fKAOZvvxLV8RjU%2FU%3D%0A&s=cef93a5fe98a3fa0fbc9df3e0d11c9bbeaee44290306834ddef30f2ba925446c
categories:
- ports-mgmt
shlibs_required:
- libpkg.so.1
shlibs_provided:
- libpkg.so.1
message: |
If you are upgrading from the old package format, first run:

  # pkg2ng


Nice. Very nice.



+MTREE_DIRS is a compatibility thing with the old pkg_tools.  It's not
needed in general as +MANIFEST can provide all that meta data itself.
It isn't going to be deprecated for at least as long as the ports tree
continues to support pkg_tools though.

Beyond that, the rest of the pkg tarball just contains a tar archive of
all the files, directories, sym-links etc to be installed by the
package.  Note that pkg(8) has no problem with creating an empty
directory for a package, unlike pkg_tools.


Excellent!


Now, while you can grovel through the details of pkg tarballs and
internal data formats like this, be aware: the format of the manifest
and the details of the meta-data included in the pkg-tarballs is subject
to change without warning as we develop pkg(8) further.  We only promise
API stability through the pkg(8) commands or for the libpkg.so library
functions; reports of breakage from usage outside those APIs will
receive little sympathy.


Understood.

So just to give you a better idea of how we manage packages here.

We've realized that if you want to package for FreeBSD (in 9 and older), you 
could get by alright if you knew how to create a +CONTENTS file from scratch. 
For RedHat, it's the SPECFIL

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 14, 2013, at 3:43 PM, Craig Rodrigues wrote:

On Sat, Jul 13, 2013 at 10:54 PM, Teske, Devin 
mailto:devin.te...@fisglobal.com>> wrote:


I assume that poudiere builds packages from ports.

Yes.


That's not how we build package repositories here (and would expect that there 
are many more like us).

How do you build packages if you are not using FreeBSD ports?

Two spectrums...

When developer A says he needs xerces, I go to:

(for an active release, such as 8.4 or 9.1)
ftp://ftp.freebsd.org/pub/FreeBSD/releases/{arch}/X.Y-RELEASE/packages/All<ftp://ftp.freebsd.org/pub/FreeBSD/releases/%7Barch%7D/X.Y-RELEASE/packages/All>

(for an in-active release, such as 8.1 or 9.0)
ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/{arch}/X.Y-RELEASE/packages/All<ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/%7Barch%7D/X.Y-RELEASE/packages/All>

Meanwhile, when either developer B or integrator A says "I have software X that 
I want to package", we use "pkgbase" to build the FreeBSD package, the RedHat 
RPM, etc.


  Do you have your own Makefiles which resemble FreeBSD ports,
or are you doing something completely different?


That's essentially what pkgbase is. Except instead of building makefiles that 
build the 3rd-party software, we rely on the 3rd-party compilation process to 
produce the binaries and have a configuration file (pkgbase.conf) which 
describes the vendor layout.

The vendor layout can be anything and do anything and prepare anything. But the 
pkgbase.conf says "fileA" in the vendor layout goes to "src/fileA" (where 
"src/" looks exactly like the un-packed tar-ball, *minus* +CONTENTS -- which is 
generated from PLIST when you say "make").

Since we're using CVS, people that are familiar with CVS would think it's quite 
cute that it does a "cvs admin -ko" on files that are pulled from the vendor 
branch (resulting in keyword expansion to retain its vendor location -- so if a 
problem happens in the field, we know that the file that should be edited -- 
based on its keyword expansion -- is over in the vendor "depend" tree which, 
upon modification, will cause a cascading rebuild of packages for releases we 
actively maintain [through a sliding window]).

So it's very similar to ports, but we need the mechanism of being able to (for 
the benefit of developer A/B and integrator A above) freeze the software 
distribution of software X.

The ports tree manages that through a collection of centrally located distfiles 
(which acts as a fall-back for when vendor distfiles dry up). But that method 
doesn't work so well for us because when we find a bug we often fix it 
ourselves in our own local branch and then communicate the fix back the vendor 
(leaving it up to them to incorporate it). The next time around on our 
migration to a new release, *if* we have made a local patch, producing a custom 
version, *and* the vendor has not taken a patch, we'll _first_ re-evaluate the 
new version with a documented test-case or (if that fails) we'll keep the patch 
in-place.

The ports tree moves too fast for us. And many times we find a bug, we end up 
performing cost/benefit analysis only to find that we can live with the bug 
*if* the problem has been fixed already. In rare cases, we'll merge a single 
patch from a future revision to get things going. But we *won't* just blindly 
take a new version simply because it fixes a bug (as it may introduce new bugs 
-- unwinding many man-hours of many testers and developers attempting to lead 
up to a release of a wholly contained full-working software entity).

I hope this helps to explain a little more about how we take FreeBSD releases, 
vet them, and them blast them out to thousands of machines over many customers 
after many months of testing.

Now to say that we don't use ports would be a lie. Sometimes a developer asks 
for a package that is not available on the release we're shipping. This 
happened recently. So I went into the ports tree, built the package on 8.1 
(which had 99% binary packages from 8.1-RELEASE -- see above URLs) and then I 
said "make package" and threw it in the repository.

ASIDE: For diligence, "make describe" iirc generates my INDEX entry to append.




We expect to start with an empty directory, go grab packages that we want (one 
by one) from FTP. Then put those tarballs into a directory. There's no 
additional step because we download the INDEX file to the repository too. Local 
repository is built.

What FTP site are you grabbing packages from, the FreeBSD ftp site (or mirrors) 
or one of your own?

We grab from either ftp.f.o or ftp-archive.f.o and plop into an NFS-based repo 
(which has a structure mirroring that of the ftp{,-archive}.f.o structure).



How are you generating the INDEX file?

Well, since we prefer to shi

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 14, 2013, at 11:50 AM, Adrian Chadd wrote:

> ... I bet you could do that. I bet you could build the rpm inside a
> linux jail and have the relevant uname bits overridden in the right
> way.
> 

There's an idea.
-- 
Devin


> On 14 July 2013 09:52, Teske, Devin  wrote:
>> 
>> On Jul 14, 2013, at 8:01 AM, Chris Rees wrote:
>> 
>>> On 14 Jul 2013, at 08:29, Teske, Devin wrote:
>>>> 
>>>> To give you an idea as to just how helpful this is...
>>>> 
>>>> Imagine the following hierarchy:
>>>> 
>>>> src/pkgbase/depend/mystuff/script1
>>>> src/pkgbase/depend/mystuff/textfile1
>>>> src/pkgbase/depend/mystuff/sourcefile.c
>>>> src/pkgbase/depend/mystuff/Makefile
>>>> 
>>>> You are a developer. You want to ship a package that contains "script1", 
>>>> "textfile1", and "binary1" (which is compiled by saying "make" to turn 
>>>> "sourcefile.c" into "binary1")
>>>> 
>>>> You want to ship 8 types of packages:
>>>> 
>>>> FreeBSD-4.11
>>>> FreeBSD-8.1 (i386)
>>>> FreeBSD-8.1 (amd64)
>>>> RedHat EL 4
>>>> RedHat EL 6 (i386)
>>>> RedHat EL 6 (x86_64)
>>>> Debian Wheezy
>>>> Debian Wheezy 64-bit
>>>> 
>>>> This is where my framework comes in-handy...
>>>> 
>>>> cd ~/src/pkgbase/freebsd/RELENG_4/category/mystuff
>>>> make
>>>> # it pulled the necessary bits from "src/pkgbase/depend/mystuff" and built 
>>>> the .tgz
>>>> 
>>>> cd ~/src/pkgbase/freebsd/RELENG_8/category/mystuff
>>>> make
>>>> # it pulled the necessary bits from the "depend" dir and built .tbz
>>>> 
>>>> cd ~/src/pkgbase/redhat/rhel4/category/sub-category/mystuff
>>>> make
>>>> # pulled in "depend" and made .rpm
>>>> 
>>>> cd ~/src/pkgbase/redhat/rhel6/category/sub-category/mystuff
>>>> make
>>>> # pulled in "depend" and made .rpm
>>>> 
>>>> etc.
>>>> 
>>>> Of course, *any* time the depend tree has binaries in it... you have to 
>>>> first do a make in there on the platform you want to ship the binary for, 
>>>> and then do "make depend" in the platform-specific tree to pull in the 
>>>> binaries. Once you've done that, you don't have to muck with the depend 
>>>> tree again unless there are changes there.
>>>> 
>>>> So, I assume that your prejudice remarks are because you haven't either 
>>>> seen (a) such a platform or (b) such a need for said platform.
>>>> 
>>>> Yeah, I could rewrite the freebsd-specific logic to use "pkg create", but 
>>>> let me tell you...
>>>> 
>>>> When you have to touch a file that needs to get shipped out to multiple 
>>>> platforms...
>>>> 
>>>> It's damned nice to be able to build the FreeBSD packages under RedHat 
>>>> *BECAUSE* the redhat RPMs can't be built under anything else (building an 
>>>> RPM on FreeBSD and attempting to install it on RedHat results in an error 
>>>> message similar to "this is an rpm for FreeBSD; go away").
>>>> 
>>>> Whereas FreeBSD will never balk about a package built on another platform.
>>>> 
>>>> It's a huge time-saving measure... not having to jump over to each/every 
>>>> unique platform to package things up *IF/WHEN* you know that there are no 
>>>> binaries in the package *or* you've already checked the pre-compiled 
>>>> binaries into the arch-specific hierarchy.
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> Or you
>>>>> can maintain the old cruft for your business -- just don't expect
>>>>> anyone else to use it, or even want to.
>>>>> 
>>>> 
>>>> 
>>>> I have no intention of making old-world packages... but I also have no 
>>>> intention of using "pkg create".
>>> 
>>> You still haven't really explained at all why you can't use libpkg.  If it 
>>> doesn't run on Debian (not tried), it's got to be easier to port it than 
>>> rewrite a hacked version, hasn't it???  At least then you'll also be 
>>> contr

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 14, 2013, at 8:01 AM, Chris Rees wrote:

> On 14 Jul 2013, at 08:29, Teske, Devin wrote:
>> 
>> To give you an idea as to just how helpful this is...
>> 
>> Imagine the following hierarchy:
>> 
>> src/pkgbase/depend/mystuff/script1
>> src/pkgbase/depend/mystuff/textfile1
>> src/pkgbase/depend/mystuff/sourcefile.c
>> src/pkgbase/depend/mystuff/Makefile
>> 
>> You are a developer. You want to ship a package that contains "script1", 
>> "textfile1", and "binary1" (which is compiled by saying "make" to turn 
>> "sourcefile.c" into "binary1")
>> 
>> You want to ship 8 types of packages:
>> 
>> FreeBSD-4.11
>> FreeBSD-8.1 (i386)
>> FreeBSD-8.1 (amd64)
>> RedHat EL 4
>> RedHat EL 6 (i386)
>> RedHat EL 6 (x86_64)
>> Debian Wheezy
>> Debian Wheezy 64-bit
>> 
>> This is where my framework comes in-handy...
>> 
>> cd ~/src/pkgbase/freebsd/RELENG_4/category/mystuff
>> make
>> # it pulled the necessary bits from "src/pkgbase/depend/mystuff" and built 
>> the .tgz
>> 
>> cd ~/src/pkgbase/freebsd/RELENG_8/category/mystuff
>> make
>> # it pulled the necessary bits from the "depend" dir and built .tbz
>> 
>> cd ~/src/pkgbase/redhat/rhel4/category/sub-category/mystuff
>> make
>> # pulled in "depend" and made .rpm
>> 
>> cd ~/src/pkgbase/redhat/rhel6/category/sub-category/mystuff
>> make
>> # pulled in "depend" and made .rpm
>> 
>> etc.
>> 
>> Of course, *any* time the depend tree has binaries in it... you have to 
>> first do a make in there on the platform you want to ship the binary for, 
>> and then do "make depend" in the platform-specific tree to pull in the 
>> binaries. Once you've done that, you don't have to muck with the depend tree 
>> again unless there are changes there.
>> 
>> So, I assume that your prejudice remarks are because you haven't either seen 
>> (a) such a platform or (b) such a need for said platform.
>> 
>> Yeah, I could rewrite the freebsd-specific logic to use "pkg create", but 
>> let me tell you...
>> 
>> When you have to touch a file that needs to get shipped out to multiple 
>> platforms...
>> 
>> It's damned nice to be able to build the FreeBSD packages under RedHat 
>> *BECAUSE* the redhat RPMs can't be built under anything else (building an 
>> RPM on FreeBSD and attempting to install it on RedHat results in an error 
>> message similar to "this is an rpm for FreeBSD; go away").
>> 
>> Whereas FreeBSD will never balk about a package built on another platform.
>> 
>> It's a huge time-saving measure... not having to jump over to each/every 
>> unique platform to package things up *IF/WHEN* you know that there are no 
>> binaries in the package *or* you've already checked the pre-compiled 
>> binaries into the arch-specific hierarchy.
>> 
>> 
>> 
>> 
>>> Or you
>>> can maintain the old cruft for your business -- just don't expect
>>> anyone else to use it, or even want to.
>>> 
>> 
>> 
>> I have no intention of making old-world packages... but I also have no 
>> intention of using "pkg create".
> 
> You still haven't really explained at all why you can't use libpkg.  If it 
> doesn't run on Debian (not tried), it's got to be easier to port it than 
> rewrite a hacked version, hasn't it???  At least then you'll also be 
> contributing back.
> 

Simple, really.

Let's take RPM for example. The RPM package format has been ported to other 
platforms.

But, I can't take archivers/rpm4 and build on RPM on FreeBSD and install it on 
RedHat.

This is because the RPM format records the platform that you "build" your RPM 
on (not the binaries, just the RPM) *into* said RPM.

This actually adds a requirement to the RPM production that the RPMs be 
produced on the platform that they will be installed-to.

Currently, no such restriction exists for the building of FreeBSD packages 
(within our system). This would have been true if we had ported pkg_create (and 
may continue to be true if we ported pkg and its ilk), but let's say for the 
sake of argument that the future of "pkg" looks bright and it gets ported to 
all sorts of systems (ported in a fashion similar to RPM) *and* we find one day 
that the +MANIFEST starts containing a target-platform (resulting in refusal to 
install a *.txz package because it was rolled on a diff

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote:

> On 14/07/2013 06:48, Teske, Devin wrote:
>> Question: Where can I learn more about the actual format of what's in
>> the new tarballs? This is going to be important not for bsdconfig,
>> but $work (we have our own build platform; I'm going to have to
>> rewrite it from mastering PLIST files to mastering YAML MANIFEST
>> files and I want to know all the gritty details).
> 
> We do need a pkg-manifest(5) man page, which can double as a
> pkg-tarball(5) page since the manifest file is where most of the
> interesting bits are.
> 
> A pkg tarball is a compressed tar archive like so:
> 
> lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz
> -rw-r--r--  0 root   wheel 530 Jan  1  1970 +COMPACT_MANIFEST
> -rw-r--r--  0 root   wheel6385 Jan  1  1970 +MANIFEST
> -rw-r--r--  0 root   wheel   17567 Jan  1  1970 +MTREE_DIRS
> -r--r--r--  0 root   wheel   19453 Jul  7 12:26
> /usr/local/etc/bash_completion.d/_pkg.bash
> -r-xr-xr-x  0 root   wheel 629 Jul  7 12:26
> /usr/local/etc/periodic/daily/400.status-pkg
> -r-xr-xr-x  0 root   wheel 823 Jul  7 12:26
> /usr/local/etc/periodic/daily/411.pkg-backup
> -r-xr-xr-x  0 root   wheel 678 Jul  7 12:26
> /usr/local/etc/periodic/daily/490.status-pkg-changes
> -r-xr-xr-x  0 root   wheel2558 Jul  7 12:26
> /usr/local/etc/periodic/security/410.pkg-audit
> -r-xr-xr-x  0 root   wheel 611 Jul  7 12:26
> /usr/local/etc/periodic/security/460.pkg-checksum
> -r--r--r--  0 root   wheel 839 Jul  7 12:26
> /usr/local/etc/pkg.conf.sample
> -r--r--r--  0 root   wheel   43432 Jul  7 12:26 /usr/local/include/pkg.h
> -r--r--r--  0 root   wheel  727558 Jul  7 12:26 /usr/local/lib/libpkg.a
> lrwxr-xr-x  0 root   wheel   0 Jul  7 12:26 /usr/local/lib/libpkg.so
> -> libpkg.so.1
> -r--r--r--  0 root   wheel 1227064 Jul  7 12:26 /usr/local/lib/libpkg.so.1
> -rw-r--r--  0 root   wheel 187 Jul  7 12:26
> /usr/local/libdata/pkgconfig/pkg.pc
> [... etc ...]
> 

Interesting. I notice that (while looking ahead to see a prefix: of /usr/local 
in the +MANIFEST), the tarball itself has files that include /usr/local in 
their path.

Does pkg handle packages where the prefix (in +MANIFEST) is "/usr/local" _but_ 
"tar tf" of the unxz'd tarball is _not_ "/usr/local"? (e.g. pkg_install style) 
or is this the new way going forward?



> There must at least be a +MANIFEST -- other meta data files are
> optional.  +COMPACT_MANIFEST is a subset of the full +MANIFEST.  They're
> both YAML documents.  +COMPACT_MANIFEST looks like this, for example:
> 
> ---
> name: pkg
> version: 1.1.4
> origin: ports-mgmt/pkg
> comment: New generation package manager
> arch: freebsd:9:x86:64
> www: 
> https://urldefense.proofpoint.com/v1/url?u=http://wiki.freebsd.org/pkgng&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=LPoff3ddLDoHE30AU6R5vgBklqxsjVAosXGjJop6uO4%3D%0A&s=30b27cf33d7e594144fc38f20bc75d84464788e715f7fc82b2495f813fd4be2d
> maintainer: port...@freebsd.org
> prefix: /usr/local
> licenselogic: single
> licenses:
> - BSD
> flatsize: 6311507
> desc: |
>  New Generation package management tool for FreeBSD
> 
>  WWW: http://wiki.freebsd.org/pkgng
> categories:
> - ports-mgmt
> shlibs_required:
> - libpkg.so.1
> shlibs_provided:
> - libpkg.so.1
> message: |
>  If you are upgrading from the old package format, first run:
> 
># pkg2ng
> 

Nice. Very nice.



> +MTREE_DIRS is a compatibility thing with the old pkg_tools.  It's not
> needed in general as +MANIFEST can provide all that meta data itself.
> It isn't going to be deprecated for at least as long as the ports tree
> continues to support pkg_tools though.
> 
> Beyond that, the rest of the pkg tarball just contains a tar archive of
> all the files, directories, sym-links etc to be installed by the
> package.  Note that pkg(8) has no problem with creating an empty
> directory for a package, unlike pkg_tools.
> 

Excellent!


> Now, while you can grovel through the details of pkg tarballs and
> internal data formats like this, be aware: the format of the manifest
> and the details of the meta-data included in the pkg-tarballs is subject
> to change without warning as we develop pkg(8) further.  We only promise
> API stability through the pkg(8) commands or for the libpkg.so library
> functions; reports of breakage from usage outside those APIs will
> receive little sympathy.
> 

Understood.

So just to give you a better idea of how we manage packages here.

We've realized that if you want to package for FreeBSD (in 9 and older), you 
could get by alright if you knew how to create 

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 13, 2013, at 9:24 PM, Peter Wemm wrote:

> On Sat, Jul 13, 2013 at 11:54 AM, Teske, Devin
>  wrote:
> 
>> So yes... I'm asking... in a HEAD world, what is the "officially supported" 
>> method of acquisition?
> 
> * pkgng will be configured to pull via http from project
> infrastructure by default.

So is this the right _first step_ patch for HEAD bsdconfig?

NOTE: Based on feedback, this will be the list of what official pkg 
infrastructure we want to support from the base. The selection would be 
communicated to pkg via $PACKAGESITE.

Index: head/usr.sbin/bsdconfig/share/media/http.subr
===
--- head/usr.sbin/bsdconfig/share/media/http.subr   (revision 25)
+++ head/usr.sbin/bsdconfig/share/media/http.subr   (working copy)
@@ -77,7 +77,9 @@ f_dialog_menu_media_http()
f_dialog_title_restore
local prompt="$msg_please_select_the_site_closest_to_you_or_other"
local menu_list="
-   'URL' '$msg_specify_some_other_http_site'
+   '$msg_main_site'  'http.freebsd.org'
+   'URL' '$msg_specify_some_other_http_site'
+   'IPv6 $msg_main_site' 'http.freebsd.org'
" # END-QUOTE
local hline="$msg_select_a_site_thats_close"
 
Index: head/usr.sbin/bsdconfig/share/media/ftp.subr
===
--- head/usr.sbin/bsdconfig/share/media/ftp.subr(revision 25)
+++ head/usr.sbin/bsdconfig/share/media/ftp.subr(working copy)
@@ -73,157 +73,7 @@ f_dialog_menu_media_ftp()
f_dialog_title_restore
local prompt="$msg_please_select_the_site_closest_to_you_or_other"
local menu_list="
-   '$msg_main_site'  'ftp.freebsd.org'
-   'URL' '$msg_specify_some_other_ftp_site'
-   '$msg_snapshots_server_japan'
- 'snapshots.jp.freebsd.org'
-   '$msg_snapshots_server_sweden'
- 'snapshots.se.freebsd.org'
-   'IPv6 $msg_main_site' 'ftp.freebsd.org'
-   ' IPv6 $msg_ireland'  'ftp3.ie.freebsd.org'
-   ' IPv6 $msg_israel'   'ftp.il.freebsd.org'
-   ' IPv6 $msg_japan''ftp2.jp.freebsd.org'
-   ' IPv6 $msg_usa'  'ftp4.us.freebsd.org'
-   ' IPv6 $msg_turkey'   'ftp2.tr.freebsd.org'
-   '$msg_primary''ftp1.freebsd.org'
-   ' $msg_primary #2''ftp2.freebsd.org'
-   ' $msg_primary #3''ftp3.freebsd.org'
-   ' $msg_primary #4''ftp4.freebsd.org'
-   ' $msg_primary #5''ftp5.freebsd.org'
-   ' $msg_primary #6''ftp6.freebsd.org'
-   ' $msg_primary #7''ftp7.freebsd.org'
-   ' $msg_primary #8''ftp8.freebsd.org'
-   ' $msg_primary #9''ftp9.freebsd.org'
-   ' $msg_primary #10'   'ftp10.freebsd.org'
-   ' $msg_primary #11'   'ftp11.freebsd.org'
-   ' $msg_primary #12'   'ftp12.freebsd.org'
-   ' $msg_primary #13'   'ftp13.freebsd.org'
-   ' $msg_primary #14'   'ftp14.freebsd.org'
-   '$msg_argentina'  'ftp.ar.freebsd.org'
-   '$msg_australia'  'ftp.au.freebsd.org'
-   ' $msg_australia #2'  'ftp2.au.freebsd.org'
-   ' $msg_australia #3'  'ftp3.au.freebsd.org'
-   '$msg_austria''ftp.at.freebsd.org'
-   ' $msg_austria #2''ftp2.at.freebsd.org'
-   '$msg_brazil' 'ftp.br.freebsd.org'
-   ' $msg_brazil #2' 'ftp2.br.freebsd.org'
-   ' $msg_brazil #3' 'ftp3.br.freebsd.org'
-   ' $msg_brazil #4' 'ftp4.br.freebsd.org'
-   ' $msg_brazil #5' 'ftp5.br.freebsd.org'
-   ' $msg_brazil #6'   

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 14, 2013, at 12:06 AM, Garrett Wollman wrote:

> In article <20130714064601$3...@grapevine.csail.mit.edu>,
> dte...@freebsd.org writes:
> 
>>> [I wrote:]
>>> It accesses the sqlite database in /var/db/pkg that was previously
>>> retrieved from the remote repository.
> 
>> Now from what you explained of pkg, I'm worried that for bsdconfig:
>> 
>> 1. Browse packages (nothing else)
>> 2. Exit bsdconfig
>> 
>> and now because the "pkg rquery" has staged a db, future "pkg" commands
>> are now influenced.
> 
> Only if you update /usr/local/etc/pkg.conf to set a permanent
> PACKAGESITE.  Otherwise, it will notice that the remote repo catalog
> you have isn't for your currently selected remote repo, and use that
> instead.
> 
>>> You really shouldn't know about the actual format of the tarballs;
>>> your time would be better spent learning the new "pkg create", "pkg
>>> register", and "pkg repo" commands.  Depending on your needs, you
>>> might want to write to the libpkg API instead.
> 
>> That won't work for us.
>> 
>> You're not going to like the answer, but it does mean that things like
>> "pkg create", "pkg register" and "pkg repo" won't work for us.
> 
> Congratulations for building your entire workflow around a horrible
> kluge straight out of 1993.

Prejudice much?


>  FreeBSD, however, is moving on.

Moving from tarballs to tarballs, yep... moving along nicely.


>  (And
> it's long past time!)

I love your enthusiastic embrace.



>  Your developers will just have to deal.

You're making the wrong argument.

Bringing in a new package system doesn't help developers or integrators. It 
doesn't matter if you switch "pkg_create" with "pkg_create". Developers like 
being able to go into a build tree and say "make" and end-up with a package.

How that build tree does it should be "sufficiently advanced enough to be 
indistinguishable from magic."

Your answer implies that this infrastructure is unnecessary, when I can indeed 
tell you that even if it is not divulged to me what goes into a package, I'll 
still end up ripping it open and creating my own build platform that doesn't 
depend on platform-specific tools.

And when the day comes that FreeBSD uses something other than tarballs, then 
I'll then force the developers to build the packages on the platform using said 
tools; but until then, why pigeon-hole the process of building a package into 
one OS when developers could care less whether their "make" uses pkg_create, 
pkg, rpm, dpkg, or what else?

To give you an idea as to just how helpful this is...

Imagine the following hierarchy:

src/pkgbase/depend/mystuff/script1
src/pkgbase/depend/mystuff/textfile1
src/pkgbase/depend/mystuff/sourcefile.c
src/pkgbase/depend/mystuff/Makefile

You are a developer. You want to ship a package that contains "script1", 
"textfile1", and "binary1" (which is compiled by saying "make" to turn 
"sourcefile.c" into "binary1")

You want to ship 8 types of packages:

FreeBSD-4.11
FreeBSD-8.1 (i386)
FreeBSD-8.1 (amd64)
RedHat EL 4
RedHat EL 6 (i386)
RedHat EL 6 (x86_64)
Debian Wheezy
Debian Wheezy 64-bit

This is where my framework comes in-handy...

cd ~/src/pkgbase/freebsd/RELENG_4/category/mystuff
make
# it pulled the necessary bits from "src/pkgbase/depend/mystuff" and built the 
.tgz

cd ~/src/pkgbase/freebsd/RELENG_8/category/mystuff
make
# it pulled the necessary bits from the "depend" dir and built .tbz

cd ~/src/pkgbase/redhat/rhel4/category/sub-category/mystuff
make
# pulled in "depend" and made .rpm

cd ~/src/pkgbase/redhat/rhel6/category/sub-category/mystuff
make
# pulled in "depend" and made .rpm

etc.

Of course, *any* time the depend tree has binaries in it... you have to first 
do a make in there on the platform you want to ship the binary for, and then do 
"make depend" in the platform-specific tree to pull in the binaries. Once 
you've done that, you don't have to muck with the depend tree again unless 
there are changes there.

So, I assume that your prejudice remarks are because you haven't either seen 
(a) such a platform or (b) such a need for said platform.

Yeah, I could rewrite the freebsd-specific logic to use "pkg create", but let 
me tell you...

When you have to touch a file that needs to get shipped out to multiple 
platforms...

It's damned nice to be able to build the FreeBSD packages under RedHat 
*BECAUSE* the redhat RPMs can't be built under anything else (building an RPM 
on FreeBSD and attempting to install it on RedHat results in an error message 
similar to "this is an rpm for FreeBSD; go away").

Whereas FreeBSD will never balk about a package built on another platform.

It's a huge time-saving measure... not having to jump over to each/every unique 
platform to package things up *IF/WHEN* you know that there are no binaries in 
the package *or* you've already checked the pre-compiled binaries into the 
arch-specific hierarchy.




>  Or you
> can maintain the old cruft for your business -- just don't expect
> anyone else to use

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 13, 2013, at 11:13 PM, Garrett Wollman wrote:

> In article <20130714054840$7...@grapevine.csail.mit.edu>,
> dte...@freebsd.org writes:
> 
>> How about rquery? What protocol does that use? and what does it talk to?
> 
> It accesses the sqlite database in /var/db/pkg that was previously
> retrieved from the remote repository.
> 

This "previously retrieved" has me worried.

One of the things that you could do with sysinstall was:

1. Browse packages (nothing else)
2. Exit sysinstall

And your system is completely unmodified and the mirror your chose is forgotten.

Now from what you explained of pkg, I'm worried that for bsdconfig:

1. Browse packages (nothing else)
2. Exit bsdconfig

and now because the "pkg rquery" has staged a db, future "pkg" commands are now 
influenced.

In which case, I think I would best off downloading the sqlite db to a 
temporary location and pointing pkg at it (if that's possible). Because I'm not 
sure that I want adhoc browsing in bsdconfig to change the default mirror (but 
maybe that's desired? I'm not sure).



>> Question: Where can I learn more about the actual format of what's in
>> the new tarballs? This is going to be important not for bsdconfig, but
>> $work (we have our own build platform; I'm going to have to rewrite it
>> from mastering PLIST files to mastering YAML MANIFEST files and I want
>> to know all the gritty details).
> 
> You really shouldn't know about the actual format of the tarballs;
> your time would be better spent learning the new "pkg create", "pkg
> register", and "pkg repo" commands.  Depending on your needs, you
> might want to write to the libpkg API instead.
> 

That won't work for us.

You're not going to like the answer, but it does mean that things like "pkg 
create", "pkg register" and "pkg repo" won't work for us.

Because FreeBSD packages are just tar-balls, I created a build-system that 
allows us to master and re-master them on *any* platform with nothing more than 
"make" and a few standard UNIX utilities such as "awk" and etc. This makes it 
extremely convenient for developers and integrators. You can feel free to check 
out the build tree on Mac OS X, make some changes to a file, check them in, 
build the package, throw it in the repos, and all from the comfort of your own 
OS.

Yes, it flies in the face of "dog-fooding", but the value-add that it brings to 
the table is quite tangible w/respect to our workflow.

So the format is still something I want to know quite well.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 13, 2013, at 8:33 PM, Craig Rodrigues wrote:

On Sat, Jul 13, 2013 at 8:18 PM, Teske, Devin 
mailto:devin.te...@fisglobal.com>> wrote:

Can you do an "rquery" on a local repository? (say, one that has been mounted 
via NFS or some other media, local or otherwise but looking like a local 
filesystem once-mounted). What would be required to get a local repository like 
that going?


(1)  Install the port:  ports-mgmgt/poudriere , and follow the instructions at: 
https://fossil.etoilebsd.net/poudriere/<https://urldefense.proofpoint.com/v1/url?u=https://fossil.etoilebsd.net/poudriere/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=ITD4dgEgVeGBJaxdFd9siyBYqr8XS3YLpKM1XB2BIlo%3D%0A&s=3f2fd1a60ccad617509daca0f40579fc7aa17cfc3a6ee176d38a4f2de3c913a0>
(2)  Build your own repository of packages

I assume that poudiere builds packages from ports.

That's not how we build package repositories here (and would expect that there 
are many more like us).

We expect to start with an empty directory, go grab packages that we want (one 
by one) from FTP. Then put those tarballs into a directory. There's no 
additional step because we download the INDEX file to the repository too. Local 
repository is built.

Is it reasonable to assume that this is still possible? We don't want to have 
to set up poudriere or build our own packages for packages that were already 
built at the time of release. (we don't track anything except security releng 
branches).



(3)  Make that repository available in a local file system or NFS


That's what we want.


I believe that "pkg" and "poudriere" address pretty much all the questions you 
have been asking,
so you just need to set up the environment so that you can do experiments to 
integrate it with bsdconfig.


But I want bsdconfig to work with local repositories without having poudriere.



PC-BSD and DragonFlyBSD have switched to using "pkg", so it is quite well 
tested.


Cool.


Baptiste and the rest of the pkgng and poudriere team have put a lot of effort 
and thought into this,
and I believe you will be pleased with the results once you migrate over to 
these tools.


I think so too. But right now a *lot* of unanswered questions.
--
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 13, 2013, at 9:24 PM, Peter Wemm wrote:

> On Sat, Jul 13, 2013 at 11:54 AM, Teske, Devin
>  wrote:
> 
>> So yes... I'm asking... in a HEAD world, what is the "officially supported" 
>> method of acquisition?
> 
> This has been answered elsewhere, but to be absolutely clear:
> 
> * 10.0 will ship with pkgng format packages.
> 

Ok. Respectfully not what I was inquiring about.


> * pkgng will be configured to pull via http from project
> infrastructure by default.
> 

Thanks. Good to know. So when you don't define PACKAGESITE, it defaults to HTTP.

How about rquery? What protocol does that use? and what does it talk to?



> * The project will not be providing pkg_install format packages via
> any method for 10.x.
> 

Yes, I know this from previous announcements (not the topic at-hand; 
respectfully).



> This doesn't stop third parties building and providing them
> unofficially but my understanding is that they won't be on any project
> operated freebsd.org sites.
> 

Question: Where can I learn more about the actual format of what's in the new 
tarballs? This is going to be important not for bsdconfig, but $work (we have 
our own build platform; I'm going to have to rewrite it from mastering PLIST 
files to mastering YAML MANIFEST files and I want to know all the gritty 
details).
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 13, 2013, at 4:51 PM, Mark Felder wrote:

> On Sat, Jul 13, 2013, at 13:54, Teske, Devin wrote:
>> 
>> 
>> If FTP access (or any of the other remote access methods) are going away
>> for HEAD pkg access, I'll need to know so I can make the appropriate
>> changes in the HEAD branch of bsdconfig.
>> 
> 
> It's simpler than you think. The new pkg uses libfetch. You can have
> your PACAKGESITE be file:// ftp:// http:// https:// ... 
> 

In bsdconfig (as one might expect), before a user installs a package, he or she 
is presented with a list of packages with descriptions (and perhaps other 
limited information).

ASIDE: For efficiency, I will actually need three things: (1) a list of all 
packages (2) their descriptions and (3) their run-time dependencies.

NOTE: I'll need run-time dependencies so that as they're checking boxes in the 
UI, I can update accordingly (has nothing to do with how the dependencies are 
handled when the packages get installed; I'll let pkg handle that when it comes 
time, but for the UI and for the "review" screen, I want the user to be 
informed and I can do that more efficiently if I have a master-list and don't 
have to make continuous queries).

That being said, I was planning on doing a "pkg rquery" to get that master list 
of [minimally] 3-pieces of information.

What protocol does that "rquery" run on? What would one have to do to set up 
their own server that can accept an "rquery"? (our customers don't have 
Internet access)

Last but not least...

Can you do an "rquery" on a local repository? (say, one that has been mounted 
via NFS or some other media, local or otherwise but looking like a local 
filesystem once-mounted). What would be required to get a local repository like 
that going?



> I do suspect that HTTP_PROXY support is probably not available as I
> recall seeing a post where someone was asking about that getting
> implemented for fetch. I'll have to research that again, though.

Thanks. Keep me up to date on that.

Neither sysinstall nor bsdinstall really give HTTP_PROXY access much thought 
(they support it, but only minimally). They just construct a raw HTTP header 
with the FTP URL and send that directly to the proxy. One cute thing it does 
(when initializing the connection) is test for Squid and if-so, appends 
";type=i" to the URL (to force binary download versus ascii).

No support for proxy-server authentication (however user/pass authentication 
for the FTP server is passed through to the proxy-server).
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 13, 2013, at 11:16 AM, Kimmo Paasiala wrote:

> On Sat, Jul 13, 2013 at 7:46 PM, Teske, Devin  
> wrote:
>> 
>> On Jul 13, 2013, at 8:17 AM, Teske, Devin wrote:
>> 
>> 
>> On Jul 13, 2013, at 1:07 AM, Baptiste Daroussin wrote:
>> 
>> On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote:
>> On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote:
>> 
>> Hi,
>> 
>> I have just committed (r253305) a change the make pkg_install not being built
>> and installed by default on HEAD.
>> 
>> If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your
>> src.conf(5)
>> 
>> [snip]
>> 
>> I for one am effected and will have to change things.
>> 
>> If you are referring to bsdconfig's package management,
>> 
>> [snip] Yes. that's what I'm talking about. [snip]
>> 
>> 
>> it is not working anyway
>> HEAD as we do not and will not provides any pkg_install for HEAD via any of 
>> the
>> usual distribution process: http, ftp, CD.
>> 
>> 
>> 
>> The FTP mirror of packages is going away? (if you said yes and pointed at a 
>> prior conversation about leading up to this, I would not be surprised -- I'm 
>> just questioning it because I don't see the value in pairing-down methods of 
>> acquisition)
>> 
>> If this is the case, what's the surviving acquisition method? A custom TCP 
>> protocol perhaps?
>> 
>> There may be those that wish to use pkg in the pkg_add manner and download 
>> things and then inspect them manually before adding them. For example, I 
>> often go the 
>> freshports.org<https://urldefense.proofpoint.com/v1/url?u=http://freshports.org/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=Ft5J2W3Nm8xze74zARHsiglLFTGOYrxAzaubbP7wvcM%3D%0A&s=cb1df85e237d1a6be19f981e8b352419dd056ea0f4e782b330cb9c7cfd15fda5>
>>  to find a package that fills a need... download it from the official FTP 
>> server(s), inspect all of them, and choose the one that best fits me need 
>> (and only then installing from the local file; tossing the rest). If I go 
>> through the "pkg" tool, I have to inspect things *after* they've been 
>> installed which is not to my satisfaction.
>> 
>> 
>> [snip
>> bsdconfig is not installed by
>> default,
>> 
>> Wrong, please see...
>> http://svnweb.freebsd.org/base?view=revision&revision=252862
>> [snip]
>> 
>> The first thing that comes to mind in reprogramming bsdconfig's package 
>> management for pkgng...
>> 
>> We have a *very* large list of FTP mirrors. This will presumably be replaced 
>> with a list of "pkg" mirrors.
>> 
>> Do we have such a list that we want to program into the base configuration 
>> of bsdconfig?
>> --
>> Devin
>> 
> 
> Come on, this only concerns 10-CURRENT.

Right; which is why -current is CC'd.


> Where is it stated that the
> FTP mirrors for FreeBSD 8/9 packages are going away?
> 

I don't know where you got the impression that I was concerned with 8/9 
packages.

I'm strictly concerned with HEAD and strictly concerned with planning for the 
future.

So yes... I'm asking... in a HEAD world, what is the "officially supported" 
method of acquisition?

I saw mention on a page that "rsync access will provided for those who want to 
mirror" but my I'm not really interested in mirroring while I would like to 
continue to be able to grab packages myself without installing them.

Maybe the answer is that "pkg" has a method of acquiring a single package 
(without dependencies) without downloading it. That would solve my personal 
workflow issue (again, I like to download tarballs, inspect them, and then 
install them from the locally downloaded file -- then if it passes validation 
tests, that downloaded file is migrated to our own distribution servers; it's a 
work-flow for validating packages -- it has less to do with how pkgng works and 
more just whether I can get packages in a fashion such as FTP or HTTP or 
whatever.

Now that aside, bsdconfig currently is a different story. To rewrite the 
packages module to work in a HEAD world for HEAD, using pkgng servers, I'm 
going to have to change the way things are done. Currently there is an 
abstraction layer for fetching packages from media (which can be FTP, HTTP, 
HTTP Proxy, NFS, Local Directory, CDROM, a DOS partition, a UFS partition, and 
[god forbid] Floppy -- with perhaps SMB/CIFS down the road). When it wants to 
add a package, it calls the routine to get the package data

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 13, 2013, at 8:17 AM, Teske, Devin wrote:


On Jul 13, 2013, at 1:07 AM, Baptiste Daroussin wrote:

On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote:
On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote:

Hi,

I have just committed (r253305) a change the make pkg_install not being built
and installed by default on HEAD.

If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your
src.conf(5)

[snip]

I for one am effected and will have to change things.

If you are referring to bsdconfig's package management,

[snip] Yes. that's what I'm talking about. [snip]


it is not working anyway
HEAD as we do not and will not provides any pkg_install for HEAD via any of the
usual distribution process: http, ftp, CD.



The FTP mirror of packages is going away? (if you said yes and pointed at a 
prior conversation about leading up to this, I would not be surprised -- I'm 
just questioning it because I don't see the value in pairing-down methods of 
acquisition)

If this is the case, what's the surviving acquisition method? A custom TCP 
protocol perhaps?

There may be those that wish to use pkg in the pkg_add manner and download 
things and then inspect them manually before adding them. For example, I often 
go the freshports.org<http://freshports.org> to find a package that fills a 
need... download it from the official FTP server(s), inspect all of them, and 
choose the one that best fits me need (and only then installing from the local 
file; tossing the rest). If I go through the "pkg" tool, I have to inspect 
things *after* they've been installed which is not to my satisfaction.


[snip
bsdconfig is not installed by
default,

Wrong, please see...
http://svnweb.freebsd.org/base?view=revision&revision=252862
[snip]

The first thing that comes to mind in reprogramming bsdconfig's package 
management for pkgng...

We have a *very* large list of FTP mirrors. This will presumably be replaced 
with a list of "pkg" mirrors.

Do we have such a list that we want to program into the base configuration of 
bsdconfig?
--
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 13, 2013, at 1:07 AM, Baptiste Daroussin wrote:

On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote:
On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote:

Hi,

I have just committed (r253305) a change the make pkg_install not being built
and installed by default on HEAD.

If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your
src.conf(5)

I think while a good move, we need to allow a window of development to re-work 
other HEAD components.

It might also be worth developing a lint-tool to make sure we get every last 
instance (both from C code and sh code) within our base before we motion to cut 
a release with this change.

I for one am effected and will have to change things.

If you are referring to bsdconfig's package management,

If you choose to read into it that way, sure. Yes. that's what I'm talking 
about.

NOTE: I can't be the only person in HEAD that's using pkg_install still.



it is not working anyway
HEAD as we do not and will not provides any pkg_install for HEAD via any of the
usual distribution process: http, ftp, CD.


I'm quite aware that changes will be needed. I'm still interested in a 
"transition wiki" if one exists.




Secondly if I'm properly reading src.conf.5

If there's a reference to "WITH_BSDCONFIG" in src.conf.5, that needs to be 
removed. I wasn't aware that it was referenced in there.


bsdconfig is not installed by
default,

Wrong, please see...
http://svnweb.freebsd.org/base?view=revision&revision=252862



which means we have a non default code depending on a now non default
code,

Wrong.


I don't see a problem here, should I?


Yes.
--
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [HEADSUP] No more pkg_install on HEAD by default

[Teske, Devin]

On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote:

> Hi,
> 
> I have just committed (r253305) a change the make pkg_install not being built
> and installed by default on HEAD.
> 
> If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your
> src.conf(5)

I think while a good move, we need to allow a window of development to re-work 
other HEAD components.

It might also be worth developing a lint-tool to make sure we get every last 
instance (both from C code and sh code) within our base before we motion to cut 
a release with this change.

I for one am effected and will have to change things.

Can you point us at a guide (or even better, a Wiki) that can smooth the 
process?
-- 
Devin

NOTE: I cut ports off the CC as they don't need to worry about this ("this" 
being heading toward cut of a RELEASE with untold uses of pkg_tools in base)

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

[CFT] sysutils/bsdconfg (0.9.0) and sysutils/sysrc (5.2)

[Teske, Devin]

Hey all,

With SVN r252862, bsdconfig(8) and sysrc(8) are alive in HEAD.

They are for the most-part vestigial organs that will be enmeshed further as 
they breathe fresh-air, but for now they certainly won't cause the build the 
break (100% shell; the only thing that gets "compiled" are the man-pages) and 
won't disturb anything else.

What's been unfurled into HEAD is exactly the same as what's just been released 
to ports:

sysutils/bsdconfig (latest version 0.9.0)
sysutils/sysrc (latest version 5.2)

My plan is to merge these to stable/9 in 3 days. Reasons for merge are stated 
in above-revision log-message (mail will be sent to releng and anybody else 
that cares to receive it or redirect it).

This e-mail is about testing.

I have been testing on 9.0-R all this time and there should be no problems with 
9.1. But if some brave souls will install the port onto 9.1-R and give it a go, 
that would be great (this is, afterall, a Call For Testing).

If you test bsdconfig on anything higher than 9.1-R (say, 9.1-STABLE), package 
management may not be kind to you (but there's a work-around, go into the 
options dialog and set your release as 9.1-RELEASE). This has to do with the 
way binary packages are stored on the FTP server (of course, if you have your 
own local repository, success depends on your ability to make the repository 
look "official" -- just as you had to with sysinstall).

No bother testing bsdconfig on anything older than 9.0-R (while sysrc requires 
only 4.x or higher).

Thanks all so much in-advance.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FreeBSD-10.0 code freeze

[Teske, Devin]

On May 27, 2013, at 6:33 AM, Oded Shanoon wrote:

> Hi,
> 
> I heard a rumor that the 10.0 code freeze will be in the end of July.
> Is that true?

No.

The release engineering team will shift focus to 9.2-R after the current focus 
of 8.4-R.



> Can anyone tell me when does it plan to be?
> 

10 will be the focal point sometime after 9.2-R.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [UPDATE] sysutils/bsdconfig snapshot

[Teske, Devin]

(replying to my own request with some additional details on what to try in this 
update)

On May 7, 2013, at 1:36 PM, Teske, Devin wrote:

> Hello,
> 
> I've taken a new snapshot of HEAD usr.sbin/bsdconfig and made it available 
> through the ports tree (as sysutils/bsdconfig). The last snapshot was quite 
> some time months ago, so a lot has changed since then.
> 

Files have moved, there's now a /usr/share/bsdconfig directory for shared 
include files, and the fake language files have been removed. That's what 
you'll notice from what the new port installs.


> Most notably, we now have the beginnings of the package management module, 
> edging ever-closer to a 1.0 release status.
> 

Of which it's important to note that this initial work is based on the old 
pkg_tools and not pkgng. We'll convert it to pkgng later, but first we need to 
make sure the framework for interacting with huge package data is usable.

One of the challenges in testing this feature is that on 9.1, there aren't any 
packages available (yet; that should change eventually -- being restored 
retroactively).

However, if you don't use the "FTP" media types, you'll get a lot further. What 
I've been using as a test is to download the 9.0-RELEASE packages/INDEX file 
and throw that into a local "packages" directory and then point the "Directory" 
media type at the parent directory where the "packages" directory was created.

For example, …

cd /tmp
mkdir packages
fetch 
ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/9.0-RELEASE/packages/INDEX

Then use the "Directory" media type and enter a value of "/tmp". The 
"packages/INDEX" file will be found in the media ("/tmp" directory) and the the 
packages interface will then be presented.

If you're already running 9.0-RELEASE, then you can choose the "FTP Passive" 
media type, then choose the second menu item title "URL" and enter the URL 
"ftp://ftp-archive.freebsd.org";.

ASIDE: /me notes that he really ought to change the "FTP" routine to use a 
default $FTPMODE of "auto" instead of "active" (currently, "FTP" means 
explicitly active, and "FTP Passive" means explicitly passive).

Once you're in the package interface, you will see:

1. List of package categories. NOTE: For a package category to be listed, it 
has to have at least one package. This list also includes a dummy category 
named "All".

2. For each category (including the "All" dummy category), the number of:

2.a. packages available in that category

2.b. packages that are installed from that category

2.c. "selected" packages (a "selected" package is one that has been been marked 
for install, re-install, un-install, or marked as a dependency of a package to 
install/re-install)

are displayed.

3. Above the category list is a "> Review" menu item. Use this to review the 
selections you've made.

Choosing a category intuitively displays the packages available from that 
category.

In the list of packages, packages that are already installed are marked with an 
"X"; those marked for install have an "I"; re-install have an "R"; un-install, 
"U"; and dependencies are marked with a "D".

Selecting a package that is not marked will turn the mark into an "I".

Selecting a package that is marked with an "X" will bring up a menu asking if 
you would like to mark it for Uninstall, Re-install, or return it back to just 
being installed.

Selecting a package that is marked as a dependency turns it into a package 
requested for installation (D mark becomes I mark). If without leaving the 
package selection screen you then re-select the same package (which was a 
dependency turned into an "I"), the mark goes away, but if you exit and return 
to the package selection menu, the mark is restored as a "D". This is because 
package dependencies are not calculated until you return to the category list 
(when counts must be updated).

When you're navigating the menus, you may notice an asterisk (*) on some menu 
items. This indicates that a page has been "cached". Unlike sysinstall which 
would spend ~10 minutes (on my hardware) processing the INDEX file, building 
the in-memory structures for all category submenus, bsdconfig only generates 
the menus you request, and does them on-the-fly.

The first time you are presented with the list of package categories, no 
category has an asterisk next to it. The first time you enter a package 
category, this is a small 1-5 second delay (depending on hardware) to generate 
the menu(s) for that category. When you return to the category list, that 
category will now have an aster

[UPDATE] sysutils/bsdconfig snapshot

[Teske, Devin]

Hello,

I've taken a new snapshot of HEAD usr.sbin/bsdconfig and made it available 
through the ports tree (as sysutils/bsdconfig). The last snapshot was quite 
some time months ago, so a lot has changed since then.

Most notably, we now have the beginnings of the package management module, 
edging ever-closer to a 1.0 release status.

I'd like to see if there are any interested folks out there that could give my 
updated version a go and provide some feedback. Any/all feedback would be 
greatly appreciated.

Just an FYI however… this code is only expected to work on 9.0-R or higher.
-- 
Cheers,
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"