Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
> On 27 September 2018 at 06:46, tech-lists wrote: > > > > So, I want to know where and when each system was compiled. > > Why lose this information by default? > > This comes down to the simple fact that our build / release process > does not currently distinguish between building a world or kernel > that's destined for a binary release, and building for a bespoke > install. Right now the release process inherently builds with knobs > set at their defaults. > > The original plan would have flipped the default on the branch as part > of the release process, but there was a concern we might miss > something so it was enabled in advance of the branch. And as further information the plan is to flip this back off in ^head after the stable/12 branch is made, and then again, flip it off in stable/12 after the releng/12.0 branch is made, thus returning everything back to how it was, except in releng branch. -- Rod Grimes rgri...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
On 27 September 2018 at 06:46, tech-lists wrote: > > So, I want to know where and when each system was compiled. > Why lose this information by default? This comes down to the simple fact that our build / release process does not currently distinguish between building a world or kernel that's destined for a binary release, and building for a bespoke install. Right now the release process inherently builds with knobs set at their defaults. The original plan would have flipped the default on the branch as part of the release process, but there was a concern we might miss something so it was enabled in advance of the branch. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
On 11/09/2018 20:35, Ed Maste wrote: On 11 September 2018 at 07:35, Tomoaki AOKI wrote: I prefer releng, rather than stable, to make it default. Binary releases requiring reproducible builds are built from release and releng branches. This might be the reasonable long-term strategy, but we don't yet have experience running through the release process with it enabled. I would like to enable it by default on the branch, at least initially, to avoid discovering issues only immediately prior to the release. Hi, Personally I think this should (after testing on -current) be enabled only where binary-only updates (for everything) are anticipated. Then again, I don't run a binary-only system despite having to manage more than 16 systems. One reason is the hardware is all different, so different things are enabled in the kernel. The other reason is that I can reduce a machines security overhead if only what is required is available. This all requires source builds. So, I want to know where and when each system was compiled. Why lose this information by default? thanks, -- J. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
> On 11 September 2018 at 07:35, Tomoaki AOKI wrote: > > I prefer releng, rather than stable, to make it default. > > Binary releases requiring reproducible builds are built from > > release and releng branches. > > This might be the reasonable long-term strategy, but we don't yet have > experience running through the release process with it enabled. I > would like to enable it by default on the branch, at least initially, > to avoid discovering issues only immediately prior to the release. I wish we had done this before ALPHA1 it would of given us a larger window to work in. There should be a set of builds Thursday, can we get this turned on for them so we can get at least a build with it before we branch? IE, commit this to ^head/ now. Then once stable/12 is branched we can turn it off in head so the developers are not disturbed. And further then once releng/12.0 is branched we can turn it off in stable/12 so that those users have status quo. This I think gives us maximal test time, including the binary upgrade bits that should get tested between each BETA and RC build. And minimal impact to developers and users. -- Rod Grimes rgri...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
On 11 September 2018 at 07:35, Tomoaki AOKI wrote: > I prefer releng, rather than stable, to make it default. > Binary releases requiring reproducible builds are built from > release and releng branches. This might be the reasonable long-term strategy, but we don't yet have experience running through the release process with it enabled. I would like to enable it by default on the branch, at least initially, to avoid discovering issues only immediately prior to the release. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
I prefer releng, rather than stable, to make it default. Binary releases requiring reproducible builds are built from release and releng branches. On Mon, 10 Sep 2018 10:56:14 -0400 Ed Maste wrote: > The FreeBSD base system is a reproducible build[1] with a minor > exception: the build metadata (timestamps, user, hostname, etc.) > included in the kernel and loader. > > With the default, non-reproducible build the kernel ident looks like: > > FreeBSD 12.0-ALPHA5 #4 r338195: Mon Jan 1 10:11:12 EDT 2018 >user@hostname:/path/to/freebsd/src > > and the loader ident: > > FreeBSD/amd64 EFI loader, Revision 1.1 > (Mon Jan 1 10:11:12 EDT 2018 user@hostname) > > With reproducible builds enabled the kernel ident looks like: > > FreeBSD 12.0-ALPHA5 r338195 > > and the loader ident: > > FreeBSD/amd64 EFI loader, Revision 1.1 > > I would like to enable the REPRODUCIBLE_BUILD knob by default for the > 12.0 release, and propose we do this by adding a step to switch the > default to the list of changes[2] that re@ commits to the branch as > part of the release process. > > [1] https://reproducible-builds.org > [2] > https://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-releng/releng-head.html > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" > -- Tomoaki AOKI ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
On 9/10/18 10:55 AM, Rodney W. Grimes wrote: >> On 9/10/18 9:51 AM, Rodney W. Grimes wrote: The FreeBSD base system is a reproducible build[1] with a minor exception: the build metadata (timestamps, user, hostname, etc.) included in the kernel and loader. With the default, non-reproducible build the kernel ident looks like: FreeBSD 12.0-ALPHA5 #4 r338195: Mon Jan 1 10:11:12 EDT 2018 user@hostname:/path/to/freebsd/src and the loader ident: FreeBSD/amd64 EFI loader, Revision 1.1 (Mon Jan 1 10:11:12 EDT 2018 user@hostname) With reproducible builds enabled the kernel ident looks like: FreeBSD 12.0-ALPHA5 r338195 and the loader ident: FreeBSD/amd64 EFI loader, Revision 1.1 I would like to enable the REPRODUCIBLE_BUILD knob by default for the 12.0 release, and propose we do this by adding a step to switch the default to the list of changes[2] that re@ commits to the branch as part of the release process. >>> >>> Why not just turn this on and leave it on? >> >> For kernels not built against a pristine tree the extra info is useful to >> have. For better or worse, kgdb also parses the path to try to find >> kernel.full (used by e.g. 'kgdb -n last'), so if you remove the path it >> won't be able to find the matching kernel using its current logic. > > So this means stable/12 users would have hassles getting kgdb to work? No, this means that if you turn this option on in HEAD and leave it always on (as I read your mail to say), then it would be a hassle for developers on head. On stable branches it would be nice to keep the info if people are building kernels that aren't stock kernels (meaning modified source trees). For release kernels, crashinfo should work fine though even with the extra information stripped. For release builds the information is not really useful, it's only ever useful if someone is building their own kernel for some reason (and even in some of those cases it isn't all that useful). -- John Baldwin ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
On 10 September 2018 at 14:52, Ed Maste wrote: > > I brought this up on -arch in 2015... That said, the kgdb -n issue was brought up in the old thread and it seems I did forget about it. I don't think we should cater too much to the needs of the deprecated in-tree kgdb, but we should make sure that this works with kgdb from the port. There are now standardized ways to specify this information that we should migrate to, rather than the current approach. Note that here I'm really only interested in what we do for FreeBSD binary releases, and the path for kgdb -n etc. isn't really relevant in this case. Src tree defaults are in scope only because release settings and src tree defaults are coupled. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
On 10 September 2018 at 13:57, Rodney W. Grimes wrote: >> >> I know a number of developers want to keep the metadata for their own >> builds at least. > > And we do not really know what the users position is on this... If users are building FreeBSD from source they can set the knob however they like. Users of our prebuilt binares/releases are the ones who benefit from reproducible builds, and they're the ones unable to set it. > I think there is more discussion to be had before we flip > this knob anyplace. I brought this up on -arch in 2015, and gave presentations on reproducible builds in FreeBSD at BSDCan 2016 and AsiaBSDCon 2017. I believe we've had ample opportunity to discuss this in the context of what makes sense in a binary release, but there are still reasonable open questions about defaults in HEAD. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
> On 10 September 2018 at 12:51, Rodney W. Grimes > wrote: > > > > Why not just turn this on and leave it on? > > I know a number of developers want to keep the metadata for their own > builds at least. And we do not really know what the users position is on this... and developers also run on stable/X, they may not like this flipped there, though IMHO we should not be catering so much to developers, while shooting users. > > We have essentially three different levels of metadata that are > arguably sensible: > > 1. Major/minor version, release/branch name, architecture > 2. Version control information > 3. Path, user, date, time, host > > And three kinds of working trees: > > 1. Non-versioned with/without modifications (e.g., a src tarball) > 2. git/svn/other checkout, without modifications > 3. git/svn/other checkout, with modifications > > What I'm proposing for 12.0 gives us 1+2 always (regardless of the > state of the tree). > > I think there's more discussion to be had on the mapping between the > tree type/state and amount of metadata to include. If we come to a > consensus I'll handle it, but don't want to hold up a change destined > for 12.0 with a broader discussion. I think there is more discussion to be had before we flip this knob anyplace. -- Rod Grimes rgri...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
> On 9/10/18 9:51 AM, Rodney W. Grimes wrote: > >> The FreeBSD base system is a reproducible build[1] with a minor > >> exception: the build metadata (timestamps, user, hostname, etc.) > >> included in the kernel and loader. > >> > >> With the default, non-reproducible build the kernel ident looks like: > >> > >> FreeBSD 12.0-ALPHA5 #4 r338195: Mon Jan 1 10:11:12 EDT 2018 > >>user@hostname:/path/to/freebsd/src > >> > >> and the loader ident: > >> > >> FreeBSD/amd64 EFI loader, Revision 1.1 > >> (Mon Jan 1 10:11:12 EDT 2018 user@hostname) > >> > >> With reproducible builds enabled the kernel ident looks like: > >> > >> FreeBSD 12.0-ALPHA5 r338195 > >> > >> and the loader ident: > >> > >> FreeBSD/amd64 EFI loader, Revision 1.1 > >> > >> I would like to enable the REPRODUCIBLE_BUILD knob by default for the > >> 12.0 release, and propose we do this by adding a step to switch the > >> default to the list of changes[2] that re@ commits to the branch as > >> part of the release process. > > > > Why not just turn this on and leave it on? > > For kernels not built against a pristine tree the extra info is useful to > have. For better or worse, kgdb also parses the path to try to find > kernel.full (used by e.g. 'kgdb -n last'), so if you remove the path it > won't be able to find the matching kernel using its current logic. So this means stable/12 users would have hassles getting kgdb to work? > crashinfo uses different logic so will still work fine (crashinfo looks > for all the things matching /boot/*/kernel and tries them all until it finds > a match). > > -- > John Baldwin > > > -- Rod Grimes rgri...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
On 10 September 2018 at 12:51, Rodney W. Grimes wrote: > > Why not just turn this on and leave it on? I know a number of developers want to keep the metadata for their own builds at least. We have essentially three different levels of metadata that are arguably sensible: 1. Major/minor version, release/branch name, architecture 2. Version control information 3. Path, user, date, time, host And three kinds of working trees: 1. Non-versioned with/without modifications (e.g., a src tarball) 2. git/svn/other checkout, without modifications 3. git/svn/other checkout, with modifications What I'm proposing for 12.0 gives us 1+2 always (regardless of the state of the tree). I think there's more discussion to be had on the mapping between the tree type/state and amount of metadata to include. If we come to a consensus I'll handle it, but don't want to hold up a change destined for 12.0 with a broader discussion. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
On 9/10/18 9:51 AM, Rodney W. Grimes wrote: >> The FreeBSD base system is a reproducible build[1] with a minor >> exception: the build metadata (timestamps, user, hostname, etc.) >> included in the kernel and loader. >> >> With the default, non-reproducible build the kernel ident looks like: >> >> FreeBSD 12.0-ALPHA5 #4 r338195: Mon Jan 1 10:11:12 EDT 2018 >>user@hostname:/path/to/freebsd/src >> >> and the loader ident: >> >> FreeBSD/amd64 EFI loader, Revision 1.1 >> (Mon Jan 1 10:11:12 EDT 2018 user@hostname) >> >> With reproducible builds enabled the kernel ident looks like: >> >> FreeBSD 12.0-ALPHA5 r338195 >> >> and the loader ident: >> >> FreeBSD/amd64 EFI loader, Revision 1.1 >> >> I would like to enable the REPRODUCIBLE_BUILD knob by default for the >> 12.0 release, and propose we do this by adding a step to switch the >> default to the list of changes[2] that re@ commits to the branch as >> part of the release process. > > Why not just turn this on and leave it on? For kernels not built against a pristine tree the extra info is useful to have. For better or worse, kgdb also parses the path to try to find kernel.full (used by e.g. 'kgdb -n last'), so if you remove the path it won't be able to find the matching kernel using its current logic. crashinfo uses different logic so will still work fine (crashinfo looks for all the things matching /boot/*/kernel and tries them all until it finds a match). -- John Baldwin ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
> The FreeBSD base system is a reproducible build[1] with a minor > exception: the build metadata (timestamps, user, hostname, etc.) > included in the kernel and loader. > > With the default, non-reproducible build the kernel ident looks like: > > FreeBSD 12.0-ALPHA5 #4 r338195: Mon Jan 1 10:11:12 EDT 2018 >user@hostname:/path/to/freebsd/src > > and the loader ident: > > FreeBSD/amd64 EFI loader, Revision 1.1 > (Mon Jan 1 10:11:12 EDT 2018 user@hostname) > > With reproducible builds enabled the kernel ident looks like: > > FreeBSD 12.0-ALPHA5 r338195 > > and the loader ident: > > FreeBSD/amd64 EFI loader, Revision 1.1 > > I would like to enable the REPRODUCIBLE_BUILD knob by default for the > 12.0 release, and propose we do this by adding a step to switch the > default to the list of changes[2] that re@ commits to the branch as > part of the release process. Why not just turn this on and leave it on? > > [1] https://reproducible-builds.org > [2] > https://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-releng/releng-head.html -- Rod Grimes rgri...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
Hi Ed, I think that sounds great. In the future, could we go even further and, by default, only emit date/user/path if the source tree is “dirty” with respect to SVN? If the build really is reproducible, that data should only be informative when building something that doesn’t match a FreeBSD SVN revision (e.g., a Git commit in a local repo or a tree with local changes). Cheers, Jon -- jonat...@freebsd.org On 10 Sep 2018, at 12:26, Ed Maste wrote: The FreeBSD base system is a reproducible build[1] with a minor exception: the build metadata (timestamps, user, hostname, etc.) included in the kernel and loader. With the default, non-reproducible build the kernel ident looks like: FreeBSD 12.0-ALPHA5 #4 r338195: Mon Jan 1 10:11:12 EDT 2018 user@hostname:/path/to/freebsd/src and the loader ident: FreeBSD/amd64 EFI loader, Revision 1.1 (Mon Jan 1 10:11:12 EDT 2018 user@hostname) With reproducible builds enabled the kernel ident looks like: FreeBSD 12.0-ALPHA5 r338195 and the loader ident: FreeBSD/amd64 EFI loader, Revision 1.1 I would like to enable the REPRODUCIBLE_BUILD knob by default for the 12.0 release, and propose we do this by adding a step to switch the default to the list of changes[2] that re@ commits to the branch as part of the release process. [1] https://reproducible-builds.org [2] https://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-releng/releng-head.html ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
On 10 September 2018 at 11:11, Jonathan Anderson wrote: > Hi Ed, > > I think that sounds great. In the future, could we go even further and, by > default, only emit date/user/path if the source tree is “dirty” with respect > to SVN? If the build really is reproducible, that data should only be > informative when building something that doesn’t match a FreeBSD SVN > revision (e.g., a Git commit in a local repo or a tree with local changes). Indeed, and I already have that support in newvers.sh: -r means build reproducibly and -R means build reproducibly if the source tree represents an unmodified checkout from a version control system. It's currently not hooked up to a src.conf knob, and we don't really have a great way to handle options with more than two states. We could have REPRODUCIBLE_BUILD always on by default, using the -R mode - something to revisit after we're done with the 12.0 process. Also, note that the build will currently not be reproducible between svn/git/tarball because the version control info is included in the ident strings. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Enabling the WITH_REPRODUCIBLE_BUILD knob for 12.0-REL
On Mon, Sep 10, 2018 at 8:58 AM Ed Maste wrote: > The FreeBSD base system is a reproducible build[1] with a minor > exception: the build metadata (timestamps, user, hostname, etc.) > included in the kernel and loader. > > With the default, non-reproducible build the kernel ident looks like: > > FreeBSD 12.0-ALPHA5 #4 r338195: Mon Jan 1 10:11:12 EDT 2018 >user@hostname:/path/to/freebsd/src > > and the loader ident: > > FreeBSD/amd64 EFI loader, Revision 1.1 > (Mon Jan 1 10:11:12 EDT 2018 user@hostname) > > With reproducible builds enabled the kernel ident looks like: > > FreeBSD 12.0-ALPHA5 r338195 > > and the loader ident: > > FreeBSD/amd64 EFI loader, Revision 1.1 > > I would like to enable the REPRODUCIBLE_BUILD knob by default for the > 12.0 release, and propose we do this by adding a step to switch the > default to the list of changes[2] that re@ commits to the branch as > part of the release process. > > [1] https://reproducible-builds.org > [2] > https://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-releng/releng-head.html Turning it on, like we turn off WITNESS, for stable branches, I think this is a good idea. The loader doesn't really need the extra stuff to function properly, so dropping it is OK. Warner ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
