Re: [OE-core] [PATCH 7/7] reproducible: Drop BUILD_REPRODUCIBLE_BINARIES variable
On Thu, 2021-10-14 at 09:46 -0400, Bruce Ashfield wrote:
> On Thu, Oct 14, 2021 at 9:45 AM Alexandre Belloni
> wrote:
> >
> > On 14/10/2021 08:38:46-0400, Bruce Ashfield wrote:
> > > On Thu, Oct 14, 2021 at 8:32 AM Richard Purdie
> > > wrote:
> > > >
> > > > On Thu, 2021-10-14 at 08:28 -0400, Bruce Ashfield wrote:
> > > > > On Thu, Oct 14, 2021 at 8:10 AM Richard Purdie
> > > > > wrote:
> > > > > >
> > > > > > We want things to be reproduicble and the variable doesn't really
> > > > > > change
> > > > > > much any more. Drop the remaining uses and make those code paths
> > > > > > always
> > > > > > active.
> > > > >
> > > > > It wasn't clear to me from reading the patch. What is the way that
> > > > > someone would now get the current timestamp into a kernel build, if
> > > > > that's the behaviour that they want ?
> > > >
> > > > With this change they probably don't get the option. If we want that to
> > > > be
> > > > configurable, we should probably move the control to a different
> > > > variable which
> > > > is focused specifically on the kernel. The hardest bit is probably
> > > > picking a
> > > > name!
> > >
> > > This is probably the most surprising feature of reproducibility for
> > > kernel developers, and a question that I've gotten multiple times
> > > ("what happened to my timestamp ?" followed by "how do I turn this
> > > off?")
> > >
> > > Most kernel developers already don't really like the "yocto workflow",
> > > and this adds another element in that category. It is common in a
> > > debug scenario to check the timestamp of the running kernel to make
> > > sure that you've actually booted the one you just built.
> > >
> >
> > But do kernel developers actually do their development using YP?
> > I definitively not. I do all my BSP work outside of any build system and
> > then I finally integrate everything once this is working/ready.
>
> Many do, yup!
I've sent out a new version of this patch with an additional one which gives
people a setting to control the timestamp for the kernel.
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156961):
https://lists.openembedded.org/g/openembedded-core/message/156961
Mute This Topic: https://lists.openembedded.org/mt/86310659/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 7/7] reproducible: Drop BUILD_REPRODUCIBLE_BINARIES variable
On Thu, Oct 14, 2021 at 9:45 AM Alexandre Belloni
wrote:
>
> On 14/10/2021 08:38:46-0400, Bruce Ashfield wrote:
> > On Thu, Oct 14, 2021 at 8:32 AM Richard Purdie
> > wrote:
> > >
> > > On Thu, 2021-10-14 at 08:28 -0400, Bruce Ashfield wrote:
> > > > On Thu, Oct 14, 2021 at 8:10 AM Richard Purdie
> > > > wrote:
> > > > >
> > > > > We want things to be reproduicble and the variable doesn't really
> > > > > change
> > > > > much any more. Drop the remaining uses and make those code paths
> > > > > always
> > > > > active.
> > > >
> > > > It wasn't clear to me from reading the patch. What is the way that
> > > > someone would now get the current timestamp into a kernel build, if
> > > > that's the behaviour that they want ?
> > >
> > > With this change they probably don't get the option. If we want that to be
> > > configurable, we should probably move the control to a different variable
> > > which
> > > is focused specifically on the kernel. The hardest bit is probably
> > > picking a
> > > name!
> >
> > This is probably the most surprising feature of reproducibility for
> > kernel developers, and a question that I've gotten multiple times
> > ("what happened to my timestamp ?" followed by "how do I turn this
> > off?")
> >
> > Most kernel developers already don't really like the "yocto workflow",
> > and this adds another element in that category. It is common in a
> > debug scenario to check the timestamp of the running kernel to make
> > sure that you've actually booted the one you just built.
> >
>
> But do kernel developers actually do their development using YP?
> I definitively not. I do all my BSP work outside of any build system and
> then I finally integrate everything once this is working/ready.
Many do, yup!
Bruce
>
> It did indeed surprise me the first time I saw that but this doesn't
> have any actual effect on my workflow. Also, the issue was already
> existing if what you were trying to build was already in the sstate
> cache (e.G from a nightly build), it would contain the time of the
> actual build.
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156959):
https://lists.openembedded.org/g/openembedded-core/message/156959
Mute This Topic: https://lists.openembedded.org/mt/86310659/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 7/7] reproducible: Drop BUILD_REPRODUCIBLE_BINARIES variable
On 14/10/2021 08:38:46-0400, Bruce Ashfield wrote:
> On Thu, Oct 14, 2021 at 8:32 AM Richard Purdie
> wrote:
> >
> > On Thu, 2021-10-14 at 08:28 -0400, Bruce Ashfield wrote:
> > > On Thu, Oct 14, 2021 at 8:10 AM Richard Purdie
> > > wrote:
> > > >
> > > > We want things to be reproduicble and the variable doesn't really change
> > > > much any more. Drop the remaining uses and make those code paths always
> > > > active.
> > >
> > > It wasn't clear to me from reading the patch. What is the way that
> > > someone would now get the current timestamp into a kernel build, if
> > > that's the behaviour that they want ?
> >
> > With this change they probably don't get the option. If we want that to be
> > configurable, we should probably move the control to a different variable
> > which
> > is focused specifically on the kernel. The hardest bit is probably picking a
> > name!
>
> This is probably the most surprising feature of reproducibility for
> kernel developers, and a question that I've gotten multiple times
> ("what happened to my timestamp ?" followed by "how do I turn this
> off?")
>
> Most kernel developers already don't really like the "yocto workflow",
> and this adds another element in that category. It is common in a
> debug scenario to check the timestamp of the running kernel to make
> sure that you've actually booted the one you just built.
>
But do kernel developers actually do their development using YP?
I definitively not. I do all my BSP work outside of any build system and
then I finally integrate everything once this is working/ready.
It did indeed surprise me the first time I saw that but this doesn't
have any actual effect on my workflow. Also, the issue was already
existing if what you were trying to build was already in the sstate
cache (e.G from a nightly build), it would contain the time of the
actual build.
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156957):
https://lists.openembedded.org/g/openembedded-core/message/156957
Mute This Topic: https://lists.openembedded.org/mt/86310659/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 7/7] reproducible: Drop BUILD_REPRODUCIBLE_BINARIES variable
On Thu, Oct 14, 2021 at 8:32 AM Richard Purdie
wrote:
>
> On Thu, 2021-10-14 at 08:28 -0400, Bruce Ashfield wrote:
> > On Thu, Oct 14, 2021 at 8:10 AM Richard Purdie
> > wrote:
> > >
> > > We want things to be reproduicble and the variable doesn't really change
> > > much any more. Drop the remaining uses and make those code paths always
> > > active.
> >
> > It wasn't clear to me from reading the patch. What is the way that
> > someone would now get the current timestamp into a kernel build, if
> > that's the behaviour that they want ?
>
> With this change they probably don't get the option. If we want that to be
> configurable, we should probably move the control to a different variable
> which
> is focused specifically on the kernel. The hardest bit is probably picking a
> name!
This is probably the most surprising feature of reproducibility for
kernel developers, and a question that I've gotten multiple times
("what happened to my timestamp ?" followed by "how do I turn this
off?")
Most kernel developers already don't really like the "yocto workflow",
and this adds another element in that category. It is common in a
debug scenario to check the timestamp of the running kernel to make
sure that you've actually booted the one you just built.
>
> You don't really want to turn off all reproducibility everywhere for just the
> kernel timestamping change.
Agreed.
And agreed on picking the name, could it be something around
KERNEL_DEBUG ? And we could even put a bbnote (that they won't see,
but that's the best I can think of) that indicates that
reproducibility is on and the timestamp is set to .. and the
opposite when reproducibility is off ?
Bruce
>
> Cheers,
>
> Richard
>
--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156950):
https://lists.openembedded.org/g/openembedded-core/message/156950
Mute This Topic: https://lists.openembedded.org/mt/86310659/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 7/7] reproducible: Drop BUILD_REPRODUCIBLE_BINARIES variable
On Thu, 2021-10-14 at 08:28 -0400, Bruce Ashfield wrote: > On Thu, Oct 14, 2021 at 8:10 AM Richard Purdie > wrote: > > > > We want things to be reproduicble and the variable doesn't really change > > much any more. Drop the remaining uses and make those code paths always > > active. > > It wasn't clear to me from reading the patch. What is the way that > someone would now get the current timestamp into a kernel build, if > that's the behaviour that they want ? With this change they probably don't get the option. If we want that to be configurable, we should probably move the control to a different variable which is focused specifically on the kernel. The hardest bit is probably picking a name! You don't really want to turn off all reproducibility everywhere for just the kernel timestamping change. Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#156949): https://lists.openembedded.org/g/openembedded-core/message/156949 Mute This Topic: https://lists.openembedded.org/mt/86310659/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 7/7] reproducible: Drop BUILD_REPRODUCIBLE_BINARIES variable
On Thu, Oct 14, 2021 at 8:10 AM Richard Purdie
wrote:
>
> We want things to be reproduicble and the variable doesn't really change
> much any more. Drop the remaining uses and make those code paths always
> active.
It wasn't clear to me from reading the patch. What is the way that
someone would now get the current timestamp into a kernel build, if
that's the behaviour that they want ?
Bruce
>
> Signed-off-by: Richard Purdie
> ---
> meta/classes/image-prelink.bbclass| 13 ++-
> meta/classes/image.bbclass| 14 +++
> meta/classes/kernel.bbclass | 54 +--
> meta/conf/bitbake.conf| 1 -
> meta/lib/oe/sstatesig.py | 2 +-
> meta/recipes-core/busybox/busybox.inc | 10 ++---
> 6 files changed, 41 insertions(+), 53 deletions(-)
>
> diff --git a/meta/classes/image-prelink.bbclass
> b/meta/classes/image-prelink.bbclass
> index 0da094a5518..8158eeaf4ca 100644
> --- a/meta/classes/image-prelink.bbclass
> +++ b/meta/classes/image-prelink.bbclass
> @@ -46,17 +46,12 @@ prelink_image () {
> dynamic_loader=${@get_linuxloader(d)}
>
> # prelink!
> - if [ "${BUILD_REPRODUCIBLE_BINARIES}" = "1" ]; then
> - bbnote " prelink: BUILD_REPRODUCIBLE_BINARIES..."
> - if [ "$REPRODUCIBLE_TIMESTAMP_ROOTFS" = "" ]; then
> - export PRELINK_TIMESTAMP=`git log -1 --pretty=%ct `
> - else
> - export
> PRELINK_TIMESTAMP=$REPRODUCIBLE_TIMESTAMP_ROOTFS
> - fi
> - ${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -am
> -N -c ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
> + if [ "$REPRODUCIBLE_TIMESTAMP_ROOTFS" = "" ]; then
> + export PRELINK_TIMESTAMP=`git log -1 --pretty=%ct `
> else
> - ${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -amR
> -N -c ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
> + export PRELINK_TIMESTAMP=$REPRODUCIBLE_TIMESTAMP_ROOTFS
> fi
> + ${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -am -N -c
> ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
>
> # Remove the prelink.conf if we had to add it.
> if [ "$dummy_prelink_conf" = "true" ]; then
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index c2f32320278..2fa69a40d10 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -651,17 +651,15 @@ ROOTFS_PREPROCESS_COMMAND +=
> "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge'
> POPULATE_SDK_PRE_TARGET_COMMAND += "${@bb.utils.contains('DISTRO_FEATURES',
> 'usrmerge', 'create_merged_usr_symlinks_sdk; ', '',d)}"
>
> reproducible_final_image_task () {
> -if [ "${BUILD_REPRODUCIBLE_BINARIES}" = "1" ]; then
> +if [ "$REPRODUCIBLE_TIMESTAMP_ROOTFS" = "" ]; then
> +REPRODUCIBLE_TIMESTAMP_ROOTFS=`git -C "${COREBASE}" log -1
> --pretty=%ct 2>/dev/null` || true
> if [ "$REPRODUCIBLE_TIMESTAMP_ROOTFS" = "" ]; then
> -REPRODUCIBLE_TIMESTAMP_ROOTFS=`git -C "${COREBASE}" log -1
> --pretty=%ct 2>/dev/null` || true
> -if [ "$REPRODUCIBLE_TIMESTAMP_ROOTFS" = "" ]; then
> -REPRODUCIBLE_TIMESTAMP_ROOTFS=`stat -c%Y
> ${@bb.utils.which(d.getVar("BBPATH"), "conf/bitbake.conf")}`
> -fi
> +REPRODUCIBLE_TIMESTAMP_ROOTFS=`stat -c%Y
> ${@bb.utils.which(d.getVar("BBPATH"), "conf/bitbake.conf")}`
> fi
> -# Set mtime of all files to a reproducible value
> -bbnote "reproducible_final_image_task: mtime set to
> $REPRODUCIBLE_TIMESTAMP_ROOTFS"
> -find ${IMAGE_ROOTFS} -print0 | xargs -0 touch -h
> --date=@$REPRODUCIBLE_TIMESTAMP_ROOTFS
> fi
> +# Set mtime of all files to a reproducible value
> +bbnote "reproducible_final_image_task: mtime set to
> $REPRODUCIBLE_TIMESTAMP_ROOTFS"
> +find ${IMAGE_ROOTFS} -print0 | xargs -0 touch -h
> --date=@$REPRODUCIBLE_TIMESTAMP_ROOTFS
> }
>
> systemd_preset_all () {
> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
> index 4acec1877e0..3dd9b4446db 100644
> --- a/meta/classes/kernel.bbclass
> +++ b/meta/classes/kernel.bbclass
> @@ -324,21 +324,20 @@ addtask bundle_initramfs after do_install before
> do_deploy
>
> kernel_do_compile() {
> unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE
> - if [ "${BUILD_REPRODUCIBLE_BINARIES}" = "1" ]; then
> - # kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH
> may not
> - # be set
> - if [ "${SOURCE_DATE_EPOCH}" = "" -o "${SOURCE_DATE_EPOCH}" =
> "0" ]; then
> - # The source directory is not necessarily a git
> repository, so we
> - # specify the git-dir to ensure that git does not
> query a
> - # repository in any parent directory.
> -
