Re: [OE-core] [PATCH v3 07/11] image_types.bbclass: improve cpio image reproducibility
> >> Is it useful to keep the option using cpio from the host? Why not > >> always use cpio-native? > > > > I guess it is a judgement call which tools to build and which tools to > use from the host. Not building tools which are guaranteed to be on the > host saves build time. > > I guess build time for cpio-native will be somewhere in the noise for > a typical build. Avoiding the extra complexity (code + test and QA) of > maintaining support for both cpio from the host and cpio-native would > seem to be far more of an issue. There's no guarantee that cpio is > available on the host (it currently needs to be installed manually if > building within the ubuntu:16.04 docker image, for example). > Looking at the HOSTTOOLS in poky/meta/conf/bitbake.conf I see "cpio" is listed among required tools. So at least for "poky" a missing cpio would/should generate an error. > > Also, the need for cpio-native with the reproducible options it is going > to be fairly rare. > > I'm not sure I agree that wanting reproducible builds is a rare > requirement but if it is then isn't eliminating a rarely used code > path an argument *in favour of* switching to a single solution which > supports both cases? Agreed. I don't have a problem with defaulting to cpio-native, it would solve a few problems as well, but I think some kind of consensus is needed first. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3 07/11] image_types.bbclass: improve cpio image reproducibility
On Thu, Aug 10, 2017 at 12:35 PM, Bystricky, Juro
wrote:
>>
>> Is it useful to keep the option using cpio from the host? Why not
>> always use cpio-native?
>
> I guess it is a judgement call which tools to build and which tools to use
> from the host. Not building tools which are guaranteed to be on the host
> saves build time.
I guess build time for cpio-native will be somewhere in the noise for
a typical build. Avoiding the extra complexity (code + test and QA) of
maintaining support for both cpio from the host and cpio-native would
seem to be far more of an issue. There's no guarantee that cpio is
available on the host (it currently needs to be installed manually if
building within the ubuntu:16.04 docker image, for example).
> Also, the need for cpio-native with the reproducible options it is going to
> be fairly rare.
I'm not sure I agree that wanting reproducible builds is a rare
requirement but if it is then isn't eliminating a rarely used code
path an argument *in favour of* switching to a single solution which
supports both cases?
>
>> >do_image_cpio[depends] += "cpio-replacement-
>> native:do_populate_sysroot"
>> >EXTRANATIVEPATH += "cpio-native"
>> >
>> > 2. Provide a way to pass custom arguments to cpio. This is done via
>> >a new variable IMAGE_CMD_CPIO. For binary reproducible cpio archives
>> >one needs to set (in local.conf)
>> >
>> >IMAGE_CMD_CPIO = "cpio --ignore-devno --reproducible "
>> >
>> > 3. A symlink is created as part of the image. Here we make sure it gets
>> the
>> >timestamps (mtime) based on the timestamp of the symlink target,
>> rather than
>> >the timestamp corresponding to the build time.
>> >
>> > Signed-off-by: Juro Bystricky
>> > ---
>> > meta/classes/image_types.bbclass | 14 --
>> > 1 file changed, 12 insertions(+), 2 deletions(-)
>> >
>> > diff --git a/meta/classes/image_types.bbclass
>> b/meta/classes/image_types.bbclass
>> > index e0368c7..50d0c07 100644
>> > --- a/meta/classes/image_types.bbclass
>> > +++ b/meta/classes/image_types.bbclass
>> > @@ -122,9 +122,17 @@ IMAGE_CMD_TAR ?= "tar"
>> > # ignore return code 1 "file changed as we read it" as other tasks(e.g.
>> do_image_wic) may be hardlinking rootfs
>> > IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf
>> ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} .
>> || [ $? -eq 1 ]"
>> >
>> > +# By default, cpio from the host is used, which can be quite old. If
>> > +# you need special parameters (like --ignore-devno --reproducible) which
>> are only
>> > +# supported by GNU cpio upstream >= 2.12, then override that default:
>> > +# IMAGE_CMD_CPIO = "cpio --ignore-devno"
>> > +# do_image_cpio[depends] += "cpio-replacement-
>> native:do_populate_sysroot"
>> > +# EXTRANATIVEPATH += "cpio-native"
>> > +
>> > +IMAGE_CMD_CPIO ?= "cpio"
>> > do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
>> > IMAGE_CMD_cpio () {
>> > - (cd ${IMAGE_ROOTFS} && find . | cpio -o -H newc
>> >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
>> > + (cd ${IMAGE_ROOTFS} && find . | ${IMAGE_CMD_CPIO} -o -H newc -v
>> >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
>> > # We only need the /init symlink if we're building the real
>> > # image. The -dbg image doesn't need it! By being clever
>> > # about this we also avoid 'touch' below failing, as it
>> > @@ -134,10 +142,12 @@ IMAGE_CMD_cpio () {
>> > if [ ! -L ${IMAGE_ROOTFS}/init ] && [ ! -e
>> ${IMAGE_ROOTFS}/init ]; then
>> > if [ -L ${IMAGE_ROOTFS}/sbin/init ] || [ -e
>> ${IMAGE_ROOTFS}/sbin/init ]; then
>> > ln -sf /sbin/init
>> ${WORKDIR}/cpio_append/init
>> > + # improve reproducibility: set the link
>> mtime to be the same as the target
>> > + touch -h -r ${IMAGE_ROOTFS}/sbin/init
>> ${WORKDIR}/cpio_append/init
>> > else
>> > touch ${WORKDIR}/cpio_append/init
>> > fi
>> > - (cd ${WORKDIR}/cpio_append && echo ./init | cpio
>> -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
>> > + (cd ${WORKDIR}/cpio_append && echo ./init |
>> ${IMAGE_CMD_CPIO} -oA -H newc -F
>> ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
>> > fi
>> > fi
>> > }
>> > --
>> > 2.7.4
>> >
>> > --
>> > ___
>> > Openembedded-core mailing list
>> > Openembedded-core@lists.openembedded.org
>> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3 07/11] image_types.bbclass: improve cpio image reproducibility
>
> Is it useful to keep the option using cpio from the host? Why not
> always use cpio-native?
>
I guess it is a judgement call which tools to build and which tools to use from
the host. Not building tools which are guaranteed to be on the host saves build
time. Also, the need for cpio-native with the reproducible options it is going
to be fairly rare.
> >do_image_cpio[depends] += "cpio-replacement-
> native:do_populate_sysroot"
> >EXTRANATIVEPATH += "cpio-native"
> >
> > 2. Provide a way to pass custom arguments to cpio. This is done via
> >a new variable IMAGE_CMD_CPIO. For binary reproducible cpio archives
> >one needs to set (in local.conf)
> >
> >IMAGE_CMD_CPIO = "cpio --ignore-devno --reproducible "
> >
> > 3. A symlink is created as part of the image. Here we make sure it gets
> the
> >timestamps (mtime) based on the timestamp of the symlink target,
> rather than
> >the timestamp corresponding to the build time.
> >
> > Signed-off-by: Juro Bystricky
> > ---
> > meta/classes/image_types.bbclass | 14 --
> > 1 file changed, 12 insertions(+), 2 deletions(-)
> >
> > diff --git a/meta/classes/image_types.bbclass
> b/meta/classes/image_types.bbclass
> > index e0368c7..50d0c07 100644
> > --- a/meta/classes/image_types.bbclass
> > +++ b/meta/classes/image_types.bbclass
> > @@ -122,9 +122,17 @@ IMAGE_CMD_TAR ?= "tar"
> > # ignore return code 1 "file changed as we read it" as other tasks(e.g.
> do_image_wic) may be hardlinking rootfs
> > IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf
> ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} .
> || [ $? -eq 1 ]"
> >
> > +# By default, cpio from the host is used, which can be quite old. If
> > +# you need special parameters (like --ignore-devno --reproducible) which
> are only
> > +# supported by GNU cpio upstream >= 2.12, then override that default:
> > +# IMAGE_CMD_CPIO = "cpio --ignore-devno"
> > +# do_image_cpio[depends] += "cpio-replacement-
> native:do_populate_sysroot"
> > +# EXTRANATIVEPATH += "cpio-native"
> > +
> > +IMAGE_CMD_CPIO ?= "cpio"
> > do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
> > IMAGE_CMD_cpio () {
> > - (cd ${IMAGE_ROOTFS} && find . | cpio -o -H newc
> >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> > + (cd ${IMAGE_ROOTFS} && find . | ${IMAGE_CMD_CPIO} -o -H newc -v
> >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> > # We only need the /init symlink if we're building the real
> > # image. The -dbg image doesn't need it! By being clever
> > # about this we also avoid 'touch' below failing, as it
> > @@ -134,10 +142,12 @@ IMAGE_CMD_cpio () {
> > if [ ! -L ${IMAGE_ROOTFS}/init ] && [ ! -e
> ${IMAGE_ROOTFS}/init ]; then
> > if [ -L ${IMAGE_ROOTFS}/sbin/init ] || [ -e
> ${IMAGE_ROOTFS}/sbin/init ]; then
> > ln -sf /sbin/init
> ${WORKDIR}/cpio_append/init
> > + # improve reproducibility: set the link
> mtime to be the same as the target
> > + touch -h -r ${IMAGE_ROOTFS}/sbin/init
> ${WORKDIR}/cpio_append/init
> > else
> > touch ${WORKDIR}/cpio_append/init
> > fi
> > - (cd ${WORKDIR}/cpio_append && echo ./init | cpio
> -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> > + (cd ${WORKDIR}/cpio_append && echo ./init |
> ${IMAGE_CMD_CPIO} -oA -H newc -F
> ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> > fi
> > fi
> > }
> > --
> > 2.7.4
> >
> > --
> > ___
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3 07/11] image_types.bbclass: improve cpio image reproducibility
On Wed, Aug 9, 2017 at 10:48 AM, Juro Bystricky
wrote:
> This patch helps to build cpio images that are binary reproducible.
> The changes are as follows:
>
> 1. By default, cpio from the host is used, which can be quite old.
>Hence we need to implement a way to use/call cpio-native, which supports
>new features needed for binary reproducibility, notably the arguments
>such as "--reproducible" and "--ignore-devno".
>This can be achieved by specifying the following (in local.conf):
Is it useful to keep the option using cpio from the host? Why not
always use cpio-native?
>do_image_cpio[depends] += "cpio-replacement-native:do_populate_sysroot"
>EXTRANATIVEPATH += "cpio-native"
>
> 2. Provide a way to pass custom arguments to cpio. This is done via
>a new variable IMAGE_CMD_CPIO. For binary reproducible cpio archives
>one needs to set (in local.conf)
>
>IMAGE_CMD_CPIO = "cpio --ignore-devno --reproducible "
>
> 3. A symlink is created as part of the image. Here we make sure it gets the
>timestamps (mtime) based on the timestamp of the symlink target, rather
> than
>the timestamp corresponding to the build time.
>
> Signed-off-by: Juro Bystricky
> ---
> meta/classes/image_types.bbclass | 14 --
> 1 file changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/image_types.bbclass
> b/meta/classes/image_types.bbclass
> index e0368c7..50d0c07 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -122,9 +122,17 @@ IMAGE_CMD_TAR ?= "tar"
> # ignore return code 1 "file changed as we read it" as other tasks(e.g.
> do_image_wic) may be hardlinking rootfs
> IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf
> ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} . ||
> [ $? -eq 1 ]"
>
> +# By default, cpio from the host is used, which can be quite old. If
> +# you need special parameters (like --ignore-devno --reproducible) which are
> only
> +# supported by GNU cpio upstream >= 2.12, then override that default:
> +# IMAGE_CMD_CPIO = "cpio --ignore-devno"
> +# do_image_cpio[depends] += "cpio-replacement-native:do_populate_sysroot"
> +# EXTRANATIVEPATH += "cpio-native"
> +
> +IMAGE_CMD_CPIO ?= "cpio"
> do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
> IMAGE_CMD_cpio () {
> - (cd ${IMAGE_ROOTFS} && find . | cpio -o -H newc
> >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> + (cd ${IMAGE_ROOTFS} && find . | ${IMAGE_CMD_CPIO} -o -H newc -v
> >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> # We only need the /init symlink if we're building the real
> # image. The -dbg image doesn't need it! By being clever
> # about this we also avoid 'touch' below failing, as it
> @@ -134,10 +142,12 @@ IMAGE_CMD_cpio () {
> if [ ! -L ${IMAGE_ROOTFS}/init ] && [ ! -e
> ${IMAGE_ROOTFS}/init ]; then
> if [ -L ${IMAGE_ROOTFS}/sbin/init ] || [ -e
> ${IMAGE_ROOTFS}/sbin/init ]; then
> ln -sf /sbin/init ${WORKDIR}/cpio_append/init
> + # improve reproducibility: set the link mtime
> to be the same as the target
> + touch -h -r ${IMAGE_ROOTFS}/sbin/init
> ${WORKDIR}/cpio_append/init
> else
> touch ${WORKDIR}/cpio_append/init
> fi
> - (cd ${WORKDIR}/cpio_append && echo ./init | cpio -oA
> -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> + (cd ${WORKDIR}/cpio_append && echo ./init |
> ${IMAGE_CMD_CPIO} -oA -H newc -F
> ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> fi
> fi
> }
> --
> 2.7.4
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
