Re: [PATCH 00/34] Make kernel build deterministic
* Artem Bityutskiy dedeki...@gmail.com wrote: On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote: On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote: Hi, this series makes it possible to build bit-identical kernel image and modules from identical sources. Of course the build is already deterministic in terms of behavior of the code, but the various timestamps embedded in the object files make it hard to compare two builds, for instance to verify that a makefile cleanup didn't accidentally change something. A prime example is /proc/config.gz, which has both a timestamp in the gzip header and a timestamp in the payload data. With this series applied, a script like this will produce identical kernels each time: Very nice stuff. Do you want to take the individual patches through one of your trees, or do you mind if the subsystem maintainers take them through theirs? But unfortunately, it is very easy to break this and for sure it'll be broken very soon. So additionally, I'd suggest: 1. Instrument checkpatch.pl and make it err or warn on timestamps. See the grandparent mail: checkpatch: Warn about usage of __DATE__, __TIME__ and __TIMESTAMP__ Thanks, Ingo ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH 00/34] Make kernel build deterministic
* Michal Marek mma...@suse.cz wrote:
Hi,
this series makes it possible to build bit-identical kernel image and
modules from identical sources. Of course the build is already
deterministic in terms of behavior of the code, but the various
timestamps embedded in the object files make it hard to compare two
builds, for instance to verify that a makefile cleanup didn't
accidentally change something. A prime example is /proc/config.gz, which
has both a timestamp in the gzip header and a timestamp in the payload
data. With this series applied, a script like this will produce
identical kernels each time:
#!/bin/bash
rm -rf /dev/shm/{source,build}{,1,2}
export KCONFIG_NOTIMESTAMP=1
export KBUILD_BUILD_TIMESTAMP='Sun May 1 12:00:00 CEST 2011'
export KBUILD_BUILD_USER=user
export KBUILD_BUILD_HOST=host
export ROOT_DEV=FLOPPY
for i in 1 2; do
mkdir /dev/shm/source
# randomize the inode order just for fun
git ls-tree -r -z --name-only HEAD | sort -R -z | xargs -0 \
cp --parents --target=/dev/shm/source
pushd /dev/shm/source
mkdir /dev/shm/build
/dev/shm/build/all.config
for opt in GCOV_KERNEL UBIFS_FS_DEBUG; do
echo # CONFIG_$opt is not set /dev/shm/build/all.config
done
make O=/dev/shm/build KCONFIG_ALLCONFIG=1 allmodconfig
make O=/dev/shm/build -j64
popd
mv /dev/shm/source /dev/shm/source$i
mv /dev/shm/build /dev/shm/build$i
done
diff -rq /dev/shm/build{1,2}
Nice!
Acked-by: Ingo Molnar mi...@elte.hu
Unfortunatelly, this cannot be used to validate indentation-only
patches, even if CONFIG_DEBUG_INFO is turned off. This is because of the
__FILE__ and __LINE__ macros used in many places. For the same reason,
the source and build directory needs to be the same, otherwise the
results will differ.
Nor can it be used to validate untrusted patches in general: a subtle change
might be introduced in a piece of code that is dependent on a .config detail
that is off for that particular build.
But in the common case it's nice to be able to have deterministic/reproducable
builds.
Thanks,
Ingo
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH 00/34] Make kernel build deterministic
On 5.4.2011 21:24, Artem Bityutskiy wrote: On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote: On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote: Hi, this series makes it possible to build bit-identical kernel image and modules from identical sources. Of course the build is already deterministic in terms of behavior of the code, but the various timestamps embedded in the object files make it hard to compare two builds, for instance to verify that a makefile cleanup didn't accidentally change something. A prime example is /proc/config.gz, which has both a timestamp in the gzip header and a timestamp in the payload data. With this series applied, a script like this will produce identical kernels each time: Very nice stuff. Do you want to take the individual patches through one of your trees, or do you mind if the subsystem maintainers take them through theirs? But unfortunately, it is very easy to break this and for sure it'll be broken very soon. I'm not so pessimistic. 34 patches and 57 files might sound like a lot, but given that this has been accumulating since day one, the cleanup should last for some time. So additionally, I'd suggest: 1. Instrument checkpatch.pl and make it err or warn on timestamps. This is patch 34/34 in this series: https://lkml.org/lkml/2011/4/5/198 2. Probably instrument linux-next to rise a warning when people break this. I'm not sure if Stephen has that much spare time, and I don't think it is necessary. I think the checkpatch check is sufficient and I'll check myself occasionally. Michal ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH 00/34] Make kernel build deterministic
On 5.4.2011 17:49, Greg KH wrote: Very nice stuff. Do you want to take the individual patches through one of your trees, or do you mind if the subsystem maintainers take them through theirs? I'd leave it up to the subsystem maintainers. I'll check once the merge window opens and send the remaining patches to Linus directly. Michal ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH 00/34] Make kernel build deterministic
On Wed, 2011-04-06 at 11:07 +0200, Michal Marek wrote: So additionally, I'd suggest: 1. Instrument checkpatch.pl and make it err or warn on timestamps. This is patch 34/34 in this series: https://lkml.org/lkml/2011/4/5/198 Yeah, great, did not notice, thanks! 2. Probably instrument linux-next to rise a warning when people break this. I'm not sure if Stephen has that much spare time, and I don't think it is necessary. I think the checkpatch check is sufficient and I'll check myself occasionally. Yes, that's fine, I just wanted to speak this out - there is a probability that someone gets excited and creates some instrumentation to kbuild to automatically detect bad things and then Stephen could easily use that. WRT not necessary - well, I think it is always better to cut bad things before they are merged, rather than afterwards. But anyway, let's forget about this. -- Best Regards, Artem Bityutskiy (Артём Битюцкий) ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
[PATCH 00/34] Make kernel build deterministic
Hi,
this series makes it possible to build bit-identical kernel image and
modules from identical sources. Of course the build is already
deterministic in terms of behavior of the code, but the various
timestamps embedded in the object files make it hard to compare two
builds, for instance to verify that a makefile cleanup didn't
accidentally change something. A prime example is /proc/config.gz, which
has both a timestamp in the gzip header and a timestamp in the payload
data. With this series applied, a script like this will produce
identical kernels each time:
#!/bin/bash
rm -rf /dev/shm/{source,build}{,1,2}
export KCONFIG_NOTIMESTAMP=1
export KBUILD_BUILD_TIMESTAMP='Sun May 1 12:00:00 CEST 2011'
export KBUILD_BUILD_USER=user
export KBUILD_BUILD_HOST=host
export ROOT_DEV=FLOPPY
for i in 1 2; do
mkdir /dev/shm/source
# randomize the inode order just for fun
git ls-tree -r -z --name-only HEAD | sort -R -z | xargs -0 \
cp --parents --target=/dev/shm/source
pushd /dev/shm/source
mkdir /dev/shm/build
/dev/shm/build/all.config
for opt in GCOV_KERNEL UBIFS_FS_DEBUG; do
echo # CONFIG_$opt is not set /dev/shm/build/all.config
done
make O=/dev/shm/build KCONFIG_ALLCONFIG=1 allmodconfig
make O=/dev/shm/build -j64
popd
mv /dev/shm/source /dev/shm/source$i
mv /dev/shm/build /dev/shm/build$i
done
diff -rq /dev/shm/build{1,2}
Unfortunatelly, this cannot be used to validate indentation-only
patches, even if CONFIG_DEBUG_INFO is turned off. This is because of the
__FILE__ and __LINE__ macros used in many places. For the same reason,
the source and build directory needs to be the same, otherwise the
results will differ. This was tested on
x86_64/{defconfig,allmodconfig,allyesconfig} and ppc64/defconfig. The
series is also available at
git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild-2.6.git
deterministic-build-v1
Michal
Michal Marek (34):
kconfig: Do not record timestamp in auto.conf and autoconf.h
kbuild: Call gzip with -n
powerpc: Call gzip with -n
kbuild: Use the deterministic mode of ar
powerpc: Use the deterministic mode of ar
kbuild: Drop unused LINUX_COMPILE_TIME and LINUX_COMPILE_DOMAIN
macros
kbuild: Allow to override LINUX_COMPILE_BY and LINUX_COMPILE_HOST
macros
initramfs: Use KBUILD_BUILD_TIMESTAMP for generated entries
x86: Allow to override the ROOT_DEV variable
cyclades: Drop __TIME__ usage
nozomi: Drop __TIME__ usage
isdn/diva: Drop __TIME__ usage
media/radio-maxiradio: Drop __TIME__ usage
media/cx231xx: Drop __TIME__ usage
baycom: Drop __TIME__ usage
nand/denali: Drop __TIME__ usage
hdlcdrv: Drop __TIME__ usage
wan/pc300: Drop __TIME__ usage
rt2x00: Drop __TIME__ usage
parport: Drop __TIME__ usage
aacraid: Drop __TIME__ usage
scsi/in2000: Drop __TIME__ usage
scsi/wd33c93: Drop __TIME__ usage
usb/u132-hcd: Drop __TIME__ usage
usb/ftdi-elan: Drop __TIME__ usage
dlm: Drop __TIME__ usage
gfs2: Drop __TIME__ usage
atm: Drop __TIME__ usage
tipc: Drop __TIME__ usage
rio: Drop __DATE__ usage
edac: Drop __DATE__ usage
pmcraid: Drop __DATE__ usage
usb/lh7a40x_udc: Drop __DATE__ usage
checkpatch: Warn about usage of __DATE__, __TIME__ and __TIMESTAMP__
Documentation/kbuild/kbuild.txt | 12 ++
arch/powerpc/boot/Makefile |2 +-
arch/powerpc/boot/wrapper|6 ++--
arch/x86/boot/Makefile |2 +-
drivers/char/cyclades.c |3 +-
drivers/char/nozomi.c|3 +-
drivers/char/rio/rioinit.c |2 +-
drivers/edac/amd76x_edac.c |2 +-
drivers/edac/amd8111_edac.c |2 +-
drivers/edac/amd8131_edac.c |2 +-
drivers/edac/cpc925_edac.c |2 +-
drivers/edac/e752x_edac.c|2 +-
drivers/edac/e7xxx_edac.c|2 +-
drivers/edac/edac_module.c |2 +-
drivers/edac/i5000_edac.c|2 +-
drivers/edac/i5400_edac.c|2 +-
drivers/edac/i7300_edac.c|2 +-
drivers/edac/i7core_edac.c |2 +-
drivers/edac/i82860_edac.c |2 +-
drivers/edac/i82875p_edac.c |2 +-
drivers/edac/i82975x_edac.c |2 +-
drivers/edac/mpc85xx_edac.h |2 +-
drivers/edac/mv64x60_edac.h |2 +-
drivers/edac/ppc4xx_edac.c |2 +-
drivers/edac/r82600_edac.c |2 +-
drivers/isdn/hardware/eicon/divasfunc.c |5 +--
drivers/media/radio/radio-maxiradio.c|3 +-
drivers/media/video/cx231xx/cx231xx-avcore.c |2 +-
drivers/mtd/nand/denali.c|3 +-
Re: [PATCH 00/34] Make kernel build deterministic
On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote: On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote: Hi, this series makes it possible to build bit-identical kernel image and modules from identical sources. Of course the build is already deterministic in terms of behavior of the code, but the various timestamps embedded in the object files make it hard to compare two builds, for instance to verify that a makefile cleanup didn't accidentally change something. A prime example is /proc/config.gz, which has both a timestamp in the gzip header and a timestamp in the payload data. With this series applied, a script like this will produce identical kernels each time: Very nice stuff. Do you want to take the individual patches through one of your trees, or do you mind if the subsystem maintainers take them through theirs? I'm happy for this to go through a single tree. James ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH 00/34] Make kernel build deterministic
Em 05-04-2011 15:16, James Bottomley escreveu: On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote: On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote: Hi, this series makes it possible to build bit-identical kernel image and modules from identical sources. Of course the build is already deterministic in terms of behavior of the code, but the various timestamps embedded in the object files make it hard to compare two builds, for instance to verify that a makefile cleanup didn't accidentally change something. A prime example is /proc/config.gz, which has both a timestamp in the gzip header and a timestamp in the payload data. With this series applied, a script like this will produce identical kernels each time: Very nice stuff. Do you want to take the individual patches through one of your trees, or do you mind if the subsystem maintainers take them through theirs? I'm happy for this to go through a single tree. Me too. With respect to the patches I was c/c (patches 13, 14, 31): Acked-by: Mauro Carvalho Chehab mche...@redhat.com Thanks, Mauro. ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH 00/34] Make kernel build deterministic
On Tue, Apr 05, 2011 at 03:29:19PM -0300, Mauro Carvalho Chehab wrote: Em 05-04-2011 15:16, James Bottomley escreveu: On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote: On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote: Hi, this series makes it possible to build bit-identical kernel image and modules from identical sources. Of course the build is already deterministic in terms of behavior of the code, but the various timestamps embedded in the object files make it hard to compare two builds, for instance to verify that a makefile cleanup didn't accidentally change something. A prime example is /proc/config.gz, which has both a timestamp in the gzip header and a timestamp in the payload data. With this series applied, a script like this will produce identical kernels each time: Very nice stuff. Do you want to take the individual patches through one of your trees, or do you mind if the subsystem maintainers take them through theirs? I'm happy for this to go through a single tree. Me too. With respect to the patches I was c/c (patches 13, 14, 31): Acked-by: Mauro Carvalho Chehab mche...@redhat.com Me too. Acked-by: Greg Kroah-Hartman gre...@suse.de on the patches I was copied on. thanks, greg k-h ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH 00/34] Make kernel build deterministic
On Tue, 2011-04-05 at 08:49 -0700, Greg KH wrote: On Tue, Apr 05, 2011 at 04:58:47PM +0200, Michal Marek wrote: Hi, this series makes it possible to build bit-identical kernel image and modules from identical sources. Of course the build is already deterministic in terms of behavior of the code, but the various timestamps embedded in the object files make it hard to compare two builds, for instance to verify that a makefile cleanup didn't accidentally change something. A prime example is /proc/config.gz, which has both a timestamp in the gzip header and a timestamp in the payload data. With this series applied, a script like this will produce identical kernels each time: Very nice stuff. Do you want to take the individual patches through one of your trees, or do you mind if the subsystem maintainers take them through theirs? But unfortunately, it is very easy to break this and for sure it'll be broken very soon. So additionally, I'd suggest: 1. Instrument checkpatch.pl and make it err or warn on timestamps. 2. Probably instrument linux-next to rise a warning when people break this. -- Best Regards, Artem Bityutskiy (Битюцкий Артём) ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
