Re: MIPS parallel build breakage
On Thu, Mar 01, 2018 at 12:07:29PM +0100, Sascha Hauer wrote: > Hi Sam and Antony, > > (Antony because you introduced this and Sam because you have intimate > knowledge with our build system ;) > > I am trying to solve a problem that annoys me for some years now. The > problem is that the MIPS defconfigs regularly fail in my autobuilder. > I usually help myself with trying it again and most of the time it > then works, but doing this for years increases the pressure to do > something. > > What happens is that the MIPS builds use include/generated/compile.h: > > > arch/mips/include/asm/pbl_macros.h:28:#include > > This often ends in: > > > In file included from arch/mips/boot/start.S:20:0: > > arch/mips/include/asm/pbl_macros.h:28:31: > > fatal error: generated/compile.h: No such file or directory > > compilation terminated. > > include/generated/compile.h is generated in common/Makefile: > > > include/generated/compile.h: FORCE > > @$($(quiet)chk_compile.h) > > $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ > > "$(UTS_MACHINE)" "$(CC) $(KBUILD_CFLAGS)" > > Other users of this file add an explicit dependency on it: > > > # dependencies on generated files need to be listed explicitly > > $(obj)/version.o: include/generated/compile.h > > This only seems to work though when the dependent file is in the same > directory, but not when the file is in arch/mips/boot/. > > I tried doing the include/generated/compile.h in Makefile instead of > common/Makefile, but then the version counting gets confused and causes > unnecessary rebuilds. > > Any ideas how to solve this? We still have no solution for this. I dropped MIPS from my autobuilder, so we won't catch any build errors on MIPS anymore :( Sascha -- Pengutronix e.K. | | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- | ___ barebox mailing list barebox@lists.infradead.org http://lists.infradead.org/mailman/listinfo/barebox
Re: MIPS parallel build breakage
On Fri, Mar 02, 2018 at 04:37:31PM +0300, Antony Pavlov wrote: > On Fri, 2 Mar 2018 13:34:01 +0100 > Sascha Hauer wrote: > > > On Fri, Mar 02, 2018 at 03:17:35PM +0300, Antony Pavlov wrote: > > > On Thu, 1 Mar 2018 12:07:29 +0100 > > > Sascha Hauer wrote: > > > > > > > Hi Sam and Antony, > > > > > > > > (Antony because you introduced this and Sam because you have intimate > > > > knowledge with our build system ;) > > > > > > > > I am trying to solve a problem that annoys me for some years now. The > > > > problem is that the MIPS defconfigs regularly fail in my autobuilder. > > > > I usually help myself with trying it again and most of the time it > > > > then works, but doing this for years increases the pressure to do > > > > something. > > > > > > > > What happens is that the MIPS builds use include/generated/compile.h: > > > > > > > > > arch/mips/include/asm/pbl_macros.h:28:#include > > > > > > > > This often ends in: > > > > > > > > > In file included from arch/mips/boot/start.S:20:0: > > > > > arch/mips/include/asm/pbl_macros.h:28:31: > > > > > fatal error: generated/compile.h: No such file or directory > > > > > compilation terminated. > > > > > > > > include/generated/compile.h is generated in common/Makefile: > > > > > > > > > include/generated/compile.h: FORCE > > > > > @$($(quiet)chk_compile.h) > > > > > $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ > > > > > "$(UTS_MACHINE)" "$(CC) $(KBUILD_CFLAGS)" > > > > > > > > Other users of this file add an explicit dependency on it: > > > > > > > > > # dependencies on generated files need to be listed explicitly > > > > > $(obj)/version.o: include/generated/compile.h > > > > > > > > This only seems to work though when the dependent file is in the same > > > > directory, but not when the file is in arch/mips/boot/. > > > > > > > > I tried doing the include/generated/compile.h in Makefile instead of > > > > common/Makefile, but then the version counting gets confused and causes > > > > unnecessary rebuilds. > > > > > > > > Any ideas how to solve this? > > > > > > I can't reproduce parallel build breakage. > > > > The failure starts here when using 6 threads or more and only when > > compiling the black-swift_defconfig. After one successful build I can > > reproduce this reliably here with: > > > > rm -f include/generated/compile.h && make -j 6 > > > > > But I see another problem: > > > the version string inside mips pbl binary is differ from the version > > > string > > > inside main barebox binary. > > > > > > I suppose that the problem is that current build system handles mips asm > > > files > > > (which are actually use generated compile.h) and c-files in a different > > > way. > > > > Ups. How do the strings differ? > > insert generated version into pbl by using mips_barebox_10h: > > barebox$ git diff > diff --git a/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h > b/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h > index 5c1537262e..6e85a7009a 100644 > --- a/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h > +++ b/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h > @@ -43,18 +43,8 @@ > .setpush > .setnoreorder > > - b __start > -nop > + mips_barebox_10h > > - /* > - On MIPS Technologies boards > - 0x1fc00010 address is reserved for BoardID > - */ > - .org0x10 > - .word 0x > - .word 0x > - > -__start: > mips_disable_interrupts > > /* cpu specific setup ... */ > barebox$ make mrproper > ... > barebox$ export ARCH=mips > barebox$ export CROSS_COMPILE=mips-linux-gnu- > barebox$ make qemu-malta_defconfig > ... > barebox$ make -j3 -s > ... > barebox$ xxd barebox-flash-image | head -n 6 > : 1000 0017 > 0010: 6261 7265 626f 7820 3230 3138 2e30 322e barebox 2018.02. > 0020: 302d 3030 3037 392d 6731 6564 6462 3064 0-00079-g1eddb0d > 0030: 3338 322d 6469 7274 7920 2330 2046 7269 382-dirty #0 Fri >^^^ > 0040: 204d 6172 2032 2031 363a 3032 3a32 3920 Mar 2 16:02:29 > 0050: 4d53 4b20 3230 3138 MSK 2018 > > #0, 16:02:29 > > barebox$ qemu-system-mips -nodefaults -M malta -m 256 \ > > -nographic -serial stdio -monitor null \ > > -bios barebox-flash-image > > > barebox 2018.02.0-00079-g1eddb0d382-dirty #1 Fri Mar 2 16:02:38 MSK 2018 > ^^^ > > #1, 16:02:38 I can imagine this also goes down to the missing dependency. UTS_VERSION contains the build number (the number behind the #) and the build date. Now if the PBL macros lack the dependency and generated/compile.h is included *before* it is re-generated whereas the files in common do have
Re: MIPS parallel build breakage
On Fri, 2 Mar 2018 13:34:01 +0100 Sascha Hauer wrote: > On Fri, Mar 02, 2018 at 03:17:35PM +0300, Antony Pavlov wrote: > > On Thu, 1 Mar 2018 12:07:29 +0100 > > Sascha Hauer wrote: > > > > > Hi Sam and Antony, > > > > > > (Antony because you introduced this and Sam because you have intimate > > > knowledge with our build system ;) > > > > > > I am trying to solve a problem that annoys me for some years now. The > > > problem is that the MIPS defconfigs regularly fail in my autobuilder. > > > I usually help myself with trying it again and most of the time it > > > then works, but doing this for years increases the pressure to do > > > something. > > > > > > What happens is that the MIPS builds use include/generated/compile.h: > > > > > > > arch/mips/include/asm/pbl_macros.h:28:#include > > > > > > This often ends in: > > > > > > > In file included from arch/mips/boot/start.S:20:0: > > > > arch/mips/include/asm/pbl_macros.h:28:31: > > > > fatal error: generated/compile.h: No such file or directory > > > > compilation terminated. > > > > > > include/generated/compile.h is generated in common/Makefile: > > > > > > > include/generated/compile.h: FORCE > > > > @$($(quiet)chk_compile.h) > > > > $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ > > > > "$(UTS_MACHINE)" "$(CC) $(KBUILD_CFLAGS)" > > > > > > Other users of this file add an explicit dependency on it: > > > > > > > # dependencies on generated files need to be listed explicitly > > > > $(obj)/version.o: include/generated/compile.h > > > > > > This only seems to work though when the dependent file is in the same > > > directory, but not when the file is in arch/mips/boot/. > > > > > > I tried doing the include/generated/compile.h in Makefile instead of > > > common/Makefile, but then the version counting gets confused and causes > > > unnecessary rebuilds. > > > > > > Any ideas how to solve this? > > > > I can't reproduce parallel build breakage. > > The failure starts here when using 6 threads or more and only when > compiling the black-swift_defconfig. After one successful build I can > reproduce this reliably here with: > > rm -f include/generated/compile.h && make -j 6 > > > But I see another problem: > > the version string inside mips pbl binary is differ from the version string > > inside main barebox binary. > > > > I suppose that the problem is that current build system handles mips asm > > files > > (which are actually use generated compile.h) and c-files in a different way. > > Ups. How do the strings differ? insert generated version into pbl by using mips_barebox_10h: barebox$ git diff diff --git a/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h b/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h index 5c1537262e..6e85a7009a 100644 --- a/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h +++ b/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h @@ -43,18 +43,8 @@ .setpush .setnoreorder - b __start -nop + mips_barebox_10h - /* - On MIPS Technologies boards - 0x1fc00010 address is reserved for BoardID - */ - .org0x10 - .word 0x - .word 0x - -__start: mips_disable_interrupts /* cpu specific setup ... */ barebox$ make mrproper ... barebox$ export ARCH=mips barebox$ export CROSS_COMPILE=mips-linux-gnu- barebox$ make qemu-malta_defconfig ... barebox$ make -j3 -s ... barebox$ xxd barebox-flash-image | head -n 6 : 1000 0017 0010: 6261 7265 626f 7820 3230 3138 2e30 322e barebox 2018.02. 0020: 302d 3030 3037 392d 6731 6564 6462 3064 0-00079-g1eddb0d 0030: 3338 322d 6469 7274 7920 2330 2046 7269 382-dirty #0 Fri ^^^ 0040: 204d 6172 2032 2031 363a 3032 3a32 3920 Mar 2 16:02:29 0050: 4d53 4b20 3230 3138 MSK 2018 #0, 16:02:29 barebox$ qemu-system-mips -nodefaults -M malta -m 256 \ > -nographic -serial stdio -monitor null \ > -bios barebox-flash-image barebox 2018.02.0-00079-g1eddb0d382-dirty #1 Fri Mar 2 16:02:38 MSK 2018 ^^^ #1, 16:02:38 > Another obvious way to fix the issue is to remove UTS_RELEASE and > UTS_VERSION from the pbl headers. Would that be an option? It isn't a __good__ option. -- Best regards, Antony Pavlov ___ barebox mailing list barebox@lists.infradead.org http://lists.infradead.org/mailman/listinfo/barebox
Re: MIPS parallel build breakage
On Fri, Mar 02, 2018 at 03:17:35PM +0300, Antony Pavlov wrote: > On Thu, 1 Mar 2018 12:07:29 +0100 > Sascha Hauer wrote: > > > Hi Sam and Antony, > > > > (Antony because you introduced this and Sam because you have intimate > > knowledge with our build system ;) > > > > I am trying to solve a problem that annoys me for some years now. The > > problem is that the MIPS defconfigs regularly fail in my autobuilder. > > I usually help myself with trying it again and most of the time it > > then works, but doing this for years increases the pressure to do > > something. > > > > What happens is that the MIPS builds use include/generated/compile.h: > > > > > arch/mips/include/asm/pbl_macros.h:28:#include > > > > This often ends in: > > > > > In file included from arch/mips/boot/start.S:20:0: > > > arch/mips/include/asm/pbl_macros.h:28:31: > > > fatal error: generated/compile.h: No such file or directory > > > compilation terminated. > > > > include/generated/compile.h is generated in common/Makefile: > > > > > include/generated/compile.h: FORCE > > > @$($(quiet)chk_compile.h) > > > $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ > > > "$(UTS_MACHINE)" "$(CC) $(KBUILD_CFLAGS)" > > > > Other users of this file add an explicit dependency on it: > > > > > # dependencies on generated files need to be listed explicitly > > > $(obj)/version.o: include/generated/compile.h > > > > This only seems to work though when the dependent file is in the same > > directory, but not when the file is in arch/mips/boot/. > > > > I tried doing the include/generated/compile.h in Makefile instead of > > common/Makefile, but then the version counting gets confused and causes > > unnecessary rebuilds. > > > > Any ideas how to solve this? > > I can't reproduce parallel build breakage. The failure starts here when using 6 threads or more and only when compiling the black-swift_defconfig. After one successful build I can reproduce this reliably here with: rm -f include/generated/compile.h && make -j 6 > But I see another problem: > the version string inside mips pbl binary is differ from the version string > inside main barebox binary. > > I suppose that the problem is that current build system handles mips asm files > (which are actually use generated compile.h) and c-files in a different way. Ups. How do the strings differ? Another obvious way to fix the issue is to remove UTS_RELEASE and UTS_VERSION from the pbl headers. Would that be an option? Sascha -- Pengutronix e.K. | | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- | ___ barebox mailing list barebox@lists.infradead.org http://lists.infradead.org/mailman/listinfo/barebox
Re: MIPS parallel build breakage
On Thu, 1 Mar 2018 12:07:29 +0100 Sascha Hauer wrote: > Hi Sam and Antony, > > (Antony because you introduced this and Sam because you have intimate > knowledge with our build system ;) > > I am trying to solve a problem that annoys me for some years now. The > problem is that the MIPS defconfigs regularly fail in my autobuilder. > I usually help myself with trying it again and most of the time it > then works, but doing this for years increases the pressure to do > something. > > What happens is that the MIPS builds use include/generated/compile.h: > > > arch/mips/include/asm/pbl_macros.h:28:#include > > This often ends in: > > > In file included from arch/mips/boot/start.S:20:0: > > arch/mips/include/asm/pbl_macros.h:28:31: > > fatal error: generated/compile.h: No such file or directory > > compilation terminated. > > include/generated/compile.h is generated in common/Makefile: > > > include/generated/compile.h: FORCE > > @$($(quiet)chk_compile.h) > > $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ > > "$(UTS_MACHINE)" "$(CC) $(KBUILD_CFLAGS)" > > Other users of this file add an explicit dependency on it: > > > # dependencies on generated files need to be listed explicitly > > $(obj)/version.o: include/generated/compile.h > > This only seems to work though when the dependent file is in the same > directory, but not when the file is in arch/mips/boot/. > > I tried doing the include/generated/compile.h in Makefile instead of > common/Makefile, but then the version counting gets confused and causes > unnecessary rebuilds. > > Any ideas how to solve this? I can't reproduce parallel build breakage. But I see another problem: the version string inside mips pbl binary is differ from the version string inside main barebox binary. I suppose that the problem is that current build system handles mips asm files (which are actually use generated compile.h) and c-files in a different way. -- Best regards, Antony Pavlov ___ barebox mailing list barebox@lists.infradead.org http://lists.infradead.org/mailman/listinfo/barebox
Re: MIPS parallel build breakage
Hi, On Thu, Mar 01, 2018 at 12:07:29PM +0100, Sascha Hauer wrote: > (Antony because you introduced this and Sam because you have intimate > knowledge with our build system ;) > > I am trying to solve a problem that annoys me for some years now. The > problem is that the MIPS defconfigs regularly fail in my autobuilder. > I usually help myself with trying it again and most of the time it > then works, but doing this for years increases the pressure to do > something. > > What happens is that the MIPS builds use include/generated/compile.h: > > > arch/mips/include/asm/pbl_macros.h:28:#include > > This often ends in: > > > In file included from arch/mips/boot/start.S:20:0: > > arch/mips/include/asm/pbl_macros.h:28:31: > > fatal error: generated/compile.h: No such file or directory > > compilation terminated. > > include/generated/compile.h is generated in common/Makefile: > > > include/generated/compile.h: FORCE > > @$($(quiet)chk_compile.h) > > $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ > > "$(UTS_MACHINE)" "$(CC) $(KBUILD_CFLAGS)" > > Other users of this file add an explicit dependency on it: > > > # dependencies on generated files need to be listed explicitly > > $(obj)/version.o: include/generated/compile.h > > This only seems to work though when the dependent file is in the same > directory, but not when the file is in arch/mips/boot/. > > I tried doing the include/generated/compile.h in Makefile instead of > common/Makefile, but then the version counting gets confused and causes > unnecessary rebuilds. > > Any ideas how to solve this? arch/mips/boot/: common/ in arch/mips/Makefile or something like that? These two are both targets of the toplevel make call, so that should work. The same for $(BOARD), I think and maybe others that include generated/compile.h. Michael -- Pengutronix e.K. | | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- | ___ barebox mailing list barebox@lists.infradead.org http://lists.infradead.org/mailman/listinfo/barebox
