On Mon, 2024-09-30 at 11:10 +0200, Yoann Congal via lists.openembedded.org wrote: > From: Yoann Congal <[email protected]> > > The current reproducibility test stops all build tasks when a single > task fails (default BitBake behavior). This means that a single build > failure prevents the reproducibility of other packages from being > tested, which is not ideal. > > To address this, we now use the --continue option of BitBake during test > builds, allowing the build process to proceed even when some tasks fail. > The failure cases are handled as gracefully as possible. > > In the event of a build failure: > - The entire reproducibility test will be considered a failure. > - The complete BitBake log will be saved in the "saved output" directory > to facilitate debugging. On the autobuilder, this log should be > available at https://autobuilder.yocto.io/pub/repro-fail/. > - The last 20 lines of the log, which should show the failing recipes, > will be displayed in the oe-selftest console. > - If BitBake fails to create the deployment directory, it will be > manually created to allow the comparison process to proceed. > > Here is what the output looks like when testing reproducibility of bash, > a failing recipe (hello-fail) and a non-reproducible recipe > (hello-norepro): > $ rm -rf ../build-master-st <snip>/yocto/repro_output/* ; > OEQA_DEBUGGING_SAVED_OUTPUT=<snip>/yocto/repro_output oe-selftest -r > reproducible.ReproducibleTests > <snip> > 2024-09-30 00:04:27,485 - oe-selftest - INFO - test_reproducible_builds > (reproducible.ReproducibleTests.test_reproducible_builds) > 2024-09-30 00:05:31,676 - oe-selftest - INFO - Non-reproducible packages > will be copied to <snip>/yocto/repro_output/oe-reproducible-20240930-aukc_3o6 > 2024-09-30 00:05:31,676 - oe-selftest - INFO - Building reproducibleA > (sstate allowed)... > 2024-09-30 00:13:08,009 - oe-selftest - INFO - Bitbake failed! but keep > going... Log: > 2024-09-30 00:13:08,009 - oe-selftest - INFO - Command 'bitbake > --continue bash hello-norepro hello-fail' returned non-zero exit status 1: > 2024-09-30 00:13:08,009 - oe-selftest - INFO - > 2024-09-30 00:13:08,009 - oe-selftest - INFO - ... (last 20 lines of > output) > 2024-09-30 00:13:08,009 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_package_write_deb: Started > 2024-09-30 00:13:08,009 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_package_qa: Started > 2024-09-30 00:13:08,009 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_populate_sysroot: Succeeded > 2024-09-30 00:13:08,009 - oe-selftest - INFO - NOTE: Running setscene > task 342 of 343 > (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_create_package_spdx_setscene) > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_create_package_spdx_setscene: Started > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_create_package_spdx_setscene: Succeeded > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: Running setscene > task 343 of 343 > (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_create_spdx_setscene) > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_create_spdx_setscene: Started > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_create_spdx_setscene: Succeeded > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_package_qa: Succeeded > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_package_write_deb: Succeeded > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: Running noexec > task 979 of 990 > (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_build) > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: Tasks Summary: > Attempted 979 tasks of which 833 didn't need to be rerun and 1 failed. > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: Generating JSON > CVE summary > 2024-09-30 00:13:08,010 - oe-selftest - INFO - Complete CVE JSON report > summary created at: > <snip>/yocto/poky/build-master-st/reproducibleA/tmp/log/cve/cve-summary.json > 2024-09-30 00:13:08,010 - oe-selftest - INFO - > 2024-09-30 00:13:08,010 - oe-selftest - INFO - Summary: 1 task failed: > 2024-09-30 00:13:08,010 - oe-selftest - INFO - > <snip>/yocto/poky/meta/recipes-core/hello-single/hello-fail_1.0.bb:do_compile > 2024-09-30 00:13:08,010 - oe-selftest - INFO - log: > <snip>/yocto/poky/build-master-st/reproducibleA/tmp/work/core2-64-poky-linux/hello-fail/1.0/temp/log.do_compile.2116040 > 2024-09-30 00:13:08,010 - oe-selftest - INFO - Summary: There was 1 > ERROR message, returning a non-zero exit code. > 2024-09-30 00:13:08,011 - oe-selftest - INFO - reproducibleA build failed. > Trying to compute built packages differences but the test will fail. > 2024-09-30 00:13:08,011 - oe-selftest - INFO - Building > reproducibleB-extended (sstate NOT allowed)... > 2024-09-30 00:40:12,933 - oe-selftest - INFO - Bitbake failed! but keep > going... Log: > 2024-09-30 00:40:12,933 - oe-selftest - INFO - Command 'bitbake > --continue bash hello-norepro hello-fail' returned non-zero exit status 1: > 2024-09-30 00:40:12,933 - oe-selftest - INFO - > 2024-09-30 00:40:12,933 - oe-selftest - INFO - ... (last 20 lines of > output) > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_package_write_deb: Started > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_package_qa: Started > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_create_spdx: Succeeded > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: Running task 978 > of 990 > (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_create_package_spdx) > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_create_package_spdx: Started > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_create_package_spdx: Succeeded > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_package_write_ipk: Succeeded > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_package_qa: Succeeded > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_package_write_deb: Succeeded > 2024-09-30 00:40:12,934 - oe-selftest - INFO - NOTE: recipe > bash-5.2.32-r0: task do_package_write_rpm: Succeeded > 2024-09-30 00:40:12,934 - oe-selftest - INFO - NOTE: Running noexec > task 979 of 990 > (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_build) > 2024-09-30 00:40:12,934 - oe-selftest - INFO - NOTE: Tasks Summary: > Attempted 979 tasks of which 2 didn't need to be rerun and 1 failed. > 2024-09-30 00:40:12,934 - oe-selftest - INFO - NOTE: Generating JSON > CVE summary > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Complete CVE JSON report > summary created at: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/log/cve/cve-summary.json > 2024-09-30 00:40:12,934 - oe-selftest - INFO - > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Summary: 1 task failed: > 2024-09-30 00:40:12,934 - oe-selftest - INFO - > <snip>/yocto/poky/meta/recipes-core/hello-single/hello-fail_1.0.bb:do_compile > 2024-09-30 00:40:12,934 - oe-selftest - INFO - log: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/work/core2-64-poky-linux/hello-fail/1.0/temp/log.do_compile.3007134 > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Summary: There were 5 > WARNING messages. > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Summary: There was 1 > ERROR message, returning a non-zero exit code. > 2024-09-30 00:40:12,934 - oe-selftest - INFO - reproducibleB-extended build > failed. Trying to compute built packages differences but the test will fail. > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Checking deb packages for > differences... > 2024-09-30 00:40:12,974 - oe-selftest - INFO - Reproducibility summary for > deb: same=52 different=1 different_excluded=0 missing=0 total=53 > unused_exclusions=[] > 2024-09-30 00:40:12,976 - oe-selftest - INFO - Checking ipk packages for > differences... > 2024-09-30 00:40:13,014 - oe-selftest - INFO - Reproducibility summary for > ipk: same=52 different=1 different_excluded=0 missing=0 total=53 > unused_exclusions=[] > 2024-09-30 00:40:13,015 - oe-selftest - INFO - Checking rpm packages for > differences... > 2024-09-30 00:40:13,047 - oe-selftest - INFO - Reproducibility summary for > rpm: same=52 different=1 different_excluded=0 missing=0 total=53 > unused_exclusions=[] > 2024-09-30 00:40:13,048 - oe-selftest - INFO - Running diffoscope > 2024-09-30 00:40:14,051 - oe-selftest - INFO - ... FAIL > 2024-09-30 00:40:14,052 - oe-selftest - INFO - Traceback (most recent call > last): > File "<snip>/yocto/poky/meta/lib/oeqa/selftest/cases/reproducible.py", > line 380, in test_reproducible_builds > self.fail('\n'.join(fails)) > AssertionError: Bitbake reproducibleA failure > Bitbake reproducibleB-extended failure > The following deb packages are different and not in exclusion list: > > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb > The following ipk packages are different and not in exclusion list: > > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/hello-norepro_1.0-r0_core2-64.ipk > The following rpm packages are different and not in exclusion list: > > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/hello-norepro-1.0-r0.core2_64.rpm > > 2024-09-30 00:40:14,052 - oe-selftest - INFO - > ====================================================================== > 2024-09-30 00:40:14,052 - oe-selftest - INFO - FAIL: > test_reproducible_builds > (reproducible.ReproducibleTests.test_reproducible_builds) > 2024-09-30 00:40:14,052 - oe-selftest - INFO - > ---------------------------------------------------------------------- > 2024-09-30 00:40:14,052 - oe-selftest - INFO - Traceback (most recent call > last): > File "<snip>/yocto/poky/meta/lib/oeqa/selftest/cases/reproducible.py", > line 380, in test_reproducible_builds > self.fail('\n'.join(fails)) > AssertionError: Bitbake reproducibleA failure > Bitbake reproducibleB-extended failure > The following deb packages are different and not in exclusion list: > > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb > The following ipk packages are different and not in exclusion list: > > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/hello-norepro_1.0-r0_core2-64.ipk > The following rpm packages are different and not in exclusion list: > > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/hello-norepro-1.0-r0.core2_64.rpm > > 2024-09-30 00:40:14,053 - oe-selftest - INFO - > ---------------------------------------------------------------------- > 2024-09-30 00:40:14,053 - oe-selftest - INFO - Ran 1 test in 2146.797s > 2024-09-30 00:40:14,053 - oe-selftest - INFO - FAILED > 2024-09-30 00:40:14,053 - oe-selftest - INFO - (failures=1) > 2024-09-30 00:40:17,620 - oe-selftest - INFO - RESULTS: > 2024-09-30 00:40:17,621 - oe-selftest - INFO - RESULTS - > reproducible.ReproducibleTests.test_reproducible_builds: FAILED (2146.57s) > 2024-09-30 00:40:17,623 - oe-selftest - INFO - SUMMARY: > 2024-09-30 00:40:17,623 - oe-selftest - INFO - oe-selftest () - Ran 1 test > in 2146.798s > 2024-09-30 00:40:17,623 - oe-selftest - INFO - oe-selftest - FAIL - > Required tests failed (successes=0, skipped=0, failures=1, errors=0) > > => Test failed but hello-norepro is displayed as non-reproducible. > > The testresult.json contains: > { > "oeselftest_debian-12_qemux86-64_20240930000424": { > "configuration": { <snip> }, > "result": { > "reproducible": { > "files": { > "package_deb": { > "different": [ > { > "reference": > "<snip>/yocto/poky/build-master-st/reproducibleA/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb", > "test": > "<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb" > } > ], > "different_excluded": [], > "missing": [], > "same": [ > <snip> > { > "reference": > "<snip>/yocto/poky/build-master-st/reproducibleA/tmp/deploy/deb/./core2-64/bash_5.2.32-r0_amd64.deb", > "test": > "<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/bash_5.2.32-r0_amd64.deb" > }, > <snip> > ] > }, > "package_ipk": { <same as deb> }, > "package_rpm": { <same as deb> } > } > }, > "reproducible.ReproducibleTests.test_reproducible_builds": { > "duration": 2146.5671875476837, > "log": "Traceback (most recent call last):\n File > \"<snip>/yocto/poky/meta/lib/oeqa/selftest/cases/reproducible.py\", line 380, > in test_reproducible_builds\n > self.fail('\\n'.join(fails))\nAssertionError: Bitbake reproducibleA > failure\nBitbake reproducibleB-extended failure\nThe following deb packages > are different and not in exclusion > list:\n<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb\nThe > following ipk packages are different and not in exclusion > list:\n<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/hello-norepro_1.0-r0_core2-64.ipk\nThe > following rpm packages are different and not in exclusion > list:\n<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/hello-norepro-1.0-r0.core2_64.rpm\n", > "status": "FAILED" > }, > "reproducible.rawlogs": { > "log": "DIFFERENT: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb\nSAME: > > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/bash_5.2.32-r0_amd64.deb\n<snip>" > } > } > } > } > > => "reproducible.ReproducibleTests.test_reproducible_builds".status is > correctly "FAILED" but the reproducibility of bash and hello-norepro is > tested. > > Signed-off-by: Yoann Congal <[email protected]> > --- > meta/lib/oeqa/selftest/cases/reproducible.py | 57 ++++++++++++++++---- > 1 file changed, 48 insertions(+), 9 deletions(-) > > diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py > b/meta/lib/oeqa/selftest/cases/reproducible.py > index 3d3f30eebc..af4e53af2f 100644 > --- a/meta/lib/oeqa/selftest/cases/reproducible.py > +++ b/meta/lib/oeqa/selftest/cases/reproducible.py > @@ -215,7 +215,7 @@ class ReproducibleTests(OESelftestTestCase): > bb.utils.mkdirhier(os.path.dirname(dest)) > shutil.copyfile(source, dest) > > - def do_test_build(self, name, use_sstate): > + def do_test_build(self, name, use_sstate, output_log=None): > capture_vars = ['DEPLOY_DIR_' + c.upper() for c in > self.package_classes] > > tmpdir = os.path.join(self.topdir, name, 'tmp') > @@ -234,11 +234,19 @@ class ReproducibleTests(OESelftestTestCase): > ''').format(package_classes=' '.join('package_%s' % c for c in > self.package_classes), > tmpdir=tmpdir) > > + bitbake_failure_count = 0 > if not use_sstate: > if self.sstate_targets: > self.logger.info("Building prebuild for %s (sstate > allowed)..." % (name)) > self.write_config(config) > - bitbake(' '.join(self.sstate_targets)) > + try: > + bitbake("--continue "+' '.join(self.sstate_targets), > + output_log=output_log, limit_exc_output=20) > + except AssertionError as e: > + bitbake_failure_count += 1 > + self.logger.info("Bitbake failed! but keep going... Log:") > + for line in str(e).split("\n"): > + self.logger.info(" "+line) > > # This config fragment will disable using shared and the sstate > # mirror, forcing a complete build from scratch > @@ -251,8 +259,21 @@ class ReproducibleTests(OESelftestTestCase): > self.write_config(config) > d = get_bb_vars(capture_vars) > # targets used to be called images > - bitbake(' '.join(getattr(self, 'images', self.targets))) > - return d > + try: > + bitbake("--continue "+' '.join(getattr(self, 'images', > self.targets)), > + output_log=output_log, limit_exc_output=20) > + except AssertionError as e: > + bitbake_failure_count += 1 > + > + self.logger.info("Bitbake failed! but keep going... Log:") > + for line in str(e).split("\n"): > + self.logger.info(" "+line) > + for c in self.package_classes: > + deploy = d['DEPLOY_DIR_' + c.upper()] > + if not os.path.exists(deploy): > + self.logger.info("Manually creating %s" % deploy) > + bb.utils.mkdirhier(deploy) > + return (d, bitbake_failure_count) > > def test_reproducible_builds(self): > def strip_topdir(s): > @@ -278,15 +299,33 @@ class ReproducibleTests(OESelftestTestCase): > # https://bugzilla.yoctoproject.org/show_bug.cgi?id=15554 > # So, the reproducibleA & reproducibleB directories are changed to > reproducibleA & reproducibleB-extended to have different size. > > - vars_A = self.do_test_build('reproducibleA', self.build_from_sstate) > - > - vars_B = self.do_test_build('reproducibleB-extended', False) > + fails = [] > + vars_list = [None, None] > + > + for i, (name, use_sstate) in enumerate( > + (('reproducibleA', self.build_from_sstate), > + ('reproducibleB-extended', False))): > + if self.save_results: > + output_log_path = os.path.join(save_dir, "bitbake_%s.log" % > name) > + output_log = bb.msg.logger_create("bitbake_%s" % name, > + > output=open(output_log_path, mode="w", encoding="utf-8"), > + preserve_handlers=True) > + (variables, bitbake_failure_count) = self.do_test_build( > + name, use_sstate, output_log=output_log) > + if bitbake_failure_count > 0: > + self.logger.info('%s build failed. Trying to compute built > packages differences but the test will fail.' % name) > + fails.append("Bitbake %s failure" % name) > + else: > + if self.save_results: > + # Remove full logs if bitbake was successful > + del output_log > + os.remove(output_log_path) > + vars_list[i] = variables > > + vars_A, vars_B = vars_list > # NOTE: The temp directories from the reproducible build are > purposely > # kept after the build so it can be diffed for debugging. > > - fails = [] > - > for c in self.package_classes: > with self.subTest(package_class=c): > package_class = 'package_' + c >
I think the change makes sense but the code above feels a bit ugly and hard to read. Rather than having to create directories and remove directories/logs and change output paths, can we not just copy the log in case of failure? Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#205102): https://lists.openembedded.org/g/openembedded-core/message/205102 Mute This Topic: https://lists.openembedded.org/mt/108732482/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
