Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
On Thu, Feb 25, 2021 at 12:05:39PM +0100, Jerome BENOIT wrote: > I was rather wondering if setting Rules-Requires-Root to yes in d/rules > will ask to bbuild to act as "needs-root" for autopkgtest. No. Rules-Requires-Root is only to tell the build scripts that some parts of the build requires real or fake root. autopkgtests are not primarily intended for running builds. Their actual purpose is to test packages as-installed. Building packages in an autopkgtest is only suitable if some extraordinary circumstance necessitates that (which is the case here as it is necessary for setting /proc/sys/kernel/unprivileged_userns_clone to 1). So it wouldn't make sense for it to reuse settings intended for build scripts. I don't know if you have seen the autopkgtest FAQ for maintainers: https://ci.debian.net/doc/file.MAINTAINERS.html And maybe take some notes on which steps you needed to take to set up your LXC autopkgtest environment. Maybe that can be fleshed out into a short tutorial for other maintainers facing similar situations. Regards, Dennis.
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
Dear Dennis, thanks for your reply. I was rather wondering if setting Rules-Requires-Root to yes in d/rules will ask to bbuild to act as "needs-root" for autopkgtest. Jerome
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
On Wed, Feb 24, 2021 at 11:01:51AM +0100, Jerome BENOIT wrote: > Dear all, > > > The autopkgtest probably will have to specify "needs-root" to set > > unprivileged_userns_clone=1 (unless the VM image already has that set > > up), but the test suite itself needn't run as root.w > Will set Rules-Requires-Root to yes in d/rules solve the issue ? You could try it out on your machine, but it probably won't: not even root can unshare with CLONE_NEWUSER in a chroot environment. Regards, Dennis.
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
Dear all, > The autopkgtest probably will have to specify "needs-root" to set > unprivileged_userns_clone=1 (unless the VM image already has that set > up), but the test suite itself needn't run as root.w Will set Rules-Requires-Root to yes in d/rules solve the issue ? Jerome
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
I just noticed that firehol has no autopkgtests yet, but since ci.debian.net can run those under LXC/qemu instead of chroot this would allow for the test suite to run. It might however be a bit of a challenge to set that up at home if troubleshooting is needed. Copying the one for root-unittests from firewalld seems like a good starting point: https://salsa.debian.org/utopia-team/firewalld/-/blob/debian/master/debian/tests/control The maintainer apparently faced a similar problem and disabled the test suite for buildd builds: https://salsa.debian.org/utopia-team/firewalld/-/commit/d24a8b8b5b29708c811bcad4b9885d1665875aca Maybe he'll be willing to help with setting this up. The autopkgtest probably will have to specify "needs-root" to set unprivileged_userns_clone=1 (unless the VM image already has that set up), but the test suite itself needn't run as root. Regards, Dennis.
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
Hi, I tried again to build the version of the package that was failing (3.1.6+ds-10) and it built *successfully* in an environment similar to the one I used for the initial test (except it was updated to the current state of unstable). In the failed build log, there was: Making check in tests make[2]: Entering directory '/<>/tests' make check-local make[3]: Entering directory '/<>/tests' ./unittest ./firehol ./fireqos ./link-balancer ./update-ipsets ./vnetbuild unshare: unshare failed: Operation not permitted make[3]: *** [Makefile:588: check-local] Error 1 make[3]: Leaving directory '/<>/tests' make[2]: *** [Makefile:471: check-am] Error 2 make[2]: Leaving directory '/<>/tests' make[1]: *** [Makefile:447: check-recursive] Error 1 make[1]: Leaving directory '/<>' dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2 make: *** [debian/rules:8: binary] Error 25 In the successful log, there's: Making check in tests make[2]: Entering directory '/<>/tests' make check-local make[3]: Entering directory '/<>/tests' echo "Unprivileged user namespaces not enabled - not running tests" Unprivileged user namespaces not enabled - not running tests make[3]: Leaving directory '/<>/tests' make[2]: Leaving directory '/<>/tests' make[2]: Entering directory '/<>' make[2]: Nothing to be done for 'check-am'. make[2]: Leaving directory '/<>' make[1]: Leaving directory '/<>' I don't understand why, in the failing build, the userns were detected as being enabled. It might be that another package (built previously) was enabling them during build (or during installation of one of its build-depends). But I find that surprising, given all builds are performed as an unprivileged user... In any case, I can confirm that setting /proc/sys/kernel/unprivileged_userns_clone to 1 makes the failure reproducible (still on a 4.19 kernel). And 3.1.7+ds-1 is also affected. Lucas
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
On Sun, Feb 21, 2021 at 07:32:19PM +0100, Paul Gevers wrote: > On 21-02-2021 14:02, Jerome BENOIT wrote: > > please consider to tag #982719 as bullseye-ignore > > given that the issue is not a package issue but > > it seems rather related to an chroot issue. > > A fresh upload from mere hours ago built successfully on our buildd > infrastructure with > Kernel: Linux 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) > amd64 (x86_64) No, it didn't. On that buildd server kernel.unprivileged_userns_clone must have been 0 which effectively disables the test suite as can be seen in the build log[1]: ... Making check in tests make[2]: Entering directory '/<>/tests' make check-local make[3]: Entering directory '/<>/tests' echo "Unprivileged user namespaces not enabled - not running tests" Unprivileged user namespaces not enabled - not running tests make[3]: Leaving directory '/<>/tests' make[2]: Leaving directory '/<>/tests' make[2]: Entering directory '/<>' make[2]: Nothing to be done for 'check-am'. make[2]: Leaving directory '/<>' make[1]: Leaving directory '/<>' ... The buildd admins need to decide if they want kernel.unprivileged_userns_clone to always be 1 or 0 before every build (or is it upon the package to specify that?). Otherwise this will introduce breakage randomly as the setting gets toggled. > Does that mean this bug is not yet fully understood? (Or did I miss > details, I didn't read the bug in *full* detail.) It's understood alright. The gist is that the test suite relies on a rather new system call named unshare(2) which cannot be used in a chroot environment when called with CLONE_NEWUSER -- at least that's what the manpage says. (I expect this issue to affect more packages in the future since unshare allows for convenient unit testing of network and firewall code without mockups). I read about an sbuild backend that uses unshare instead of schroot which could help, but the manpage called it "experimental". All this somewhat makes this an issue with the buildd environment itself, thus raising the question of how this bug should be handled. By disabling the test suite and reuploading, or through a bullseye-ignore? The latter would have the advantage of keeping this on the table for when others run into this. Regards, Dennis. 1: https://buildd.debian.org/status/fetch.php?pkg=firehol=all=3.1.7%2Bds-1=1613917812=1
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
Hi, On 21-02-2021 14:02, Jerome BENOIT wrote: > please consider to tag #982719 as bullseye-ignore > given that the issue is not a package issue but > it seems rather related to an chroot issue. A fresh upload from mere hours ago built successfully on our buildd infrastructure with Kernel: Linux 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) amd64 (x86_64) Does that mean this bug is not yet fully understood? (Or did I miss details, I didn't read the bug in *full* detail.) Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
Dear Paul Gevers, please consider to tag #982719 as bullseye-ignore given that the issue is not a package issue but it seems rather related to an chroot issue. Best regards, Jerome
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
On Sun, Feb 21, 2021 at 12:35:13PM +0100, Jerome BENOIT wrote: > I think that the issue is actally not a firehol issue. Correct. > But I cannot figure out to where the issue can be redirected. > For now, I am reluctant to neutralize the tests. Then you should ask Paul Gevers or another assistant from the Release Team[1] to tag this as bullseye-ignore. Regards, Dennis. 1: https://wiki.debian.org/Teams/ReleaseTeam
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
Dear Dennis, thanks for your message. I think that the issue is actally not a firehol issue. But I cannot figure out to where the issue can be redirected. For now, I am reluctant to neutralize the tests. @Lucas: do you have any hint ? Thanks, Jeromr -- Jerome BENOIT | calculus+at-rezozer^dot*net https://qa.debian.org/developer.php?login=calcu...@rezozer.net AE28 AE15 710D FF1D 87E5 A762 3F92 19A6 7F36 C68B
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
I think since all tests depend on the newns tool working, you have to disable the entire test suite (I attached a proposal patch, but didn't test it) and then test it. But you should still get confirmation from someone more knowledgable in this that what I wrote here is actually correct and that disabling the test suite is the proper way (IIRC test suites should be disabled only as a last resort, but I could be wrong here). Regards, Dennis. P.S.: I noticed that you didn't include me nor Lucas in To: or Cc:. Remember: messages to 982...@bugs.debian.org are only sent to maintainers (you) and people who actively subscribe (rarely done), so you need to explicitly add others with To: or use X-Debbugs-CC: (preferred) to reach them reliably. Not everyone checks bug web pages, esp. release team members as they deal with /a lot/ of bugs. Read up on X-Debbugs-CC: if you haven't used it in a while. firehol-nocheck.patch.gz Description: application/gzip
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
Hi Dennis, thanks for confirming. May I simply neutralize the involved test temporarily ? Cheers, Jerome
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
Control: tag -1 moreinfo confirmed I could reproduce this, but only in a bullseye chroot build environment with a running buster (4.19) kernel. I haven't tried with a bullseye kernel + bullseye chroot. The build log in the bug report states similarly: Kernel: Linux 4.19.0-6-cloud-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) amd64 (x86_64) I'd also like to point out this snippet from the ERRORS section of the manpage for unshare(2): EPERM (since Linux 3.9) CLONE_NEWUSER was specified in flags and the caller is in a chroot environment (i.e., the caller's root direc‐ tory does not match the root directory of the mount namespace in which it resides). This is important because the test suite executes /bin/unshare with -r in firehol-3.1.6+ds/tests/tools/newns which then calls unshare(CLONE_NEWUSER): $ strace -e trace=unshare unshare -r unshare -n -m /bin/date unshare(CLONE_NEWUSER) = 0 unshare(CLONE_NEWNS|CLONE_NEWNET) = 0 In fact under all kernels I've tried (4.19, 5.10) this command even when run as root always fails with "Operation not permitted": chroot /chrootenvdir /bin/unshare -r /bin/date My understanding of the interaction of chroot and unshare(2)/namespaces in general is incomplete, but this to me indicates the test suite of firehol in its current form simply cannot be run in a chroot environment (at least on 4.19 kernels, but probably also later ones) and should be disabled until the unshare backend for sbuild has matured enough to be used on the build servers. @Jerome: Could you try to get someone from either the Debian Kernel Team or the sbuild maintainers to take a look at this? If you get no answer there then make sure notify the release team as they may have to waive this one. Regards, Dennis.
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
Dear Lucas, thanks for your report. I can not reproduce the issue on a Sid schroot a the time of writing. Please, can you confirm the issue on your side ? Best wishes, Jerome -- Jerome BENOIT | calculus+at-rezozer^dot*net https://qa.debian.org/developer.php?login=calcu...@rezozer.net AE28 AE15 710D FF1D 87E5 A762 3F92 19A6 7F36 C68B
Bug#982719: firehol: FTBFS: dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2
Source: firehol Version: 3.1.6+ds-10 Severity: serious Justification: FTBFS on amd64 Tags: bullseye sid ftbfs Usertags: ftbfs-20210213 ftbfs-bullseye Hi, During a rebuild of all packages in sid, your package failed to build on amd64. Relevant part (hopefully): > make[3]: Entering directory '/<>/tests' > ./unittest ./firehol ./fireqos ./link-balancer ./update-ipsets ./vnetbuild > unshare: unshare failed: Operation not permitted > make[3]: *** [Makefile:588: check-local] Error 1 > make[3]: Leaving directory '/<>/tests' > make[2]: *** [Makefile:471: check-am] Error 2 > make[2]: Leaving directory '/<>/tests' > make[1]: *** [Makefile:447: check-recursive] Error 1 > make[1]: Leaving directory '/<>' > dh_auto_test: error: make -j1 check VERBOSE=1 returned exit code 2 The full build log is available from: http://qa-logs.debian.net/2021/02/13/firehol_3.1.6+ds-10_unstable.log A list of current common problems and possible solutions is available at http://wiki.debian.org/qa.debian.org/FTBFS . You're welcome to contribute! If you reassign this bug to another package, please marking it as 'affects'-ing this package. See https://www.debian.org/Bugs/server-control#affects If you fail to reproduce this, please provide a build log and diff it with me so that we can identify if something relevant changed in the meantime. About the archive rebuild: The rebuild was done on EC2 VM instances from Amazon Web Services, using a clean, minimal and up-to-date chroot. Every failed build was retried once to eliminate random failures.
