Bug#1039941: debhelper: intermittent dh_missing error
Control: tags -1 wontfix Control: retitle -1 debhelper: Parallel d/rules not fully supported Sven Joachim: [...] I think I am beginning to understand where the race condition is. In full builds of ncurses it is possible for 'make' to process the install-arch and binary-indep targets in parallel, which can cause new files to appear in debian/tmp after "dh_missing -i" has processed the debian/.debhelper/generated/*/installed-by-* files, but before it actually reads the contents of debian/tmp. Those files will then be falsely reported as not installed. Probably such a situation is relatively unusual, and I guess dh_missing has not been designed to handle it. Will see how best to work around it in ncurses. Sven Hi Sven Thanks for the report and diagnosing the problem. Indeed, debhelper was not never built for arbitrary parallelization of dh commands or interactions between the commands. The `dh_missing` tool is a lot more susceptible to the problem as it tries to do a "global view" analysis A proper fix in your case is to split the binary-X targets, inject a synchronization barrier and run dh_missing on the other side of it. I have no clue how to do that reliably in Make while keeping the target dependencies correct. Unfortunately, there is not much I can do to fix this in debhelper as debhelper is 80% married to d/rules being a Makefile with arbitrary code execution injected "anywhere" and then 20% of features that need this not to be the case. Indeed, it is a bug/misfeature, but I do not see a way to solve it. Best regards, Niels
Bug#1039941: debhelper: intermittent dh_missing error
Control: tags -1 - unreproducible On 2023-07-01 17:18 +0200, Sven Joachim wrote: > Control: tags -1 + unreproducible > > On 2023-06-29 21:06 +0200, Sven Joachim wrote: > >> Package: debhelper >> Version: 13.11.4 >> Severity: normal >> >> In the Salsa CI for ncurses a rather strange failure happens in the >> reprotest job[1]: >> >> , >> | dh_missing -i --fail-missing >> | dh_missing: warning: usr/lib32/libncurses.so exists in debian/tmp but is >> not installed to anywhere (related file: >> "debian/tmp/usr/lib/x86_64-linux-gnu/libncurses.so") >> | dh_missing: warning: usr/lib32/libncursesw.so exists in debian/tmp but is >> not installed to anywhere (related file: >> "debian/tmp/usr/lib/x86_64-linux-gnu/libncursesw.so") >> | dh_missing: error: missing files, aborting >> ` >> >> I recall that I had seen this error occasionally in my local builds, but >> was (and still am) not able to reproduce it :-(. Maybe there is also an >> error in the ncurses package that I do not see. Any help is appreciated. > > Small update: the reprotest job in ncurses failed four times on 2023-06-29, I > rescheduled the last one (for commit 1c832772[1]) on 2023-06-30 and it > succeeded. Commit 78b0c63b[2] should sidestep this problem in the > future. That was successful, but after some changes to ncurses' debian/rules today I got another dh_missing failure in the reprotest job[1]. > Clearly there is some kind of race condition involved here, but I have > not been able to figure out what it is, and will probably not spend more > time on it unless somebody gives me a hand. I think I am beginning to understand where the race condition is. In full builds of ncurses it is possible for 'make' to process the install-arch and binary-indep targets in parallel, which can cause new files to appear in debian/tmp after "dh_missing -i" has processed the debian/.debhelper/generated/*/installed-by-* files, but before it actually reads the contents of debian/tmp. Those files will then be falsely reported as not installed. Probably such a situation is relatively unusual, and I guess dh_missing has not been designed to handle it. Will see how best to work around it in ncurses. Sven 1. https://salsa.debian.org/debian/ncurses/-/jobs/4400129
Bug#1039941: debhelper: intermittent dh_missing error
Control: tags -1 + unreproducible On 2023-06-29 21:06 +0200, Sven Joachim wrote: > Package: debhelper > Version: 13.11.4 > Severity: normal > > In the Salsa CI for ncurses a rather strange failure happens in the > reprotest job[1]: > > , > | dh_missing -i --fail-missing > | dh_missing: warning: usr/lib32/libncurses.so exists in debian/tmp but is > not installed to anywhere (related file: > "debian/tmp/usr/lib/x86_64-linux-gnu/libncurses.so") > | dh_missing: warning: usr/lib32/libncursesw.so exists in debian/tmp but is > not installed to anywhere (related file: > "debian/tmp/usr/lib/x86_64-linux-gnu/libncursesw.so") > | dh_missing: error: missing files, aborting > | > | While detecting missing files, dh_missing noted some files with a > similar name to those > | that were missing. This error /might/ be resolved by replacing > references to the > | missing files with the similarly named ones that dh_missing found - > assuming the content > | is identical. > | > | As an example, you might want to replace: > | * debian/tmp/usr/lib/x86_64-linux-gnu/libncurses.so > | with: > | * usr/lib32/libncurses.so > | in a file in debian/ or as argument to one of the dh_* tools called > from debian/rules. > | (Note it is possible the paths are not used verbatim but instead > directories > | containing or globs matching them are used instead) > | > | Alternatively, add the missing file to debian/not-installed if it > cannot and should not > | be used. > | > | The following debhelper tools have reported what they installed (with > files per package) > | * dh_install: lib32ncurses-dev (21), lib32ncurses6 (8), lib32ncursesw6 > (8), lib32tinfo6 (4), lib64ncurses-dev (1), lib64ncurses6 (0), lib64ncursesw6 > (0), lib64tinfo6 (0), libncurses-dev (29), libncurses6 (8), libncursesw6 (8), > libncursesw6-udeb (2), libtinfo6 (4), libtinfo6-udeb (2), ncurses-base (46), > ncurses-bin (4), ncurses-doc (1), ncurses-examples (1), ncurses-term (1) > | * dh_installdocs: lib32ncurses-dev (0), lib32ncurses6 (0), > lib32ncursesw6 (0), lib32tinfo6 (0), lib64ncurses-dev (0), lib64ncurses6 (0), > lib64ncursesw6 (0), lib64tinfo6 (0), libncurses-dev (0), libncurses6 (0), > libncursesw6 (0), libtinfo6 (0), ncurses-base (1), ncurses-bin (0), > ncurses-doc (3), ncurses-examples (0), ncurses-term (0) > | If the missing files are installed by another tool, please file a bug > against it. > | When filing the report, if the tool is not part of debhelper itself, > please reference the > | "Logging helpers and dh_missing" section from the "PROGRAMMING" guide > for debhelper (10.6.3+). > | (in the debhelper package: /usr/share/doc/debhelper/PROGRAMMING.gz) > | Be sure to test with dpkg-buildpackage -A/-B as the results may vary > when only a subset is built > | If the omission is intentional or no other helper can take care of this > consider adding the > | paths to debian/not-installed. > | make: *** [debian/rules:489: binary-indep] Error 25 > | make: *** Waiting for unfinished jobs > ` > > The offending two files are created in the install-arch target in > debian/rules, installed in the first two lines of the > lib32ncurses-dev.install file, and since "make" even finished the > binary-arch target I could download the artifacts[2] and convince myself > that they landed in the lib32ncurses-dev package as intended. > > I recall that I had seen this error occasionally in my local builds, but > was (and still am) not able to reproduce it :-(. Maybe there is also an > error in the ncurses package that I do not see. Any help is appreciated. Small update: the reprotest job in ncurses failed four times on 2023-06-29, I rescheduled the last one (for commit 1c832772[1]) on 2023-06-30 and it succeeded. Commit 78b0c63b[2] should sidestep this problem in the future. Clearly there is some kind of race condition involved here, but I have not been able to figure out what it is, and will probably not spend more time on it unless somebody gives me a hand. 1. https://salsa.debian.org/debian/ncurses/-/commit/1c832772028 2. https://salsa.debian.org/debian/ncurses/-/commit/78b0c63b5a8
Bug#1039941: debhelper: intermittent dh_missing error
Package: debhelper Version: 13.11.4 Severity: normal In the Salsa CI for ncurses a rather strange failure happens in the reprotest job[1]: , | dh_missing -i --fail-missing | dh_missing: warning: usr/lib32/libncurses.so exists in debian/tmp but is not installed to anywhere (related file: "debian/tmp/usr/lib/x86_64-linux-gnu/libncurses.so") | dh_missing: warning: usr/lib32/libncursesw.so exists in debian/tmp but is not installed to anywhere (related file: "debian/tmp/usr/lib/x86_64-linux-gnu/libncursesw.so") | dh_missing: error: missing files, aborting | | While detecting missing files, dh_missing noted some files with a similar name to those | that were missing. This error /might/ be resolved by replacing references to the | missing files with the similarly named ones that dh_missing found - assuming the content | is identical. | | As an example, you might want to replace: |* debian/tmp/usr/lib/x86_64-linux-gnu/libncurses.so | with: |* usr/lib32/libncurses.so | in a file in debian/ or as argument to one of the dh_* tools called from debian/rules. | (Note it is possible the paths are not used verbatim but instead directories | containing or globs matching them are used instead) | | Alternatively, add the missing file to debian/not-installed if it cannot and should not | be used. | | The following debhelper tools have reported what they installed (with files per package) |* dh_install: lib32ncurses-dev (21), lib32ncurses6 (8), lib32ncursesw6 (8), lib32tinfo6 (4), lib64ncurses-dev (1), lib64ncurses6 (0), lib64ncursesw6 (0), lib64tinfo6 (0), libncurses-dev (29), libncurses6 (8), libncursesw6 (8), libncursesw6-udeb (2), libtinfo6 (4), libtinfo6-udeb (2), ncurses-base (46), ncurses-bin (4), ncurses-doc (1), ncurses-examples (1), ncurses-term (1) |* dh_installdocs: lib32ncurses-dev (0), lib32ncurses6 (0), lib32ncursesw6 (0), lib32tinfo6 (0), lib64ncurses-dev (0), lib64ncurses6 (0), lib64ncursesw6 (0), lib64tinfo6 (0), libncurses-dev (0), libncurses6 (0), libncursesw6 (0), libtinfo6 (0), ncurses-base (1), ncurses-bin (0), ncurses-doc (3), ncurses-examples (0), ncurses-term (0) | If the missing files are installed by another tool, please file a bug against it. | When filing the report, if the tool is not part of debhelper itself, please reference the | "Logging helpers and dh_missing" section from the "PROGRAMMING" guide for debhelper (10.6.3+). | (in the debhelper package: /usr/share/doc/debhelper/PROGRAMMING.gz) | Be sure to test with dpkg-buildpackage -A/-B as the results may vary when only a subset is built | If the omission is intentional or no other helper can take care of this consider adding the | paths to debian/not-installed. | make: *** [debian/rules:489: binary-indep] Error 25 | make: *** Waiting for unfinished jobs ` The offending two files are created in the install-arch target in debian/rules, installed in the first two lines of the lib32ncurses-dev.install file, and since "make" even finished the binary-arch target I could download the artifacts[2] and convince myself that they landed in the lib32ncurses-dev package as intended. I recall that I had seen this error occasionally in my local builds, but was (and still am) not able to reproduce it :-(. Maybe there is also an error in the ncurses package that I do not see. Any help is appreciated. 1. https://salsa.debian.org/debian/ncurses/-/jobs/4378348 2. https://salsa.debian.org/debian/ncurses/-/jobs/4378348/artifacts/download
