Bug#1010608: openldap: Flaky test test063-delta-multiprovider
Control: severity -1 important On Wed, Dec 28, 2022 at 10:32:30PM +0100, Paul Gevers wrote: Then not running the script at all is an improvement over the current situation. Flaky tests are bad. Until a better solution is found, how about skipping the test? I have uploaded -3 with the flaky test disabled. I'm downgrading the bug, but not closing it right now - I'm still optimistic about finding a proper solution (likely not for bookworm, though).
Bug#1010608: openldap: Flaky test test063-delta-multiprovider
On Wed, Dec 28, 2022 at 10:32:30PM +0100, Paul Gevers wrote: Then not running the script at all is an improvement over the current situation. Flaky tests are bad. Until a better solution is found, how about skipping the test? Not ideal, but yeah, probably an improvement over shipping a flaky test in stable. Thanks for the reminder, I'll try to upload it soon.
Bug#1010608: openldap: Flaky test test063-delta-multiprovider
Hi Ryan, On Fri, 6 May 2022 13:04:54 -0700 Ryan Tandy wrote: However I'm not comfortable proposing changes to the script if I can't validate them. Then not running the script at all is an improvement over the current situation. Flaky tests are bad. Until a better solution is found, how about skipping the test? Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1010608: openldap: Flaky test test063-delta-multiprovider
On Fri, May 06, 2022 at 01:04:54PM -0700, Ryan Tandy wrote: > Control: tag -1 help > > Hi Adrian, Hi Ryan, > On Thu, May 05, 2022 at 02:54:14PM +0300, Adrian Bunk wrote: > > https://tests.reproducible-builds.org/debian/rbuild/unstable/i386/openldap_2.5.11+dfsg-1.rbuild.log.gz > > I'm afraid this link has been superseded by the new upload (which built > successfully & reproducibly). Just to confirm, you're saying that it failed > for the same reason as the amd64 build? > > > Using ldapadd to populate server 2... > > Using ldapsearch to read all the entries from server 1... > > Using ldapsearch to read all the entries from server 2... > > Using ldapsearch to read all the entries from server 3... > > Using ldapsearch to read all the entries from server 4... > > Comparing retrieved entries from server 1 and server 2... > > Comparing retrieved entries from server 1 and server 3... > > test failed - server 1 and server 3 databases differ this was from the reproducible build log. It is the same reason, except that it was "server 3" instead of "server 4" in the "test failed" line. >... > thanks, > Ryan cu Adrian
Bug#1010608: openldap: Flaky test test063-delta-multiprovider
Control: tag -1 help Hi Adrian, On Thu, May 05, 2022 at 02:54:14PM +0300, Adrian Bunk wrote: https://tests.reproducible-builds.org/debian/rbuild/unstable/i386/openldap_2.5.11+dfsg-1.rbuild.log.gz I'm afraid this link has been superseded by the new upload (which built successfully & reproducibly). Just to confirm, you're saying that it failed for the same reason as the amd64 build? Using ldapadd to populate server 2... Using ldapsearch to read all the entries from server 1... Using ldapsearch to read all the entries from server 2... Using ldapsearch to read all the entries from server 3... Using ldapsearch to read all the entries from server 4... Comparing retrieved entries from server 1 and server 2... Comparing retrieved entries from server 1 and server 3... test failed - server 1 and server 3 databases differ I looked at this script, and I think I see how this part might be fragile: *if* I'm reading correctly, it waits for server 1 to receive the changes, but then I think it proceeds with the comparison immediately, and could fail if server 3 or 4 was slower. https://git.openldap.org/openldap/openldap/-/blob/master/tests/scripts/test063-delta-multiprovider#L309-359 This is also different from the previous section (lines 264-294) which waits a flat $SLEEP1 seconds (default: 7) for changes to be synced. However I'm not comfortable proposing changes to the script if I can't validate them. I could really use some help figuring out how to reproduce this failure. I would need to have just server 3 or 4 affected by some slowdown - and not sure what kind, whether CPU or network or disk. I guess I'll start by seeing if I can use tc to add latency to just the specific port... thanks, Ryan
Bug#1010608: openldap: Flaky test test063-delta-multiprovider
--On Thursday, May 5, 2022 3:54 PM +0300 Adrian Bunk wrote: Source: openldap Version: 2.5.11+dfsg-1 Severity: seriou Tags: ftbfs X-Debbugs-Cc: Philipp Kern https://buildd.debian.org/status/fetch.php?pkg=openldap=amd64=2. 5.12%2Bdfsg-1=1651720566=0 https://tests.reproducible-builds.org/debian/rbuild/unstable/i386/openlda p_2.5.11+dfsg-1.rbuild.log.gz ... Starting test063-delta-multiprovider for mdb... running defines.sh Initializing server configurations... Starting server 1 on TCP/IP port 9011... Using ldapsearch to check that server 1 is running... Using ldapadd for context on server 1... Starting server 2 on TCP/IP port 9012... Using ldapsearch to check that server 2 is running... Starting server 3 on TCP/IP port 9013... Using ldapsearch to check that server 3 is running... Starting server 4 on TCP/IP port 9014... Using ldapsearch to check that server 4 is running... Using ldapadd to populate server 1... Waiting 7 seconds for syncrepl to receive changes... Using ldapsearch to read all the entries from server 1... Using ldapsearch to read all the entries from server 2... Using ldapsearch to read all the entries from server 3... Using ldapsearch to read all the entries from server 4... Comparing retrieved entries from server 1 and server 2... Comparing retrieved entries from server 1 and server 3... Comparing retrieved entries from server 1 and server 4... Using ldapadd to populate server 2... Using ldapsearch to read all the entries from server 1... Using ldapsearch to read all the entries from server 2... Using ldapsearch to read all the entries from server 3... Using ldapsearch to read all the entries from server 4... Comparing retrieved entries from server 1 and server 2... Comparing retrieved entries from server 1 and server 3... test failed - server 1 and server 3 databases differ test063-delta-multiprovider failed for mdb after 28 seconds The test suite is heavily timing dependent. If you're building in a resource constrainted environment, you'll need to adjust the timers accordingly. --Quanah
Bug#1010608: openldap: Flaky test test063-delta-multiprovider
Source: openldap Version: 2.5.11+dfsg-1 Severity: seriou Tags: ftbfs X-Debbugs-Cc: Philipp Kern https://buildd.debian.org/status/fetch.php?pkg=openldap=amd64=2.5.12%2Bdfsg-1=1651720566=0 https://tests.reproducible-builds.org/debian/rbuild/unstable/i386/openldap_2.5.11+dfsg-1.rbuild.log.gz ... > Starting test063-delta-multiprovider for mdb... running defines.sh Initializing server configurations... Starting server 1 on TCP/IP port 9011... Using ldapsearch to check that server 1 is running... Using ldapadd for context on server 1... Starting server 2 on TCP/IP port 9012... Using ldapsearch to check that server 2 is running... Starting server 3 on TCP/IP port 9013... Using ldapsearch to check that server 3 is running... Starting server 4 on TCP/IP port 9014... Using ldapsearch to check that server 4 is running... Using ldapadd to populate server 1... Waiting 7 seconds for syncrepl to receive changes... Using ldapsearch to read all the entries from server 1... Using ldapsearch to read all the entries from server 2... Using ldapsearch to read all the entries from server 3... Using ldapsearch to read all the entries from server 4... Comparing retrieved entries from server 1 and server 2... Comparing retrieved entries from server 1 and server 3... Comparing retrieved entries from server 1 and server 4... Using ldapadd to populate server 2... Using ldapsearch to read all the entries from server 1... Using ldapsearch to read all the entries from server 2... Using ldapsearch to read all the entries from server 3... Using ldapsearch to read all the entries from server 4... Comparing retrieved entries from server 1 and server 2... Comparing retrieved entries from server 1 and server 3... test failed - server 1 and server 3 databases differ > test063-delta-multiprovider failed for mdb after 28 seconds (exit 1) make[4]: *** [Makefile:303: mdb-mod] Error 1
