Re: Build failures on master?!
On 22/12/2019 22:26, Dr. Matthias St. Pierre wrote: >> With apologies for being a week behind on mail, but I didn't see any >> commits in the past week that look like they were targetted to fix external >> tests, and I see (E.g.) >> https://travis-ci.org/openssl/openssl/jobs/628003784?utm_medium=notification_source=github_status >> succeeding (as well as my local build with the krb5 tests). Are the >> external tests still failing? > > I don't really understand what's going on, but it looks like job 19 fails > intermittently: Job #30814.19 (the one you > pointet out) succeeded, but #30827.19 failed. And afterwards #30846.19 > succeeded again. > > - #30814.19 (two days ago): SUCCEEDED > > https://travis-ci.org/openssl/openssl/jobs/628003784?utm_medium=notification_source=github_status > > > - #30827.19 (23 hours ago): FAILED > > https://travis-ci.org/openssl/openssl/jobs/628205488?utm_medium=notification_source=github_status > > Test Summary Report > --- > 95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1) > Failed test: 1 AFAICT the krb5 test failure on master seems to be environmental. We get this failure: make[5]: Entering directory '/home/travis/build/openssl/openssl/krb5/src/lib/rpc/unit-test' rm -f krb5cc_rpc_test_* ../../../kadmin/testing/scripts/env-setup.sh ../../../kadmin/testing/scripts/start_servers RPC_TEST_SRVTAB=/tmp/rpc_test_v5srvtab.$$ ; export RPC_TEST_SRVTAB ; \ trap "echo Failed, cleaning up... ; rm -f $RPC_TEST_SRVTAB ; ../../../kadmin/testing/scripts/env-setup.sh ../../../kadmin/testing/scripts/stop_servers ; trap '' 0 ; exit 1" 0 1 2 3 14 15 ; \ if ../../../kadmin/testing/scripts/env-setup.sh \ runtest --debug --srcdir . --host x86_64-pc-linux-gnu SERVER=./server CLIENT=./client \ KINIT=../../../clients/kinit/kinit \ KDESTROY=../../../clients/kdestroy/kdestroy \ PRIOCNTL_HACK=0 VALGRIND="" \ PASS="tcp" --tool rpc_test ; \ then \ echo Cleaning up... ; \ rm -f $RPC_TEST_SRVTAB krb5cc_rpc_test_* ; \ ../../../kadmin/testing/scripts/env-setup.sh ../../../kadmin/testing/scripts/stop_servers ; \ trap 0 ; exit 0 ; \ else exit 1 ; fi WARNING: Couldn't find tool init file Test Run By travis on Thu Dec 19 08:07:45 2019 Native configuration is x86_64-pc-linux-gnu === rpc_test tests === Schedule of variations: unix Running target unix Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target. Using /usr/share/dejagnu/config/unix.exp as generic interface file for target. Using ./config/unix.exp as tool-and-target-specific interface file. TOP=/home/travis/build/openssl/openssl/krb5/src/kadmin Running pass `tcp' ... Running ./rpc_test.0/expire.exp ... FAIL: tcp: expire: client 1: unexpected return status 2, should be 0. FAIL: tcp: expire: client 2: unexpected return status 2, should be 0. FAIL: tcp: expire: client 3: unexpected return status 2, should be 0. Running ./rpc_test.0/fullrun.exp ... FAIL: tcp: full run: client fullrun: unexpected return status 2, should be 0. FAIL: tcp: fullrun: echo test: expected 11 dots, got 1 FAIL: tcp: fullrun: expected four bad verifiers, got 0 Running ./rpc_test.0/gsserr.exp ... FAIL: tcp: gss err: timeout waiting for server output === rpc_test Summary === # of expected passes2 # of unexpected failures7 ./client version ./server version Failed, cleaning up... Makefile:678: recipe for target 'unit-test-body' failed However, if I run the same test locally with the same config options the krb5 tests pass. This is the equivalent section of the log: make[5]: Entering directory '/home/matt/dev/openssl-write/krb5/src/lib/rpc/unit-test' rm -f krb5cc_rpc_test_* ../../../kadmin/testing/scripts/env-setup.sh ../../../kadmin/testing/scripts/start_servers RPC_TEST_SRVTAB=/tmp/rpc_test_v5srvtab.$$ ; export RPC_TEST_SRVTAB ; \ trap "echo Failed, cleaning up... ; rm -f $RPC_TEST_SRVTAB ; ../../../kadmin/testing/scripts/env-setup.sh ../../../kadmin/testing/scripts/stop_servers ; trap '' 0 ; exit 1" 0 1 2 3 14 15 ; \ if ../../../kadmin/testing/scripts/env-setup.sh \ runtest --debug --srcdir . --host x86_64-pc-linux-gnu SERVER=./server CLIENT=./client \ KINIT=../../../clients/kinit/kinit \ KDESTROY=../../../clients/kdestroy/kdestroy \ PRIOCNTL_HACK=0 VALGRIND="" \ PASS="tcp" --tool rpc_test ; \ then \ echo Cleaning up... ; \ rm -f $RPC_TEST_SRVTAB krb5cc_rpc_test_* ; \ ../../../kadmin/testing/scripts/env-setup.sh ../../../kadmin/testing/scripts/stop_servers ; \ trap 0 ; exit 0 ; \ else exit 1 ; fi WARNING: Couldn't find tool init file Test run by matt on Mon Dec 23 12:26:44 2019 Native configuration is x86_64-pc-linux-gnu === rpc_test tests === Schedule of variations: unix Running target unix Using /usr/share/dejagnu/baseboards/unix.exp as board description
RE: Build failures on master?!
> With apologies for being a week behind on mail, but I didn't see any > commits in the past week that look like they were targetted to fix external > tests, and I see (E.g.) > https://travis-ci.org/openssl/openssl/jobs/628003784?utm_medium=notification_source=github_status > succeeding (as well as my local build with the krb5 tests). Are the > external tests still failing? I don't really understand what's going on, but it looks like job 19 fails intermittently: Job #30814.19 (the one you pointet out) succeeded, but #30827.19 failed. And afterwards #30846.19 succeeded again. - #30814.19 (two days ago): SUCCEEDED https://travis-ci.org/openssl/openssl/jobs/628003784?utm_medium=notification_source=github_status - #30827.19 (23 hours ago): FAILED https://travis-ci.org/openssl/openssl/jobs/628205488?utm_medium=notification_source=github_status Test Summary Report --- 95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1) Failed test: 1 Non-zero exit status: 1 95-test_external_pyca.t (Wstat: 256 Tests: 1 Failed: 1) Failed test: 1 Non-zero exit status: 1 Files=3, Tests=3, 225 wallclock secs ( 0.67 usr 0.05 sys + 112.06 cusr 29.38 csys = 142.16 CPU) Result: FAIL Makefile:2893: recipe for target '_tests' failed make[1]: *** [_tests] Error 1 make[1]: Leaving directory '/home/travis/build/openssl/openssl' Makefile:2891: recipe for target 'tests' failed make: *** [tests] Error 2 ** FAILED -- MAKE TEST - #30846.19 (2 hours ago) SUCCEEDED https://travis-ci.org/openssl/openssl/jobs/628466771?utm_medium=notification_source=github_status There are other tests failing, too. If you look at the build history, you will hardly see any successful builds on master and 1.1.1. https://travis-ci.org/openssl/openssl/builds?utm_medium=notification_source=github_status Matthias
Re: Build failures on master?!
On Sun, Dec 15, 2019 at 08:25:02PM +, Dr. Matthias St. Pierre wrote: > Gentle reminder: it's almost a month since a started this thread, but the > external pyca and krb5 > tests are still failing. Apart from complicating the review process, it also > happens that people With apologies for being a week behind on mail, but I didn't see any commits in the past week that look like they were targetted to fix external tests, and I see (E.g.) https://travis-ci.org/openssl/openssl/jobs/628003784?utm_medium=notification_source=github_status succeeding (as well as my local build with the krb5 tests). Are the external tests still failing? Thanks, Ben
AW: Build failures on master?!
Gentle reminder: it's almost a month since a started this thread, but the external pyca and krb5 tests are still failing. Apart from complicating the review process, it also happens that people are noticing the failures and are hesitating to update to the current tip of master [1]. IMHO chronically failing tests should be deactivated and an issue created for fixing them. Matthias [1] https://github.com/openssl/openssl/issues/9866#issuecomment-565714221
Re: Build failures on master?!
On Mon, Nov 18, 2019 at 10:12:52PM +, Matt Caswell wrote: > > > On 18/11/2019 21:48, Dr. Matthias St. Pierre wrote: > > The last 19 commits on https://github.com/openssl/openssl/commits/master, > > starting from Nov 14 have a red cross from the CIs. What's going on again? > > My personal impression is that builds on master are failing 50% of the time. > > The builds on master have been failing for *much* longer than that. They > should be succeeded on the PRs, but its the extended tests that fail. > > > > > It is really tedious to check pull requests for build errors just to find > > that those errors > > are well known failures from master. In particular, the krb5 an pyca tests > > are notoriously > > failing. Are there any plans to fix them or disable them? > > The build failures mostly come from the external tests (or that was the > case I haven't checked recently). The problem with these is that, on > occasion we need the maintainers of these packages to help us track down > a problem. Or in some cases we actually need them to make a change (for > example because something changed on master which revealed that they > were relying on some behaviour in master which they really shouldn't be > relying on). > > Once one of them is failing then multiple errors can build up and they > get masked by the earlier failure. > > I'll try and find some time to take another pass at the current > failures. I suspect the answer might be to disable some of the external > tests. Please let me know if you want me to look at the krb5 tests (or pester folks at upstream there to do so). I'm in Singapore for the IETF this week and have been behind on, well, everything. -Ben
Re: AW: Build failures on master?!
On 18/11/2019 22:29, Dr. Matthias St. Pierre wrote: >>> The last 19 commits on https://github.com/openssl/openssl/commits/master, >>> starting from Nov 14 have a red cross from the CIs. What's going on again? >>> My personal impression is that builds on master are failing 50% of the time. >> >> The builds on master have been failing for *much* longer than that. They >> should be succeeded on the PRs, but its the extended tests that fail. > > Yes I know. I was just counting the last consecutive failures. What actually > wonders > me is the fact that the tests do not fail throughout. In between there are > always short > periods where the tests succced. Why is this? Really? I don't see that. The travis dashboard shows red for as far back as I checked: https://travis-ci.org/openssl/openssl/builds Matt > > The reason why I issued this cry for help is because after watching it for a > long time I > got the impression that due to the ongoing failures everybody has become dull > and > indifferent agains the red crosses. > > Matthias >
AW: Build failures on master?!
> > The last 19 commits on https://github.com/openssl/openssl/commits/master, > > starting from Nov 14 have a red cross from the CIs. What's going on again? > > My personal impression is that builds on master are failing 50% of the time. > > The builds on master have been failing for *much* longer than that. They > should be succeeded on the PRs, but its the extended tests that fail. Yes I know. I was just counting the last consecutive failures. What actually wonders me is the fact that the tests do not fail throughout. In between there are always short periods where the tests succced. Why is this? The reason why I issued this cry for help is because after watching it for a long time I got the impression that due to the ongoing failures everybody has become dull and indifferent agains the red crosses. Matthias
Re: Build failures on master?!
On 18/11/2019 21:48, Dr. Matthias St. Pierre wrote: > The last 19 commits on https://github.com/openssl/openssl/commits/master, > starting from Nov 14 have a red cross from the CIs. What's going on again? > My personal impression is that builds on master are failing 50% of the time. The builds on master have been failing for *much* longer than that. They should be succeeded on the PRs, but its the extended tests that fail. > > It is really tedious to check pull requests for build errors just to find > that those errors > are well known failures from master. In particular, the krb5 an pyca tests > are notoriously > failing. Are there any plans to fix them or disable them? The build failures mostly come from the external tests (or that was the case I haven't checked recently). The problem with these is that, on occasion we need the maintainers of these packages to help us track down a problem. Or in some cases we actually need them to make a change (for example because something changed on master which revealed that they were relying on some behaviour in master which they really shouldn't be relying on). Once one of them is failing then multiple errors can build up and they get masked by the earlier failure. I'll try and find some time to take another pass at the current failures. I suspect the answer might be to disable some of the external tests. Matt > > 95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1) > 95-test_external_pyca.t (Wstat: 256 Tests: 1 Failed: 1) > > Clean builds on master should have highest priority. Because if there are too > many red > crosses, nobody cares about them anymore. > > Matthias > >
Re: Build failures on master?!
On Mon, Nov 18, 2019 at 09:48:38PM +, Dr. Matthias St. Pierre wrote: > The last 19 commits on https://github.com/openssl/openssl/commits/master, > starting from Nov 14 have a red cross from the CIs. What's going on again? I have filed 2 issues on Nov 9 that that caused the CIs to fail, that that haven't been closed yet. > My personal impression is that builds on master are failing 50% of the time. > > It is really tedious to check pull requests for build errors just to find > that those errors > are well known failures from master. In particular, the krb5 an pyca tests > are notoriously > failing. Are there any plans to fix them or disable them? > > 95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1) > 95-test_external_pyca.t (Wstat: 256 Tests: 1 Failed: 1) > > Clean builds on master should have highest priority. Because if there are too > many red > crosses, nobody cares about them anymore. Note that they are also broken in the 1.1.1-stable branch. At least pyca is something you should be able to fix with a new upstream version. Kurt
Build failures on master?!
The last 19 commits on https://github.com/openssl/openssl/commits/master, starting from Nov 14 have a red cross from the CIs. What's going on again? My personal impression is that builds on master are failing 50% of the time. It is really tedious to check pull requests for build errors just to find that those errors are well known failures from master. In particular, the krb5 an pyca tests are notoriously failing. Are there any plans to fix them or disable them? 95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1) 95-test_external_pyca.t (Wstat: 256 Tests: 1 Failed: 1) Clean builds on master should have highest priority. Because if there are too many red crosses, nobody cares about them anymore. Matthias
Re: Build failures on master
This keeps happening because the standard tests don't include no-fips or no-pic or no-module (the latter two will disable fips too), so we keep not hitting this hurdle in PRs unless we have extended tests run (done by having '[extended tests]' in the commit message). We should add a configuration with 'no-fips' among our standard tests, so this becomes automatically visible in PRs. The issue with the tests? Doing FIPS tests without checking if 'fips' is disabled or not. Cheers, Richard On Sat, 28 Sep 2019 22:25:04 +0200, Dr. Matthias St. Pierre wrote: > > Hi, > > since my last commit on master did not build successfully, I checked the CIs. > It turned out that a lot of tests are failing, and they have been for quite a > while now. > > https://travis-ci.org/openssl/openssl/builds/590874649?utm_source=github_status_medium=notification > > Apart from the well known trio boringssl, krb5, and pyca, there are some new > failing tests. > It would be nice if we could put some effort into fixing them in order to get > back to having > stable successful builds on master again. The build failures on master also > affect the > pull request CI builds and if the failure persists, people stop checking > their red crosses. > > Matthias > > > -- > > https://travis-ci.org/openssl/openssl/jobs/590874663 > > 80-test_ssl_old.t(Wstat: 256 Tests: 6 Failed: 1) > Failed test: 3 > Non-zero exit status: 1 > > > https://travis-ci.org/openssl/openssl/jobs/590874664 > > Test Summary Report > --- > 95-test_external_boringssl.t (Wstat: 256 Tests: 1 Failed: 1) > Failed test: 1 > Non-zero exit status: 1 > 95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1) > Failed test: 1 > Non-zero exit status: 1 > 95-test_external_pyca.t (Wstat: 256 Tests: 1 Failed: 1) > Failed test: 1 > Non-zero exit status: 1 > > https://travis-ci.org/openssl/openssl/jobs/590874669 > > Test Summary Report > --- > 30-test_evp.t(Wstat: 256 Tests: 19 Failed: 1) > Failed test: 1 > Non-zero exit status: 1 > 30-test_evp_fetch_prov.t (Wstat: 5376 Tests: 34 Failed: 21) > Failed tests: 1, 9-18, 25-34 > Non-zero exit status: 21 > > https://travis-ci.org/openssl/openssl/jobs/590874670 > > Test Summary Report > --- > 30-test_evp.t(Wstat: 256 Tests: 19 Failed: 1) > Failed test: 1 > Non-zero exit status: 1 > 30-test_evp_fetch_prov.t (Wstat: 5376 Tests: 34 Failed: 21) > Failed tests: 1, 9-18, 25-34 > Non-zero exit status: 21 > -- Richard Levitte levi...@openssl.org OpenSSL Project http://www.openssl.org/~levitte/
Build failures on master
Hi, since my last commit on master did not build successfully, I checked the CIs. It turned out that a lot of tests are failing, and they have been for quite a while now. https://travis-ci.org/openssl/openssl/builds/590874649?utm_source=github_status_medium=notification Apart from the well known trio boringssl, krb5, and pyca, there are some new failing tests. It would be nice if we could put some effort into fixing them in order to get back to having stable successful builds on master again. The build failures on master also affect the pull request CI builds and if the failure persists, people stop checking their red crosses. Matthias -- https://travis-ci.org/openssl/openssl/jobs/590874663 80-test_ssl_old.t(Wstat: 256 Tests: 6 Failed: 1) Failed test: 3 Non-zero exit status: 1 https://travis-ci.org/openssl/openssl/jobs/590874664 Test Summary Report --- 95-test_external_boringssl.t (Wstat: 256 Tests: 1 Failed: 1) Failed test: 1 Non-zero exit status: 1 95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1) Failed test: 1 Non-zero exit status: 1 95-test_external_pyca.t (Wstat: 256 Tests: 1 Failed: 1) Failed test: 1 Non-zero exit status: 1 https://travis-ci.org/openssl/openssl/jobs/590874669 Test Summary Report --- 30-test_evp.t(Wstat: 256 Tests: 19 Failed: 1) Failed test: 1 Non-zero exit status: 1 30-test_evp_fetch_prov.t (Wstat: 5376 Tests: 34 Failed: 21) Failed tests: 1, 9-18, 25-34 Non-zero exit status: 21 https://travis-ci.org/openssl/openssl/jobs/590874670 Test Summary Report --- 30-test_evp.t(Wstat: 256 Tests: 19 Failed: 1) Failed test: 1 Non-zero exit status: 1 30-test_evp_fetch_prov.t (Wstat: 5376 Tests: 34 Failed: 21) Failed tests: 1, 9-18, 25-34 Non-zero exit status: 21