Re: Build failures on master?!

2019-12-23 Thread Matt Caswell



On 22/12/2019 22:26, Dr. Matthias St. Pierre wrote:
>> With apologies for being a week behind on mail, but I didn't see any
>> commits in the past week that look like they were targetted to fix external
>> tests, and I see (E.g.)
>> https://travis-ci.org/openssl/openssl/jobs/628003784?utm_medium=notification_source=github_status
>> succeeding (as well as my local build with the krb5 tests).  Are the
>> external tests still failing?
> 
> I don't really understand what's going on, but it looks like job 19 fails 
> intermittently: Job #30814.19 (the one you 
> pointet out) succeeded, but #30827.19 failed. And afterwards #30846.19 
> succeeded again.
> 
> - #30814.19 (two days ago): SUCCEEDED
>   
> https://travis-ci.org/openssl/openssl/jobs/628003784?utm_medium=notification_source=github_status
> 
> 
> - #30827.19 (23 hours ago): FAILED
>   
> https://travis-ci.org/openssl/openssl/jobs/628205488?utm_medium=notification_source=github_status
> 
> Test Summary Report
> ---
> 95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1)
>   Failed test:  1

AFAICT the krb5 test failure on master seems to be environmental. We get
this failure:

make[5]: Entering directory
'/home/travis/build/openssl/openssl/krb5/src/lib/rpc/unit-test'
rm -f krb5cc_rpc_test_*
../../../kadmin/testing/scripts/env-setup.sh
../../../kadmin/testing/scripts/start_servers
RPC_TEST_SRVTAB=/tmp/rpc_test_v5srvtab.$$ ; export RPC_TEST_SRVTAB ; \
trap "echo Failed, cleaning up... ; rm -f $RPC_TEST_SRVTAB ;
../../../kadmin/testing/scripts/env-setup.sh
../../../kadmin/testing/scripts/stop_servers   ; trap '' 0 ; exit 1" 0 1
2 3 14 15 ; \
if ../../../kadmin/testing/scripts/env-setup.sh \
runtest --debug --srcdir . --host x86_64-pc-linux-gnu SERVER=./server
CLIENT=./client \
KINIT=../../../clients/kinit/kinit \
KDESTROY=../../../clients/kdestroy/kdestroy \
PRIOCNTL_HACK=0 VALGRIND="" \
PASS="tcp" --tool rpc_test  ; \
then \
echo Cleaning up... ; \
rm -f $RPC_TEST_SRVTAB krb5cc_rpc_test_* ; \
../../../kadmin/testing/scripts/env-setup.sh
../../../kadmin/testing/scripts/stop_servers   ; \
trap 0 ; exit 0 ; \
else exit 1 ; fi
WARNING: Couldn't find tool init file
Test Run By travis on Thu Dec 19 08:07:45 2019
Native configuration is x86_64-pc-linux-gnu

=== rpc_test tests ===

Schedule of variations:
unix

Running target unix
Using /usr/share/dejagnu/baseboards/unix.exp as board description file
for target.
Using /usr/share/dejagnu/config/unix.exp as generic interface file for
target.
Using ./config/unix.exp as tool-and-target-specific interface file.
TOP=/home/travis/build/openssl/openssl/krb5/src/kadmin
Running pass `tcp' ...
Running ./rpc_test.0/expire.exp ...
FAIL: tcp: expire: client 1: unexpected return status 2, should be 0.
FAIL: tcp: expire: client 2: unexpected return status 2, should be 0.
FAIL: tcp: expire: client 3: unexpected return status 2, should be 0.
Running ./rpc_test.0/fullrun.exp ...
FAIL: tcp: full run: client fullrun: unexpected return status 2, should
be 0.
FAIL: tcp: fullrun: echo test: expected 11 dots, got 1
FAIL: tcp: fullrun: expected four bad verifiers, got 0
Running ./rpc_test.0/gsserr.exp ...
FAIL: tcp: gss err: timeout waiting for server output

=== rpc_test Summary ===

# of expected passes2
# of unexpected failures7
./client version 
./server version 
Failed, cleaning up...
Makefile:678: recipe for target 'unit-test-body' failed



However, if I run the same test locally with the same config options the
krb5 tests pass. This is the equivalent section of the log:


make[5]: Entering directory
'/home/matt/dev/openssl-write/krb5/src/lib/rpc/unit-test'
rm -f krb5cc_rpc_test_*
../../../kadmin/testing/scripts/env-setup.sh
../../../kadmin/testing/scripts/start_servers
RPC_TEST_SRVTAB=/tmp/rpc_test_v5srvtab.$$ ; export RPC_TEST_SRVTAB ; \
trap "echo Failed, cleaning up... ; rm -f $RPC_TEST_SRVTAB ;
../../../kadmin/testing/scripts/env-setup.sh
../../../kadmin/testing/scripts/stop_servers   ; trap '' 0 ; exit 1" 0 1
2 3 14 15 ; \
if ../../../kadmin/testing/scripts/env-setup.sh \
runtest --debug --srcdir . --host x86_64-pc-linux-gnu SERVER=./server
CLIENT=./client \
KINIT=../../../clients/kinit/kinit \
KDESTROY=../../../clients/kdestroy/kdestroy \
PRIOCNTL_HACK=0 VALGRIND="" \
PASS="tcp" --tool rpc_test  ; \
then \
echo Cleaning up... ; \
rm -f $RPC_TEST_SRVTAB krb5cc_rpc_test_* ; \
../../../kadmin/testing/scripts/env-setup.sh
../../../kadmin/testing/scripts/stop_servers   ; \
trap 0 ; exit 0 ; \
else exit 1 ; fi
WARNING: Couldn't find tool init file
Test run by matt on Mon Dec 23 12:26:44 2019
Native configuration is x86_64-pc-linux-gnu

=== rpc_test tests ===

Schedule of variations:
unix

Running target unix
Using /usr/share/dejagnu/baseboards/unix.exp as board description 

RE: Build failures on master?!

2019-12-22 Thread Dr. Matthias St. Pierre
> With apologies for being a week behind on mail, but I didn't see any
> commits in the past week that look like they were targetted to fix external
> tests, and I see (E.g.)
> https://travis-ci.org/openssl/openssl/jobs/628003784?utm_medium=notification_source=github_status
> succeeding (as well as my local build with the krb5 tests).  Are the
> external tests still failing?

I don't really understand what's going on, but it looks like job 19 fails 
intermittently: Job #30814.19 (the one you 
pointet out) succeeded, but #30827.19 failed. And afterwards #30846.19 
succeeded again.

- #30814.19 (two days ago): SUCCEEDED
  
https://travis-ci.org/openssl/openssl/jobs/628003784?utm_medium=notification_source=github_status


- #30827.19 (23 hours ago): FAILED
  
https://travis-ci.org/openssl/openssl/jobs/628205488?utm_medium=notification_source=github_status

Test Summary Report
---
95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
95-test_external_pyca.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=3, Tests=3, 225 wallclock secs ( 0.67 usr  0.05 sys + 112.06 cusr 
29.38 csys = 142.16 CPU)
Result: FAIL
Makefile:2893: recipe for target '_tests' failed
make[1]: *** [_tests] Error 1
make[1]: Leaving directory '/home/travis/build/openssl/openssl'
Makefile:2891: recipe for target 'tests' failed
make: *** [tests] Error 2
** FAILED -- MAKE TEST

- #30846.19 (2 hours ago) SUCCEEDED
  
https://travis-ci.org/openssl/openssl/jobs/628466771?utm_medium=notification_source=github_status

There are other tests failing, too. If you look at the build history, you will 
hardly see any successful builds on
master and 1.1.1.

https://travis-ci.org/openssl/openssl/builds?utm_medium=notification_source=github_status


Matthias





Re: Build failures on master?!

2019-12-21 Thread Benjamin Kaduk
On Sun, Dec 15, 2019 at 08:25:02PM +, Dr. Matthias St. Pierre wrote:
> Gentle reminder: it's almost a month since a started this thread, but the 
> external pyca and krb5 
> tests are still failing. Apart from complicating the review process, it also 
> happens that people

With apologies for being a week behind on mail, but I didn't see any
commits in the past week that look like they were targetted to fix external
tests, and I see (E.g.)
https://travis-ci.org/openssl/openssl/jobs/628003784?utm_medium=notification_source=github_status
succeeding (as well as my local build with the krb5 tests).  Are the
external tests still failing?

Thanks,

Ben


AW: Build failures on master?!

2019-12-15 Thread Dr. Matthias St. Pierre
Gentle reminder: it's almost a month since a started this thread, but the 
external pyca and krb5 
tests are still failing. Apart from complicating the review process, it also 
happens that people
are noticing the failures and are hesitating to update to the current tip of 
master [1].

IMHO chronically failing tests should be deactivated and an issue created for 
fixing them.

Matthias

[1] https://github.com/openssl/openssl/issues/9866#issuecomment-565714221



Re: Build failures on master?!

2019-11-21 Thread Benjamin Kaduk
On Mon, Nov 18, 2019 at 10:12:52PM +, Matt Caswell wrote:
> 
> 
> On 18/11/2019 21:48, Dr. Matthias St. Pierre wrote:
> > The last 19 commits on https://github.com/openssl/openssl/commits/master,
> > starting from Nov 14 have a red cross from the CIs. What's going on again?
> > My personal impression is that builds on master are failing 50% of the time.
> 
> The builds on master have been failing for *much* longer than that. They
> should be succeeded on the PRs, but its the extended tests that fail.
> 
> > 
> > It is really tedious to check pull requests for build errors just to find 
> > that those errors
> > are well known failures from master. In particular, the krb5 an pyca tests 
> > are notoriously
> > failing. Are there any plans to fix them or disable them?
> 
> The build failures mostly come from the external tests (or that was the
> case I haven't checked recently). The problem with these is that, on
> occasion we need the maintainers of these packages to help us track down
> a problem. Or in some cases we actually need them to make a change (for
> example because something changed on master which revealed that they
> were relying on some behaviour in master which they really shouldn't be
> relying on).
> 
> Once one of them is failing then multiple errors can build up and they
> get masked by the earlier failure.
> 
> I'll try and find some time to take another pass at the current
> failures. I suspect the answer might be to disable some of the external
> tests.

Please let me know if you want me to look at the krb5 tests (or pester
folks at upstream there to do so).  I'm in Singapore for the IETF this week
and have been behind on, well, everything.

-Ben


Re: AW: Build failures on master?!

2019-11-18 Thread Matt Caswell



On 18/11/2019 22:29, Dr. Matthias St. Pierre wrote:
>>> The last 19 commits on https://github.com/openssl/openssl/commits/master,
>>> starting from Nov 14 have a red cross from the CIs. What's going on again?
>>> My personal impression is that builds on master are failing 50% of the time.
>>
>> The builds on master have been failing for *much* longer than that. They
>> should be succeeded on the PRs, but its the extended tests that fail.
> 
> Yes I know. I was just counting the last consecutive failures. What actually 
> wonders
> me is the fact that the tests do not fail throughout. In between there are 
> always short
> periods where the tests succced. Why is this?

Really? I don't see that. The travis dashboard shows red for as far back
as I checked:

https://travis-ci.org/openssl/openssl/builds

Matt

> 
> The reason why I issued this cry for help is because after watching it for a 
> long time I
> got the impression that due to the ongoing failures everybody has become dull 
> and
> indifferent agains the red crosses.
> 
> Matthias
> 


AW: Build failures on master?!

2019-11-18 Thread Dr. Matthias St. Pierre
> > The last 19 commits on https://github.com/openssl/openssl/commits/master,
> > starting from Nov 14 have a red cross from the CIs. What's going on again?
> > My personal impression is that builds on master are failing 50% of the time.
> 
> The builds on master have been failing for *much* longer than that. They
> should be succeeded on the PRs, but its the extended tests that fail.

Yes I know. I was just counting the last consecutive failures. What actually 
wonders
me is the fact that the tests do not fail throughout. In between there are 
always short
periods where the tests succced. Why is this?

The reason why I issued this cry for help is because after watching it for a 
long time I
got the impression that due to the ongoing failures everybody has become dull 
and
indifferent agains the red crosses.

Matthias



Re: Build failures on master?!

2019-11-18 Thread Matt Caswell



On 18/11/2019 21:48, Dr. Matthias St. Pierre wrote:
> The last 19 commits on https://github.com/openssl/openssl/commits/master,
> starting from Nov 14 have a red cross from the CIs. What's going on again?
> My personal impression is that builds on master are failing 50% of the time.

The builds on master have been failing for *much* longer than that. They
should be succeeded on the PRs, but its the extended tests that fail.

> 
> It is really tedious to check pull requests for build errors just to find 
> that those errors
> are well known failures from master. In particular, the krb5 an pyca tests 
> are notoriously
> failing. Are there any plans to fix them or disable them?

The build failures mostly come from the external tests (or that was the
case I haven't checked recently). The problem with these is that, on
occasion we need the maintainers of these packages to help us track down
a problem. Or in some cases we actually need them to make a change (for
example because something changed on master which revealed that they
were relying on some behaviour in master which they really shouldn't be
relying on).

Once one of them is failing then multiple errors can build up and they
get masked by the earlier failure.

I'll try and find some time to take another pass at the current
failures. I suspect the answer might be to disable some of the external
tests.

Matt

> 
>   95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1)
>   95-test_external_pyca.t (Wstat: 256 Tests: 1 Failed: 1)
> 
> Clean builds on master should have highest priority. Because if there are too 
> many red
> crosses, nobody cares about them anymore.
> 
> Matthias
> 
> 


Re: Build failures on master?!

2019-11-18 Thread Kurt Roeckx
On Mon, Nov 18, 2019 at 09:48:38PM +, Dr. Matthias St. Pierre wrote:
> The last 19 commits on https://github.com/openssl/openssl/commits/master,
> starting from Nov 14 have a red cross from the CIs. What's going on again?

I have filed 2 issues on Nov 9 that that caused the CIs to fail,
that that haven't been closed yet.

> My personal impression is that builds on master are failing 50% of the time.
> 
> It is really tedious to check pull requests for build errors just to find 
> that those errors
> are well known failures from master. In particular, the krb5 an pyca tests 
> are notoriously
> failing. Are there any plans to fix them or disable them?
> 
>   95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1)
>   95-test_external_pyca.t (Wstat: 256 Tests: 1 Failed: 1)
> 
> Clean builds on master should have highest priority. Because if there are too 
> many red
> crosses, nobody cares about them anymore.

Note that they are also broken in the 1.1.1-stable branch.

At least pyca is something you should be able to fix with a new
upstream version.


Kurt



Build failures on master?!

2019-11-18 Thread Dr. Matthias St. Pierre
The last 19 commits on https://github.com/openssl/openssl/commits/master,
starting from Nov 14 have a red cross from the CIs. What's going on again?
My personal impression is that builds on master are failing 50% of the time.

It is really tedious to check pull requests for build errors just to find that 
those errors
are well known failures from master. In particular, the krb5 an pyca tests are 
notoriously
failing. Are there any plans to fix them or disable them?

  95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1)
  95-test_external_pyca.t (Wstat: 256 Tests: 1 Failed: 1)

Clean builds on master should have highest priority. Because if there are too 
many red
crosses, nobody cares about them anymore.

Matthias




Re: Build failures on master

2019-09-29 Thread Richard Levitte
This keeps happening because the standard tests don't include no-fips
or no-pic or no-module (the latter two will disable fips too), so we
keep not hitting this hurdle in PRs unless we have extended tests run
(done by having '[extended tests]' in the commit message).  We should
add a configuration with 'no-fips' among our standard tests, so this
becomes automatically visible in PRs.

The issue with the tests?  Doing FIPS tests without checking if 'fips'
is disabled or not.

Cheers,
Richard

On Sat, 28 Sep 2019 22:25:04 +0200,
Dr. Matthias St. Pierre wrote:
> 
> Hi,
> 
> since my last commit on master did not build successfully, I checked the CIs.
> It turned out that a lot of tests are failing, and they have been for quite a 
> while now.
> 
> https://travis-ci.org/openssl/openssl/builds/590874649?utm_source=github_status_medium=notification
> 
> Apart from the well known trio boringssl, krb5, and pyca, there are some new 
> failing tests.
> It would be nice if we could put some effort into fixing them in order to get 
> back to having
> stable successful builds on master again. The build failures on master also 
> affect the
> pull request CI builds and if the failure persists, people stop checking 
> their red crosses.
> 
> Matthias
> 
> 
> --
> 
> https://travis-ci.org/openssl/openssl/jobs/590874663
> 
> 80-test_ssl_old.t(Wstat: 256 Tests: 6 Failed: 1)
>   Failed test:  3
>   Non-zero exit status: 1
> 
> 
> https://travis-ci.org/openssl/openssl/jobs/590874664
> 
> Test Summary Report
> ---
> 95-test_external_boringssl.t (Wstat: 256 Tests: 1 Failed: 1)
>   Failed test:  1
>   Non-zero exit status: 1
> 95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1)
>   Failed test:  1
>   Non-zero exit status: 1
> 95-test_external_pyca.t (Wstat: 256 Tests: 1 Failed: 1)
>   Failed test:  1
>   Non-zero exit status: 1
> 
> https://travis-ci.org/openssl/openssl/jobs/590874669
> 
> Test Summary Report
> ---
> 30-test_evp.t(Wstat: 256 Tests: 19 Failed: 1)
>   Failed test:  1
>   Non-zero exit status: 1
> 30-test_evp_fetch_prov.t (Wstat: 5376 Tests: 34 Failed: 21)
>   Failed tests:  1, 9-18, 25-34
>   Non-zero exit status: 21
> 
> https://travis-ci.org/openssl/openssl/jobs/590874670
> 
> Test Summary Report
> ---
> 30-test_evp.t(Wstat: 256 Tests: 19 Failed: 1)
>   Failed test:  1
>   Non-zero exit status: 1
> 30-test_evp_fetch_prov.t (Wstat: 5376 Tests: 34 Failed: 21)
>   Failed tests:  1, 9-18, 25-34
>   Non-zero exit status: 21
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Build failures on master

2019-09-28 Thread Dr. Matthias St. Pierre
Hi,

since my last commit on master did not build successfully, I checked the CIs.
It turned out that a lot of tests are failing, and they have been for quite a 
while now.

https://travis-ci.org/openssl/openssl/builds/590874649?utm_source=github_status_medium=notification

Apart from the well known trio boringssl, krb5, and pyca, there are some new 
failing tests.
It would be nice if we could put some effort into fixing them in order to get 
back to having
stable successful builds on master again. The build failures on master also 
affect the
pull request CI builds and if the failure persists, people stop checking their 
red crosses.

Matthias


--

https://travis-ci.org/openssl/openssl/jobs/590874663

80-test_ssl_old.t(Wstat: 256 Tests: 6 Failed: 1)
  Failed test:  3
  Non-zero exit status: 1


https://travis-ci.org/openssl/openssl/jobs/590874664

Test Summary Report
---
95-test_external_boringssl.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
95-test_external_pyca.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1

https://travis-ci.org/openssl/openssl/jobs/590874669

Test Summary Report
---
30-test_evp.t(Wstat: 256 Tests: 19 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
30-test_evp_fetch_prov.t (Wstat: 5376 Tests: 34 Failed: 21)
  Failed tests:  1, 9-18, 25-34
  Non-zero exit status: 21

https://travis-ci.org/openssl/openssl/jobs/590874670

Test Summary Report
---
30-test_evp.t(Wstat: 256 Tests: 19 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
30-test_evp_fetch_prov.t (Wstat: 5376 Tests: 34 Failed: 21)
  Failed tests:  1, 9-18, 25-34
  Non-zero exit status: 21