Re: Travis in solid red mode again

2020-02-07 Thread Matt Caswell



On 03/02/2020 19:19, Matthias St. Pierre wrote:
> Mission accomplished, Travis just turned green again on master  :-)

Every one of the last 12 master jobs have failed to complete
successfully due to a travis timeout. Apart from the build that is
timing out (enable-msan), all the other builds in the job are green.

Matt

> 
> https://travis-ci.org/openssl/openssl/builds/645563965?utm_source=github_status_medium=notification
> 
> 
> Thanks to Matt, Richard and everybody else who participated in fixing it.
> 
> 
> Matthias
> 
> 
> 


Re: Travis in solid red mode again

2020-02-03 Thread Matthias St. Pierre

Mission accomplished, Travis just turned green again on master  :-)

https://travis-ci.org/openssl/openssl/builds/645563965?utm_source=github_status_medium=notification

Thanks to Matt, Richard and everybody else who participated in fixing it.


Matthias





Re: Travis in solid red mode again

2020-02-03 Thread Salz, Rich
  
  >  Could we add a CI check for PRs that confirms that the target branch is
green in travis?

Looking at 
https://docs.travis-ci.com/user/notifications/#conditional-notifications and 
https://config.travis-ci.com/ref/notifications/email it seems like it should be 
fairly easy configure builds to do "send email to openssl-commits when builds 
on master fail"


 



Re: Travis in solid red mode again

2020-02-03 Thread Salz, Rich
>Oh, actually, maybe I'm wrong. Is it just Appveyor that posts failures
and not Travis?
  
Yes.  You need to config travis to mail failures, as I pointed out in my 
earlier message.




RE: Travis in solid red mode again

2020-02-01 Thread Dr. Matthias St. Pierre


> -Original Message-
> From: Matt Caswell 
> Sent: Sunday, February 2, 2020 12:58 AM
> To: Dr Paul Dale ; Dr. Matthias St. Pierre 
> 
> Cc: Salz, Rich ; openssl-project@openssl.org
> Subject: Re: Travis in solid red mode again
> 
> 
> 
> On 01/02/2020 10:39, Dr Paul Dale wrote:
> > I thought I was subscribed but don’t seem to see the failures.
> > I do get the (very many) PR activity emails….
> 
> Oh, actually, maybe I'm wrong. Is it just Appveyor that posts failures
> and not Travis?

I see both CI's posting in January:

https://mta.openssl.org/pipermail/openssl-commits/2020-January/thread.html

Broken: openssl/openssl#30961 (master - 2de5a5f)   Travis CI
Broken: openssl/openssl#30962 (OpenSSL_1_1_1-stable - 10e166a)   Travis CI
Build failed: openssl master.30389   AppVeyor
Build completed: openssl OpenSSL_1_1_1-stable.30390   AppVeyor
Fixed: openssl/openssl#30966 (master - e7b834b)   Travis CI
Fixed: openssl/openssl#30967 (OpenSSL_1_1_1-stable - 2c52a36)   Travis CI
Build failed: openssl master.30394   AppVeyor
Build completed: openssl OpenSSL_1_1_1-stable.30395   AppVeyor
Build failed: openssl master.30405   AppVeyor
Build completed: openssl master.30406   AppVeyor
Build failed: openssl master.30411   AppVeyor
Build failed: openssl master.30412   AppVeyor



Re: Travis in solid red mode again

2020-02-01 Thread Matt Caswell



On 01/02/2020 10:39, Dr Paul Dale wrote:
> I thought I was subscribed but don’t seem to see the failures.
> I do get the (very many) PR activity emails….

Oh, actually, maybe I'm wrong. Is it just Appveyor that posts failures
and not Travis?

Matt

> 
> 
> Pauli
> -- 
> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
> Phone +61 7 3031 7217
> Oracle Australia
> 
> 
> 
> 
>> On 1 Feb 2020, at 8:35 pm, Dr. Matthias St. Pierre
>> mailto:matthias.st.pie...@ncp-e.com>>
>> wrote:
>>
>>> -Original Message-
>>> On 01/02/2020 00:34, Salz, Rich wrote:

> Could we add a CI check for PRs that confirms that the target branch is
    green in travis?

 Looking at
 https://docs.travis-ci.com/user/notifications/#conditional-notifications
 and https://config.travis-
>>> ci.com/ref/notifications/email
>>>  it seems like it should be
>>> fairly easy configure builds to do "send email to openssl-commits
>>> when builds on
>>> master fail"
>>>
>>> We already have that.
>>
>> Maybe we should make it a requirement that at least all OTC members
>> subscribe to openssl-commits?
>>
>> Matthias
>>
> 


Re: Travis in solid red mode again

2020-02-01 Thread Matt Caswell



On 01/02/2020 11:38, Dr. Matthias St. Pierre wrote:
>> I thought I was subscribed but don’t seem to see the failures.
> 
>> I do get the (very many) PR activity emails….
> 
>  
> 
> You are right, those “[openssl]  update” mails generate a lot
> 
> of noise. Do we really need them? If not, we could just deactivate them.

I regularly look at these, so would like to keep them.

Matt


> 
> Alternatively, we could move the CI failures to a separate openssl-ci list.
> 
>  
> 
> Matthias
> 
>  
> 


RE: Travis in solid red mode again

2020-02-01 Thread Dr. Matthias St. Pierre
> I thought I was subscribed but don’t seem to see the failures.
> I do get the (very many) PR activity emails….

You are right, those “[openssl]  update” mails generate a lot
of noise. Do we really need them? If not, we could just deactivate them.
Alternatively, we could move the CI failures to a separate openssl-ci list.

Matthias



Re: Travis in solid red mode again

2020-02-01 Thread Dr Paul Dale
I thought I was subscribed but don’t seem to see the failures.
I do get the (very many) PR activity emails….


Pauli
-- 
Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
Phone +61 7 3031 7217
Oracle Australia




> On 1 Feb 2020, at 8:35 pm, Dr. Matthias St. Pierre 
>  wrote:
> 
>> -Original Message-
>> On 01/02/2020 00:34, Salz, Rich wrote:
>>> 
 Could we add a CI check for PRs that confirms that the target branch is
>>>green in travis?
>>> 
>>> Looking at 
>>> https://docs.travis-ci.com/user/notifications/#conditional-notifications 
>>> and https://config.travis-
>> ci.com/ref/notifications/email it seems like it should be fairly easy 
>> configure builds to do "send email to openssl-commits when builds on
>> master fail"
>> 
>> We already have that.
> 
> Maybe we should make it a requirement that at least all OTC members subscribe 
> to openssl-commits?
> 
> Matthias
> 



RE: Travis in solid red mode again

2020-02-01 Thread Dr. Matthias St. Pierre
> -Original Message-
> On 01/02/2020 00:34, Salz, Rich wrote:
> >
> >   >  Could we add a CI check for PRs that confirms that the target branch is
> > green in travis?
> >
> > Looking at 
> > https://docs.travis-ci.com/user/notifications/#conditional-notifications 
> > and https://config.travis-
> ci.com/ref/notifications/email it seems like it should be fairly easy 
> configure builds to do "send email to openssl-commits when builds on
> master fail"
> 
> We already have that.

Maybe we should make it a requirement that at least all OTC members subscribe 
to openssl-commits?

Matthias



Re: Travis in solid red mode again

2020-02-01 Thread Matt Caswell



On 01/02/2020 00:34, Salz, Rich wrote:
>   
>   >  Could we add a CI check for PRs that confirms that the target branch is
> green in travis?
> 
> Looking at 
> https://docs.travis-ci.com/user/notifications/#conditional-notifications and 
> https://config.travis-ci.com/ref/notifications/email it seems like it should 
> be fairly easy configure builds to do "send email to openssl-commits when 
> builds on master fail"

We already have that.

Matt


Re: Travis in solid red mode again

2020-01-31 Thread Matt Caswell



On 31/01/2020 22:55, Matt Caswell wrote:
> 
> 
> On 31/01/2020 17:07, Matthias St. Pierre wrote:
>>
>> After a much to short green period, Travis has turned to solid red mode
>> again:
>>
>> https://travis-ci.org/openssl/openssl/builds?utm_medium=notification_source=github_status
>>
>>
>> This fact causes a lot of pain for reviewers, because every failing job
>> needs to be checked manually.
>> I would be happy to fix some of the failures if I could, but in most
>> cases I have no clue what the problem is,
>> because I can't reproduce the issue locally. Could we try to fix them in
>> a joint effort?  For your convenience,
>> I added a summary of all failing jobs of Build #31837 below.
> 
> Trial fix for at least some of these issues in #10989. We'll see what
> travis makes of it.

It fixed most of the test failures but not the krb5 one. I have another
trial fix in #10992 (which includes the fix in #10989).

Matt


> 
>>
>> Since this is not the first time that this happens, I think it is
>> overdue that the OTC discusses those constant
>> build failures and proposes some measures to raise the priority for
>> fixing CI errors. One drastic possibility
>> would be to allow only CI fixing pull requests to be merged as long as
>> master and the stable branches are red.
>> But I am sure there is a golden middle way between such an extreme
>> sanction and our current indifference.
>>
> 
> Part of the issue is that the master branch travis failures are not
> visible in the PRs because the master branch runs the full tests
> (including the extended tests), whereas the PRs don't run the extended
> tests by default. However running the extended tests takes too long.
> 
> Could we add a CI check for PRs that confirms that the target branch is
> green in travis?
> 
> Matt
> 
> 
>>
>> Regards,
>>
>> Matthias
>>
>>
>>
>> 
>>
>>
>> Last Successful Build: #31583 (10 days ago!)
>> https://travis-ci.org/openssl/openssl/builds/639939072?utm_medium=notification_source=github_status
>>
>>
>> Recent Build: #31837
>> https://travis-ci.org/openssl/openssl/builds/644238908?utm_medium=notification_source=github_status
>>
>>
>>
>> SSL TESTS
>> =
>>
>> *
>> https://travis-ci.org/openssl/openssl/jobs/644238919?utm_medium=notification_source=github_status
>>
>> *
>> https://travis-ci.org/openssl/openssl/jobs/644238926?utm_medium=notification_source=github_status
>>
>> *
>> https://travis-ci.org/openssl/openssl/jobs/644238929?utm_medium=notification_source=github_status
>>
>>
>>>   Test Summary Report
>>> ---
>>>   80-test_ssl_new.t    (Wstat: 256 Tests: 30 Failed: 1)
>>>     Failed test:  4
>>>     Non-zero exit status: 1
>>>   80-test_ssl_old.t    (Wstat: 256 Tests: 6 Failed: 1)
>>>     Failed test:  2
>>>     Non-zero exit status: 1
>>
>>
>>
>> BORINGSSL AND KRB5
>> ==
>>
>> *
>> https://travis-ci.org/openssl/openssl/jobs/644238927?utm_medium=notification_source=github_status
>>
>>
>>> Test Summary Report
>>> ---
>>
>>> 95-test_external_boringssl.t (Wstat: 256 Tests: 1 Failed: 1)
>>>    Failed test:  1
>>>    Non-zero exit status: 1
>>> 95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1)
>>>    Failed test:  1
>>>    Non-zero exit status: 1
>>
>>
>>
>> 95-test_external_boringssl.t:
>> ---
>>
>>     grep 'go.*FAILED' t
>>     go test: FAILED (SSL3-Client-ClientAuth-RSA)
>>     go test: FAILED (SSL3-Server-ClientAuth-RSA)
>>     go test: FAILED (ClientAuth-Sign-RSA-SSL3)
>>     go test: FAILED (ClientAuth-InvalidSignature-RSA-SSL3)
>>     go test: FAILED (CertificateVerificationSucceed-Server-SSL3-Sync)
>>     go test: FAILED
>> (CertificateVerificationSucceed-Server-SSL3-Sync-SplitHandshakeRecords)
>>     go test: FAILED
>> (CertificateVerificationSucceed-Server-SSL3-Sync-PackHandshakeFlight)
>>     go test: FAILED (CertificateVerificationSucceed-Server-SSL3-Async)
>>     go test: FAILED
>> (CertificateVerificationSucceed-Server-SSL3-Async-SplitHandshakeRecords)
>>     go test: FAILED
>> (CertificateVerificationSucceed-Server-SSL3-Async-PackHandshakeFlight)
>>
>>
>>
>> 95-test_external_krb5.t:
>> ---
>>
>>     LD_LIBRARY_PATH=`echo -L../../.

Re: Travis in solid red mode again

2020-01-31 Thread Matt Caswell



On 31/01/2020 17:07, Matthias St. Pierre wrote:
> 
> After a much to short green period, Travis has turned to solid red mode
> again:
> 
> https://travis-ci.org/openssl/openssl/builds?utm_medium=notification_source=github_status
> 
> 
> This fact causes a lot of pain for reviewers, because every failing job
> needs to be checked manually.
> I would be happy to fix some of the failures if I could, but in most
> cases I have no clue what the problem is,
> because I can't reproduce the issue locally. Could we try to fix them in
> a joint effort?  For your convenience,
> I added a summary of all failing jobs of Build #31837 below.

Trial fix for at least some of these issues in #10989. We'll see what
travis makes of it.

> 
> Since this is not the first time that this happens, I think it is
> overdue that the OTC discusses those constant
> build failures and proposes some measures to raise the priority for
> fixing CI errors. One drastic possibility
> would be to allow only CI fixing pull requests to be merged as long as
> master and the stable branches are red.
> But I am sure there is a golden middle way between such an extreme
> sanction and our current indifference.
> 

Part of the issue is that the master branch travis failures are not
visible in the PRs because the master branch runs the full tests
(including the extended tests), whereas the PRs don't run the extended
tests by default. However running the extended tests takes too long.

Could we add a CI check for PRs that confirms that the target branch is
green in travis?

Matt


> 
> Regards,
> 
> Matthias
> 
> 
> 
> 
> 
> 
> Last Successful Build: #31583 (10 days ago!)
> https://travis-ci.org/openssl/openssl/builds/639939072?utm_medium=notification_source=github_status
> 
> 
> Recent Build: #31837
> https://travis-ci.org/openssl/openssl/builds/644238908?utm_medium=notification_source=github_status
> 
> 
> 
> SSL TESTS
> =
> 
> *
> https://travis-ci.org/openssl/openssl/jobs/644238919?utm_medium=notification_source=github_status
> 
> *
> https://travis-ci.org/openssl/openssl/jobs/644238926?utm_medium=notification_source=github_status
> 
> *
> https://travis-ci.org/openssl/openssl/jobs/644238929?utm_medium=notification_source=github_status
> 
> 
>>  Test Summary Report
>> ---
>>  80-test_ssl_new.t    (Wstat: 256 Tests: 30 Failed: 1)
>>    Failed test:  4
>>    Non-zero exit status: 1
>>  80-test_ssl_old.t    (Wstat: 256 Tests: 6 Failed: 1)
>>    Failed test:  2
>>    Non-zero exit status: 1
> 
> 
> 
> BORINGSSL AND KRB5
> ==
> 
> *
> https://travis-ci.org/openssl/openssl/jobs/644238927?utm_medium=notification_source=github_status
> 
> 
>> Test Summary Report
>> ---
> 
>> 95-test_external_boringssl.t (Wstat: 256 Tests: 1 Failed: 1)
>>   Failed test:  1
>>   Non-zero exit status: 1
>> 95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1)
>>   Failed test:  1
>>   Non-zero exit status: 1
> 
> 
> 
> 95-test_external_boringssl.t:
> ---
> 
>     grep 'go.*FAILED' t
>     go test: FAILED (SSL3-Client-ClientAuth-RSA)
>     go test: FAILED (SSL3-Server-ClientAuth-RSA)
>     go test: FAILED (ClientAuth-Sign-RSA-SSL3)
>     go test: FAILED (ClientAuth-InvalidSignature-RSA-SSL3)
>     go test: FAILED (CertificateVerificationSucceed-Server-SSL3-Sync)
>     go test: FAILED
> (CertificateVerificationSucceed-Server-SSL3-Sync-SplitHandshakeRecords)
>     go test: FAILED
> (CertificateVerificationSucceed-Server-SSL3-Sync-PackHandshakeFlight)
>     go test: FAILED (CertificateVerificationSucceed-Server-SSL3-Async)
>     go test: FAILED
> (CertificateVerificationSucceed-Server-SSL3-Async-SplitHandshakeRecords)
>     go test: FAILED
> (CertificateVerificationSucceed-Server-SSL3-Async-PackHandshakeFlight)
> 
> 
> 
> 95-test_external_krb5.t:
> ---
> 
>     LD_LIBRARY_PATH=`echo -L../../../lib | sed -e "s/-L//g" -e "s/
> /:/g"` KRB5_CONFIG=../../../config-files/krb5.conf LC_ALL=C ./t_nfold
>     N-fold
>     Input:    "basch"
>     192-Fold: 1aab6b42 964b98b2 1f8cde2d 2448ba34 55d7862c 9731643f
>     Input:    "eichin"
>     192-Fold: 65696368 696e4b73 2b4b1b43 da1a5b99 5a58d2c6 d0d2dcca
>     Input:    "sommerfeld"
>     192-Fold: 2f7a9855 7c6ee4ab adf4e711 92dd442b d4ff5325 a5def75c
>     Input:    "MASSACHVSETTS INSTITVTE OF TECHNOLOGY"
>     192-Fold: db3b0d8f 0b061e60 3282b308 a5084122 9ad798fa b9540c1b
>     RFC tests:
&g

Travis in solid red mode again

2020-01-31 Thread Matthias St. Pierre



After a much to short green period, Travis has turned to solid red mode again:

https://travis-ci.org/openssl/openssl/builds?utm_medium=notification_source=github_status

This fact causes a lot of pain for reviewers, because every failing job needs 
to be checked manually.
I would be happy to fix some of the failures if I could, but in most cases I 
have no clue what the problem is,
because I can't reproduce the issue locally. Could we try to fix them in a 
joint effort?  For your convenience,
I added a summary of all failing jobs of Build #31837 below.

Since this is not the first time that this happens, I think it is overdue that 
the OTC discusses those constant
build failures and proposes some measures to raise the priority for fixing CI 
errors. One drastic possibility
would be to allow only CI fixing pull requests to be merged as long as master 
and the stable branches are red.
But I am sure there is a golden middle way between such an extreme sanction and 
our current indifference.


Regards,

Matthias






Last Successful Build: #31583 (10 days ago!)
https://travis-ci.org/openssl/openssl/builds/639939072?utm_medium=notification_source=github_status

Recent Build: #31837
https://travis-ci.org/openssl/openssl/builds/644238908?utm_medium=notification_source=github_status


SSL TESTS
=

* 
https://travis-ci.org/openssl/openssl/jobs/644238919?utm_medium=notification_source=github_status
* 
https://travis-ci.org/openssl/openssl/jobs/644238926?utm_medium=notification_source=github_status
* 
https://travis-ci.org/openssl/openssl/jobs/644238929?utm_medium=notification_source=github_status

>  Test Summary Report
> ---
>  80-test_ssl_new.t    (Wstat: 256 Tests: 30 Failed: 1)
>    Failed test:  4
>    Non-zero exit status: 1
>  80-test_ssl_old.t    (Wstat: 256 Tests: 6 Failed: 1)
>    Failed test:  2
>    Non-zero exit status: 1



BORINGSSL AND KRB5
==

* 
https://travis-ci.org/openssl/openssl/jobs/644238927?utm_medium=notification_source=github_status

> Test Summary Report
> ---

> 95-test_external_boringssl.t (Wstat: 256 Tests: 1 Failed: 1)
>   Failed test:  1
>   Non-zero exit status: 1
> 95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1)
>   Failed test:  1
>   Non-zero exit status: 1



95-test_external_boringssl.t:
---

    grep 'go.*FAILED' t
    go test: FAILED (SSL3-Client-ClientAuth-RSA)
    go test: FAILED (SSL3-Server-ClientAuth-RSA)
    go test: FAILED (ClientAuth-Sign-RSA-SSL3)
    go test: FAILED (ClientAuth-InvalidSignature-RSA-SSL3)
    go test: FAILED (CertificateVerificationSucceed-Server-SSL3-Sync)
    go test: FAILED 
(CertificateVerificationSucceed-Server-SSL3-Sync-SplitHandshakeRecords)
    go test: FAILED 
(CertificateVerificationSucceed-Server-SSL3-Sync-PackHandshakeFlight)
    go test: FAILED (CertificateVerificationSucceed-Server-SSL3-Async)
    go test: FAILED 
(CertificateVerificationSucceed-Server-SSL3-Async-SplitHandshakeRecords)
    go test: FAILED 
(CertificateVerificationSucceed-Server-SSL3-Async-PackHandshakeFlight)



95-test_external_krb5.t:
---

    LD_LIBRARY_PATH=`echo -L../../../lib | sed -e "s/-L//g" -e "s/ /:/g"` 
KRB5_CONFIG=../../../config-files/krb5.conf LC_ALL=C ./t_nfold
    N-fold
    Input:    "basch"
    192-Fold: 1aab6b42 964b98b2 1f8cde2d 2448ba34 55d7862c 9731643f
    Input:    "eichin"
    192-Fold: 65696368 696e4b73 2b4b1b43 da1a5b99 5a58d2c6 d0d2dcca
    Input:    "sommerfeld"
    192-Fold: 2f7a9855 7c6ee4ab adf4e711 92dd442b d4ff5325 a5def75c
    Input:    "MASSACHVSETTS INSTITVTE OF TECHNOLOGY"
    192-Fold: db3b0d8f 0b061e60 3282b308 a5084122 9ad798fa b9540c1b
    RFC tests:

    ...

    Generating random keyblock: . . . OK
    Creating opaque key from keyblock: . . . OK
    Encrypting (c): . . . Failed: Cannot allocate memory <===ALLOC FAILURE==
    Aborted (core dumped) 
<===CORE DUMPED==
    Makefile:684: recipe for target 'check-unix' failed
    make[5]: *** [check-unix] Error 134
    make[5]: Leaving directory 
'/home/travis/build/openssl/openssl/krb5/src/lib/crypto/crypto_tests'
    Makefile:1107: recipe for target 'check-recurse' failed
    make[4]: *** [check-recurse] Error 1
    make[4]: Leaving directory 
'/home/travis/build/openssl/openssl/krb5/src/lib/crypto'
    Makefile:992: recipe for target 'check-recurse' failed
    make[3]: *** [check-recurse] Error 1
    make[3]: Leaving directory '/home/travis/build/openssl/openssl/krb5/src/lib'
    Makefile:1533: recipe for target 'check-recurse' failed
    make[2]: *** [check-recurse] Error 1
    make[2]: Leaving directory '/home/travis/build/openssl/openssl/krb5/src'
    ../../util/shlib_wrap.sh ../recipes/95-test_external_krb5_data/krb5.sh => 2
    not ok 1 - running krb5 tests