Re: Improving the CI experience

[Georges Racinet]

On 9/29/20 10:54 AM, Raphaël Gomès wrote:
>
>
> On 9/28/20 8:12 PM, Augie Fackler wrote:
>>
>>
>>> On Sep 28, 2020, at 10:43, Raphaël Gomès >> > wrote:
>>>
>>> Hi all,
>>>
>>> As you may know, we (Octobus) and other Mercurial contributors have
>>> been using
>>> the Heptapod [1] CI to vet our patches pre-submission against 7
>>> variants of
>>> Mercurial:
>>>   - Python 2 pure
>>>   - Python 3 pure
>>>   - Python 2 python+c
>>>   - Python 3 python+c
>>>   - Python 2 rust+c
>>>   - Python 3 rust+c
>>>   - Python 2 chg
>>>   - (also code checks and Rust unit tests, of course)
>>>
>>> All variants above are run on x86_64 Debian machines. Our CI does
>>> not (yet) have
>>> the MacOS, Windows, or FreeBSDoptionsthat the Mercurial Buildbot
>>> has, but
>>> does test more variants on Linux.
>>>
>>> Nevertheless, it catches a lot of the mistakes that we all make
>>> sometimes, 
>>> before we even send the patches upstream: regressions in Python 3,
>>> API breakage
>>> with Rust, bad formatting, and other bugs of various nature. It does
>>> so way
>>> before it touches the `committed` repo, which means that if I screw
>>> up my
>>> patch, it only has to bother me.
>>>
>>> The value of such a system is highly dependent on it being
>>> trustworthy: flaky
>>> tests do exist, but they should be addressed and fixed in a timely
>>> manner and
>>> my CI being red should mean that *I* missed something. Yet, I have
>>> heard the
>>> phrase "I don't even look at the pipelines anymore, they're always
>>> broken for
>>> something I haven't done".
>>>
>>> Most of the time it's something minor: I've even been guilty of
>>> breaking the CI
>>> myself in my last patch series (oops) due to bad formatting!
>>> However, these
>>> issues should be few and far between: the developer experience
>>> suffers greatly
>>> and we grow to accept that the tests are always broken, don't bother
>>> to run
>>> them at all, etc.
>>>
>>> # Proposal
>>>
>>> I thought a start to fixing this situation would be for me to setup 
>>> an automation to send a small email to this mailing-list every time
>>> either the
>>> `stable` or `default` branches (that track `hg-committed`) are
>>> broken, with a link 
>>> to the pipeline results.
>>> This will not catch everything (only a subset of the possible
>>> targets is tested), but
>>> I think this would be an improvement for all contributors, even
>>> those that don't
>>> use Heptapod.
>>>
>>> What do you all think?
>>
>> I'm open to it. Do you think the CI that you're running on heptapod
>> is faster than the buildbot? Should we keep the buildbot or shut it down?
>>
> Faster in terms of latency, almost certainly, at least for now because
> of the number of workers, and probably in terms of the actual speed of
> the runners, though I'm just making a guess.
>
Looking at a random build on buildbot [1], it took about 2 hours to run
the full test suite without option and with --pure.

In terms of usefulness to the community as a whole, I think we probably
want to define the full latency as the full time it takes between a push
to hg-committed to available results and potential alerts on the mailing
lists.

We can split that in

1. the time it takes for the changesets to arrive in Heptapod.

    That part is not automatic yet, we'd need to improve on that and
that's fairly easy.

    The most efficient would be for hg-committed to send a trigger in a
hook. The service listening to that trigger would then pull from
hg-committed and push to Heptapod.

    Raphaël or I could submit this to mercurial-infra, what do you think ?

2. the time it takes for workers to take the jobs.

    That part depends on how busy the CI system is. In practice, the
jobs start right away more often than not, but we can get 15-20 minutes
of wait sometimes. This is good enough for us now, we have lots of
options if that becomes a problem.

3. the time it takes to run the jobs.

    The equivalent in Heptapod CI of the buildbot build mentioned above
would be the test-py2 and test-py2-pure jobs. They run in parallel in
less than half an hour each. Picking again randomly a pipeline [2], I
see the full set of 11 jobs complete in 25 minutes.

    Some jobs are much faster (Rust cargo tests about 2 minutes, check
tests about 4 minutes). I don't know yet if it would be easy to get fast
feedback about them, just saying there's room for follow-up improvements.

[1] https://buildbot.mercurial-scm.org/builders/hg%20tests/builds/3078

[2] https://foss.heptapod.net/octobus/mercurial-devel/-/pipelines/10794

> However, the Buildbot still has value since it covers cases that our
> CI still does not cover and not everyone is using Heptapod, so let's
> keep it around for a while.
>
Yes these are complementary. However, if we get to the point where the
Heptapod pipelines run after each push to hg-committed and it works
smoothly, we could consider lightening the weight on the buildbot by
removing redundant builds.

> To 

Re: Improving the CI experience

[Raphaël Gomès]

On 9/28/20 8:12 PM, Augie Fackler wrote:



On Sep 28, 2020, at 10:43, Raphaël Gomès > wrote:


Hi all,

As you may know, we (Octobus) and other Mercurial contributors have 
been using
the Heptapod [1] CI to vet our patches pre-submission against 7 
variants of

Mercurial:
- Python 2 pure
- Python 3 pure
- Python 2 python+c
- Python 3 python+c
- Python 2 rust+c
- Python 3 rust+c
- Python 2 chg
- (also code checks and Rust unit tests, of course)

All variants above are run on x86_64 Debian machines. Our CI does not 
(yet) have

the MacOS, Windows, or FreeBSDoptionsthat the Mercurial Buildbot has, but
does test more variants on Linux.

Nevertheless, it catches a lot of the mistakes that we all make 
sometimes,
before we even send the patches upstream: regressions in Python 3, 
API breakage
with Rust, bad formatting, and other bugs of various nature. It does 
so way

before it touches the `committed` repo, which means that if I screw up my
patch, it only has to bother me.

The value of such a system is highly dependent on it being 
trustworthy: flaky
tests do exist, but they should be addressed and fixed in a timely 
manner and
my CI being red should mean that *I* missed something. Yet, I have 
heard the
phrase "I don't even look at the pipelines anymore, they're always 
broken for

something I haven't done".

Most of the time it's something minor: I've even been guilty of 
breaking the CI
myself in my last patch series (oops) due to bad formatting! However, 
these
issues should be few and far between: the developer experience 
suffers greatly
and we grow to accept that the tests are always broken, don't bother 
to run

them at all, etc.

# Proposal

I thought a start to fixing this situation would be for me to setup
an automation to send a small email to this mailing-list every time 
either the
`stable` or `default` branches (that track `hg-committed`) are 
broken, with a link

to the pipeline results.
This will not catch everything (only a subset of the possible targets 
is tested), but
I think this would be an improvement for all contributors, even those 
that don't

use Heptapod.

What do you all think?


I'm open to it. Do you think the CI that you're running on heptapod is 
faster than the buildbot? Should we keep the buildbot or shut it down?


Faster in terms of latency, almost certainly, at least for now because 
of the number of workers, and probably in terms of the actual speed of 
the runners, though I'm just making a guess.


However, the Buildbot still has value since it covers cases that our CI 
still does not cover and not everyone is using Heptapod, so let's keep 
it around for a while. To further drive the point home, I'm not 
comfortable recommending Heptapod for the project or anything of that 
sort yet, even ruling out personal preferences for workflow. Maybe we 
can have that discussion in a few months, who knows.


The fact that the Buildbot runs *after* the patches have landed makes me 
sad, but that's an issue for another day probably!



Raphaël

[1] https://foss.heptapod.net/octobus/mercurial-devel/-/pipelines
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org 


https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel



___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: Improving the CI experience

[Augie Fackler]

> On Sep 28, 2020, at 10:43, Raphaël Gomès  wrote:
> 
> Hi all,
> 
> As you may know, we (Octobus) and other Mercurial contributors have been using
> the Heptapod [1] CI to vet our patches pre-submission against 7 variants of
> Mercurial:
>   - Python 2 pure
>   - Python 3 pure
>   - Python 2 python+c
>   - Python 3 python+c
>   - Python 2 rust+c
>   - Python 3 rust+c
>   - Python 2 chg
>   - (also code checks and Rust unit tests, of course)
> 
> All variants above are run on x86_64 Debian machines. Our CI does not (yet) 
> have
> the MacOS, Windows, or FreeBSD options that the Mercurial Buildbot has, but
> does test more variants on Linux.
> 
> Nevertheless, it catches a lot of the mistakes that we all make sometimes, 
> before we even send the patches upstream: regressions in Python 3, API 
> breakage
> with Rust, bad formatting, and other bugs of various nature. It does so way
> before it touches the `committed` repo, which means that if I screw up my
> patch, it only has to bother me.
> 
> The value of such a system is highly dependent on it being trustworthy: flaky
> tests do exist, but they should be addressed and fixed in a timely manner and
> my CI being red should mean that *I* missed something. Yet, I have heard the
> phrase "I don't even look at the pipelines anymore, they're always broken for
> something I haven't done".
> 
> Most of the time it's something minor: I've even been guilty of breaking the 
> CI
> myself in my last patch series (oops) due to bad formatting! However, these
> issues should be few and far between: the developer experience suffers greatly
> and we grow to accept that the tests are always broken, don't bother to run
> them at all, etc.
> 
> # Proposal
> 
> I thought a start to fixing this situation would be for me to setup 
> an automation to send a small email to this mailing-list every time either the
> `stable` or `default` branches (that track `hg-committed`) are broken, with a 
> link 
> to the pipeline results. 
> This will not catch everything (only a subset of the possible targets is 
> tested), but
> I think this would be an improvement for all contributors, even those that 
> don't
> use Heptapod.
> 
> What do you all think?

I'm open to it. Do you think the CI that you're running on heptapod is faster 
than the buildbot? Should we keep the buildbot or shut it down?

> Raphaël
> 
> [1]  https://foss.heptapod.net/octobus/mercurial-devel/-/pipelines 
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Improving the CI experience

[Raphaël Gomès]

Hi all,

As you may know, we (Octobus) and other Mercurial contributors have been 
using

the Heptapod [1] CI to vet our patches pre-submission against 7 variants of
Mercurial:
- Python 2 pure
- Python 3 pure
- Python 2 python+c
- Python 3 python+c
- Python 2 rust+c
- Python 3 rust+c
- Python 2 chg
- (also code checks and Rust unit tests, of course)

All variants above are run on x86_64 Debian machines. Our CI does not 
(yet) have

the MacOS, Windows, or FreeBSDoptionsthat the Mercurial Buildbot has, but
does test more variants on Linux.

Nevertheless, it catches a lot of the mistakes that we all make sometimes,
before we even send the patches upstream: regressions in Python 3, API 
breakage

with Rust, bad formatting, and other bugs of various nature. It does so way
before it touches the `committed` repo, which means that if I screw up my
patch, it only has to bother me.

The value of such a system is highly dependent on it being trustworthy: 
flaky
tests do exist, but they should be addressed and fixed in a timely 
manner and

my CI being red should mean that *I* missed something. Yet, I have heard the
phrase "I don't even look at the pipelines anymore, they're always 
broken for

something I haven't done".

Most of the time it's something minor: I've even been guilty of breaking 
the CI

myself in my last patch series (oops) due to bad formatting! However, these
issues should be few and far between: the developer experience suffers 
greatly

and we grow to accept that the tests are always broken, don't bother to run
them at all, etc.

# Proposal

I thought a start to fixing this situation would be for me to setup
an automation to send a small email to this mailing-list every time 
either the
`stable` or `default` branches (that track `hg-committed`) are broken, 
with a link

to the pipeline results.
This will not catch everything (only a subset of the possible targets is 
tested), but
I think this would be an improvement for all contributors, even those 
that don't

use Heptapod.

What do you all think?
Raphaël

[1] https://foss.heptapod.net/octobus/mercurial-devel/-/pipelines
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel