Re: [Nix-dev] Continuous Integration

[Ericson, John]

>
> If https://github.com/NixOS/hydra/pull/277 is merged, then there is one
> step left to get automatic PR management: An input plugin which fetches
> information about open PRs on a repo. I'll write it if 277 ever gets in,
> but without 277 or something like it there's no point.
>

Well, it go merged! I don't actually know anything about the relevant APIs,
but shouldn't github be able to push this information to hydra? I'd think
notifying on newly-opened PRs and changes to master would be sufficient.

On Thu, Apr 28, 2016 at 3:42 AM, Domen Kožar <do...@dev.si> wrote:

> That's a good point, but I don't see how a separate tab would contrain
> hydra from adding extra input parameters to the eval. It's just a perl
> "controller" for UI separation.
>
> /cc aszlig
>
> On Thu, Apr 28, 2016 at 11:37 AM, <s...@shealevy.com> wrote:
>
>> IMO putting the logic completely into hydra will be too limiting. If
>> hydra just gives a nix expression some info about the PRs, then the nix
>> expression can do arbitrary things (e.g. only build PRs from trusted users,
>> or that don't change stdenv, or whatever). If it's a jobset tab then you
>> have to add every possible variation into the hydra UI.
>>
>>
>>
>> - Original Message -
>> From:
>> "Domen Kožar" <do...@dev.si>
>>
>> To:
>> "Shea Levy" <s...@shealevy.com>
>> Cc:
>> "Graham Christensen" <gra...@grahamc.com>, "Ericson John" <
>> john_eric...@brown.edu>, "nix-dev" <nix-dev@lists.science.uu.nl>
>> Sent:
>> Thu, 28 Apr 2016 11:32:53 +0100
>>
>> Subject:
>> Re: [Nix-dev] Continuous Integration
>>
>>
>> I was talking to Aszlig about this and it would be best if PRs would be a
>> tab on the jobset. Since jobset defines the inputs, it would be tested for
>> PRs against that specific branch.
>>
>> On Thu, Apr 28, 2016 at 11:30 AM, <s...@shealevy.com> wrote:
>>
>>> It can set pull request statuses (e.g. see
>>> https://github.com/shlevy/hydra-github-status-test/pull/3), but
>>> currently you have to manually create the jobset corresponding to the PR
>>> and ensure that the relevant jobs are captured by the status plugin.
>>>
>>> If https://github.com/NixOS/hydra/pull/277 is merged, then there is one
>>> step left to get automatic PR management: An input plugin which fetches
>>> information about open PRs on a repo. I'll write it if 277 ever gets in,
>>> but without 277 or something like it there's no point.
>>>
>>>
>>> - Original Message -
>>> From:
>>> "Graham Christensen" <gra...@grahamc.com>
>>>
>>> To:
>>> "Domen Kožar" <do...@dev.si>
>>> Cc:
>>> "Ericson John" <john_eric...@brown.edu>, "nix-dev" <
>>> nix-dev@lists.science.uu.nl>
>>> Sent:
>>> Wed, 27 Apr 2016 18:23:55 -0500
>>> Subject:
>>> Re: [Nix-dev] Continuous Integration
>>>
>>>
>>>
>>>
>>>
>>> Domen Kožar <do...@dev.si> writes:
>>>
>>> > It doesn't build PRs, just updates the status.
>>> >
>>>
>>> Does this mean it can build specific branches (master, release-16.04..)
>>> and set the commit status on those commits? ie: doesn't have anything to
>>> do with pull request statuses?
>>>
>>> Best,
>>> Graham
>>> ___
>>> nix-dev mailing list
>>> nix-dev@lists.science.uu.nl
>>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>
>>>
>>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Continuous Integration

[Domen Kožar]

That's a good point, but I don't see how a separate tab would contrain
hydra from adding extra input parameters to the eval. It's just a perl
"controller" for UI separation.

/cc aszlig

On Thu, Apr 28, 2016 at 11:37 AM, <s...@shealevy.com> wrote:

> IMO putting the logic completely into hydra will be too limiting. If hydra
> just gives a nix expression some info about the PRs, then the nix
> expression can do arbitrary things (e.g. only build PRs from trusted users,
> or that don't change stdenv, or whatever). If it's a jobset tab then you
> have to add every possible variation into the hydra UI.
>
>
>
> - Original Message -
> From:
> "Domen Kožar" <do...@dev.si>
>
> To:
> "Shea Levy" <s...@shealevy.com>
> Cc:
> "Graham Christensen" <gra...@grahamc.com>, "Ericson John" <
> john_eric...@brown.edu>, "nix-dev" <nix-dev@lists.science.uu.nl>
> Sent:
> Thu, 28 Apr 2016 11:32:53 +0100
>
> Subject:
> Re: [Nix-dev] Continuous Integration
>
>
> I was talking to Aszlig about this and it would be best if PRs would be a
> tab on the jobset. Since jobset defines the inputs, it would be tested for
> PRs against that specific branch.
>
> On Thu, Apr 28, 2016 at 11:30 AM, <s...@shealevy.com> wrote:
>
>> It can set pull request statuses (e.g. see
>> https://github.com/shlevy/hydra-github-status-test/pull/3), but
>> currently you have to manually create the jobset corresponding to the PR
>> and ensure that the relevant jobs are captured by the status plugin.
>>
>> If https://github.com/NixOS/hydra/pull/277 is merged, then there is one
>> step left to get automatic PR management: An input plugin which fetches
>> information about open PRs on a repo. I'll write it if 277 ever gets in,
>> but without 277 or something like it there's no point.
>>
>>
>> - Original Message -----
>> From:
>> "Graham Christensen" <gra...@grahamc.com>
>>
>> To:
>> "Domen Kožar" <do...@dev.si>
>> Cc:
>> "Ericson John" <john_eric...@brown.edu>, "nix-dev" <
>> nix-dev@lists.science.uu.nl>
>> Sent:
>> Wed, 27 Apr 2016 18:23:55 -0500
>> Subject:
>> Re: [Nix-dev] Continuous Integration
>>
>>
>>
>>
>>
>> Domen Kožar <do...@dev.si> writes:
>>
>> > It doesn't build PRs, just updates the status.
>> >
>>
>> Does this mean it can build specific branches (master, release-16.04..)
>> and set the commit status on those commits? ie: doesn't have anything to
>> do with pull request statuses?
>>
>> Best,
>> Graham
>> ___
>> nix-dev mailing list
>> nix-dev@lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Continuous Integration

[shea]

IMO putting the logic completely into hydra will be too limiting. If
hydra just gives a nix expression some info about the PRs, then the
nix expression can do arbitrary things (e.g. only build PRs from
trusted users, or that don't change stdenv, or whatever). If it's a
jobset tab then you have to add every possible variation into the
hydra UI.

- Original Message -
From:
 "Domen Kožar" <do...@dev.si>

To:
"Shea Levy" <s...@shealevy.com>
Cc:
"Graham Christensen" <gra...@grahamc.com>, "Ericson John"
<john_eric...@brown.edu>, "nix-dev" <nix-dev@lists.science.uu.nl>
Sent:
Thu, 28 Apr 2016 11:32:53 +0100
Subject:
Re: [Nix-dev] Continuous Integration

I was talking to Aszlig about this and it would be best if PRs would
be a tab on the jobset. Since jobset defines the inputs, it would be
tested for PRs against that specific branch.

On Thu, Apr 28, 2016 at 11:30 AM, <s...@shealevy.com [1]>
 wrote:
It can set pull request statuses (e.g.
see https://github.com/shlevy/hydra-github-status-test/pull/3 [2]),
but currently you have to manually create the jobset corresponding to
the PR and ensure that the relevant jobs are captured by the status
plugin.

If https://github.com/NixOS/hydra/pull/277 [3] is merged, then there
is one step left to get automatic PR management: An input plugin which
fetches information about open PRs on a repo. I'll write it if 277
ever gets in, but without 277 or something like it there's no point.

- Original Message -
From:
 "Graham Christensen" <gra...@grahamc.com [4]>

To:
"Domen Kožar" <do...@dev.si [5]>
Cc:
"Ericson John" <john_eric...@brown.edu [6]>, "nix-dev"
<nix-dev@lists.science.uu.nl [7]>
Sent:
Wed, 27 Apr 2016 18:23:55 -0500
Subject:
Re: [Nix-dev] Continuous Integration

 Domen Kožar <do...@dev.si [8]> writes:

 > It doesn't build PRs, just updates the status.
 >

 Does this mean it can build specific branches (master,
release-16.04..)
 and set the commit status on those commits? ie: doesn't have anything
to
 do with pull request statuses?

 Best,
 Graham

 ___
 nix-dev mailing list
nix-dev@lists.science.uu.nl [9]
http://lists.science.uu.nl/mailman/listinfo/nix-dev [10]

 

Links:
--
[1] mailto:s...@shealevy.com
[2] https://github.com/shlevy/hydra-github-status-test/pull/3
[3] https://github.com/NixOS/hydra/pull/277
[4] mailto:gra...@grahamc.com
[5] mailto:do...@dev.si
[6] mailto:john_eric...@brown.edu
[7] mailto:nix-dev@lists.science.uu.nl
[8] mailto:do...@dev.si
[9] mailto:nix-dev@lists.science.uu.nl
[10] http://lists.science.uu.nl/mailman/listinfo/nix-dev

___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Continuous Integration

[Domen Kožar]

I was talking to Aszlig about this and it would be best if PRs would be a
tab on the jobset. Since jobset defines the inputs, it would be tested for
PRs against that specific branch.

On Thu, Apr 28, 2016 at 11:30 AM, <s...@shealevy.com> wrote:

> It can set pull request statuses (e.g. see
> https://github.com/shlevy/hydra-github-status-test/pull/3), but currently
> you have to manually create the jobset corresponding to the PR and ensure
> that the relevant jobs are captured by the status plugin.
>
> If https://github.com/NixOS/hydra/pull/277 is merged, then there is one
> step left to get automatic PR management: An input plugin which fetches
> information about open PRs on a repo. I'll write it if 277 ever gets in,
> but without 277 or something like it there's no point.
>
>
> - Original Message -
> From:
> "Graham Christensen" <gra...@grahamc.com>
>
> To:
> "Domen Kožar" <do...@dev.si>
> Cc:
> "Ericson John" <john_eric...@brown.edu>, "nix-dev" <
> nix-dev@lists.science.uu.nl>
> Sent:
> Wed, 27 Apr 2016 18:23:55 -0500
> Subject:
> Re: [Nix-dev] Continuous Integration
>
>
>
>
>
> Domen Kožar <do...@dev.si> writes:
>
> > It doesn't build PRs, just updates the status.
> >
>
> Does this mean it can build specific branches (master, release-16.04..)
> and set the commit status on those commits? ie: doesn't have anything to
> do with pull request statuses?
>
> Best,
> Graham
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Continuous Integration

[shea]

It can set pull request statuses (e.g.
see https://github.com/shlevy/hydra-github-status-test/pull/3), but
currently you have to manually create the jobset corresponding to the
PR and ensure that the relevant jobs are captured by the status
plugin.

If https://github.com/NixOS/hydra/pull/277 is merged, then there is
one step left to get automatic PR management: An input plugin which
fetches information about open PRs on a repo. I'll write it if 277
ever gets in, but without 277 or something like it there's no point.

- Original Message -
From: "Graham Christensen" 
To:"Domen Kožar" 
Cc:"Ericson John" , "nix-dev"

Sent:Wed, 27 Apr 2016 18:23:55 -0500
Subject:Re: [Nix-dev] Continuous Integration

 Domen Kožar  writes:

 > It doesn't build PRs, just updates the status.
 >

 Does this mean it can build specific branches (master,
release-16.04..)
 and set the commit status on those commits? ie: doesn't have anything
to
 do with pull request statuses?

 Best,
 Graham
 ___
 nix-dev mailing list
 nix-dev@lists.science.uu.nl
 http://lists.science.uu.nl/mailman/listinfo/nix-dev

___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Continuous Integration

[Graham Christensen]

Domen Kožar  writes:

> It doesn't build PRs, just updates the status.
>

Does this mean it can build specific branches (master, release-16.04..)
and set the commit status on those commits? ie: doesn't have anything to
do with pull request statuses?

Best,
Graham
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Continuous Integration

[Domen Kožar]

It doesn't build PRs, just updates the status.

On Wed, Apr 27, 2016 at 9:44 PM, Ericson, John 
wrote:

> Support for hydra to build github PR has been added for a few weeks now in
> https://github.com/NixOS/hydra/pull/280 . What's the next step for
> actually using this with nixpkgs and hydra.nixos.org?
>
> On Tue, Feb 23, 2016 at 4:46 PM, Ericson, John 
> wrote:
>
>> > S3 and Hydra PR support
>>
>> I asked Domen earlier on IRC, but in case anyone else is interested,
>> is there any way to contribute to these? If this will go most the way
>> towards continuously updating channel(s) I'll give an arm and a leg.
>>
>> > IMHO it's very often a fault of some builds or tests failing (sometimes
>> > transiently). > IMHO it's very often a fault of some builds or tests
>> failing (sometimes
>> > transiently).
>>
>> Well my proposal very much does matter if honestly-failing
>> builds/tests are a problem---let's not have one bad commit hold up
>> good commits. If most of the failures are transitive however, well
>> that's very unfortunate. I hear bad things about timeouts but would
>> love more details on the transitive errors we face and what can be
>> done about them.
>>
>> John
>>
>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Continuous Integration

[Ericson, John]

Support for hydra to build github PR has been added for a few weeks now in
https://github.com/NixOS/hydra/pull/280 . What's the next step for actually
using this with nixpkgs and hydra.nixos.org?

On Tue, Feb 23, 2016 at 4:46 PM, Ericson, John 
wrote:

> > S3 and Hydra PR support
>
> I asked Domen earlier on IRC, but in case anyone else is interested,
> is there any way to contribute to these? If this will go most the way
> towards continuously updating channel(s) I'll give an arm and a leg.
>
> > IMHO it's very often a fault of some builds or tests failing (sometimes
> > transiently). > IMHO it's very often a fault of some builds or tests
> failing (sometimes
> > transiently).
>
> Well my proposal very much does matter if honestly-failing
> builds/tests are a problem---let's not have one bad commit hold up
> good commits. If most of the failures are transitive however, well
> that's very unfortunate. I hear bad things about timeouts but would
> love more details on the transitive errors we face and what can be
> done about them.
>
> John
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Continuous Integration

[Ericson, John]

> S3 and Hydra PR support

I asked Domen earlier on IRC, but in case anyone else is interested,
is there any way to contribute to these? If this will go most the way
towards continuously updating channel(s) I'll give an arm and a leg.

> IMHO it's very often a fault of some builds or tests failing (sometimes
> transiently). > IMHO it's very often a fault of some builds or tests failing 
> (sometimes
> transiently).

Well my proposal very much does matter if honestly-failing
builds/tests are a problem---let's not have one bad commit hold up
good commits. If most of the failures are transitive however, well
that's very unfortunate. I hear bad things about timeouts but would
love more details on the transitive errors we face and what can be
done about them.

John
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Continuous Integration

[Vladimír Čunát]

IMHO it's very often a fault of some builds or tests failing (sometimes
transiently). Of course, the lag also doesn't help to resolve these
issues quickly, but more people actually fixing that stuff might have
more impact than HW and infrastructural changes.

--Vladimir




smime.p7s
Description: S/MIME Cryptographic Signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Continuous Integration

[Domen Kožar]

There are currently on-going improvements to Hydra to upload built packages
to S3 cache instead of central Hydra machine. That should speed up the
bulids a lot. Another thing on horizon are SSDs for the central machine.
That should get evaluation
times down to dozen of minutes instead of hours.

On the other hand, handling test breakages, there's much we can do. There
is some unfinished work getting Hydra to build PRs, meantime we can improve
travis-ci integration to fail less often.

On Mon, Feb 22, 2016 at 10:34 PM, Ericson, John 
wrote:

> As everyone knows, while updates to nixpkgs roll in 'round the clock,
> the unstable channels are usually stuck for days or weeks before
> lurching forward. In other words, we don't do continuous integration.
> This does not seem sustainable as nixpkgs grows larger and changes
> more rapidly.
>
> When I asked about this on IRC previously, I was told that the problem
> was intractable while we only had one mac mini, but now we have more
> Darwin build machines, and I believe the Darwin nixpkgs infrastructure
> has also matured leading to less mass-rebuilds. Thus, I hope there is
> no longer a fatal imbalance between our Linux and Darwin needs and
> capabilities.
>
> So perhaps now is good time to consider: can we commit to ensuring
> that nixos-unsable == nixpkgs-unstable == master? In particular this
> means ensuring that every PR passes before merging, and rebuilding
> previously-passing PRs whenever master changes. With most build
> systems, this would unleash a thundering heard of rebuilding PRs every
> time master changes, but since we cache very well and most PRs
> commute, I believe this won't be a problem. Mass-rebuilds of course
> violate this, but they are a bottleneck no matter what we do.
>
> Note that Travis doesn't rebuild previously-passing PRs. Also, we'd
> probably want some way to bless a bunch of PRs so they are
> automatically merged as soon as their builds pass. I'd love to
> integrate Hydra with github directly, but in the meantime the Rust
> community wrote https://github.com/barosl/homu to address these
> issues, and it might be worth looking at. [I follow Rust development
> so I've seen it in action. But I haven't used it myself.]
>
> What I think this boils down to is the difference between between
> master and the *-unstable channels branches dictating hydra's queue,
> and the set of open PRs doing so. The order in which PRs get merged is
> mostly-arbitrary, and thus the linearity that imposes is mostly
> worthless: Hydra must sign off on one commit before moving on to the
> next even if the failing commit doesn't interact with the succeeding
> commits at all.
>
> Finally, there is more that I don't know than I do know about hydra
> and the status quo, so I hope above all else this kicks off a good
> discussion.
>
> John
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev