Re: [Nix-dev] Travis Testing Needs Rethinking

[Wout Mertens]

Help on https://github.com/madjar/nox/issues/38 is appreciated methinks :-)

On Wed, Mar 2, 2016, 2:46 AM Herwig Hochleitner 
wrote:

> 2016-02-14 19:44 GMT+01:00 zimbatm :
>
>> What if master was always the latest successful hydra build ? People
>> could rebase on top and have cached builds. Nox could also have cached
>> builds.
>>
>
> +1 for having Travis build against the latest successful hydra build.
> Don't care whether that's channels/nixos-unstable or a repurposed master.
>
> This way, most packages timing out on travis, would be those that trigger
> mass-rebuilds, which need to go to staging anyway. It should make the flag
> much more useful.
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
-- 

Wout.
(typed on mobile, excuse terseness)
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Travis Testing Needs Rethinking

[Herwig Hochleitner]

2016-02-14 19:44 GMT+01:00 zimbatm :

> What if master was always the latest successful hydra build ? People could
> rebase on top and have cached builds. Nox could also have cached builds.
>

+1 for having Travis build against the latest successful hydra build. Don't
care whether that's channels/nixos-unstable or a repurposed master.

This way, most packages timing out on travis, would be those that trigger
mass-rebuilds, which need to go to staging anyway. It should make the flag
much more useful.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Travis Testing Needs Rethinking

[Tobias Pflug]

> On 16 Feb 2016, at 00:06, Александр Цамутали  wrote:
> 
> 14.02.2016, 21:44, "zimbatm" :
>> What if master was always the latest successful hydra build ? People could 
>> rebase on top and have cached builds. Nox could also have cached builds.
>> 
>> Instead of merging a PR we would instruct Hydra to queue the build. If the 
>> build is successful hydra would merge it into master and close the PR. 
>> Otherwise it would amend the PR with the build errors. To instruct Hydra we 
>> could annotate a PR with a custom message like: "hydra go!".
>> 
>> I believe that this would also help to spot regressions faster. Right now 
>> Darwin tends to get breaking chances because Nox only tests against Nixos. 
>> And since the builds would be done against all OSes we would also be able to 
>> merge all the unstable channels.
> 
> +1 for passing every change through CI before merging. I don't know about 
> features of Travis CI and if it's possible to implement this with it. But I 
> know that OpenStack Infra uses Zuul for such tasks.
> 
> 

I still lack insight since I haven't been using nix for that long but this 
clearly seems like a goal that should be aimed for
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Travis Testing Needs Rethinking

[Александр Цамутали]

14.02.2016, 21:44, "zimbatm" :
> What if master was always the latest successful hydra build ? People could 
> rebase on top and have cached builds. Nox could also have cached builds.
>
> Instead of merging a PR we would instruct Hydra to queue the build. If the 
> build is successful hydra would merge it into master and close the PR. 
> Otherwise it would amend the PR with the build errors. To instruct Hydra we 
> could annotate a PR with a custom message like: "hydra go!".
>
> I believe that this would also help to spot regressions faster. Right now 
> Darwin tends to get breaking chances because Nox only tests against Nixos. 
> And since the builds would be done against all OSes we would also be able to 
> merge all the unstable channels.

+1 for passing every change through CI before merging. I don't know about 
features of Travis CI and if it's possible to implement this with it. But I 
know that OpenStack Infra uses Zuul for such tasks.


-- 
Александр Цамутали
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Travis Testing Needs Rethinking

[zimbatm]

What if master was always the latest successful hydra build ? People could
rebase on top and have cached builds. Nox could also have cached builds.

Instead of merging a PR we would instruct Hydra to queue the build. If the
build is successful hydra would merge it into master and close the PR.
Otherwise it would amend the PR with the build errors. To instruct Hydra we
could annotate a PR with a custom message like: "hydra go!".

I believe that this would also help to spot regressions faster. Right now
Darwin tends to get breaking chances because Nox only tests against Nixos.
And since the builds would be done against all OSes we would also be able
to merge all the unstable channels.


On Sat, 13 Feb 2016 at 19:24 Kevin Cox  wrote:

> On Sat, 2016-02-13 at 17:03 +, Adam Russell wrote:
>
> Wouldn't this all be less of an issue if the build on Hydra wasn't behind
> by weeks? Should we talk about how to improve that? Personally I don't even
> know how to navigate or interpret Hydra when I go look at it.
>
>
> I don't think Hydra is actually behind much, the channels are but IIUC
> most builds are actually completing in a couple of hours. (Actually looking
> now I see some 2 day old builds but nothing too terrible
> http://hydra.nixos.org/status).
>
> Getting faster hydra builds might make this less of an issue but there
> will always be a delay. Of course if that delay is very small it won't
> matter much.
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Travis Testing Needs Rethinking

[Arseniy Seroka]

There are several things we need to check in a PR even if it builds
successfully.
On 14 Feb 2016 21:44, "zimbatm"  wrote:

> What if master was always the latest successful hydra build ? People could
> rebase on top and have cached builds. Nox could also have cached builds.
>
> Instead of merging a PR we would instruct Hydra to queue the build. If the
> build is successful hydra would merge it into master and close the PR.
> Otherwise it would amend the PR with the build errors. To instruct Hydra we
> could annotate a PR with a custom message like: "hydra go!".
>
> I believe that this would also help to spot regressions faster. Right now
> Darwin tends to get breaking chances because Nox only tests against Nixos.
> And since the builds would be done against all OSes we would also be able
> to merge all the unstable channels.
>
>
> On Sat, 13 Feb 2016 at 19:24 Kevin Cox  wrote:
>
>> On Sat, 2016-02-13 at 17:03 +, Adam Russell wrote:
>>
>> Wouldn't this all be less of an issue if the build on Hydra wasn't behind
>> by weeks? Should we talk about how to improve that? Personally I don't even
>> know how to navigate or interpret Hydra when I go look at it.
>>
>>
>> I don't think Hydra is actually behind much, the channels are but IIUC
>> most builds are actually completing in a couple of hours. (Actually looking
>> now I see some 2 day old builds but nothing too terrible
>> http://hydra.nixos.org/status).
>>
>> Getting faster hydra builds might make this less of an issue but there
>> will always be a delay. Of course if that delay is very small it won't
>> matter much.
>>
>> ___
>> nix-dev mailing list
>> nix-dev@lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Travis Testing Needs Rethinking

[Domen Kožar]

Thanks - I think it's worth a try. Could you open an issue? I also have
some ideas how to dodge logs limit.

On Sat, Feb 13, 2016 at 1:41 PM, Kevin Cox  wrote:

> Hello,
>
> I have been trying to submit a pull request for quite some time but keep
> getting bitten by the Travis testing. The problem boils down to a couple of
> things. Firstly travis has build limits both in terms of log size and build
> time. These are expected and while there is possibility of getting them
> raised (at least for log size) I'm going to assume that these are the
> bounds we have to work within.
>
> The reason that this is a problem is because of the way nox (the review
> tool) works. Nox merges your new changes into the master branch then
> attempts to build your PR. The problem with this is that anything you
> depend on that has recently been changed will have to be rebuilt from
> source as they aren't yet in a binary cache. When you have a sufficiently
> complex package you almost always have dependencies that have recently
> changed so you end up rebuilding a lot of unchanged packages which will
> often send you over your resource quotas. This is especially true as these
> complex packages are often close to the limits on their own.
>
> To combat this problem I propose that packages are tested on their current
> branch. To determine what has changed simple identify the changes since the
> latest common ancestor of the pull request and the master branch. This
> means that you can allow your PR to get slightly behind master and now all
> of your dependencies are in the binary cache and don't eat up your
> resources.
>
> The apparent downside is that you aren't testing on the latest code so
> there may have been incompatible changes in the mean time that appear on
> merge. However since there is often a couple of days delay between Travis
> passing and merging of PRs anyways so I don't see this as much added risk.
> If desired an error or warning can be added when a PR is "too far" behind
> master to limit this problem as well. In the rare case of conflict it will
> be caught on the master branch, either by testing master in a similar way
> (although finding the "base" commit might be non-trivial and you still have
> the resource usage issue) or when they appear on hydra. Thanks to git and
> Github reverting PRs that cause breakage is literally two clicks so
> anything that causes a merge conflict can be reverted and the submitter
> notified so that it can be fixed up to be re-merged.
>
> I think that the overall value proposition is a benefit as it will make
> merging PRs much less painful because build times will be down which
> prevents travis killing builds and enables faster feedback and iteration.
>
> TL;DR building PRs based off of latest master often takes too many
> resources for Travis so we should base CI builds off of the PR branch.
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Travis Testing Needs Rethinking

[Tuomas Tynkkynen]

On 02/13/2016 03:41 PM, Kevin Cox wrote:

>
> TL;DR building PRs based off of latest master often takes too many
> resources for Travis so we should base CI builds off of the PR branch.
>

100% agreed. While we're on the topic of Travis and pull requests, our 
contribution guide currently contains this as the last step:

 > Rebase you branch against current master.

IMHO this is actively harmful and shouldn't be done, because:

- Sending a pull request for a commit hash X should imply a semantic 
meaning of 'I have personally tested that the exact commit X works'. 
Currently we are essentially encouraging contributors to send us 
untested stuff.
- If Kevin's suggestion wrt. building only against the PR branch is 
implemented, rebasing against master means that the likelihood of 
dependencies of the current Travis build not being yet built by Hydra 
increases, leading to Travis timeouts yet again.
- If master has been broken (e.g. tarball evaluation fails) by someone 
else's change in the meantime, this totally unrelated breakage causes 
Travis failures for all those pull requests where the rebase-to-master 
dance was done.

So, any objections for removing that part from the contribution guide 
and perhaps suggesting rebase only in case of GitHub reporting conflicts?
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Travis Testing Needs Rethinking

[Adam Russell]

Wouldn't this all be less of an issue if the build on Hydra wasn't behind
by weeks? Should we talk about how to improve that? Personally I don't even
know how to navigate or interpret Hydra when I go look at it.

On Sat, Feb 13, 2016 at 10:48 AM Jakob Gillich  wrote:

> I assume PR branch refers to unstable? Or even a stable branch? I don't
> really see how testing against non-master is better when you're submitting
> something for master.
>
> On Sat, Feb 13, 2016, at 02:41 PM, Kevin Cox wrote:
>
> Hello,
>
> I have been trying to submit a pull request for quite some time but keep
> getting bitten by the Travis testing. The problem boils down to a couple of
> things. Firstly travis has build limits both in terms of log size and build
> time. These are expected and while there is possibility of getting them
> raised (at least for log size) I'm going to assume that these are the
> bounds we have to work within.
>
> The reason that this is a problem is because of the way nox (the review
> tool) works. Nox merges your new changes into the master branch then
> attempts to build your PR. The problem with this is that anything you
> depend on that has recently been changed will have to be rebuilt from
> source as they aren't yet in a binary cache. When you have a sufficiently
> complex package you almost always have dependencies that have recently
> changed so you end up rebuilding a lot of unchanged packages which will
> often send you over your resource quotas. This is especially true as these
> complex packages are often close to the limits on their own.
>
> To combat this problem I propose that packages are tested on their current
> branch. To determine what has changed simple identify the changes since the
> latest common ancestor of the pull request and the master branch. This
> means that you can allow your PR to get slightly behind master and now all
> of your dependencies are in the binary cache and don't eat up your
> resources.
>
> The apparent downside is that you aren't testing on the latest code so
> there may have been incompatible changes in the mean time that appear on
> merge. However since there is often a couple of days delay between Travis
> passing and merging of PRs anyways so I don't see this as much added risk.
> If desired an error or warning can be added when a PR is "too far" behind
> master to limit this problem as well. In the rare case of conflict it will
> be caught on the master branch, either by testing master in a similar way
> (although finding the "base" commit might be non-trivial and you still have
> the resource usage issue) or when they appear on hydra. Thanks to git and
> Github reverting PRs that cause breakage is literally two clicks so
> anything that causes a merge conflict can be reverted and the submitter
> notified so that it can be fixed up to be re-merged.
>
> I think that the overall value proposition is a benefit as it will make
> merging PRs much less painful because build times will be down which
> prevents travis killing builds and enables faster feedback and iteration.
>
> TL;DR building PRs based off of latest master often takes too many
> resources for Travis so we should base CI builds off of the PR branch.
> *___*
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
> Email had 2 attachments:
>
>- signature.asc
>  1k (application/pgp-signature)
>- smime.p7s
>  8k (application/x-pkcs7-signature)
>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Travis Testing Needs Rethinking

[Kevin Cox]

On Sat, 2016-02-13 at 17:48 +0100, Jakob Gillich wrote:
> I assume PR branch refers to unstable? Or even a stable branch? I
> don't really see how testing against non-master is better when you're
> submitting something for master.
The PR will still be based of master but master as of a couple of
hours/days ago. That way master has had a little bit of time to settle.
The problem is that once you merge it in you get the very latest master
which often has a number of non-uploaded packages.



signature.asc
Description: This is a digitally signed message part


smime.p7s
Description: S/MIME cryptographic signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Travis Testing Needs Rethinking

[Kevin Cox]

On Sat, 2016-02-13 at 17:03 +, Adam Russell wrote:
> Wouldn't this all be less of an issue if the build on Hydra wasn't
> behind by weeks? Should we talk about how to improve that? Personally
> I don't even know how to navigate or interpret Hydra when I go look
> at it.
> 
I don't think Hydra is actually behind much, the channels are but IIUC
most builds are actually completing in a couple of hours. (Actually
looking now I see some 2 day old builds but nothing too terrible http:/
/hydra.nixos.org/status).
Getting faster hydra builds might make this less of an issue but there
will always be a delay. Of course if that delay is very small it won't
matter much.


signature.asc
Description: This is a digitally signed message part


smime.p7s
Description: S/MIME cryptographic signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev