On Tue, Dec 13, 2016 at 01:11:37PM +0100, Martin Babinsky wrote: > On 12/13/2016 01:07 PM, Fraser Tweedale wrote: > > On Tue, Dec 13, 2016 at 09:41:40AM +0100, Martin Babinsky wrote: > > > Hi list, > > > > > > https://github.com/freeipa/freeipa/pull/177 was recently merged despite > > > causing nearly half of the tests in our Travis CI gating to fail. This > > > broke > > > Travis CI for all other PR that were rebased after this merge, causing > > > false > > > negative errors everywhere. > > > > > > Fraser reverted the offending commits in > > > https://github.com/freeipa/freeipa/pull/329 which restored Travis to > > > original state (never mind PEP8 errors they were in the original code > > > already). > > > > > > Regarding this issues I have two questions: > > > > > > a) > > > > > > should I merge https://github.com/freeipa/freeipa/pull/329 and thus revert > > > the breakage in order to unblock other contributors? Given the current > > > traffic I think it is sufficient to wait for us to investigate and > > > produce a > > > fix. If not, please scream loudly. > > > > > Definitely not, I am using this PR as a playground to poke the CI in > > various ways. Also, proper fix would be best :) See also my other > > mail (I would have replied here but fetchmail was not fetching mail > > and I missed it until just now >_<). > > > > > b) > > > > > > what can we improve to make the results of CI more visible to > > > contributors? > > > I think that I should sit down with Martin 2 and investigate the > > > possibility > > > to send notifications about negative CI results (sufficient IMO) to the > > > mailing list. > > > > > Or the commit author. > > > > It would be *great* if the test job, in event of failure, would > > collect all the obvious logs and dump them somewhere as artifacts. > > > > That would be great but AFAIK Travis CI does support only archiving > artifacts into a running AWS instance[1] which we do not have and would have > to buy, run and maintain ourselves. For now we have to inspect the log dumps > generated in the job output. > Do the containers not have internet access? That is all you really need, surely.
The log dumps are not enough... `tail -n 5000' is not enough :) Even if there is no other way besides S3, it might still be worth it. > > > In the meanwhile I would like to ask all reviewers to carefully check the > > > output of failed Travis CI runs. If the job fails, you will see the > > > results > > > at the very end of the log. There are two sections: PEP8 errors and test > > > output. You can expand both of them to see what went wrong and report it > > > to > > > the PR author if necessary. > > > > > > The reviewer and author can then use the very same tool used in CI [1] to > > > reproduce the failures locally. Using '--no-cleanup' option during the > > > run > > > [2] leaves behind a running container which you can attach to and > > > investigate further. > > > > > > [1] https://github.com/freeipa/ipa-docker-test-runner > > > [2] > > > https://github.com/freeipa/ipa-docker-test-runner/blob/master/README.md > > > > > > If you have any additional questions/suggestions about Travis feel free to > > > contact me. > > > > > > -- > > > Martin^3 Babinsky > > > > > > -- > > > Manage your subscription for the Freeipa-devel mailing list: > > > https://www.redhat.com/mailman/listinfo/freeipa-devel > > > Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code > > [1] https://docs.travis-ci.com/user/uploading-artifacts/ > > -- > Martin^3 Babinsky -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code