Re: [Freeipa-devel] CI: exporting test runner output
On Thu, Jan 05, 2017 at 09:38:03AM +0100, Tomas Krizek wrote: > On 01/05/2017 09:25 AM, Fraser Tweedale wrote: > > On Thu, Jan 05, 2017 at 08:53:14AM +0100, Martin Babinsky wrote: > >> On 01/05/2017 08:06 AM, Fraser Tweedale wrote: > >>> Hi all, > >>> > >>> Although it has been discussed before and met with some skepticism, > >>> here is a POC that exporting test runner output to, e.g. a pastebin, > >>> does work: > >>> > >>> - experimental commit: https://github.com/freeipa/freeipa/pull/370 > >>> - example paste: https://paste.fedoraproject.org/520085/ > >>> (it is gzipped for reasons discussed in the PR) > >>> > >>> I think we should proceed with getting these artifacts out of Travis > >>> and stored somewhere (it doesn't have to be > >>> paste.fedoraproject.org). ``tail -n 5000`` of the log file has > >>> proven to be not enough to diagnose all failures. > >>> > >> Wow this is great, why have I not thought about it beforehand? > Seems like a great feature. Thanks, Fraser! > >> We can reduce the log size if we truncate everything before ERRORS/FAILURES > >> output of pytest run (we leave the log as it is if the fail occurs before > >> this stage), that should shave off considerable amount of cruft from the > >> paste unless somebody sends a PR that breaks all out tests :D. > >> > >>> If we stick with paste.fedoraproject.org, we can send to a > >>> "project-specific" namespace e.g. > >>> https://paste.fedoraproject.org/~freeipa, so that we do not clutter > >>> up the main archive (I think). > >>> I was wrong. All "project" pastes appear in main namespace as well as project namespace. Not sure if by design or not. > >>> A few questions for discussion: > >>> > >>> 1. Stick with fpaste or not? If so, use "~freeipa" namespace? > >>>(Keep in mind that the size limitation that exists for fpaste, > >>>which requires compressing the artifact, may not be a problem > >>>elsewhere). > >>> > >>> 2. Export log always, or only if the build job failed? > >>> > >> I would also paste the output to "freeipa" or even better "freeipa-travis" > >> namespace and only send it if the job fails. > >> > > I might go with "freeipa-ci". > +1 > Unfortunately fpaste can't handle this. Has to be all-alpha. So we can use "freeipaci" but given the constraint I would rather just use "freeipa". I shall file a fedora-infra ticket to see if this can be addressed. > >>> 3. Should pasted logs expire? If so, what should TTL be? > >>> > >> IMHO yes, but TTL is hard to determine, since the author of the PR may not > >> be present to review the results immediately (because he is on PTO etc.). I > >> think we should set TTL to something like 1 week and as a fallback keep > >> tailing the CI results log. > >> > > 1 week sounds reasonable. We can change it later if we need to. > I actually wouldn't mind extending this to something like 2-4 weeks. In > some cases it might be useful to have access to older logs (PTOs, or > simply to just view the history for some reason). Is there any downside > to keeping the logs for a bit longer? > Not really. I was thinking server diskspace is logs were very big but now that we're compressing I don't think it matters. 4 weeks, sure why not :) > >>> 4. Should we continue to `tail -n 5000` the log as we currently do, > >>>or just rely on exported log? > If you're talking about the log in the travis web interface, I would > keep it. It's easily accessible from the browser. > >>> Thanks, > >>> Fraser > >> Fraser, are you OK with waiting with this effort until we push > >> https://github.com/freeipa/freeipa/pull/361 ?. I will just do some more > >> adjustments there (like result log trimming) and it should be pushed ASAP. > >> > > Yes, I was aware that there would be conflicts with this PR. I > > don't mind waiting. Thanks for your input. > > > > Cheers, > > Fraser > > > -- > Tomas Krizek > > -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] CI: exporting test runner output
On 01/05/2017 09:25 AM, Fraser Tweedale wrote: > On Thu, Jan 05, 2017 at 08:53:14AM +0100, Martin Babinsky wrote: >> On 01/05/2017 08:06 AM, Fraser Tweedale wrote: >>> Hi all, >>> >>> Although it has been discussed before and met with some skepticism, >>> here is a POC that exporting test runner output to, e.g. a pastebin, >>> does work: >>> >>> - experimental commit: https://github.com/freeipa/freeipa/pull/370 >>> - example paste: https://paste.fedoraproject.org/520085/ >>> (it is gzipped for reasons discussed in the PR) >>> >>> I think we should proceed with getting these artifacts out of Travis >>> and stored somewhere (it doesn't have to be >>> paste.fedoraproject.org). ``tail -n 5000`` of the log file has >>> proven to be not enough to diagnose all failures. >>> >> Wow this is great, why have I not thought about it beforehand? Seems like a great feature. Thanks, Fraser! >> We can reduce the log size if we truncate everything before ERRORS/FAILURES >> output of pytest run (we leave the log as it is if the fail occurs before >> this stage), that should shave off considerable amount of cruft from the >> paste unless somebody sends a PR that breaks all out tests :D. >> >>> If we stick with paste.fedoraproject.org, we can send to a >>> "project-specific" namespace e.g. >>> https://paste.fedoraproject.org/~freeipa, so that we do not clutter >>> up the main archive (I think). >>> >>> A few questions for discussion: >>> >>> 1. Stick with fpaste or not? If so, use "~freeipa" namespace? >>>(Keep in mind that the size limitation that exists for fpaste, >>>which requires compressing the artifact, may not be a problem >>>elsewhere). >>> >>> 2. Export log always, or only if the build job failed? >>> >> I would also paste the output to "freeipa" or even better "freeipa-travis" >> namespace and only send it if the job fails. >> > I might go with "freeipa-ci". +1 >>> 3. Should pasted logs expire? If so, what should TTL be? >>> >> IMHO yes, but TTL is hard to determine, since the author of the PR may not >> be present to review the results immediately (because he is on PTO etc.). I >> think we should set TTL to something like 1 week and as a fallback keep >> tailing the CI results log. >> > 1 week sounds reasonable. We can change it later if we need to. I actually wouldn't mind extending this to something like 2-4 weeks. In some cases it might be useful to have access to older logs (PTOs, or simply to just view the history for some reason). Is there any downside to keeping the logs for a bit longer? >>> 4. Should we continue to `tail -n 5000` the log as we currently do, >>>or just rely on exported log? If you're talking about the log in the travis web interface, I would keep it. It's easily accessible from the browser. >>> Thanks, >>> Fraser >> Fraser, are you OK with waiting with this effort until we push >> https://github.com/freeipa/freeipa/pull/361 ?. I will just do some more >> adjustments there (like result log trimming) and it should be pushed ASAP. >> > Yes, I was aware that there would be conflicts with this PR. I > don't mind waiting. Thanks for your input. > > Cheers, > Fraser > -- Tomas Krizek -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] CI: exporting test runner output
On Thu, Jan 05, 2017 at 08:53:14AM +0100, Martin Babinsky wrote: > On 01/05/2017 08:06 AM, Fraser Tweedale wrote: > > Hi all, > > > > Although it has been discussed before and met with some skepticism, > > here is a POC that exporting test runner output to, e.g. a pastebin, > > does work: > > > > - experimental commit: https://github.com/freeipa/freeipa/pull/370 > > - example paste: https://paste.fedoraproject.org/520085/ > > (it is gzipped for reasons discussed in the PR) > > > > I think we should proceed with getting these artifacts out of Travis > > and stored somewhere (it doesn't have to be > > paste.fedoraproject.org). ``tail -n 5000`` of the log file has > > proven to be not enough to diagnose all failures. > > > Wow this is great, why have I not thought about it beforehand? > > We can reduce the log size if we truncate everything before ERRORS/FAILURES > output of pytest run (we leave the log as it is if the fail occurs before > this stage), that should shave off considerable amount of cruft from the > paste unless somebody sends a PR that breaks all out tests :D. > > > If we stick with paste.fedoraproject.org, we can send to a > > "project-specific" namespace e.g. > > https://paste.fedoraproject.org/~freeipa, so that we do not clutter > > up the main archive (I think). > > > > A few questions for discussion: > > > > 1. Stick with fpaste or not? If so, use "~freeipa" namespace? > >(Keep in mind that the size limitation that exists for fpaste, > >which requires compressing the artifact, may not be a problem > >elsewhere). > > > > 2. Export log always, or only if the build job failed? > > > I would also paste the output to "freeipa" or even better "freeipa-travis" > namespace and only send it if the job fails. > I might go with "freeipa-ci". > > 3. Should pasted logs expire? If so, what should TTL be? > > > IMHO yes, but TTL is hard to determine, since the author of the PR may not > be present to review the results immediately (because he is on PTO etc.). I > think we should set TTL to something like 1 week and as a fallback keep > tailing the CI results log. > 1 week sounds reasonable. We can change it later if we need to. > > 4. Should we continue to `tail -n 5000` the log as we currently do, > >or just rely on exported log? > > > > Thanks, > > Fraser > > > > Fraser, are you OK with waiting with this effort until we push > https://github.com/freeipa/freeipa/pull/361 ?. I will just do some more > adjustments there (like result log trimming) and it should be pushed ASAP. > Yes, I was aware that there would be conflicts with this PR. I don't mind waiting. Thanks for your input. Cheers, Fraser -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] CI: exporting test runner output
On 01/05/2017 08:06 AM, Fraser Tweedale wrote: Hi all, Although it has been discussed before and met with some skepticism, here is a POC that exporting test runner output to, e.g. a pastebin, does work: - experimental commit: https://github.com/freeipa/freeipa/pull/370 - example paste: https://paste.fedoraproject.org/520085/ (it is gzipped for reasons discussed in the PR) I think we should proceed with getting these artifacts out of Travis and stored somewhere (it doesn't have to be paste.fedoraproject.org). ``tail -n 5000`` of the log file has proven to be not enough to diagnose all failures. Wow this is great, why have I not thought about it beforehand? We can reduce the log size if we truncate everything before ERRORS/FAILURES output of pytest run (we leave the log as it is if the fail occurs before this stage), that should shave off considerable amount of cruft from the paste unless somebody sends a PR that breaks all out tests :D. If we stick with paste.fedoraproject.org, we can send to a "project-specific" namespace e.g. https://paste.fedoraproject.org/~freeipa, so that we do not clutter up the main archive (I think). A few questions for discussion: 1. Stick with fpaste or not? If so, use "~freeipa" namespace? (Keep in mind that the size limitation that exists for fpaste, which requires compressing the artifact, may not be a problem elsewhere). 2. Export log always, or only if the build job failed? I would also paste the output to "freeipa" or even better "freeipa-travis" namespace and only send it if the job fails. 3. Should pasted logs expire? If so, what should TTL be? IMHO yes, but TTL is hard to determine, since the author of the PR may not be present to review the results immediately (because he is on PTO etc.). I think we should set TTL to something like 1 week and as a fallback keep tailing the CI results log. 4. Should we continue to `tail -n 5000` the log as we currently do, or just rely on exported log? Thanks, Fraser Fraser, are you OK with waiting with this effort until we push https://github.com/freeipa/freeipa/pull/361 ?. I will just do some more adjustments there (like result log trimming) and it should be pushed ASAP. -- Martin^3 Babinsky -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
