Re: orphaning Taskotron-related packages

[Kamil Paral]

On Wed, Nov 25, 2020 at 2:29 PM Josef Skladanka  wrote:

> > Orphaning python-mongoquery and retiring everything else makes sense to
> > me.
> >
> > Tim
>
> +1
>

OK, I'll make it happen.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org

Re: orphaning Taskotron-related packages

[Kamil Paral]

On Mon, Nov 16, 2020 at 8:27 AM František Zatloukal 
wrote:

> The "correct" way to do this is to use orphaning procedure. This way,
> anybody interested will have enough time to take over the to-be removed
> packages.
>

"When Fedora maintainers do not want or are not able to maintain a package
any longer, they can orphan or retire
 the
package. When they think that the package is still useful for Fedora, they
should orphan it. Then other maintainers that are interested in maintaining
it, can take ownership of this package. In case the package is no longer
useful for Fedora, e.g. because it was renamed, upstream does not exist
anymore, then it should be retired."
https://fedoraproject.org/wiki/Orphaned_package_that_need_new_maintainers

As I said, I think it makes sense to go for orphaning in case of
mongoquery, and retiring for taskotron packages. But as you note:


>
> Also, all packages that are orphaned for 6+ weeks get retired. This will
> happen before F34, so there is no harm in doing this through orphans.
>

Yes, it should be the same anyway, if we do it right now. I just know that
sometimes the retirement process does not happen automatically and the
packages linger, and that's why I suggested retiring the upstream-dead
packages right away. I see no benefit in a two-step process, only potential
issues. But I don't care, really.

Anyway, what do you think about the actual proposal, and not the
technicalities?
Tim, Josef, do you have anything to say here?
Thanks.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org

Re: orphaning Taskotron-related packages

[Kamil Paral]

Note: The email subject should have said "retiring" instead of "orphaning".
There is little reason to orphan them, retiring is the right approach here.
Perhaps except for mongoquery, somebody else could be interested in
maintaining that, so that one should be orphaned instead.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org

orphaning Taskotron-related packages

[Kamil Paral]

Hello,
our Taskotron-related repositories (with exceptions like resultsdb) have
been marked as EOL and archived more than half a year ago. I believe it's
time to also tidy up our rpm packages in Fedora.

Looking at our group ownership [1] I have identified the following rpms
which could be retired:
* libtaskotron - nothing depends on it
* taskotron-trigger - nothing depends on it
* python-mongoquery - only taskotron-trigger depends on it
* execdb - nothing depends on it

I think we could have a light discussion regarding libtaskotron. The git
repo is marked as EOL and therefore the package should go away. However,
it's easy to flip the git repo back to a live state and it's not that
trivial to bring back an rpm package. So we should at least briefly
consider whether there is anything in that repo that could be useful in
some of our other current efforts. When looking at the code [2], I think
there are a few modules that could be potentially useful elsewhere, like
koji_utils/bodhi_utils/rpm_utils, perhaps a method or two from
os_utils/file_utils, and a perhaps a few directives. The problem is that
the code is not currently designed to work as a standalone library, but as
a runner - it expects a config file present, it expects certain directories
present, etc. The directives are exceptionally horrible if you want to use
them just by importing and not through the runner. It would require a
fairly big rewrite. Also we could drop ca. 70%-80% of the current codebase
(the runner stuff and unuseful modules).

If Tim said he would like to use some of that code in his CI efforts, or
Josef et al. said they would use some of that for oraculum or packager
dashboard etc, we might not retire libtaskotron, but restructure it into
something more useful. I'd be fine helping with that. But, even if we
wanted to do that, I'd honestly consider leaving libtaskotron as it is, and
rather creating a new package instead, something like 'libfedoraqa' or
similar. And only the few useful modules would be transferred there. It
feels to me much cleaner than reusing the libtaskotron name but gutting it
heavily.

What are your thoughts? Do you see any near-term use of some of the
existing code in libtaskotron? Or any other mentioned packages? Or can we
just drop them and bring them back if we ever need them? (If we don't have
an immediate use, there is a high probability we won't ever need them
again).

Thanks,
Kamil

[1] https://src.fedoraproject.org/group/qa-tools-sig
[2] https://pagure.io/taskotron/libtaskotron/blob/develop/f/libtaskotron
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org

Re: current maintainer of lmbench3

[Kamil Paral]

On Sun, Nov 1, 2020 at 9:53 PM Ryan Kosta  wrote:

> Dear QA tool development,
>
> During this week's kernel testday, I encountered a few interesting
> compilation errors with lmbench. Nothing to warrent major concern but
> their was a variable or two that had the wrong types, enough to offer a
> patch or two.
>
> When looking for lmbench I found the official intel github repository
> for it online. The readme was consistent with the readme in the source
> from the testcase stating the version as 2alpha8. yet the directory was
> named lmbench3.
>
> For this testcase did you clone lmbench directly from the intel github
> repository or is their an internal Fedora repository in which it came
> from?
>

Hey Ryan,
for reference, this is the test day and the test case we're talking about,
correct?
https://fedoraproject.org/wiki/Test_Day:2020-10-26_Kernel_5.9_Test_Week
https://fedoraproject.org/wiki/QA:Testcase_kernel_regression

We don't own the kernel-tests test suite, that's owned by the kernel team.
You can file issues or ask questions in the project repository:
https://pagure.io/kernel-tests
or using Kernel team's communication channels, i.e. their IRC or a mailing
list:
https://fedoraproject.org/wiki/Kernel

Hope that helps :)
Kamil
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org

Re: Poor first time experience for potential contributors interested in release validation

[Kamil Paral]

On Thu, Aug 6, 2020 at 5:16 AM Jonathan Watt  wrote:

> I originally reported this issue at
> https://bugzilla.redhat.com/show_bug.cgi?id=1862703 but was asked to
> re-report it here:
>
> I was using whatcanidoforfedora.org to see what areas I could contribute
> to. I selected "Validate Releases" and it took me to the QA wiki:
>
>   https://fedoraproject.org/wiki/QA/Join#release-validation
>
> That section says:
>
>   For information on getting and installing pre-releases, see .
>

Actually, it's in this section:
https://fedoraproject.org/wiki/QA/Join#Testing_Fedora_pre-releases
where it talks about Alpha and Beta releases. And since we have none for
F33 yet, the get-prerelease page is empty. At least I hope it will come
alive again once we have a Beta compose (Alphas are no longer produced).
The wording could be improved regarding this, yes.

What is applicable right now is Rawhide testing, release validation testing
(on Rawhide) and test days. We'll be branching F33 from Rawhide during the
next week or so, I believe, so then it will become F33 testing instead of
Rawhide testing.

As Ben said in Bugzilla, the best place to talk about this is the test list
[1] (this is not the test list), or you can create tickets for us in our
tracker [2].

Thanks for your interest in helping us!

[1]
https://lists.fedoraproject.org/archives/list/t...@lists.fedoraproject.org/
[2] https://pagure.io/fedora-qa
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org

Re: delete fedora-qa/qa-ansible project?

[Kamil Paral]

On Mon, May 4, 2020 at 12:10 PM Kamil Paral  wrote:

> Hello,
> Fedora Infra Ansible is now available on Pagure [1], which means people
> can finally submit and review pull requests. That means we no longer have a
> use case for our own fedora-qa/qa-ansible mirror [2], which we created just
> for the purpose of easily reviewing our PRs to Fedora Infra Ansible.
>
> Does anyone have any particular argument for keeping our qa-ansible
> project around? Unless somebody shouts, I'll delete that project.
>

Project deleted.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org

delete fedora-qa/qa-ansible project?

[Kamil Paral]

Hello,
Fedora Infra Ansible is now available on Pagure [1], which means people can
finally submit and review pull requests. That means we no longer have a use
case for our own fedora-qa/qa-ansible mirror [2], which we created just for
the purpose of easily reviewing our PRs to Fedora Infra Ansible.

Does anyone have any particular argument for keeping our qa-ansible project
around? Unless somebody shouts, I'll delete that project.

Thanks,
Kamil

[1] https://pagure.io/fedora-infra/ansible
[2] https://pagure.io/fedora-qa/qa-ansible/
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org

Re: seeing issues querying results by nvr/item in resultsdb

[Kamil Paral]

On Thu, Apr 9, 2020 at 12:46 AM Tim Flink  wrote:

> I'm working on something that looks for certain results in resultsdb
> and I'm getting some strange results. I'm hoping someone here can point
> out what I'm doing wrong.
>
> I'm hitting an issue where the data I'm getting back from resultsdb via
> the api isn't matching what I see in the frontend. The result is
> clearly in the db (I can get the data if I query the result ID) but
> when I search for the result using one of its keywords, I get an empty
> set back. What's even stranger is that this isn't consistent - some
> results submitted around the same time return the results I expect.
>
> I wrote up a quick script to show what I'm seeing:
>
> https://gist.github.com/tflink/1a1ed4263fbeb56b04442a948770965f
>
> It includes the output I'm seeing at the moment as I suspect it'll
> start working as soon as someone else looks at it :)
>
> I've tried this on the resultsdb production machine and I'm seeing the
> same results. I feel like I'm missing something obvious here so if you
> see it, I'd appreciate the help.
>
> Thanks,
>
> Tim
>

I'm not sure I completely understood your description. However, I did
notice that you're searching for
switchboard-plug-about.2.6.2-1.fc31
when you should be searching for
switchboard-plug-about-2.6.2-1.fc31
(a dot replaced by a dash)

That could solve the mystery, hopefully :)

Its results then show up just fine for me in the frontend:
https://taskotron.fedoraproject.org/resultsdb/results?item:like=switchboard-plug-about-2.6.2-1.fc31*
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org

Re: Git forge requirements discussion

[Kamil Paral]

On Wed, Feb 12, 2020 at 10:35 AM Kamil Paral  wrote:

> On Tue, Feb 11, 2020 at 10:42 PM Ben Cotton  wrote:
>
>> On Tue, Feb 11, 2020 at 4:37 PM Matthew Miller 
>> wrote:
>> >
>> > Can you put the voting/polls idea in the form of a user story?
>> >
>> I think #44 on the list I sent covers that:
>> As a Fedora team member, I can vote +1/abstain/-1 on issues so that I
>> can approve or deny proposals.
>>
>
> We would actually need several polls per ticket, so that we can vote on
> Beta/Final Blocker/FE from a single discussion (and not 4 separate
> tickets). I'll add that user story.
>
>

Done, replied to the devel thread. Feel free to add whatever you think is
missing.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org

Re: Git forge requirements discussion

[Kamil Paral]

On Tue, Feb 11, 2020 at 10:42 PM Ben Cotton  wrote:

> On Tue, Feb 11, 2020 at 4:37 PM Matthew Miller 
> wrote:
> >
> > Can you put the voting/polls idea in the form of a user story?
> >
> I think #44 on the list I sent covers that:
> As a Fedora team member, I can vote +1/abstain/-1 on issues so that I
> can approve or deny proposals.
>

We would actually need several polls per ticket, so that we can vote on
Beta/Final Blocker/FE from a single discussion (and not 4 separate
tickets). I'll add that user story.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org

Re: Git forge requirements discussion

[Kamil Paral]

On Wed, Jan 22, 2020 at 10:00 AM Adam Williamson 
wrote:

> Hey folks!
>
> There's a post on the Community Blog:
>
> https://communityblog.fedoraproject.org/git-forge-requirements
>
> and a thread on devel@:
>
>
> https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/YKA74LI7RU4PEQF3BOEL35MTVIRVHYUM/
>
> explaining that CPE is considering 'the future of a git forge solution
> which will be run by the CPE team on behalf of the Fedora Community',
> i.e. possibly ditching Pagure. The named contenders so far are Pagure,
> Gitlab and Github.
>
> The devel@ thread is going to be the usual tire fire, but I think we
> should write up something on this and send it to the Council (who are
> supposed to represent the interests of Fedora 'communities' in this
> process). Obviously we use Pagure for project hosting, issue tracking,
> and we're also working on the new async blockerbugs process, which
> relies on integration with Pagure.
>
> Can folks (especially those working on the blockerbugs stuff) read
> through the posts and think about what requirements we have here? I
> don't think it's going to be useful to have a big argument about the
> specific contenders here (see devel@ for that!) but it would be good to
> write up what QA actually needs in a forge. I'll try and gather the
> feedback and organize it into something to send to Council.
>

(Replying here and not devel list to consolidate our thoughts first)

Hmm,
I don't think we have some very special requirements. For the planned
blocker review async discussion, we need API to be able to add comments and
edit the ticket description, ideally also adjust ticket tags, milestones or
some other properties. We need a webhook support to get notified of changes
or access to a message bus. Nothing unusual I think. Or, of course, native
support for voting/polls (ideally multiple per a single ticket) would be
even better :-)

As for general requirements, or at least very nice to have's, I'd probably
repeat what was already mentioned in devel list: open source, with a good
integration to Fedora core services (the account system, messaging).
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org

upgrade your libtaskotron virtualenv to python3

[Kamil Paral]

Hi taskotron developers,

we've recently switched to using python3 by default in libtaskotron and all
taskotron tasks. So if you have your local virtualenv to be used with
libtaskotron, I suggest you re-create it with python3 [1] instead of
python2, because that's how libtaskotron is going to get used from now on.

Cheers,
Kamil

[1]
https://qa.fedoraproject.org/docs/libtaskotron/latest/devguide.html#installing-a-development-environment
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org

Re: Please update rpmlint on taskotron

[Kamil Paral]

On Mon, Jul 9, 2018 at 2:40 PM, Miro Hrončok  wrote:

> All Python 3 packages now get bytceode related errors with Python 3.7.
>
> See for example https://taskotron.fedoraprojec
> t.org/artifacts/all/7653ca56-8370-11e8-bbcf-525400fc9f92/
> tests.yml/python-libsass-0.14.5-1.fc29.log
>
> rpmlint was fixed to work with Python 3.7 bytecode in 1.10-14.
>
> Note that form the logs it seems you run fc28 rpmlint on rawhide builds.
> Is that OK? I've told spot that the 3.7 related fixes are not needed on
> F28, but are they?
>


Hi Miro,

we currently run all our tests on F28 machines. That was mainly done to
increase reliability, especially compared to running on Rawhide clients. So
yes, if you update rpmlint on F28, this problem will disappear.

We have the capacity to run a particular test on a machine with system
version matching the package disttag. For example if the test needed to
install the package, that would be necessary. But we always considered
rpmlint to be a generic test that can be independent from the system
environment. If it is not much of a problem, it would be great if you could
make sure the F28 rpmlint works well even for Rawhide packages. I think it
matches the developer use case, where they can easily run rpmlint from
their stable Fedora release workstation on their freshly built Rawhide
packages.

Thanks,
Kamil
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org/message/L5KODIQWFXJU525YJY2DODYKNH2YINZI/

Re: git clone ... Permission denied

[Kamil Paral]

On Tue, Apr 17, 2018 at 2:36 PM, Normand  wrote:

> Since today, I have a problem accessing the openqa git tree from pagure.
> I do not understand what I am doing wrong.
>
> any suggestion ?
>
> ===
> $git clone -v https://pagure.io/fedora-qa/os-autoinst-distri-fedora.git
> Cloning into 'os-autoinst-distri-fedora'...
> fatal: unable to access 'https://pagure.io/fedora-qa/o
> s-autoinst-distri-fedora.git/': Failed to connect to pagure.io port 443:
> Permission denied
> ===
>

Probably just a temporary pagure issue. Try again later. Btw, it works for
me at the moment.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: qa-ansible - repo for PRs against fedora infra ansible

[Kamil Paral]

On Wed, Feb 21, 2018 at 7:49 PM, Kevin Fenzi <ke...@scrye.com> wrote:

> On 02/21/2018 05:54 AM, Kamil Paral wrote:
> > Hello,
> >
> > I've created a new project and cloned all Fedora Infra ansible into it:
> > https://pagure.io/fedora-qa/qa-ansible
> >
> > The idea is to have a repo against which we can create a PR and review it
> > comfortably. Once reviewed, the patch of course needs to go to the real
> > infra ansible repo. But this should help for patches where you want other
> > people to have a look at it before pushing.
> >
> > If you want to use it, I recommend adding this as another remote in your
> > existing infra ansible checkout, that's probably the most comfortable
> > setup, at least for me.
>
> Note that we have plans to make the ansible repo avaiable in public for
> PR's, but have been holding off on this until Patricks git backend is
> finished. That should let us have it multiple places and stay in sync.
>

Great news, thanks. Looking forward to it :)
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

qa-ansible - repo for PRs against fedora infra ansible

[Kamil Paral]

Hello,

I've created a new project and cloned all Fedora Infra ansible into it:
https://pagure.io/fedora-qa/qa-ansible

The idea is to have a repo against which we can create a PR and review it
comfortably. Once reviewed, the patch of course needs to go to the real
infra ansible repo. But this should help for patches where you want other
people to have a look at it before pushing.

If you want to use it, I recommend adding this as another remote in your
existing infra ansible checkout, that's probably the most comfortable
setup, at least for me.

Cheers,
Kamil
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Making ResultsDB dumps public?

[Kamil Paral]

On Mon, Jan 15, 2018 at 3:25 PM, Tim Flink <tfl...@redhat.com> wrote:

> On Sun, 14 Jan 2018 14:19:54 -0800
> Kevin Fenzi <ke...@scrye.com> wrote:
>
> > On 01/11/2018 07:44 AM, Kamil Paral wrote:
> > > On Wed, Nov 1, 2017 at 7:48 PM, Kevin Fenzi <ke...@scrye.com
> > > <mailto:ke...@scrye.com>> wrote:
> > >
> > > On 10/24/2017 03:36 AM, Kamil Paral wrote:
> > > > On Sun, Oct 8, 2017 at 12:14 AM, Kevin Fenzi <ke...@scrye.com
> > > > <mailto:ke...@scrye.com>> wrote:
> > > >> They should be there starting tomorrow.
> > > >>
> > > >
> > > >
> > > > I still don't see any dumps in there. Is it still being
> > > > worked on?
> > >
> > > I seem to have made a typo in the script that does this. ;(
> > >
> > > Sent in a freeze break request with a fix
> > >
> > >
> > > I and Josef looked at it today, and it seems it contains some very
> > > old data - from Jan 2017 to April 2017. We failed to find the
> > > script that generates the dump in infra ansible, so we can't really
> > > debug what's wrong. Can you please look at it or give us some
> > > pointers?
> >
> > Odd.
> >
> > It's
> > https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/
> batcave/files/public-db-copy.sh?id=778dae3
> > which runs on batcave01 and runs:
> > https://infrastructure.fedoraproject.org/cgit/ansible.git/plain/scripts/
> public-db-copy?id=778dae3
> >
> > scp db-qa01.qa.fedoraproject.org:/resultsdb-$(date +%F).dump.xz
> > /srv/web/infra/db-dumps/resultsdb.dump.xz
> >
> > and on db-qa01.qa:/backups:
> >
> > -rw-r--r--. 1 postgres postgres 142589024 Apr 12  2016
> > resultsdb-2016-04-12.dump.xz
> > -rw-r--r--. 1 postgres postgres  50184088 Jun 20  2017
> > resultsdb-2017-06-20.dump.xz
> > -rw-r--r--. 1 postgres postgres  50367072 Jan 13 01:59
> > resultsdb-2018-01-13.dump.xz
> > -rw-r--r--. 1 postgres postgres  50367072 Jan 14 01:12
> > resultsdb-2018-01-14.dump.xz
> >
> > It claims to be the dump from today... so not sure whats going on.
>
> The database for production resultsdb is on db-qa02.qa instead of
> db-qa01.qa.
>
> From my read of those scripts, I suspect that is the cause of the
> public dumps being old.
>

Thanks for the fix, it's working properly now (Josef checked).
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: 2017-10-16 @ 14:00 UTC - Fedora QA Devel Meeting

[Kamil Paral]

Dne 16. 10. 2017 6:36 napsal uživatel "Tim Flink" :

# Fedora QA Devel Meeting
# Date: 2017-10-16
# Time: 14:00 UTC
(https://fedoraproject.org/wiki/Infrastructure/UTCHowto)
# Location: #fedora-meeting-1 on irc.freenode.net


https://fedoraproject.org/wiki/QA:Qadevel-20171016

If you have any additional topics, please reply to this thread or add
them in the wiki doc.

Tim


Proposed Agenda
===

Announcements and Information
-
  - Please list announcements or significant information items below so
the meeting goes faster

Tasking
---
  - Does anyone need tasks to do?

Potential Other Topics
--

  - deployment of ansiblize branches

Open Floor
--
  - TBD

___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org



I'm on PTO today, won't attend.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Need help ?

[Kamil Paral]

On Fri, Sep 1, 2017 at 10:57 PM, Ellen OSullivan  wrote:

> I'm a newbie, would like to help out with something.  I'm dev/qa
> automation at work but fairly new to Linux.  Willing to clean the floors,
> make the coffee :-)
>


Hi Ellen, if you familiarize yourself with testing proposed updates [1] and
providing feedback, and will then want to help out in development tasks
related to that area, we'd really appreciate if somebody could fix the
fedora-gooey-karma tool (packaged in Fedora, just broken atm) [2]. It broke
when Bodhi [3] changed its API to version 2 and nobody had time to fix
f-g-k. Just an idea.

Cheers and see you around,
Kamil


[1] https://fedoraproject.org/wiki/QA:Updates_Testing
[2] https://pagure.io/fedora-qa/fedora-gooey-karma
[3] https://bodhi.fedoraproject.org/
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Upcoming changes for rpmgrill

[Kamil Paral]

On Thu, Aug 31, 2017 at 1:05 AM, Róman Joost  wrote:

> Heya,
>
> just wanted to give everyone a heads up for some of the upcoming changes
> in a new rpmgrill release.
>
> The rpmgrill tool now exits with an exit code of 1 in case of test
> failures, 0 if all tests ran successfully. While this might sound like a
> trivial thing, I thought I better share it before the release, since
> this will obviously be picked up by CI systems the tool might be running
> in.
>
> PS: Changelog: https://github.com/default-to-
> open/rpmgrill#v0-31-unreleased
>
> Kind Regards,
> --
> Róman Joost
>


Thanks for the heads up. We'll need to adjust task-rpmgrill, currently it
doesn't take this into account.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Deprecating rpmgrill-fetch-build and rpmgrill-analyze-local

[Kamil Paral]

On Wed, Aug 23, 2017 at 9:00 AM, Róman Joost  wrote:

> Hi,
>
> The rpmgrill tool ships with two commands which I think can be
> deprecated:
>
> rpmgrill-fetch-build in favour of `koji download-task`
> rpmgrill-analyze-local in favour of rpmgrill-unpack; rpmgrill unpacked
>
> Does anybody use these binaries and has disagreements?
>

Hi, we don't seem to be using any of those two commands in
https://pagure.io/taskotron/task-rpmgrill

So it shouldn't be a problem for us.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Outdated clamav database in Taskotron

[Kamil Paral]

>
> > > So maybe the ClamAV definitions should be treated similarly? In
> > > a separate package which gets updated on a regular interval to pull in
> > > the latest data?
> > >
> >
> > That would be the best solution here, yes. Could someone please file an
> RFE
> > against clamav?
> Done:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=147



The clamav maintainer doesn't seem to have the time needed to provide a
regularly updated clamav-data package. So today I tried to update the
rpmgrill task with running freshclam before running rpmgrill, using
clamav.upjs.sk mirror. That mirror is very fast when I try it (as opposed
to the default mirror). However, when running it in our infra, I hit
"WARNING: Mirror 158.197.16.70 is not synchronized." errors with
clamav.upjs.sk mirror [1] and extremely long times with the default mirror
(my patience ran out after 5 minutes).

So, sorry, I give up. Unless we can convince someone to maintain a
regularly updated clamav-data package, or provide a reliable and fast
clamav mirror (that is willing to be hit very often from our infra), I'm
afraid we can't offer an updated clamav database for our tests.

[1] https://paste.fedoraproject.org/paste/hqewIGsDK8XFn9O6B0bPPQ/raw
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Outdated clamav database in Taskotron

[Kamil Paral]

On Wed, Aug 2, 2017 at 2:44 AM, Dan Callaghan <dcall...@redhat.com> wrote:

> Excerpts from Róman Joost's message of 2017-08-02 10:18 +10:00:
> > Dear Petr,
> >
> > On Tue, Aug 01, 2017 at 01:05:34PM +0200, Petr Pisar wrote:
> > > On Tue, Aug 01, 2017 at 11:59:41AM +0200, Kamil Paral wrote:
> > > > thanks for the report. In the task, we just install rpmgrill and run
> it. If
> > > > rpmgrill reports outdated clamav results, it seems that's something
> that
> > > > should be fixed in rpmgrill itself (it could depend on clamav-update
> and
> > > > update the virus db before running the virus check). Can you please
> report
> > > > a bug against rpmgrill and post the link here?
> > >
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1477130
> > Many thanks for the report. I haven't looked into the specifics, but I'm
> > not sure what we envision rpmgrill should be doing here. Should it run a
> > freshclam every invokation of rpmgrill? From what I see, that can take
> > quite a bit of time.
>

That was my initial idea, yes. But now that I tried it, I see two problems:
a) freshclam needs to run as root, therefore rpmgrill executed under a
standard user can't update it anyway. It could warn about it, and trigger
the update if running under root (taskotron case), but it's not a generic
solution which makes sure the results are up-to-date.
b) The database refresh is SLOW. I expected just a few seconds. But here's
my experience:
https://paste.fedoraproject.org/paste/3oGdfMpOP84~rhQm34AZvw/raw
Three minutes full of timing out. This can probably be much longer, if
servers are in even a worse condition some day.


> >
> > Maybe what could be done tho is a systemd timer installed with the
> > package which runs freshclam every now and than?
>
> This might not work well if rpmgrill is invoked as part of some system
> which creates a fresh VM from scratch and then deletes it shortly after
> (think single-use slaves with Jenkins OpenStack Cloud plugin, for
> example). The freshclam timer will likely never trigger before rpmgrill
> is run.
>

That's Taskotron case, yes. After looking more into this, I guess I
wouldn't object updating clamav database before running rpmgrill. The
problem I have with it is how slow it is. We run rpmgrill on every new koji
builds, that means very frequently. Each run is performed on a clean
machine, meaning each rpmgrill execution takes 3+ minutes longer just
because of clamav servers being horrible. I'd definitely add a timeout and
kill the process if it did not finish in 5 minutes or so, but even the
usual e.g. 3 minutes of extra execution time (mostly idle time) is
something I don't like.


>
> One possible thing which might help is if rpmgrill could warn or even
> fail, if it detects that it's being run with "outdated" ClamAV
> definitions. Not sure how old you want to consider "outdated", or if
> there is an easy way to check it... At worst I guess something like, if
> modtime of ClamAV definitions is more than 4 weeks in the past give an
> error? Do we know how frequently the definitions are updated?
>
> The other thing is that this idea of "download some data from the
> internet in order to make this package work" is not a good approach. It
> breaks in exactly the scenario I mentioned above, where a freshly
> installed copy of the package is not actually usable. The pciids and
> usbids database used to be like this too (shipping some old version of
> the data, plus a cron job to pull down updates from the internet) but
> nowadays we have the hwdata package which just gets updated with the
> latest definitions once per month. This is a much nicer solution because
> it means you can install a machine using only Fedora packages (or
> a freshly built disk image) and it already has the data it needs,
> without then going back to some random server on the internet.
>

Very much agreed.


>
> So maybe the ClamAV definitions should be treated similarly? In
> a separate package which gets updated on a regular interval to pull in
> the latest data?
>

That would be the best solution here, yes. Could someone please file an RFE
against clamav?
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

all projects moved to pagure

[Kamil Paral]

Hello,

we've moved all our projects from bitbucket (and github, in case of
testcloud) to pagure. You'll need to update the remote sources for (or
re-clone) all projects that you have checked out and were hosted outside of
pagure. You can find the new locations of our projects in this list:
https://pagure.io/group/taskotron
or this list:
https://pagure.io/group/fedora-qa

I also created a Taskotron umbrella project here:
https://pagure.io/taskotron

The issues and diffs for most of these projects are still hosted in
Phabricator, but we're going to transfer all of them to Pagure in the
following days. Also, we'll need to update all the links scattered
everywhere, that hasn't been done yet.

If there's anything unclear, please ask.

Thanks,
Kamil
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Taskotron: depcheck task replaced by rpmdeplint

[Kamil Paral]

If you are a Fedora packager, you might be interested to know that in
Taskotron  we replaced the depcheck
 task with
rpmdeplint 
task. So if there are any dependency issues with the new update you submit
to Bodhi , you’ll see that as
*dist.rpmdeplint* failure (in the *Automated Tests* tab). The failure logs
should look very similar to the depcheck ones (basically, the logs contain
the errors dnf would spit out if it tried to install that package), so
there should be no transitioning effort needed.

If you listen for depcheck results somehow, i.e. in FMN
, make sure to update your
rules to listen for *dist.rpmdeplint* instead. We have updated the default
filters in FMN, so if you haven’t changed them, you should receive
notifications for failures in rpmdeplint (and also upgradepath and
abicheck) for submitted updates owned by you.

The reason for this switch is that we wanted to get rid of custom
dependency checking (done directly on top of libsolv), and use an existing
tool for that instead. That saves us time, we don’t need to study all the
pitfalls of dependency resolution, and we benefit from someone else
maintaining and developing the tool (that doesn’t mean we won’t send
patches if needed). rpmdeplint  offered
exactly what we were looking for.

We will decommission depcheck task from Taskotron execution in the next few
days, if there are no issues. Rpmdeplint results are already being
published for all proposed updates.

If you have any questions, please ask in comments or reach us at #fedora-qa
 freenode irc channel or
qa-devel

(or test

or devel
)
mailing list.


(You can view this announcement also at: https://kparal.wordpress.com/2
017/06/22/taskotron-depcheck-task-replaced-by-rpmdeplint/ )
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: 2017-04-24 @ 15:00 UTC - Fedora QA Devel Meeting

[Kamil Paral]

On Mon, Apr 24, 2017 at 7:06 AM, Tim Flink  wrote:

> # Fedora QA Devel Meeting
> # Date: 2017-04-24
> # Time: 14:00 UTC
> (https://fedoraproject.org/wiki/Infrastructure/UTCHowto)
> # Location: #fedora-meeting-1 on irc.freenode.net


I won't be around today, sorry.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Re-Scheduling Jobs for Taskotron as a User

[Kamil Paral]

On Tue, Apr 18, 2017 at 2:35 PM, Tim Flink  wrote:

> One of the things that seems like it is and will be a pain point for
> folks writing package-specific tasks is how to work through the times
> when there was an issue in the task and things didn't run well. At the
> moment, the only solution we have is to re-build the affected package
> or to pester an admin to re-run the trigger - neither of which is an
> ideal answer.
>
> I was thinking about how to improve this in the near future and am
> wondering about adding a "reschedule" button to execdb jobs:
>

Execdb web is not searchable, people will never find their job. It makes a
lot of sense to have the button in execdb, because that's where we want to
show people crashed jobs (those will not appear neither in resultsdb nor in
Bodhi). So thumbs up to that idea. But the search seems to be a
prerequisite to this. Or we need to make sure crashed jobs are sent out
with direct links to execdb to the relevant maintainers, if that turns out
to be easier.


>   1. authenticated user clicks on "reschedule" button
>

Authentication itself sounds like a non-trivial issue. I wonder if we can
avoid it at least temporarily. Maybe allow anyone to click the button, but
limit the number of times it can be pressed (e.g. max 5 times)? Also
introduce a timeout after the press (max once per hour since the last
re-run), and don't allow to re-run old jobs (e.g. older than 24 hours)?

Maybe authentication will be easier after all :) Maybe we don't even need
to check package permissions in the beginning, just making sure the person
has a valid FAS (cla+1) account.



>   2. execdb makes an api call to the buildmaster to find the parameters
>  which were used for that task
>   3. using the data from 2), execdb starts a new job for just that
>  item and item type
>
> I'm not thrilled at the idea of code duplication here between trigger
> and execdb but compared to figuring out a web interface for trigger,
> this seems like a more tractable solution for the short/medium term.
>
> Thoughts?
>
> Tim
>
> ___
> qa-devel mailing list -- qa-devel@lists.fedoraproject.org
> To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
>
>
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: per-arch tasks strategy in Taskotron

[Kamil Paral]

> Getting ppc64 or ppc64le machines should be only matter of filling a
> ticket with fedora infra, these arches are part of the fedora cloud.
> aarch64 is going to added soon.

Thanks for info. The problem is not that much in having hardware (I think), but 
in preparing tailored disk images for Taskotron. It's currently quite 
error-prone and costs us a lot of time to maintain. That's why we don't want to 
expand beyond x86_64 for the moment, we have more pressing matters. But I 
imagine there could be some collaboration with alternate arches SIGs in the 
future, them providing machines and disk images, and us running the tasks on 
it. Probably not in the nearest future, though.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: per-arch tasks strategy in Taskotron

[Kamil Paral]

> Hello,
> 
> I want to consult our current strategy regarding per-arch tasks. We currently
> have no such tasks, all of them are generic (they can run on any host system
> and test any single or multiple item arches). I want to convert
> task-rpmdeplint [1] to arch-specific, to avoid a race condition [2]. Also, I
> believe per-arch tasks will be necessary for supporting dist-git tasks
> (we'll need to assume that they are arch-specific by default). I found
> several issues in our current implementation.
> 
> First, I submitted a diff for making --arch cmdline argument a single value
> instead of a multiple value list:
> https://phab.qa.fedoraproject.org/D1171
> That has the benefit of ${arch} being a simple string, and that makes it
> easier in formula to operate with it - for example pass it to a bash script
> that needs to do something. It also makes possible to easily extend
> architecture list that is passed to e.g. koji directive. That is important
> for tasks dealing with multilib (rpmdeplint).
> 
> My view is that we should simplify things in order to keep our sanity and
> reasonable implementation, at the expense of some performance optimization
> (running x86_64 and i386 tasks at the same time, because hardware allows
> that). So --arch will be a single value, and that will determine what
> architecture should the task check. So if an arch-specific task wants to run
> for armhfp, i386 and x86_64, trigger will schedule 3 different jobs on
> particular hosts, and they will run independently. The task will be able to
> determine whether it's arch-specific or generic in its formula (or
> scheduling database in the future), therefore deciding whether it's executed
> once or multiple times (with different --arch arguments). It will also be
> able to say which arches it supports or doesn't support (so that we don't
> run it on e.g. armhfp even if we have armhfp machines, if the task doesn't
> support it). If we need performance optimizations in the future, we can do
> that by allowing the task to not be tied to a particular arch host (so that
> we can execute armhfp rpmdeplint on x86_64 minion). But those runs will be
> always independent (one job for x86_64, one job for armhfp, one job for
> i386).
> 
> Does that sound reasonable? If agreed, I really want to push the
> aforementioned change before dist-git tasks take off, since then we'd be
> breaking our formula API (and workflow expectations) for a lot of people.

I got a silent approval from Josef and Martin who are too scared to respond 
publicly. So I pushed the patch.

> Second, we don't have any non-x86_64 minions yet. So even if we support the
> above, we are currently able to execute just x86_64 arch-specific tasks. I
> assume we'll get some arm builder with armhfp buildslaves in the future, but
> that will take time. i386 is a different story. We are able to create the
> minions (we just don't do it yet), and we even have a single i386 builder
> (staying idle all the time). However, i386 switched to an alternate arch
> since F26, so I'm not really sure if we want to invest a lot of time into it
> (we should obviously focus primary arches first). Also, creating minions for
> i386 means data storage requirements doubled, and number of issues to debug
> doubled. I wonder whether it's a worthy investment of time at this very
> moment. Our architecture should obviously be prepared to handle
> arch-dependent tasks (especially from formula/API perspective, so that we
> don't force all users to amend their code in the future), but I'm wondering
> whether we should actually *have* multiple arches support right now, or
> rather keep it simple and tell everyone that only x86_64 is supported during
> the pilot.

Again, the general feel from behind the scenes seems to be to not support i386 
minions right now. So I'll go ahead with claiming to everyone that we only 
support x86_64 tasks for the moment (even though the framework should be more 
or less prepared to handle multiple arches).

The generic tasks (like rpmlint, rpmgrill, abicheck) will continue to download, 
test and submit results even to for other architectures (per "supported_arches" 
in taskotron.yaml, which is currently set to ['x86_64', 'i386', 'armhfp']), 
because those tasks are not minion-dependent.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

per-arch tasks strategy in Taskotron

[Kamil Paral]

Hello,

I want to consult our current strategy regarding per-arch tasks. We currently 
have no such tasks, all of them are generic (they can run on any host system 
and test any single or multiple item arches). I want to convert task-rpmdeplint 
[1] to arch-specific, to avoid a race condition [2]. Also, I believe per-arch 
tasks will be necessary for supporting dist-git tasks (we'll need to assume 
that they are arch-specific by default). I found several issues in our current 
implementation.

First, I submitted a diff for making --arch cmdline argument a single value 
instead of a multiple value list:
https://phab.qa.fedoraproject.org/D1171
That has the benefit of ${arch} being a simple string, and that makes it easier 
in formula to operate with it - for example pass it to a bash script that needs 
to do something. It also makes possible to easily extend architecture list that 
is passed to e.g. koji directive. That is important for tasks dealing with 
multilib (rpmdeplint).

My view is that we should simplify things in order to keep our sanity and 
reasonable implementation, at the expense of some performance optimization 
(running x86_64 and i386 tasks at the same time, because hardware allows that). 
So --arch will be a single value, and that will determine what architecture 
should the task check. So if an arch-specific task wants to run for armhfp, 
i386 and x86_64, trigger will schedule 3 different jobs on particular hosts, 
and they will run independently. The task will be able to determine whether 
it's arch-specific or generic in its formula (or scheduling database in the 
future), therefore deciding whether it's executed once or multiple times (with 
different --arch arguments). It will also be able to say which arches it 
supports or doesn't support (so that we don't run it on e.g. armhfp even if we 
have armhfp machines, if the task doesn't support it). If we need performance 
optimizations in the future, we can do that by allowing the task to not be tied 
to a particular arch host (so that we can execute armhfp rpmdeplint on x86_64 
minion). But those runs will be always independent (one job for x86_64, one job 
for armhfp, one job for i386).

Does that sound reasonable? If agreed, I really want to push the aforementioned 
change before dist-git tasks take off, since then we'd be breaking our formula 
API (and workflow expectations) for a lot of people.


Second, we don't have any non-x86_64 minions yet. So even if we support the 
above, we are currently able to execute just x86_64 arch-specific tasks. I 
assume we'll get some arm builder with armhfp buildslaves in the future, but 
that will take time. i386 is a different story. We are able to create the 
minions (we just don't do it yet), and we even have a single i386 builder 
(staying idle all the time). However, i386 switched to an alternate arch since 
F26, so I'm not really sure if we want to invest a lot of time into it (we 
should obviously focus primary arches first). Also, creating minions for i386 
means data storage requirements doubled, and number of issues to debug doubled. 
I wonder whether it's a worthy investment of time at this very moment. Our 
architecture should obviously be prepared to handle arch-dependent tasks 
(especially from formula/API perspective, so that we don't force all users to 
amend their code in the future), but I'm wondering whether we should actually 
*have* multiple arches support right now, or rather keep it simple and tell 
everyone that only x86_64 is supported during the pilot.

WDYT?

If we decide to go with i386 minions right now, we'll need support in trigger. 
It will either read the formula for a new field that we add, or it will have 
hardcoded value that it should run task-rpmdeplint (and any dist-git task) with 
multiple architectures. So it will schedule two jobs for each such task, one 
with --arch x86_64, and one with --arch i386. Can somebody with trigger 
experience estimate whether this is a difficult change to do or not? Also, do 
you see any further changes required in other tools?



[1] I mostly ignore task-depcheck, because it's going away. But the same issue 
would apply also there.
[2] https://phab.qa.fedoraproject.org/T894
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: New automated test coverage: openQA tests of critical path updates

[Kamil Paral]

> On Thu, 2017-03-02 at 04:31 -0500, Kamil Paral wrote:
> > > The job can - and already does - log the exact packages it actually
> > > got, but I don't think there's an easy way for it to take the
> > > 'last_modified' date for the update at the time it does the download.
> > 
> > I don't know how you download the rpms, but a single python call can
> > do that (http get and parse the json). Again, to prevent race
> > conditions, it would be good to do the call before and after
> > downloading the rpms and compare the timestamp. These race conditions
> > occur surprisingly often once you start executing hundreds/thousands
> > tasks a day.
> > 
> > But if this is easier done in the scheduler, I think that's totally fine.
> 
> During test execution, we can only really type stuff into the console.
> We try to keep the amount of typing-into-consoles we do to a minimum,
> too, as the more there is, the more likely it is openQA will choke on a
> keypress and fail. (Though these tests already involve quite a lot of
> typing, can't avoid it.) The test just uses the Bodhi CLI client to
> download the packages.

Sure, I'm not saying this needs to happen during the actual test. That seems 
silly, if we can do the same thing in the scheduler (initial timestamp) and in 
the reporter (end timestamp).

> 
> I mean, it's not impossible, we *could* just type in a curl / Python
> one-liner (or use something like httpie to hit the API to get it). I'm
> just questioning whether it's worth the effort.

It's not necessary now, in the "development" phase. But once we want gate on 
it, I think it's very important (if we want our gating mechanics to be 
reliable).
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: New automated test coverage: openQA tests of critical path updates

[Kamil Paral]

> > There's one important thing we need to do first, though. Bodhi ID
> > doesn't identify the thing tested uniquely, because Bodhi updates are
> > mutable (and the ID is kept). So Bodhi (or any gating tools) can't
> > rely on just retrieving the latest result for a particular Bodhi ID
> > and trust that result. It might be old and no longer reflect the
> > current state. We need to extend bodhi_update results with
> > "timestamp" key in extra data, that will report the "last_modified"
> > time of the Bodhi update tested. And Bodhi (or any other tool) must
> > not only query for item=$bodhi_id=bodhi_update, but also for
> > =$timestamp. Only with this we can be sure we've really
> > tested particular Bodhi update.
> 
> I'm not so sure it's really necessary, and doing it is actually tricky
> for openQA. Only the openQA job itself knows what packages it actually
> tested, and it doesn't have an easy way to get the associated
> timestamp. The scheduler could easily get the timestamp at the time the
> job was created, or at the time the job completed, but that will never
> be 100% reliable, because the job actually goes and does the download
> somewhere in between those two times.

This problem is not exclusive to openqa, it affects all tasks that test bodhi 
updates and download the included rpms (there's always a race condition 
window). For openqa, I see two options here:

a) record the timestamp in the scheduler when the job is created and use it. 
Either it will be correct, or if the race condition happens, it will publish a 
result based on testing newer packages with an older timestamp. That's slightly 
incorrect, but not really a problem. Because the update edit event scheduled 
another openqa run, and that will publish an up-to-date result. So there's no 
harm done.

b) record the timestamp in the scheduler when the job is created, and when the 
job is finished. If they don't match, ignore the result, don't publish it. The 
update edit event scheduled another openqa run anyway. Again, no harm done and 
we didn't populate resultsdb with an incorrect result. (This is similar to what 
we do in certain taskotron tasks - if we detect that a bodhi update state 
doesn't match at the time when we publish results, we print it into the logs 
and skip them.)

> 
> The job can - and already does - log the exact packages it actually
> got, but I don't think there's an easy way for it to take the
> 'last_modified' date for the update at the time it does the download.

I don't know how you download the rpms, but a single python call can do that 
(http get and parse the json). Again, to prevent race conditions, it would be 
good to do the call before and after downloading the rpms and compare the 
timestamp. These race conditions occur surprisingly often once you start 
executing hundreds/thousands tasks a day.

But if this is easier done in the scheduler, I think that's totally fine.

> 
> OTOH, I don't think it's really too bad just to show the 'most recent'
> results. That should usually only be out of date for a few minutes
> after an update is edited. It might be possible to do a 'tests
> running...' spinner when there are jobs scheduled or running for the
> update in question, even.

You're assuming here that the new task will finish successfully. It will often 
not. From my experience, network is the bane of automated testing. Bodhi will 
time out, koji will time out, they will return http 5xx errors, etc. Taskotron 
tasks are plagued with it (at least dozens such failures a day). That's why I 
try to detect the race condition and either not record it at all, or record it 
with the older timestamp, which is safe - you don't mislead people/tools when 
looking at the results. The worst thing to happen here is that a result is 
missing for a long time. And people will then complain (and we start 
investigating) or they'll use the "request re-testing" button, which we'll have 
to provide sooner or later (because all systems are imperfect).

Of course I'm not saying we need to have this *now*. But I think it's necessary 
for gating updates.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: New automated test coverage: openQA tests of critical path updates

[Kamil Paral]

> Hi folks!
> 
> I am currently rolling out some changes to the Fedora openQA deployment
> which enable a new testing workflow. From now on, a subset of openQA
> tests should be run automatically on every critpath update, both on
> initial submission and on any edit of the update.
> 
> For the next little while, at least, this won't be incredibly visible.
> openQA sends out fedmsgs for all tests, so you can sign up for FMN
> notifications to learn about these results. They'll also be
> discoverable from the openQA web UI - https://openqa.fedoraproject.org
> . The results are also being forwarded to ResultsDB, so they'll be
> visible via ResultsDB API queries and the ResultsDB web UI. But for
> now, that's it...I think.
> 
> Our intent is to set up the necessary bits so that these results will
> show up in the Bodhi web UI alongside the results for relevant
> Taskotron tests. There's an outside possibility that Bodhi is actually
> already set up to find these results in ResultsDB, in which case
> they'll just suddenly start showing up in Bodhi - we should know about
> that soon enough. :) But most likely Bodhi will need a bit of a tweak
> to find them. 

Let me add a bit of a technical background here. Bodhi web UI now queries 
ResultsDB for all available testcases, and then asks for 
item=$item=koji_build for all these testcases. So the new results won't be 
visible (unless you start reporting them per build).

Our depcheck and upgradepath tasks report both type=koji_build (for each SRPM) 
and type=bodhi_update (for each bodhi update). Both these tools internally 
process RPMs or builds, and then query Bodhi at the end and collate the results 
to bodhi_update results. We want to get rid of that collating, because it has 
numerous issues:
1. It's slow, because Bodhi is very slow to respond
2. It often causes the task to fail, because Bodhi often returns 500 errors
3. It's prone to race conditions. It happens often that a Bodhi update is 
edited between the task start and end, changing included builds.
4. It's unnecessary, because Bodhi knows all this information, and can collate 
the data itself, without any network issues or race conditions.

So while we still publish type=bodhi_update for compatibility reasons (I'm not 
sure whether some part of bodhi backend still might use this data), but want to 
get rid of it.

So my first thought was to recommend you to also publish just type=koji_build 
results and finish this transition. But then I realized that's wrong. OpenQA 
operates completely different than the aforementioned tasks do. We operate on 
builds, and can distinguish which build is causing issues. Collating them into 
bodhi_update results is just a convenience measure for the consumer. But you 
operate on the whole update as a set. You can't distinguish which build of the 
update caused the issues, you just know that some of them did. So the smallest 
unit for you is bodhi update, and you should report results as such.

The way forward is, I believe, to extend Bodhi to query both type=koji_build 
for all included builds and collate the results (if needed), and also query 
type=bodhi_update and shows those results as well. Because different tasks 
operate on different type of data, which influences how they publish the 
results. And both use cases are valid.

There's one important thing we need to do first, though. Bodhi ID doesn't 
identify the thing tested uniquely, because Bodhi updates are mutable (and the 
ID is kept). So Bodhi (or any gating tools) can't rely on just retrieving the 
latest result for a particular Bodhi ID and trust that result. It might be old 
and no longer reflect the current state. We need to extend bodhi_update results 
with "timestamp" key in extra data, that will report the "last_modified" time 
of the Bodhi update tested. And Bodhi (or any other tool) must not only query 
for item=$bodhi_id=bodhi_update, but also for =$timestamp. Only 
with this we can be sure we've really tested particular Bodhi update.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Trigger changes - call for comments

[Kamil Paral]

> Hey, gang!

> As with the ExecDB, I took some time to try and formalize what I think is to
> be done with Trigger in the near-ish future.
> Since it came to my attention, that the internal G-Docs can not be accessed
> outside of RH, this time, it is shared from my personal account - hopefully
> more people will be enabled to read and comment on the document.
> Without further ado -
> https://docs.google.com/document/d/1BEWdgm0jO4p5DsTO4vntwUumgLZGtU5lBaBX5av7MGQ/edit?usp=sharing

Again, looks reasonable to me. I added one comment about caching data for 
rescheduled jobs. 

Btw, the "current state" description is really good and it'd be great to have 
it just copy pasted into some trigger doc. Because AFAIK we don't have any 
trigger docs at the moment. 
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Taskotron CI in Taskotron

[Kamil Paral]

> So, the repo now has working PoC
> https://pagure.io/taskotron/task-taskotron-ci
> Readme contains example on how to run the task.
> Works on my setup, and I'd be glad if somebody else tried.

I tried, and it didn't eat my laptop. Also it worked. Good job. 

/me throws cookies at jskladan 
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Taskotron CI in Taskotron

[Kamil Paral]

> On Tue, 2017-02-14 at 14:46 -0500, Kamil Paral wrote:
> > > The only problem with this kind of testing is, that we still don't really
> > > have a good way to test trigger, as it is tied to external events. My
> > > idea
> > > here was that I could add something like wiki edit consumer, and trigger
> > > tasks off of that, making that one "triggering" edit from inside the
> > > testsuite.
> > 
> > In my experience, some fedmsg notifications are quite delayed from
> > time to time, easily by several hours. But I don't know if the delay
> > is in actual fedmsg bus, or just in FMN (which I use to get notified
> > on IRC). I suspect rather FMN. But if that turned out to be a fedmsg
> > delay, that might make this approach impractical (or at least less
> > reliable). So something to consider and possibly investigate.
> 
> I believe it's FMN. I've noticed the same delay with FMN notifications,
> but I've never noticed any delay with the actual fedmsg notifications
> used by all the stuff we've built around release validation (compose
> complete fedmsgs, openQA test complete fedmsgs etc.)
> 
> > I think it would be better to fake the input data. Either by having
> > our own fedmsg producer and/or bus (I don't know precisely how fedmsg
> > works and how hard is to set up something like that),
> 
> Are you aware of fedmsg-dg-replay? It's a fairly easy way to 'replay'
> fedmsgs for testing. All you need (IIRC) is the fedmsg-relay service
> running on the same system, and you can run
> 'fedmsg-dg-replay --msg-id ' and the message will be
> replayed on the local bus (if the message is .prod. or .stg. , this
> will be changed to .dev. , and the message will not be signed).
> 
> I don't remember exactly how the taskotron trigger code works, but it
> shouldn't be too hard to configure a trigger to accept unsigned
> messages with .dev. topics, for the purpose of testing with -replay. In
> the fedmsg consumers I've written I have a pattern of providing three
> consumers, one which listens for .prod. messages, one for .stg.
> messages, and one which listens for for .dev. messages and is intended
> for testing with fedmsg-dg-replay.

Great info, thanks.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Taskotron CI in Taskotron

[Kamil Paral]

> The only problem with this kind of testing is, that we still don't really
> have a good way to test trigger, as it is tied to external events. My idea
> here was that I could add something like wiki edit consumer, and trigger
> tasks off of that, making that one "triggering" edit from inside the
> testsuite.

In my experience, some fedmsg notifications are quite delayed from time to 
time, easily by several hours. But I don't know if the delay is in actual 
fedmsg bus, or just in FMN (which I use to get notified on IRC). I suspect 
rather FMN. But if that turned out to be a fedmsg delay, that might make this 
approach impractical (or at least less reliable). So something to consider and 
possibly investigate. 

I think it would be better to fake the input data. Either by having our own 
fedmsg producer and/or bus (I don't know precisely how fedmsg works and how 
hard is to set up something like that), or by making some code adjustments in 
trigger that will allow us to bypass the fedmsg reception and replace it with 
our own fedmsg, but cover as much surrounding code as possible. 
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Libtaskotron - allow non-cli data input

[Kamil Paral]

> On Wed, Feb 8, 2017 at 2:26 PM, Kamil Paral < kpa...@redhat.com > wrote:

> > > This is what I meant - keeping item as is, but being able to pass another
> > > structure to the formula, which can then be used from it. I'd still like
> > > to
> > > keep the item to a single string, so it can be queried easily in the
> > > resultsdb. The item should still represent what was tested. It's just
> > > that
> > > I
> > > want to be able to pass arbitrary data to the formulae, without the need
> > > for
> > > ugly hacks like we have seen with the git commits lately.
> > 
> 

> > So, the question is now how much we want the `item` to uniquely identify
> > the
> > item under test. Currently we mostly do (rpmlint, rpmgrill) and sometimes
> > don't (depcheck, because item is NVR, but the full ID is NEVRA, and we
> > store
> > arch in the results extradata section).
> 

> I still kind of believe that the `item` should be chosen with great respect
> to what actually is the item under test, but it also really depends on what
> you want to do with it later on. Note that the `item` is actually a
> convention (yay, more water to adamw's "if we only had some awesome new
> project" mill), and is not enforced in any way. I believe that there should
> be firm rules (once again - conventions) on what the item is for each "well
> known" item type, so you can kind-of assume that if you query for
> `item=foo=koji_build` you are getting the results related to that
> build.
> As we were discussing privately with the item types (I'm not going to go into
> much detail here, but for the rest of you guys - I'm contemplating making
> the types more general, and using more of the 'metadata' to store additional
> spefics - like replacing `type=koji_build` with `type=build, source=koji`,
> or `type=build, source=brew` - on the high level, you know that a
> package/build was tested, and you don't really care where it came from, but
> you sometimes might care, and so there is the additional metadata stored. We
> could even have more types stored for one results, or I don't know... It's
> complicated), the idea behind item is that it should be a reasonable value,
> that carries the "what was tested" information, and you will use the other
> "extra-data" fields to provide more details (like we kind-of want to do with
> arch, but we don't really..). The reason for it to be 'reasonable value" and
> not "superset of all values that we have" is to make the general querying a
> bit more straightforward.

> > If we have structured input data, what happens to `item` for
> > check_modulemd?
> > Currently it is "namespace/module#commithash". Will it stay the same, and
> > they'll just avoid parsing it because we'll also provide ${data.namespace},
> > ${data.module} and ${data.hash}? Or will the `item` be perhaps just
> > "module"
> > (and the rest will be stored as extradata)? What happens when we have a
> > generic git_commit type, and the source can be an arbitrary service? Will
> > have some convention to use item as "giturl#commithash"?
> 

> Once again - whatever makes sense as the item. For me that would be the
> Repo/SHA combo, with server, repo, branch, and commit in extradata.
> And it comes to "storing as much relevant metadata as possible" once again.
> The thing is, that as long as stuff is predictable, it almost does not
> matter what it is, and it once again points out how good of an idea is the
> conventions stuff. I believe that we are now storing much less metadata in
> resultsdb than we should, and it is caused mostly by the fact that
> - we did not really need to use the results much so far
> - it is pretty hard to pass data into libtaskotron, and querying all the
> services all the time, to get the metadata, is/was deemed a bad idea - why
> do it ourselves, if the consumer can get it himself. They know that it is
> koji_build, so they can query koji.

> There is a fine balance to be struck, IMO, so we don't end up storing "all
> the data" in resultsdb. But I believe that the stuff relevant for the result
> consumption should be there.

> > Because the ideal way would be to store the whole item+data structure as
> > item
> > in resultsdb. But that's hard to query for humans, so we want a simple
> > string as an identifier.
> 

> This, for me, is once again about being predictable. As I said above, I still
> think that `item` should be a reasonable identifier, but not necessary a
> superset of all the info. That is what the extra data is for. Talking
> about...

> > But s

Re: making test suites work the same way

[Kamil Paral]

> Well, after more discussions with kparal, we are still unsure about the
> "right" way to tackle this.
> Our current call would be:
> 1) sync requirements.txt versions with fedora (mostly done)
> 2) allow --system-site-packages in the test_env
> 3) do `pip install -r requirements.txt` (with possible flags to enforce
> versions) to the makefile virtualenv creation step
> 4) add info to readme, that testing needs installation of packages from pypi,
> and that some of them need compilation
> 4-1) put together a list of packages that need to be installed (the
> python-foobar kind, not -devel + gcc) to the system, in order to "skip" the
> stuff that needs to be compiled

> Sounds reasonable, Kamil? Others?

I went back and forth on this. I thought it would be a really simple change, 
and as usual, it seems more pain than gain. So, I went forward with this: 
1. add tox.ini to projects to allow simple test suite execution with `pytest` 
(non-controversial) 
2. configure tox.ini to print out test coverage (non-controversial) 
3. remove --system-site-packages from all places (readme, makefile) for those 
projects, that can be *fully* installed from pypi *without any compilation* 
(hopefully non-controversial). 
4. keep (or add) --system-site-packages to readme/makefile for the remaining 
projects, and add readme info how to deal with pypi compilation or local rpm 
installation 

What Josef mentioned is that he wouldn't try to replicate a perfect environment 
directly on dev machine, because that's a lot of work. Instead, use the current 
non-perfect environment on dev machines (which should be fine most of the time 
anyway) and have a separate CI service (hopefully in the future) with more 
strict environment configuration. I guess that's the most practical solution. 

We might even want to reopen the question how to version deps in 
requirements.txt vs spec file, but I'd keep that for a separate thread, if 
needed. 

My current patches for resultsdb projects are these: 
https://phab.qa.fedoraproject.org/D1114 
https://phab.qa.fedoraproject.org/D1116 
https://phab.qa.fedoraproject.org/D1117 
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

welcome Fedora Quality Planet

[Kamil Paral]

Hello,

I've set up a Quality sub-planet for Fedora Planet, available here:
http://fedoraplanet.org/quality/

The purpose is to have a single place where people can read and subscribe to 
quality-related news and information. Here's an introductory blogpost with more 
details and instructions how to join:
https://kparal.wordpress.com/2017/01/31/welcome-fedora-quality-planet/

Anyone is welcome to join and help us in publishing interesting and useful 
information!

Thanks a lot,
Kamil
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Proposal: Migrating More Git Projects to Pagure

[Kamil Paral]

> This came up in the qadevel meeting today and I wanted to put a bit
> more detail out.
> 
> Bitbucket was never intended to be the long-term home for our git
> projects - I think we're about the only folks in Fedora using it and
> it's not free software. As fedorahosted is closed down, we need to find
> a new home for blockerbugs but I figure that now is as good of a time
> as any to get all of our git projects in the same place.
> 
> I'm proposing the following moves:
> 
> * Move all Taskotron projects to pagure.io using the taskotron group:
>- pagure.io/taskotron/libtaskotron
>- pagure.io/taskotron/resultsdb
>- etc.
> 
> * Move blockerbugs under the existing fedora-qa namespace in pagure:
>- pagure.io/fedora-qa/blockerbugs
> 
> I'm not sure if there are any plans for the openqa stuff that currently
> lives on bitbucket but it'd be nice to see that moved as well.
> 
> Any objections, comments, concerns?

 (thumbs up)
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Enabling running Taskotron tasks on Koji scratch builds

[Kamil Paral]

> Couldn't we use the task ID as the primary identifier but use the srpm
> for readability sake since there is no "build" NVR for scratch builds?

Systems like new hotness will need to query according to the task ID, though 
(to get reliable results). So we're talking about hacking just resultsdb 
*frontend* here, e.g. by having "item: task_id" and "item_display: nvr" in the 
results yaml. I don't like it much. Searching in the frontend would have to 
search both in item and item_display.

Or we could use our existing "note" field to put NEVR inside, and it would be 
easily visible in the frontend without any ugly hacks. People would have to 
know that if they want to search by NEVR for scratch builds, they need to put 
the NEVR in the note search field (we'd have to add it).

I assume you proposed using task id as primary identifier just to scratch 
builds (but not standard builds). If you also mean it for standard builds, then 
scheduling tasks locally starts to be quite inconvenient (for each build you 
want to run your task on, you need to find its task id first; it's hard to 
re-run something from shell history). We would also be changing something 
that's a de-facto standard in our infra (using NEVR as a unique identifier 
across projects).


> Either that or make the primary ID a tuple of (srpm name, koji task
> start time) either stored that way or rendered as a string e.g
> foo-1.2-3.fc99.src.rpm built at 201701092100 (MMDDHHMM) would become
> foo-1.2-3.fc99-201701092100.

The same problem with inconvenient local execution. The command line starts to 
be hard to read (in tutorials, etc).

Also, the srpm name doesn't have to be anything reasonable, Koji will happily 
accept anything (it doesn't care, for scratch builds). So we can easily receive 
"1.srpm" or "my-new-patch.srpm" from the fedmsg (we actually tried this [1]). 
Deriving any useful info from this input is probably a mistake. We would have 
to download the srpm and look into the included spec file to reliably decide 
which package this is related to.

[1] 
https://apps.fedoraproject.org/datagrepper/id?id=2017-2fe001d2-32d4-434a-b3a9-29ab31eebbb0_raw=true=extra-large

> 
> > During a discussion with Kamil, a few solutions were mentioned (none
> > of them is pretty):
> > 
> > 1. We can ask koji developers if there is a way to add a method that
> > would return all koji scratch builds for a given NVR - we would then
> > take the latest one and work with it.
> 
> How would we get the NVR if there's no build identifier on scratch
> builds? Are you talking about the SRPM name in the fedmsg?

Yes. Not from the fedmsg, but from the Koji task (the fedmsg gets this value 
from the srpm filename included in the Koji task, I believe). But see the 
problem described above with srpm file naming (can be arbitrary). So even 
better would be if Koji could search and return scratch build info containing 
the metadata of the srpm or from the spec file included.

But I'm honestly not sure it is a good idea even if they wanted to implement 
it. More on that below.

> 
> > 2. We can use "koji download-task" which works for both. That would
> > mean koji_build item type would eat koji task IDs instead of NVRs.
> > This would lead to having koji task IDs in resultsdb instead of NVR's
> > which kills readability. Unless libtaskotron finds the NVR from the
> > koji task ID in the case of "real" build" and stores it in a field in
> > resultsdb...Also, we need NVRs for fedmsgs. So add code to the fedmsg
> > layer that would take care of somehow adding a NVR to fedmsg of
> > completed scratch builds tasks...
> 
> Can't we tell the difference between an NVR and a task ID just by the
> form that the content takes? Wouldn't the following work:
> 
>   1. attempt to parse input as NVR, if there are no issues, assume that
>  it's a build ID
>   2. if the NVR parse fails, check to see if the input is an int. if
>  assume that it's a koji task ID

The reverse might be easier, but yes, that's of course possible. There are 
other caveats, though.

> 
> It'd mean that our koji downloading code would get more complicated but
> it seems like nothing that more unit tests couldn't mitigate. Unless
> I'm mis-thinking here, the changes needed to pull this off in trigger
> would be even smaller.

I'm afraid our code could get infested with if clauses pretty soon. This is not 
just about downloading rpms in the koji directive. We have more koji 
build-related functionality and we'll sure get more in the future. For example, 
we currently support "download_latest_stable", but that might be very tricky 
for scratch builds with "1.srpm" files (which package is that?). If we ever 
needed to work with package epoch, it is unavailable for scratch builds (since 
there is no build info object to be returned). Another example, we have 
dist-git directive. Without clear identification of the package you can't use 
dist-git directive. Even if we download 1.srpm and look into it, you 

Re: Please Test Staging Phabricator

[Kamil Paral]

> > For a start Ipsilon tells me it's some entirely foreign third-party
> > domain - 'monikra.me' - that wants access to all my personal
> > information, which is a bit unsettling. I went ahead and let it have
> > it (For Science!) and got:
> 
> FWIW, monikra.me is a service that puiterwijk made for federated login

monikra.me is now down (HTTP 502).
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

autoqa migrated to pagure

[Kamil Paral]

I moved autoqa (yes, autoqa) from fedorahosted to pagure:
https://pagure.io/fedora-qa/autoqa

Wiki pages are exported in wiki/ directory. Just in case somebody would need it 
some day :-)

Cheers,
Kamil
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: stats-bodhi license

[Kamil Paral]

> On Thu, 2016-11-03 at 14:41 -0700, Adam Williamson wrote:
> > I'm back working on moving fedora-qa to Pagure. I'm now dealing with
> > the 'stats' scripts, and there's a problem: it appears that stats-bodhi
> > has never actually been properly licensed. It has no license header or
> > license text, and AFAICS, never has. I can't simply declare it to be
> > F/OSS licensed, as I didn't write it.
> > 
> > Can Lukas or Kamil give us a license declaration for this code? Thanks!
> 
> There's one other thing without an explicit license, leftover-tagged-
> builds.sh , which was written by Kamil. It's pretty trivial (and maybe
> no longer needed), but if you could declare a license that'd be good.

I think putting GPL2+ on them is a reasonable choice.

The leftover-tagged-builds.sh was I think written during some time that we had 
a lot of issues with Bodhi. I'd probably still keep it there, but yes, we 
haven't needed it lately.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Proposal to move things from fedora-qa.git to Pagure

[Kamil Paral]

> OK, this is all done now, and fedora-qa.git is a mere empty shell with
> a MOVED.md text file. End of an era!

Thanks, Adam.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: New ExecDB

[Kamil Paral]

> On Tue, Oct 11, 2016 at 1:14 PM, Kamil Paral < kpa...@redhat.com > wrote:

> > > Proposal looks good to me, I don't have any strong objections.
> > 
> 

> > > 1. If you don't like blame: UNIVERSE, why not use blame: TESTBENCH?
> > 
> 
> > > 2. I think that having enum values in details in crash structure would be
> > > better, but I don't have strong opinion either way.
> > 
> 

> > For consistency checking, yes. But it's somewhat inflexible. If the need
> > arises, I imagine the detail string can be in json format (or
> > semicolon-separated keyvals or something) and we can store several useful
> > properties in there, not just one.
> 

> I'd rather do the key-value thing as we do in ResultsDB than storing plalin
> Json. Yes the new Postgres can do it (and can also search it to some
> extent), but it is not all-mighty, and has its own problems.

Sure, that would be even better (but you didn't sound like supporting that idea 
a few days back). So having "state", "blame" and "keyvals" would work well, in 
my opinion. 

> > E.g. not only that Koji call failed, but what was its HTTP error code. Or
> > not
> > that dnf install failed, but also whether it was the infamous "no more
> > mirror to try" error or a dependency error. I don't want to misuse that to
> > store loads of data, but this could be useful to track specific issues we
> > have hard times to track currently (e.g. our still existing depcheck issue,
> > that happens only rarely and it's difficult for us to get a list of tasks
> > affected by it). With this, we could add a flag "this is related to problem
> > XYZ that we're trying to solve".
> 

> I probably understand, what you want, but I'd rather have a specified set of
> values, which will/can be acted upon. Maybe changing the structure to
> `{state, blame, cause, details}`, where the `cause` is still an enum of
> known values but details is freeform, but strictly used for humans?

Hm, I can't actually imagine any case where freeform text *strictly* for humans 
would get too useful. But I can imagine cases where the field is useful for 
humans *and* machines. As an example, we currently have several known causes 
why minions are not properly created. Linking to backingstores might fail, the 
libvirt VM creation or startup might fail, and we might time out while waiting 
for the machine. Those are 3 different reasons for the same overall problem in 
"MINION_CREATION". So, I imagine we can store it like this: 

state: CRASHED 
blame: TASKOTRON 
keyvals: 
step: MINION_CREATION 
reason: TIMED_OUT 

or like this: 

state: CRASHED 
blame: TASKOTRON 
cause: MINION_CREATION 
keyvals: 
reason: TIMED_OUT 

All of that is human readable and machine parseable. The top level keys are 
enums in the database, the keyvals are defined in libtaskotron and the database 
doesn't care. Of course, keyvals are keyvals, so we can easily add something 
like "human_description: foo" if we need it and use it for some use cases, or 
even "time_elapsed: 70" or whatever. It's keyvals, so we can easily ignore the 
ones we don't care about, and use them only in specific cases, or whatever. 
They are flexible and can be easily extended. 

And now we finally get to the point. When Tim asks "do you know how often 
backing stores linking fails during minion creation?" (as he really did in 
T833), we can say "here's a simple query to find out!". Or, if it is not 
implemented yet, we can say "let's add new 'reason: BACKINGSTORE_LINKING' and 
let's see in a month!". Because currently we can't really say anything and the 
best thing to do is to grep all logs, or make some custom hack on our servers. 

> So we can "CRASHED->THIRDPARTY->UNKNOWN->"text of the exception" for example,

I understand it's just an example, but in this case it would be probably 
CRASHED->UNKNOWN->UNKNOWN, because if you don't know to cause, you can't blame 
anybody. 

> or "CRASHED->TASKOTRON->NETWORK->"dnf - no more mirrors to try".

And here the blame would be THIRDPARTY, since it's NETWORK :-) 

I'd prefer keyvals over this, because you can't fit anything useful for 
querying to the free-form details field (or, as you said, you'd rather avoid 
that). The fact that I know it crashed because of network might not be enough 
information to debug issues, decide rescheduling, etc. 

> I'd rather act on a known set of values, then have code like:

You probably meant 'than', right? I'm a bit confused here. 

> if ('dnf' in detail and 'no more mirrors' in detail) or ('DNF' in detail and
> 'could not connect' in detail)

Do the example with keyvals I posted above look better? I'd say using the 
keyvals with 

Re: New ExecDB

[Kamil Paral]

> Proposal looks good to me, I don't have any strong objections.

> 1. If you don't like blame: UNIVERSE, why not use blame: TESTBENCH?
> 2. I think that having enum values in details in crash structure would be
> better, but I don't have strong opinion either way.

For consistency checking, yes. But it's somewhat inflexible. If the need 
arises, I imagine the detail string can be in json format (or 
semicolon-separated keyvals or something) and we can store several useful 
properties in there, not just one. E.g. not only that Koji call failed, but 
what was its HTTP error code. Or not that dnf install failed, but also whether 
it was the infamous "no more mirror to try" error or a dependency error. I 
don't want to misuse that to store loads of data, but this could be useful to 
track specific issues we have hard times to track currently (e.g. our still 
existing depcheck issue, that happens only rarely and it's difficult for us to 
get a list of tasks affected by it). With this, we could add a flag "this is 
related to problem XYZ that we're trying to solve". 
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: New ExecDB

[Kamil Paral]

> With ResultsDB and Trigger rewrite done, I'd like to get started on ExecDB.

> The current ExecDB is more of a tech-preview, that was to show that it's
> possible to consume the push notifications from Buildbot. The thing is, that
> the code doing it is quite a mess (mostly because the notifications are
> quite a mess), and it's directly tied not only to Buildbot, but quite
> probably to the one version of Buildbot we currently use.
> I'd like to change the process to a style, where ExecDB provides an API, and
> Buildbot (or possibly any other execution tool we use in the future) will
> just use that to switch the execution states.

> ExecDB should be the hub, in which we can go to search for execution state
> and statistics of our jobs/tasks. The execution is tied together via UUID,
> provided by ExecDB at Trigger time. The UUID is passed around through all
> the stack, from Trigger to ResultsDB.

> The process, as I envision it, is:
> 1) Trigger consumes FedMsg
> 2) Trigger creates a new Job in ExecDB, storing data like FedMsg message id,
> and other relevant information (to make rescheduling possible)
> 3) ExecDB provides the UUID, marks the Job s SCHEDULED and Trigger then
> passes the UUID, along with other data, to Buildbot.
> 4) Buildbot runs runtask, (sets ExecDB job to RUNNING)
> 5) Libtaskotron is provided the UUID, so it can then be used to report
> results to ResultsDB.
> 6) Libtaskotron reports to ResultsDB, using the UUID as the Group UUID.
> 7) Libtaskotron ends, creating a status file in a known location
> 8) The status file contains a machine-parsable information about the runtask
> execution - either "OK" or a description of "Fault" (network failed, package
> to be installed did not exist, koji did not respond... you name it)
> 9) Buidbot parses the status file, and reports back to ExecDB, marking the
> Job either as FINISHED or CRASHED (+details)

> This will need changes in Buildbot steps - a step that switches the job to
> RUNNING at the beginnning, and a step that handles the FINISHED/CRASHED
> switch. The way I see it, this can be done via a simple CURL or HTTPie call
> from the command line. No big issue here.

> We should make sure that ExecDB stores data that:
> 1) show the execution state
> 2) allow job re-scheduling
> 3) describe the reason the Job CRASHED

> 1 is obviously the state. 2 I think can be satisfied by storing the Fedmsg
> Message ID and/or the Trigger-parsed data, which are passed to Buildbot.
> Here I'd like to focus on 3:

> My initial idea was to have SCHEDULED, RUNNING, FINISHED states, and four
> crashed states, to describe where the fault was:
> - CRASHED_TASKOTRON for when the error is on "our" side (minion could not be
> started, git repo with task not cloned...)
> - CRASHED_TASK to use when there's an unhandled exception in the Task code
> - CRASHED_RESOURCES when network is down, etc
> - CRASHED_OTHER whenever we are not sure

> The point of the crashed "classes" is to be able to act on different kind of
> crash - notify the right party, or even automatically reschedule the job, in
> the case of network failure, for example.

> After talking this through with Kamil, I'd rather do something slightly
> different. There would only be one CRASHED state, but the job would contain
> additional information to
> - find the right person to notify
> - get more information about the cause of the failure
> To do this, we came up with a structure like this:
> {state: CRASHED, blame: [TASKOTRON, TASK, UNIVERSE], details: "free-text-ish
> description"}

> The "blame" classes are self-describing, although I'd love to have a better
> name for "UNIVERSE".

I was thinking about this and what about "blame: THIRD_PARTY" (or THIRDPARTY)? 
I think that best described the distinction of us (taskotron authors), them 
(task authors) and anyone else (servers, networks, etc). 

I'd also like to add "blame: UNKNOWN" to distinguish third parties we can 
identify (koji, bodhi) from errors we have no idea what caused them. This will 
allow us to more easily spot new or infrequent crashes. Alternatively, the 
"blame" field can be null/none, that can have the same meaning. But "unknown" 
is probably more descriptive (and "none" can be converted to "unknown" when 
saving this to the database). 

> We might want to add more, should it make sense, but my main focus is to find
> the right party to notify.
> The "details" field will contain the actual cause of the failure (in the case
> we know it), and although I have it marked as free-text, I'd like to have a
> set of values defined in docs, to keep things consistent.

> Doing this, we could record that "Koji failed, timed out" (and blame
> UNIVERSE, and possibly reschedule) or "DNF failed, package not found" (blame
> TASK if it was in the formula, and notify the task maintained), or "Minion
> creation failed" (and blame TASKOTRON, notify us, I guess).

> Implementing the crash clasification will obviously take some time, but it
> can be 

Re: Resultsdb v2.0 - API docs

[Kamil Paral]

> On Tue, Sep 27, 2016 at 6:06 PM, Kamil Paral < kpa...@redhat.com > wrote:

> > ...
> 
> > What are the use cases? I can think of one - yesterday Adam mentioned he
> > would like to save manual test results into resultsdb (using a frontend).
> > That would have no ExecDB entry (no UUID). Is that a problem in the current
> > design? This also means we would probably not create a group for this
> > result
> > - is that also OK?
> 

> Having no ExecDB entry is not a problem, although it provides global UUID for
> our execution, the UUID from ExecDB is not necessary at all for ResultsDB
> (or the manual-testing-frontend). The point of ExecDB's UUID is to be able
> to tie together the whole automated run from the point of Trigger to the
> ResultsDB. But ResultsDB can (and does, if used that way) create Group UUIDs
> on its own. So we could still create a groups for the manual tests - e.g.
> per build - if we wanted to, the groups are made to be more usable (and
> easier to use) than the old jobs. But we definitely could do without them,
> just selecting the right results would (IMHO) be a bit more complicated
> without the groups.

> The thing here (which I guess is not that obvious) is, that there are
> different kinds of UUIDS, and that you can generate "non-random" ones, based
> on namespace and name- this is what we're going to use in OpenQA, for
> example, where we struggled with the "old"design of ResultsDB (you needed to
> create the Job during trigger time, and then propagate the id, so it's
> available in the end, at report time). We are going to use something like
> `uuid.uuid3("OpenQA in Fedora", "Build Fedora-Rawhide-20160928.n.0")`
> (pseudocode to some extent), to create the same group UUID for the same
> build. This approach can be easily replicated anywhere, to provide canonical
> UUIDs, if needed.

> Hope that I was at least a bit on topic :)

Very much. Thanks for an exhaustive answer. 

> So, what's the decision? I know I can "guesstimate", but I'd like to see a 
> group consensus before I actually start coding. 

I'll just summarize here that we discussed this during Monday's qa-devel 
meeting and reached consensus that keeping ref_url in the group (as it used to 
be) is the current way forward. 
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: What to do with fedora-qa (fedorahosted is dying)

[Kamil Paral]

> > Phabricator works well with remote repos, and I don't think there's
> > any strong advantage in using local Phab repos. On Pagure we will get
> > more visibility for the projects, easy forking, etc. Also certain
> > simple repos (like the one above) can be completely fine with Pagure
> > issue tracker and thus don't need to be configured in Phab (the UI is
> > more difficult there for filing new bugs). I'd go with Pagure, for
> > all our projects.
> 
> In general I agree, I'd be happy for us to move pretty much everything
> into pagure. The only potential issues I see are:
> 
> 1) What about issue tracking for the projects where we currently use
> Phab? For e.g., if we want to keep tracking issues/tasks in Phab
> exclusively, can we disable Pagure issue tracking (and make sure people
> can easily find their way to Phab issue tracking from Pagure?)

You can disable issues in project settings. I don't think you can add a note to 
redirect people elsewhere, we can post a RFE or we can simply make that 
information visible in project README (which is shown by default).

The question whether we want to actually disable issues for certain projects or 
have several places to file them (pagure and phab) is a different matter. I 
guess we'll decide that on case-by-case basis.

> 
> 2) Similarly for pull requests - do we want to have parallel workflows,
> accepting both Pagure pull requests and Phab diffs? Or if we don't, can
> we disable Pagure PRs while providing sufficient breadcrumbs to get
> people into the Phab workflow?

Yes, PRs can be disabled in settings as well. Again, I think this might be 
different for each of our projects and we'll probably decide individually.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: What to do with fedora-qa (fedorahosted is dying)

[Kamil Paral]

> We still have a few miscellaneous things hosted in:
> 
> https://git.fedorahosted.org/cgit/fedora-qa.git
> 
> since fedorahosted is dying next February, what should we do with them?

Move to Pagure. I can do it right away if nobody objects, I've already set up a 
group:
https://pagure.io/group/fedora-qa

> Is this the point where we should finally decide whether to use
> Phabricator's built-in repository support or Pagure for this stuff and
> the stuff we currently host on bitbucket?

Phabricator works well with remote repos, and I don't think there's any strong 
advantage in using local Phab repos. On Pagure we will get more visibility for 
the projects, easy forking, etc. Also certain simple repos (like the one above) 
can be completely fine with Pagure issue tracker and thus don't need to be 
configured in Phab (the UI is more difficult there for filing new bugs). I'd go 
with Pagure, for all our projects.

A slightly off topic, Adam, would you happen to know why Fedora doesn't 
self-host Pagure under say https://apps.fedoraproject.org/pagure , which would 
allow e.g. FAS group integration (unlike now, when we need to mirror our FAS 
groups inside pagure.io)?
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: RFC: URLs for docs and released bits from CI

[Kamil Paral]

> > I submitted a revision today which makes some slight modification to
> > how the doit.py script is building docs, rpms and tarballs.
> > 
> > https://phab.qadevel.cloud.fedoraproject.org/D1012
> > 
> > My intention is to get this deployed to a non-staging setup soon but I
> > wanted to reach out to see if anyone had an issue for the URLs that
> > would end up being used.
> > 
> > https://baseurl/docs/
> > 
> > This path would contain docs for  built per branch for
> > non-master branches and per-release-version when triggered on the
> > master branch. The 'latest' symlink will always be updated to point at
> > the most recent build on the master branch
> > 
> > 
> > https://baseurl/releases/
> > 
> > Pretty much the same thing as for docs but with the rpms, srpms and
> > tarball(s) created during the CI build.
> > 
> > Does this seem sane to everyone? Any comments/questions/suggestions? I'd
> > like to get the auto-docs functionality deployed soon - hopefully right
> > after F25 beta freeze is over.
> > 
> 
> Just wondering if URLs would looked better like this:
> 
> https://{docs,releases}.baseurl/
> 
> (assuming there's no technical issues with that).

I like the original proposal. Just visit https://baseurl/docs/ and you see the 
project listing automagically through apache.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: 2016-09-14 @ 14:00 UTC - QA Tools Video "Standup" Meeting

[Kamil Paral]

> # QA Tools Video "Standup" Meeting
> # Date: 2016-09-14
> # Time: 14:00 UTC
> (https://fedoraproject.org/wiki/Infrastructure/UTCHowto)
> # Location: https://bluejeans.com/2509415264
> 
> One discussion that came up after Flock was that doing a video meeting
> would be worth trying to see how it works for us in terms of
> collaborating.
> 
> In that vein, we're going to try this and see how well it works. This
> will be in bluejeans which does require flash and/or plugins to work.
> I'm not overly tied to it, this is just the easiest option for now :)
> 
> https://bluejeans.com/2509415264
> 
> The agenda is pretty simple: everyone takes a turn saying what they're
> working on this week and what, if anything, is keeping them from making
> progress.
> 
> Hope to see people there.
> 
> Tim

Hi Tim,

please add this to fedocal next time, I noticed the email only today (and only 
because others told me).

Thanks.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: "500 Internal Server Error" when trying to submit a new blocker bug

[Kamil Paral]

> > Whenever I'm trying to submit a new blocker bug¹, I'm getting a HTTP 500
> > status code error page. I've tried these browsers:
> > * one recent release of Firefox (heavily configured)
> > * one vanilla Fedora Firefox release
> > * one Firefox LTS based release
> > 
> > With all of these I'm getting the error page and am unable to propose
> > blocker
> > bugs as a result. I remember that this worked on the last release cycle, I
> > think I proposed some F24 blocker bugs.
> > 
> > ¹ https://qa.fedoraproject.org/blockerbugs/propose_bug
> 
> Sorry for the troubles, Christian, we will look into this.
> 
> In the meantime, can you please propose the blockers manually?
> https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Manual_process
> (simply add "Blocks: FinalBlocker" to the bug and some justification as a
> comment)
> 
> Thanks!
> 

I created a bug about this here:
https://phab.qadevel.cloud.fedoraproject.org/T831
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: "500 Internal Server Error" when trying to submit a new blocker bug

[Kamil Paral]

> Whenever I'm trying to submit a new blocker bug¹, I'm getting a HTTP 500
> status code error page. I've tried these browsers:
> * one recent release of Firefox (heavily configured)
> * one vanilla Fedora Firefox release
> * one Firefox LTS based release
> 
> With all of these I'm getting the error page and am unable to propose blocker
> bugs as a result. I remember that this worked on the last release cycle, I
> think I proposed some F24 blocker bugs.
> 
> ¹ https://qa.fedoraproject.org/blockerbugs/propose_bug

Sorry for the troubles, Christian, we will look into this.

In the meantime, can you please propose the blockers manually?
https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Manual_process
(simply add "Blocks: FinalBlocker" to the bug and some justification as a 
comment)

Thanks!
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

libtaskotron: new mock needed to run the test suite

[Kamil Paral]

Please note I've bumped the requirements for mock in libtaskotron and removed 
some workarounds we had for bugs in the older version. Please make sure to run
$ git pull
$ pip install -r requirements.txt

otherwise the test suite might not pass for you the next time you run it and 
the errors are very cryptic.

Cheers,
Kamil
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: rpmgrill in taskotron

[Kamil Paral]

> Hey Ralph (ccing qa-devel),
> 
> just to let you know, I added some code to Koji directive to download build
> logs. The patch is posted here:
> https://phab.qadevel.cloud.fedoraproject.org/D916
> (you can use 'arch patch D916' to apply the diff in your libtaskotron
> checkout)
> 
> The corresponding changes for task-rpmgrill are in feature/buildlog branch:
> https://bitbucket.org/fedoraqa/task-rpmgrill/branch/feature/buildlog

I created task-rpmgrill project in Phabricator, and now the patches are 
available here:
https://phab.qadevel.cloud.fedoraproject.org/D920
https://phab.qadevel.cloud.fedoraproject.org/D921
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

rpmgrill in taskotron

[Kamil Paral]

Hey Ralph (ccing qa-devel),

just to let you know, I added some code to Koji directive to download build 
logs. The patch is posted here:
https://phab.qadevel.cloud.fedoraproject.org/D916
(you can use 'arch patch D916' to apply the diff in your libtaskotron checkout)

The corresponding changes for task-rpmgrill are in feature/buildlog branch:
https://bitbucket.org/fedoraqa/task-rpmgrill/branch/feature/buildlog

I could have implemented shuffling the build.log files around in 
run_rpmgrill.py, but it seemed easier to have it in a bash script. I moved all 
the rpmgrill* commands in there as well, and used our new bash directive to 
execute it (it still has rough edges, we're working on that). But having it in 
a bash script is not necessary, if you prefer to do it all in python, we 
definitely don't have to merge the code as it is :)

I also removed your failsafe decorator, but again, it's just a suggestion. The 
first reason was that since I moved most of the setup code and rpmgrill 
execution into the bash directive, the decorator didn't have any affect on it 
anyway. The second reason is that we seem to be of an opinion that ResultsDB 
should only contain real results (passed, failed), and crashes and similar 
execution statuses should be only present in ExecDB [1] (which, however, cannot 
be searched yet). This is all up to discussion, and our current solutions have 
a lot of rough edges, so if you want to continue sending CRASHED to ResultsDB 
in case of errors, we have no problem with it, and we can convert task-rpmgrill 
once we're 100% sure how we want things to look like and have it implemented 
properly. Please note, though, that we currently receive emails for all tests 
that crash hard, so we will be able to spot any task-rpmgrill issues even if 
not wrapped in the failsafe decorator.

If you have any questions or concerns, please ask :)

Cheers,
Kamil

[1] https://taskotron.fedoraproject.org/execdb/
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Task Result Namespaces

[Kamil Paral]

> > > Now, since we maintain task-dockerautotest, it should go into
> > > qa.*,
> > 
> > Agreed. Once we have dist-git checks implemented, we can ask docker
> > maintainers to keep it and maintain it in their repo. If they don't
> > want, we can keep it in our domain (same as anyone will be able to
> > run any test against any package in their personal space).
> 
> It seems really weird to have one effectively package-specific in qa.*
> but there are worse things.

I mentioned that just for the case where docker maintainers would not want to 
maintain it. A slight variation would be `qa.pkg.docker.autotest`, to make 
things look more structured.

Of course, if we lift up the restriction "namespace should denote ownership" 
(which I assumed and you didn't), we can easily add our task-dockerautotest git 
repo to the trusted list for `pkg.docker` and then report to 
`pkg.docker.autotest`. Sounds reasonable?

> > > 2. create additional dist.* namespace where tasks like abicheck
> > >and/or rpmgrill would be, or
> > 
> > What is the use case for "dist."? Which tasks would go there and
> > which would not? Is this just about "scratch" not sounding that good?
> 
> I'm pretty anti-fas-name on the release blocking checks (will detail
> below) so in my mind, dist.* is a good place to put things which need
> to be run on all the packages/updates/composes etc. - something that is
> pretty much global. Other ideas on how to split that stuff up are
> welcome, though.

OK, I see the point, considering your aversion to putting those tests under 
`fas.user/group`. But there is a difference. While we can make sure nobody 
messes up your `fas.user/group` namespace and results, we can't do that in 
`dist` (unless we introduce subnames). So `dist` will be a shared namespace 
among everyone who we'll grant access to. It's like a public FTP dir - I'm a 
bit wary about that.

> 
> Having scratch.* be the catch-all also allows us to say "scratch.*
> probably isn't important" while having another option for where to put
> results. I think of it as kinda similar to the concept of a scratch
> build in koji.

Yes, I see it as the same concept.

> > Personally, I always considered our namespaces to have two functions:
> > 1. show ownership - it's clear who owns qa.depcheck, or
> > fas.ksinny.abicheck, or group.workstation.gnome-startup
> 
> Why is it important to show ownership? In some cases (mostly
> non-release-blocking stuff), I can kind of see a case for this but for
> abicheck in particular, I don't see the point. We don't encode the
> maintainer(s) into package names for fedora, why should we for
> checks/tests/tasks?

We're hoping to be running tens and hundreds of tasks for tens and hundreds 
different people. I'd like to make it obvious which task is whose and where to 
direct any feedback, questions, bug reports, etc. I don't want to receive bug 
reports for tasks we don't even know they exist. When I make an analogy with 
github - you have repos named as person_name/repo_name and all your feedback 
goes either to the repo or to the person, you don't send it to the github team.

If we don't encode it in the namespace, how do you envisage people will learn 
the task maintainer? For packages in Fedora we have a packagedb to learn about 
the maintainers, and we have a separate component in Bugzilla for each of the 
package. What are we going to do for taskotron tasks? Are we going to have a 
task database and point to the database from the result details page from 
ResultsDB? If the task details shows you the git url as plain 
http://myserver/task.git, how do you learn who owns/maintains the repo? How do 
we pair FAS usernames with it? How do we even pair git source url with result 
namespace (currently we only submit result namespace when reporting, but do not 
submit git source url).

This is the problem I'm interested in and I assumed solving it with namespaces 
is easy and elegant. I have no problem with solving it in a different way (e.g. 
task database), but it seems it's going to be harder. (But maybe it's a better 
solution in the long run.)

> 
> From a more (perhaps overly) cynical PoV, I suspect that we're going to
> get the lion's share of complaints about failing tasks, no matter what
> the namespace happens to be.

Yes, I was trying to preempt this as much as possible by setting good defaults.

> 
> I don't really have an issue with the group names. If that name changes
> or the task is handed off to some other group, I think that there will
> be bigger changes afoot where changing filters etc. aren't the biggest
> concern.
> 
> FWIW, I don't remember the concept of ownership coming up in the
> original discussions of namespaces - mostly "security" and being able
> to tell which results are more important in terms of release blocking
> vs. package blocking.

Then it means it was just my assumption developed over the course of time :)

> 
> > 2. increase
> > security and eliminate mistakes - by allowing 

Re: Task Result Namespaces

[Kamil Paral]

> Hi,
> 
> we have deployed task result namespaces support a while ago and put
> our checks (depcheck, upgradepath, rpmlint) into the 'qa' namespace.
> With newly added tasks like task-abicheck and task-dockerautotest,
> we weren't sure where to put them and so we used the scratch
> namespace which is supposed to be used for testing purposes. Now
> with those "3rd party" tasks deployed to production, we need to
> decide what namespaces should they and other future tasks belong in.
> 
> The starting namespaces, and maintainers of tasks belonging to that
> namespace, follows:
> qa.* - Fedora QA
> pkgs..* -  maintainers
> fas..* - 
> fasgroup..* - fas members belonging to 
> scratch.* - anyone
> 
> Now, since we maintain task-dockerautotest, it should go into qa.*,

Agreed. Once we have dist-git checks implemented, we can ask docker maintainers 
to keep it and maintain it in their repo. If they don't want, we can keep it in 
our domain (same as anyone will be able to run any test against any package in 
their personal space).

> I am not sure though where does task-abicheck belong. I see three
> options here:
> 1. we can create fas group and put it there,

More on this below.

> 2. create additional dist.* namespace where tasks like abicheck
>and/or rpmgrill would be, or

What is the use case for "dist."? Which tasks would go there and which would 
not? Is this just about "scratch" not sounding that good?

> 3. change semantics of qa.* to 'anything Fedora QA maintains or
>important, not package-specific, tasks".

Personally, I always considered our namespaces to have two functions:
1. show ownership - it's clear who owns qa.depcheck, or fas.ksinny.abicheck, or 
group.workstation.gnome-startup
2. increase security and eliminate mistakes - by allowing certain people or 
groups to write results only into their own namespace, we reduce attack vectors 
(random people can't send fake results for qa.depcheck) and we eliminate 
mistakes (people will not create a thousand test cases in "qa." by accident)

I didn't intend namespaces to reflect task importance in any way, e.g. to put 
all gating tasks into "qa.". It sounds tempting, but it's likely to violate the 
two functions mentioned above. For release critical tasks, it's possible that 
we will take their ownership and maintain them, but I don't think it's going to 
happen for all of them. I don't see a problem in Bodhi or some releng tools 
monitoring qa.depcheck, qa.upgradepath and fas.ksinny.abicheck. It's all about 
trust, and the exact testcase name doesn't matter. You can't trust a random 
person that he/she will maintain the task and quickly fix all issues, but once 
you know who you're dealing with and that he/she is willing and capable of 
doing that work, it doesn't matter whether the namespace is "qa." or "fas.".

Of course there's the issue of people coming and going, and it would be nice to 
have the testcase name changing as little as possible. For that reason, I 
imagine that really important tasks will move to some group ownership, where 
people can maintain it and the namespace doesn't need to change. Let's imagine 
"group.workstation" or "group.dnf" (which can match fas groups, or be created 
manually, or both - the implementation is up to discussion).

So, personally, what I would *not* do is to place task-abicheck into "qa." 
namespace while having Dodji or Sinny still maintain it. That breaks the two 
primary rules of namespaces (as I see it). We should either move it to "qa." 
and start maintaining it ourselves, or put it into Sinny's/Dodji's personal 
namespace and let them maintain it. This is nothing personal against Sinny or 
Dodji (you two are doing great work), and it's not that I don't trust them, I'm 
using a concrete example but trying to show the principle in general.

The only thing is that "fas." is not implemented yet and that's why it is in 
"scratch." now. I don't see a problem with it, and we can move it once we 
implement "fas.". Once abicheck proves to be useful and reliable (or before, 
once we agree on what we really want), we can suggest them to come up with some 
maintenance fas group so that the namespace doesn't need to change, in case 
they want somebody else to take it over.

Of course you might have a different opinion on what the namespaces are useful 
for, and what are their important a unimportant features. Please feel free to 
disagree with me and present your views. I do not feel strongly about this, I 
was just trying to describe what I currently see as the best (safest, least 
error-prone, least maintenance) way forward. Thanks.

PS: There's one case where I can imagine we should do a different solution. If 
we wanted to show abicheck results in Bodhi *asap* and were concerned about the 
risks of displaying results from "scratch." namespace, then it would make sense 
to create some separate temporary namespace, place abicheck into in (and set up 
permissions so that it's not publicly accessible) and 

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Kamil Paral]

> > Speaking of lists, you and Tim were mentioning white/blacklists, also
> > critpath set. So what is the ideal set of packages we should run on
> > initially? All packages? Only critpath packages? Only libraries included
> > in critpath? In case we should run just on libraries, any good
> > recommendation how to recognize that (better than matching "lib*" in
> > package name)? We would need to make the decision without downloading the
> > package, but I guess we could query koji or repo metadata if necessary.
> 
> With latest commit [1] in libabigail taskotron task, ABI comparison
> will occur only on shared libraries because we call abipkgdiff with
> --dso-only option. So, if a package doesn't contains any library file,
> it won't run abi checks on files available in package.There is a
> related bug[2] which has already been fixed and will be available in
> next libabigail release. So, there is no need to look specifically for
> lib* package. Also, packages providing libraries may not start/contain
> *lib* (e.g. elfutils) in its name.

I don't see any libraries in elfutils, they are in elfutils-libs, byt yes, 
you're right, we can't rely on package names.

> 
> So, let's say we initially start with packages available in
> critpath[3], then we don't have to worry whether a package provides
> any shared library or not. 

Let me try this another way. Let's assume we can detect whether a certain 
package contains an *.so (or *.so.N+) file. Can we use this to decide which 
packages to run libabigail on? Or does libabigail run checks on more files than 
just *.so (now that we're using --dso-only)?

I'm asking, because I expect this request ("run my task on packages containing 
shared libraries/files or certain other kind") is going to be quite common in 
the future and I think we should cover it somehow (and I have an idea how). I 
understand that currently we can run libabigail on everything, and if there are 
no libraries inside, nothing bad happens. But it's not very efficient, and 
that's why I'm interested to learn how exactly to distinguish packages 
libabigail is useful for from packages libabigail just skips.

Thanks.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Kamil Paral]

> Hello Kamil,
> 
> Sorry for replying late.  I subscribed to this list, but for a reason the
> emails are still not being delivered to me.  I clicked on some more buttons
> in the interface right now, so we'll see.  In the mean time, I am using the
> web interface to reply, so please forgive the awkward formatting that might
> come out of this.

Hello Dodji, welcome.

> I think the test systems must have that kind of amount of memory.  For most
> packages, the memory consumption is reasonable.  But for some packages where
> the combination of ELF binary size and uncompressed debug info size is big,
> then, well, the tool is going to require a lot of memory.  For instance, it
> takes 13+GB of memory to compare the ABI of two instances of the libjvm.so
> library.

I've read the backlog of your discussion with tflink, but I'm not sure what the 
conclusion is. I believe we can't assign 15GB RAM to all our test VMs (and we 
can't tie a specific task to a specific set of VMs at the moment). So I guess 
we will increase our test VMs memory to some "reasonable" amount and let the 
few extremely large packages crash with OOM. (That reminds me we should make 
sure to disable swap in VMs, otherwise that would kill the host). Do you have 
any advice what this "reasonable" amount of RAM should be? 4 GB? 6 GB? So that 
95% of tasks work OK, and just the extreme ones crash. We can then add those to 
a blacklist.

Speaking of lists, you and Tim were mentioning white/blacklists, also critpath 
set. So what is the ideal set of packages we should run on initially? All 
packages? Only critpath packages? Only libraries included in critpath? In case 
we should run just on libraries, any good recommendation how to recognize that 
(better than matching "lib*" in package name)? We would need to make the 
decision without downloading the package, but I guess we could query koji or 
repo metadata if necessary.

We will need to implement white/blacklisting, ideally I'd like you to have the 
control over it, not us (so e.g. defining that in task.yaml formula). We can 
probably implement critpath support (recognizing what is in critpath and what 
is not), I'm sure it will be needed in other tasks in the future as well.

Kamil
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Kamil Paral]

> Thanks for the quick review. I have addressed review comments and added new
> diff at https://phab.qadevel.cloud.fedoraproject.org/D817?id=2081 . 

Thanks, looks good. I have no further concerns regarding the task code.

> I am not
> sure if I updated diff in right way but this is what I ended up by following
> option "Update diff" available in right side :)

Whatever works. Phabricator is not completely suited for reviewing patches from 
unknown projects, so it can be a bit cumbersome. But I needed a place where I 
can add comments to any line of the script (I think it's not possible to do on 
github). If we need to review any further changes, we can use Phab or github, 
doesn't matter.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Kamil Paral]

Hi Sinny,

I have created a review here:
https://phab.qadevel.cloud.fedoraproject.org/D813

I used our Phabricator, because it makes it easy to comment on particular 
lines. You can log in using your FAS email [1] and reply there, or here in the 
mailing list, it doesn't matter. I'm pleasantly surprised that despite our very 
lacking documentation, you managed to create a very good Taskotron task :-)

One additional thing that I noticed today - when I ran the abipkgdiff 
comparison on lyx package, it used up to 6 GB RAM. That is really high and I 
believe our test clients don't have that many available. I don't know how 
abipkgdiff works and whether it is able to adjust its memory requirements based 
on the amount of available system memory (so that it would use less memory on a 
less equipped system and still work properly). Do you know if this is going to 
be a problem or not? And do you know what is the peak memory usage for very big 
packages (e.g. what about comparing two kernel packages)? Is there any solution 
that we could use to limit the memory consumption of abipkgdiff? 

Thanks,
Kamil


[1] Please note we've been experiencing some problems with authentication 
recently, so it might not work for you:
https://phab.qadevel.cloud.fedoraproject.org/T761
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Kamil Paral]

> Hi all,

> Last year, discussion happened around "Checking the ABI of packages submitted
> to the updates-testing Fedora repository" on Fedora devel ML [1].

> We felt that taskotron[2] will be the best place to run automatic ABI checks
> for a new package update pushed in bodhi for testing against latest stable
> package. If any ABI incompatibility is detected, we will provide package
> maintainer feedback to review ABI changes, similar to how rpmlint feedback
> is given. This will help to ship stable ABI for Fedora packages, reduced
> build failures which is seen later in packages depending on it.

> I have created a libabigail [3] taskotron task which can be integrated in
> taskotron to perform automatic ABI checks for new bodhi updates. Right now,
> this task downloads all rpms with debuginfo for specified package update and
> corresponding latest stable package available for that release. Further, it
> runs libabigail's abipkgdiff tool for ABI checks. It tells PASSED if no ABI
> change occurred, otherwise FAILED and ABI changes log can be reviewed to
> ensure changes are ok to go or not.

> Source code of libabigail taskotron task can be accessed from github[4].
> Some sample output of libabigail task run on my local machine:
> * http://fpaste.org/349740/
> * http://fpaste.org/349741/
> * http://fpaste.org/349761/

> It will be great if someone can review libabigail task and provide feedback.
> Also, I would love to know further steps required to integrate it with
> taskotron?

> Thanks,
> Sinny

> [1]
> https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/BBWQWPSYA66F2T4VVOLY3BGWW43W6K6C/#TDUMSBDHQDW6A4WBTRFYYH7WMQGNWLAZ
> [2] http://libtaskotron.readthedocs.org/en/develop/
> [3] https://sourceware.org/libabigail/
> [4] https://github.com/sinnykumari/task-libabigail


Hello Sinny, 
thanks for your work and sorry for late response. I'll review your taskotron 
task and let you know if there's something that should be changed or not. 
Afterwards, we will start mirroring your git repo on our taskotron servers, and 
patch our taskotron-trigger to know about task-libabigail. We can then execute 
it on every new Koji build (or Bodhi update, your choice). Your task will 
probably be the first one that we will execute regularly while not being 
written and maintained directly by us, so if there are any rough edges in the 
process, I apologize in advance :-)

You'll need to have two branches in your git:
master - this will be used on our production server 
https://taskotron.fedoraproject.org/
develop - this will be used on our dev and staging server 
http://taskotron-dev.fedoraproject.org/ and 
https://taskotron.stg.fedoraproject.org/

We will regularly pull your repo, probably with a cron job for now. The cron 
job is not yet written and the periodicity is not yet decided, but you can 
track it here:
https://phab.qadevel.cloud.fedoraproject.org/T767

You need to decide whether it is better to run libabigain against every new 
Koji build, or just against every new Bodhi update. From a quick look, I think 
it makes more sense to run libabigail on every new Koji build, so that people 
can see the results even before creating the update (that requires looking into 
ResultsDB manually at the moment). If we run it on every Koji build, the 
results will still show up in Bodhi - Bodhi should query ResultsDB and show the 
results for those particular builds present in the update. (We might need to 
teach Bodhi about libabigail existence, I'm not sure). Ultimately it's your 
choice, what makes more sense for your check.

Please also create a wiki page at 
https://fedoraproject.org/wiki/Taskotron/Tasks/libabigail similar to these
https://fedoraproject.org/wiki/Taskotron/Tasks/depcheck
https://fedoraproject.org/wiki/Taskotron/Tasks/upgradepath
linked from https://fedoraproject.org/wiki/Taskotron/Tasks .
We try to have at least some basic documentation and FAQ for our checks in 
there. Currently it's not very discoverable (we should link to it at least from 
ResultsDB, which we currently don't) and the location can change, but at least 
it's a link we can give to people when asking basic questions about one of our 
tasks. Also, since you're going to maintain the task and not us, please include 
some "Contact" section where to post feedback or report bugs (e.g. github 
issues page). If people ask us about the task and we don't know the answer, we 
will point them to that wiki page.

Can we please continue this discussion in qa-devel [1] mailing list? We can 
discuss more implementation details in there, and I'll post my review findings 
in there as well.

Thanks,
Kamil

[1] 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org/
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Workboards and Sprints

[Kamil Paral]

> On Tue, 23 Feb 2016 14:44:14 -0700
> Tim Flink  wrote:
> 
> After the qadevel meeting today and trying it out in staging, I'm
> moving forward with a plan that nobody has rejected and I'm choosing to
> believe that means folks are OK with it :)
> 
> This means that I'm going to start making the following changes in phab:
> 
>  * removing workboards for existing projects
>  * creating a "Planning" project with a new workboard
>- workboard columns: backlog, Next Tasks, In Progress, Ready for
>  Completion
>  * tagging tasks into the new Planning project and organizing the tasks
>into priority order.

I wonder, when creating/editing a ticket, is it possible to assign it to a 
particular planning phase directly? I can assign it to a particular subproject 
in libtaskotron (e.g. I can set "Tags: libtaskotron (Dist-Git Style Tasks)"), 
but it seems I can only set "Tags: planning" (the overall project), and then I 
need to go to the workboard, find it in Backlog, and drag and drop it somewhere.

Is that how it is supposed to work, or am I missing something?

Thanks.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: 2016-02-29 @ 15:00 UTC - Fedora QA Devel Meeting

[Kamil Paral]

The same applies to me today, sorry, can't make it. 

Top posting for consistency, as Josef says :) 

- Original Message -

> Sorry, but I have some errands today (again) and I won't be able to attend.

> 2016-02-29 6:52 GMT+01:00 Tim Flink < tfl...@redhat.com > :

> > # Fedora QA Devel Meeting
> 
> > # Date: 2016-02-29
> 
> > # Time: 15:00 UTC
> 
> > ( https://fedoraproject.org/wiki/Infrastructure/UTCHowto )
> 
> > # Location: #fedora-meeting-1 on irc.freenode.net
> 

> > Don't miss the special Leap Year edition of the Fedora qadevel meeting
> 
> > - February 29th doesn't happen every year!
> 

> > On a more serious note, there are a few items up for discussion but I'm
> 
> > hoping they won't take too long.
> 

> > Please put announcements and information under the "Announcements and
> 
> > Information" section of the wiki page for this meeting:
> 

> > https://phab.qadevel.cloud.fedoraproject.org/w/meetings/20160229-fedoraqadevel/
> 

> > Tim
> 

> > Proposed Agenda
> 
> > ===
> 

> > Announcements and Information
> 
> > -
> 
> > - Please list announcements or significant information items below so
> 
> > the meeting goes faster
> 

> > Workboards and Sprints
> 
> > --
> 

> > Discuss and possibly start on the discussion started on list last week
> 

> > Tasking
> 
> > ---
> 
> > - Does anyone need tasks to do?
> 

> > Potential Other Topics
> 
> > --
> 

> > - dist-git style tasks
> 

> > Open Floor
> 
> > --
> 
> > - TBD
> 

> > ___
> 
> > qa-devel mailing list
> 
> > qa-devel@lists.fedoraproject.org
> 
> > http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org
> 

> ___
> qa-devel mailing list
> qa-devel@lists.fedoraproject.org
> http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: 2016-02-22 @ 15:00 UTC - Fedora QA Devel Meeting

[Kamil Paral]

> # Fedora QA Devel Meeting
> # Date: 2016-02-22
> # Time: 15:00 UTC
> (https://fedoraproject.org/wiki/Infrastructure/UTCHowto)
> # Location: #fedora-meeting-1 on irc.freenode.net

Sorry, I won't attend the meeting today, I have an important errand in the city.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Example Suggestions for Automation Workshop

[Kamil Paral]

> I'm preparing for an automation workshop at DevConf in February and I'm
> trying to think of good example cases to show off what Taskotron can do.
> 
> At the moment, I figure that I'll do at least 1 easy task and 1 not so
> easy but not crazy task.
> 
> Rpmlint is usually my go-to for a trivial/example task but I was hoping
> to use something that's a bit more package specific. The things I've
> thought of so far are:
> 
>  - Put together some package specific stuff for a package that we have
>some control over (testcloud, yourls etc.)
> 
>  - Find some other guinea pig package that just works well for
>package-specific tasks since the triggering will be artificial for
>the demos

Do you have any ideas what those package specific checks could be? Because all 
I can think of is running some test suite which can't be used during build time 
in Koji, but we have nothing like that in our own packages, we would have to 
find and use some other package (and study and write a wrapper for that suite).

> 
>  - See if there are any existing test cases which would automate easily.

Not sure if you mean release validation test cases, but we can most probably 
rule them out, because they all require testing with that specific TC/RC (or an 
installed system that matches those package versions), which we can't offer at 
the moment - not without nested virt or some heavy modifications in the 
environment setup process.

So I guess you mean package test cases (everything that starts with "Package "):
https://fedoraproject.org/wiki/Category:Test_Cases

And this is really a great idea. I have went through all of those test cases 
and the following seem simple enough and easily automated:
https://fedoraproject.org/wiki/QA:Testcase_ClamAV
https://fedoraproject.org/wiki/QA:Testcase_Django_smoketest
https://fedoraproject.org/wiki/QA:Testcase_HTTPd
https://fedoraproject.org/wiki/QA:Testcase_MySQL
https://fedoraproject.org/wiki/QA:Testcase_clone_distgit_repository
https://fedoraproject.org/wiki/QA:Testcase_download_upstream_sources
https://fedoraproject.org/wiki/QA:Testcase_build_from_distgit_with_mock
https://fedoraproject.org/wiki/QA:Testcase_Rkhunter

I think that picking one of them could be a nice demonstration of what we are 
aiming for with Taskotron. The only downside is that we would be showing them 
something they still can't do themselves (since we're not still open to 
arbitrary third-party task submissions, nor we support storing tasks in 
distgit-like system). But as a demonstration of what is hopefully coming in a 
reasonably close future, I think this would serve well.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Building Images for Taskotron Disposable Clients

[Kamil Paral]

> As we get closer to putting disposable clients into production, we need
> a way to have updated images for those clients. I don't think this is
> news to anyone since the topic has come up several times before but now
> there's a bit more urgency :)
> 
> In my mind, we have the following requirements:
>  - Produces qcow2 images that work with testcloud
>  - can be run in an automated way
>  - allows adding/changing/customizing packages contained in image
>  - allows arbitrary repos to be specified
> 
> and the following "nice to have" things:
>  - can build branched and rawhide images
>  - builds images from scratch using only things provided by releng
>  - written in python
>  - builds more than qcow2 for some future-proofing

qemu-img can convert between many image formats, so native support in that tool 
is not that important, I think.

>  - can run well in a VM
> 
> Is there anything that I missed?

The image should be compatible with guestfish, so that we can e.g. copy in some 
files without rebuilding the image from scratch. Might be useful for e.g. 
additional ssh keys (we have cloud-init for that at the moment, but if we had 
some troubles with it or we needed something it doesn't support, this would be 
an alternative way). I'm not fully sure what the requirements are, but I think 
guestfish can work with almost anything, including LVM, so unless the tool 
creates some crazy partition layout, it should work with everything.


> 
> As far as I know, we're looking at two options right now:
> taskotron-vmbuilder and imagefactory. I've put together a list of
> the pros and cons that I know of for both tools. Thoughts on which
> direction to take would be appreciated.
> 
> Tim
> 
> 
> taskotron-vmbuilder [1] is a PoC system kparal built around
> virt-builder [2]. Images are specified in a yaml file and instead of
> building those images from scratch "It takes cleanly prepared, digitally
> signed OS templates and customizes them".
> 
> [1] https://bitbucket.org/fedoraqa/taskotron-vmbuilder
> [2] http://libguestfs.org/virt-builder.1.html
> 
> pros:
>  - already does almost everything we need

To be fair, there have been some issues regarding SELinux, and I'm not sure 
they are sorted yet. The SELinux contexts of files inside the image were not 
set properly and one more reboot with autorelabel was needed. Might be fixed 
now, or not, I haven't tried for a long time. With anaconda, we're not likely 
to hit these kind of issues (we'll hit different ones).

>  - fits all requirements
>  - builds quickly
>  - well supported
> 
> cons:
>  - requires blobs which are out of our control
>* yes, I know who does the work behind virt-builder. My concern
>  isn't with him, it's the general concept that I don't like. This
>  also gets into the fact that we would have pretty much no control
>  over timing of release for the base images.

All the tools required to create that "blob" - or image template (as they call 
it) - are open source and in Fedora, from what I see, so we can host our own. 
virt-builder man page says:
"For serious virt-builder use, you may want to create your own repository of 
templates."

This is how to create the template:
http://libguestfs.org/virt-builder.1.html#create-the-templates
For new stable Fedora releases, we can a single install manually, and use 
virt-sysprep on the image to have it ready. For Rawhide and maybe even for 
Branched, we might want to prepare a fresh new template more often, i.e. 
automate that. This is how libguestfs project does it:
https://github.com/libguestfs/libguestfs/blob/master/builder/website/fedora.sh

So you might see this as a combination of imagefactory and virt-builder-style 
process. The image is installed clean using anaconda once in a time (but very 
rarely), and most of the time just the prepared template is adjusted (updated 
with new packages), because it's much faster.

I'm not saying this is better or worse than alternatives, I just don't think 
this "blob" argument is quite right - we'd probably create and host our own 
templates, not rely on upstream one.


>  - limited support for rawhide and branched releases

There's limited (or no) support for it in the upstream repo, that's correct.

But if we host our own repo, according to the documentation and source code, it 
seems that as long as anaconda can install it, it should be possible to create 
an image for it. Which sounds as the same situation as with imagefactory. (Of 
course with the additional requirement that virt-* tools have to work in 
Rawhide/Branched).


>  - limited support for non-server spins

I'm not really sure what you mean, we can install any package set we want, so 
the only difference would be in the filesystem layout? The upstream templates 
seems to have only @core installed, in our own images we could adjust even that.

>  - output images are large in size

This is interesting. Theoretically I see no reason why official Cloud images 
should be smaller 

Re: Building Images for Taskotron Disposable Clients

[Kamil Paral]

> My vote would be for imagefactory. In my mind, it comes down to the blob
> vs. from-scratch thing, using the same tools that releng does and the
> fact that imagefactory is completely written in python.
> 
> Whether we use the API or not is a different question, though :)

I think both tools are capable of doing what we need and the amount of required 
work will be similar. The blob argument disappears if we create and host our 
own templates. The collaboration factor is important and having some kind of 
API is also a good thing (even though I haven't seen it yet and I don't know 
what we would use it for). I don't think we will patch that too much (if we 
need to submit nontrivial patches, we're likely using a wrong tool, I don't 
think we should be spreading ourselves even thinner than we are now), so the 
implementation language is not a big deal, I think. Hosting and downloading 
might be easier with virt-builder, because it already supports querying remote 
image templates index and downloading selected files.

There is one view angle that we should consider as well, I think, which is the 
task-developer use case. For our production use case, it doesn't really matter 
how fast the compose process it or how large the disk images are. But how are 
the task developers going to use them? If they're going to download them (and 
periodically update them, even several times per week), we want the images as 
small as possible. If they're going to build them locally, we want them built 
fast and with small download sizes (the virt-builder template saves some data 
from downloading, only updates are downloaded; also there's a question whether 
we could cache the downloaded packages somehow with some of these tools). Or 
will we offer both approaches, let them pick what works best for them?

If we intended to mostly build those images on dev's computers, I'd probably 
prefer virt-builder. But my current impression is that local building will be a 
secondary option, and we'll primarily offer pre-created images for download 
(even downloading them automatically). Which makes sense, it's easier for the 
dev, and less error-prone. So in that light (assuming no one has different 
plans in mind), it doesn't really matter which technology we choose to build 
it. Image size is a factor here, though. I don't have any real numbers here, it 
would be very interesting to see the same image (same package set, same 
filesystem size) built by both tools and compare the output size (ideally even 
after running zerofree on them and compressing them). My guess is that they 
should have the same size (what would cause a difference?), but we might be 
surprised. Do we have a volunteer to test this? :)
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: Building Images for Taskotron Disposable Clients

[Kamil Paral]

> > >  - may not run well in a VM (would need nested virt)
> > 
> > This is the same as in virt-builder, it also needs virt support.
> > Originally I thought it doesn't, but it does. It can still be used
> > without hw virt support (unlike anaconda, that would just be
> > impossible performance-wise), but it's much much much slower and I
> > don't think we would want to go that route (building an image 30
> > minutes instead of 3 minutes).
> 
> It sounds like we're going to need a bare metal solution either way,
> then.

I bet my old socks that we'll end up with nested virt eventually. It's not a 
years-proven technology, but it worked well for me locally, OpenQA is now 
running on it as well inside Fedora Infra, and many projects (not just ours) 
seem to gravitate towards it, because bare metal is just "too much trouble"TM.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: openQA in infra

[Kamil Paral]

> so we can run openQA in Ze Cloud, with KVM, and it can work and
> complete a test.

What's Ze Cloud? Is it using nested virt? We might also want to use nested virt 
in Taskotron, I'm very interested in any issues related to it you run into.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: openQA in infra

[Kamil Paral]

> > so we can run openQA in Ze Cloud, with KVM, and it can work and
> > complete a test.
> 
> What's Ze Cloud?

Now I know, you fake French!
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: 2015-07-22 @ 21:00 - Taskotron Outage and qadevel Migration

[Kamil Paral]

 It took significantly longer than I was hoping but Taskotron
 dev/stg/prod has been updated and qadevel has been migrated to a new
 machine.
 
 There should be no big changes here, everything user-facing should be
 business as usual.
 
 Please let me know if you notice something not quite working right.

The Phabricator emails have a new sender address 
phabrica...@fedoraproject.org. Not sure it's intentional. If it is, would 
phabrica...@qa.fedoraproject.org be better?

None of our repos are getting synced, they report errors like these:
Initialization ErrorPull of LTRN failed: git fetch failed with error 
#255: stdout: stderr:error: cannot open FETCH_HEAD: Permission denied 

I think it's the long standing issue where newly created repos can't create 
directories and we need to do it manually. I guess it will be some kind of 
directory permission issue.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Weak dependencies come to Fedora

[Kamil Paral]

Not sure if you noticed, but this can possibly affect some of our tools or 
workflows, so I wanted to highlight it - it seems that weak dependencies are 
now approved in Fedora:
http://fedoramagazine.org/5tftw-2015-07-09/
https://fedoraproject.org/wiki/Packaging:WeakDependencies

We now have 4 more RPM dependency types to look out for :-) The first thing 
that comes to mind is depcheck, so I created a task for it here:
https://phab.qadevel.cloud.fedoraproject.org/T524

Hopefully this will not affect our other tools too much. If you know about 
something else that is likely to break, shout.

Thanks,
Kamil
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: openQA vs. Rawhide

[Kamil Paral]

 There's a neat GTK+ debugging thing called the GtkInspector which you
 can launch from any GTK+ 3 app with shift+ctrl+D. From that you can

Or Ctrl+Shift+I, if you find that easier to remember (I as inspector) :-)

 look at Properties then GtkSettings, and look at the gtk-xft-*
 settings. I noticed two significant differences between anaconda
 running from boot.iso or KDE live and anaconda running from Workstation
 live, both at 1024x768 in a VM. gtk-xft-hintstyle is 'medium' in
 Workstation but 'full' in boot.iso/KDE, and gtk-xft-dpi is 98304 in
 Workstation but 98401 in boot.iso/KDE. That value is a DPI multiplied
 by 1024; Workstation's is exactly 96 (98304/1024 = 96), boot.iso/KDE
 is...a little bit more than 96.
 
 Basically we figured out that for some reason the defaults that should
 be set by /etc/X11/Xresources don't seem to actually be merged into X,
 in boot.iso and KDE live image at least. So GDK (part of GTK+) falls
 back on its internal defaults for hinting style and display resolution
 (DPI). Its default for hinting style, in the absence of any external
 indication, is 'hintfull', and its default for DPI is to actually
 calculate the 'correct' DPI for the display. It turns out that a KVM at
 1024x768 reports its monitor's vertical size as 203mm; if you
 reproduce GDK's calculation with 768 pixels and 203mm:
 
 768 (pixel height) * 25.4 (inches to MM) / 203 (height in MM) * 1024
 (GDK's multiplication factor) ~= 98401 (actually a bit lower but it
 gets rounded to the nearest integer)
 
 and that's where the weird 98401 value comes from.

What, that's a one hell of a debug story. Awesome job.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: Coding Style

[Kamil Paral]

 After the meeting earlier today, I wanted to make a slightly different
 proposal.
 
  - if flake8 config in arcanist can be so configured, set max line
width to 120 instead of 79

On my 1920px monitor, two windows with 100 columns fit perfectly next to each 
other (11px font). Is there a strong desire to go over 100 cols? It would make 
file comparisons less practical. And 100 (well, 99) is still PEP8-approved.

I have no problems exceeding that limit in certain specific cases. For example, 
because we often bundle documentation with the source code, and we format it 
with rst, it might happen that a large block of text is indented far right and 
it makes it very long. Or there are some code samples which can be wrapped but 
it makes them hard to read. In such cases I have no problem having these 
particular lines go to 120 (or even more, if needed). But this would be very 
exceptional and almost never applied to regular code - no need to worry about 
linter exceptions discussions, I think.

PS: It seems that flake8 can't be configured in arcanist at the moment. The 
patch was written [1] but abandoned and never pushed [2]. It seems quite 
simple, though, so we could finish that up, if desired. The other approach is 
to disable line length checking in flake8 and use generic ArcanistTextLinter 
[3] for it, it's even mentioned in the docs [4]. That should be less work.

[1] https://secure.phabricator.com/D10512
[2] 
https://github.com/phacility/arcanist/blob/master/src/lint/linter/ArcanistFlake8Linter.php
[3] 
https://github.com/phacility/arcanist/blob/master/src/lint/linter/ArcanistTextLinter.php
[4] 
https://secure.phabricator.com/book/phabricator/article/arcanist_extending_lint/#arcanisttextlinter

 
  - default to no on lint exceptions - the idea is to avoid spending
time debating whether an exception is worth it or not.

If I read this correctly as we don't have a strict 0-exceptions policy, but 
they should be very rare and if you propose one, you should have a very good 
reason to do so, I think that's fine and we'll see how that works in the next 
months.

 
  - until we get the codebase compliant if you touch a file, fix the
lint errors even if those errors are not part of what you're
changing

Even though this will make our files a bit consistent (modified parts looking a 
bit different than unmodified parts), I don't mind much and I think that this 
approach is fine for the moment. Especially before we find out if we're happy 
with the new approach. It will also minimize the changes between develop and 
disposable-develop.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: Coding Style

[Kamil Paral]

 First of all I'd suggest to move our codebase to strict PEP8 (or
 as-strict-as-possible), so we can have see how our code looks like, when
 PEP8 compliant.
 For starters, we could just plain use autopep8 -
 https://pypi.python.org/pypi/autopep8/
 How about that?

For the record, Josef put up the diff here:
https://phab.qadevel.cloud.fedoraproject.org/differential/diff/1036/

As the most vocal naysayer during the IRC meeting, I have went through all of 
those changes (and I have read PEP8 again), and I have to say the changes are 
not that bad. I have feared that the code would look similarly to what Josef 
quoted in the email, but PEP8 is actually saner than I expected and allows much 
better stylistic formatting.

Which in fact touches one of the concerns I have, that it's hard to realize how 
to write the code pretty and still comply to PEP8, but it's easy to realize how 
to write the code ugly and comply. At least that's my experience.

I still have a few pet peeves about PEP8, for example check.py:189-192 or 
config_defaults.py:63. I think human being can use whitespace to make code much 
more readable (the code is in a monospace font for a reason), and PEP8's I 
know better approach of enforced conformity quite bothers me. But I understand 
that can be highly subjective, and for some people consistency is above 
readability, for me it is the other way around.

That said, if everyone else happily supports this idea, I don't object against 
a trial run in which we drop all custom configuration from .arclint and enable 
pep8 checking. We will see how it works out. I don't think it will boost our 
productivity, I think it will just move the wasted time from reviews (let's 
admit it, almost none in the past months) to the patch preparation process and 
multiply it, but maybe I'm wrong.

PS: Speaking of line length, PEP8 actually allows going up to 100 chars [1], 
and most of our existing codebase meets that, so I don't see that as much of a 
problem. The comments and docstrings still need to be wrapped at 72 chars, 
though, so I'm curious to see if we start to enforce that.
[1] https://www.python.org/dev/peps/pep-0008/#maximum-line-length
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: Coding Style

[Kamil Paral]

  Back to everyone's favorite topic - coding style and enforcement of
  that style.

And I thought we were doing so great by leaving it behind...

  
  I'm not picking on Josef here - I'm sure I've submitted code recently
  with lint errors, this was just the review I was looking at which
  triggered the idea:
  
  https://phab.qadevel.cloud.fedoraproject.org/D389

As I already mentioned in the review, the way how Phab displays lint errors is 
a problem in this particular case - the diff is hard to read. That is something 
I'd like to avoid in the future.

  
  Since the last discussion about coding style ended in a long discussion
  about PEP8, the bits we may not like about the standard and the
  exceptions that we'd want, I'm proposing that we use strict PEP8 with
  almost no exceptions. 

The big hammer solution! ;-)

  I'm not saying that that PEP8 is perfect or
  that there aren't parts of it I would like to tweak but
   - it's a well known standard, easy for non-regular contributors to
 understand and use
   - style checking tools tend to use PEP8 out of the box
   - if we don't allow exceptions, there will be less time spent on
 discussing details instead of being productive :)

The opposite is true for me. My experience with lint warnings in Phab (before 
we loosened the rules) is that I spent a considerable time rewriting and 
reformatting the code until the damn linter finally shut up (I'd probably need 
stronger words to demonstrate my temper at that time:)). Also it made the code 
slightly less readable sometimes (a personal view), which annoyed me. I lose no 
productivity on seeing an occasional lint warning in diff review (D389 is a 
slight exception in a very long time).

That being said, I try to minimize the number of lint warnings with my every 
patch, unless I consider that specific rule particularly braindead.

 
  To be more specific, I am proposing the following:
   - all QA devel projects be have flake8 as the linter in arc config
   - no code be accepted with lint errors unless there is absolutely no
 other way to get around it
   - until we get our entire codebase PEP8 compliant, if you touch a
 file, fix the lint errors even if those errors are not part of what
 you're changing
  
  Any strong objections to starting this? Any strong objections should
  have an alternative proposal and a justification why that proposal is
  worth deviating from a well known and established standard.
  

I have an alternative proposal:

1. The Phab diff should not contain too many lint warnings without a good 
enough reason, in order to keep it readable.
2. If it does, ask the author to fix it.
3. If the author is reluctant to change his coding style, and you have no 
strong objection to it per se (you just want the Phab diff to be readable), 
let's disable the particular lint check.

In this case it would involve asking Josef to stop putting spaces between 
parameter keyvals, or disabling unexpected spaces around keyword / parameter 
equals check. It could also involve disabling continuation line over-indented 
for hanging indent check, because that's another very common warning we just 
ignore.

The only problem will be when two people have strong and conflicting opinions 
on how to code should look. Is this the case? If not, then if we implement the 
solution describe above, I believe we're not going to be discussing code style 
in another year or two, judging by the past.


 
 By saying we should use strict PEP8, does that mean you want to get
 rid of the linter settings we have in .arclint or leave it there?

Strict PEP8 would mean all the configured exceptions would be purged.



Oh boy, this will be a long discussion once again.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: Fedmsg Emitting

[Kamil Paral]

  We talked with Martin about this in length some time ago, and I
  raised the question of different consumers. I see two groups here -
  machines and humans. If I understand you correctly, what you propose
  up there is to hardcode the system to fit human preferences. If I
  misunderstood it, then the whole rest of the mail is based on wrong
  assumptions, but it's still an interesting topic :)
 
 I think of it more as trying to ensure consistent behavior in the data
 that we emit to external entities, whether that's a machine or a human.

I misunderstood then, I thought it was more than that. In that case we can just 
discuss deduplication of depcheck-like results and forget about the rest of my 
email (or keep it at hand for future discussions of that topic).

 
 In my mind, the core of the issue is that we're taking what are
 essentially per-repo checks and hacking them until they emulate a
 per-build or per-update check as closely as we can make it.
 
 This means that we end up with a lot of duplicate results every time
 that the check is run because we have to run it as check(repo) instead
 of check(build) or check(update).
 
 Since we're trying to emulate a per-build or per-update check (in the
 case of upgradepath and depcheck) I don't see a reason why a human or
 machine consumer would want or need the duplicate results beyond how
 we emulate the per-build or per-update check that external entities are
 expecting.

I think the most confusing part is that depcheck is really a repo check, but we 
try to _report it_ per build (let's disregard updates, we plan to get rid of 
them). I think that the same thing as you're saying.
I understand the motivation for not duplicating those results, even though I 
consider both ways to be consistent, just in a slightly different way.

Let's consider a different repo check, let's say valid_metadata. We would run 
this check every day, once a new repo is pushed. The item would be e.g. 
fedora-22-update, with some arch, only the time of running/reporting would 
differ, every day.
We don't want to deduplicate this (there's nothing to deduplicate, there's one 
result per day). Running our naive deduplication algorithm (ignoring same 
results from the same check on the same item) would incorrectly ignore all 
results from the second execution forward (until the result would change).
How do we plan to distinguish valid_metadata from depcheck, to know which 
results to deduplicate and which not to?

I can imagine adding a timestamp to item or hardcoding some check differences 
to resultsdb, and they both seem very unclean and unmaintainable. Am I missing 
some simple and clean solution?

 
 FWIW, I don't think we're going to have many more of these emulation
 checks. I also think that we can get away with an emulation mode for
 resultsdb that handles checks like upgradepath and depcheck differently
 than other results so that they behave like other one result per
 change checks but I'm not the one writing that code. 

I'd very much like to avoid this, it just adds a bunch of complexity and makes 
task writing much harder because of documentation full of special modes and 
exceptions.

 If martin thinks
 that's not a tractable solution, I'm game for other options.
 
  When targeting humans, I believe we will cut off some use cases for
  machines, which can benefit from duplicated (and thus very
  up-to-date) information. Some ideas from top of my head describing
  what duplicated messages allow:
  * For some checks like depcheck, the machine (i.e. Bodhi) can not
  only display the outcome for a certain package, but also the time
  when this package was last tested (might be a very interesting piece
  of information, was it OK yesterday, or 14 days ago and not run
  since?).
  * Or maybe show a graph of all the outcomes in the last week - so
  that you can see that it passed 10 times and failed once, and decide
  that the failure was probably a random fluke which is not worth
  investigating further.
 
 What would another interpretation of failing once and then passing for
 all subsequent runs be? I can't think of a situation other than bugs in
 the check that would be anything other than pass.

Race conditions, perhaps. Issues with particular input data. Depends on the 
check.

 
  * If the message passes through another system (e.g. Bodhi, Koji),
  the system in question can e.g. allows users to configure how they
  want to receive results - whether duplicated or deduplicated, how
  much deduplicated, how often, etc. This is mostly true for email, RSS
  or some other communication channels, because fedmsg bus itself is
  not configurable per individual users' needs.
 
 I'm not sure there would be enough people interested in receiving that
 kind of data to make it worth the effort supplying it through bodhi or
 fedmsg would entail. If we were going to have a bunch of checks that
 were emulating per-event checks, then maybe but for upgradepath and
 depcheck? I just can't 

Re: Fedmsg Emitting

[Kamil Paral]

  Before we start sending fedmsgs we need to discuss a few things. We
  don't have to find solutions to all these problems, just keep them in
  mind when designing the solution we're going to start with:
  
  1. How often do we send fedmsg
  a) per-task
  b) per-update
  c) per-build
...
 
 That leaves us with c)

Seems reasonable to me.

 
  I guess c) allows to easier filtering in FMN.
 
 c) not only allows for easier filtering in FMN but it's also more
 compatible with how I think that releng would like to see build gating
 done. Assuming that we eventually get into the rawhide space, we'll
 have to start emitting stuff per-build anyways :)
 
 I'm of the opinion that c) is going to be best here. In the past, we've
 done a lot of results on a per-update basis but unless I'm forgetting
 something, we could transition to more of a per-build system.
 
 For example - depcheck processes updates - if one build in that update
 fails, the whole update fails. While I think that this the best choice,
 I also think that logic should be handled in bodhi instead of us trying
 to emulate what bodhi is doing. As far as I know, this is happening
 with bodhi2 - they're assuming that we'll be emitting per-build fedmsgs
 and the logic for failing/passing an update will lie in bodhi and not
 rely on our emulation of bodhi's processes.

That's great to hear.

 
  2. Who do we target: users, systems or both
  
  The issue here is with tasks that repeatedly test one update.
  Currently we check if there's a bodhi update comment with the same
  result already and if so, we don't post the comment again. To do
  something like that with fedmsgs we'd have to have a code running
  somewhere that would check against its database whether an incoming
  result is a duplicate or not. The question is where the code would
  run. Bodhi comes to mind since it already has information about
  updates and so is good for tasks that work with bodhi updates.
  However, there might be tasks that work with something else, like
  composes. In this case we'd probably have the code on taskotron
  systems.
 
 I think that how we handle scheduling of some of our current checks
 (depcheck and upgradepath) is a byproduct of trying to make a
 repo-level check look like a build/update-level check. I can't think of
 many more tasks that would run into the same problem of repeated runs.

I agree that depcheck and upgradepath are somewhat special here. Once Fedora 
Infra sees how many results we publish daily, especially during freeze periods 
(when there are lots of packaging pending stable), they might ask us to come up 
with a better solution, I'm afraid. I'm not sure if there are better ways to 
handle it, either way, there will probably always be some kind of check that 
will require this kind of constant re-running. But it seems reasonable to 
assume that it will be a small minority in the overall task pool.

 
 For the majority of tasks, I see the process as being similar to:
 
   1. trigger task $x for $y
   2. run task $x with $y as input
   3. report result for $x($y)
 
 With this, we'd be running $x for each $y and the reporting would only
 happen for each unique ($x, $y) assuming that something wasn't
 rescheduled or forced to re-run.
 
 I think it would be best to have consistent behavior for our fedmsg
 emitting. If most tasks will only emit fedmsgs once, we should take our
 minority tasks that emit more than one fedmsg per item and deduplicate
 before the messages are emitted.

Or, you can say that most tasks emit fedmsgs always (even though that means 
just once), and therefore the minority tasks should also emit it always :) I 
agree with having a consistent behavior. But I think it's possible to find a 
solution side-stepping this. See below.

 
  So if we target systems we'd just send all results in fedmsgs and let
  the systems consume them and do whatever they want to do with them
  (e.g. bodhi can squash all the tasks relevant to specific update and
  notify the maintainer of the package via fedmsg about the result). If
  we target users, we'd have to have some logic to limit rate of fedmsgs
  ourselves but that would mean hiding some of the results (although
  duplicates) from the world.
 
 I'd like to see us do the deduplication in resultsdb (assuming that's
 where the fedmsg emission will be happening). I think that we already
 have a table for items and I don't think that keeping track of
 is_emitted and the last state emitted (so we can track changes in
 state) would be too bad. Then again, I'm not the one working in the
 code and I could be wrong :)

We talked with Martin about this in length some time ago, and I raised the 
question of different consumers. I see two groups here - machines and humans. 
If I understand you correctly, what you propose up there is to hardcode the 
system to fit human preferences. If I misunderstood it, then the whole rest of 
the mail is based on wrong assumptions, but it's still an interesting topic 

watching projects in Phabricator, Herald rules simplified

[Kamil Paral]

Hello everyone,

in the last week I've spent a bit of time investigating the possibilities of 
how to watch for ticket/revision updates in Phabricator, and found out that 
some new features in Phab greatly simplify this. I have updated our Phab wiki 
page and included most common use cases:

https://fedoraproject.org/wiki/QA/Phabricator

In a nutshell:
* you can now start watching projects which automatically sends you all tasks 
(tickets) updates. No more Herald rules needed. Unfortunately, this does not 
apply for revisions (patches).
* you can now easily subscribe to tickets without a project defined (usually 
bug reports). This way you can help us classify and triage them and make sure 
they go unnoticed (that unfortunately happened in the past).
* Taskotron lovers can now easily watch for all Taskotron-related new revisions 
(patches) by using the repos-taskotron meta-project [1]. We will make sure that 
this meta-project is always linked with all taskotron-related repositories, and 
you don't need to update your Herald rules for every new Taskotron sub-project.
* You can now easily decide between a permanent item subscription or just 
one-time notification by choosing either Add emails to CC name or Send an 
email to name.

I have tested it for the last week and everything seems to work properly for 
me. If you have some troubles, ask.

Kamil

[1] https://phab.qadevel.cloud.fedoraproject.org/project/profile/23/
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: To RHEL or Not to RHEL?

[Kamil Paral]

 * Will need to be more diligent about keeping dev/stg on
   updates-testing so that we don't get any nasty surprises in production

I don't have much advice about the other points, but this one caught my 
attention. Do we really need to use updates-testing for dev/stg? That might be 
quite problematic, because anyone can submit anything, no matter how broken, 
into updates-testing. Wouldn't be a safer approach to update dev daily (and stg 
e.g. every other day) from stable updates? And production would be updated 
weekly or bi-weekly (or however often we need it), with the exception of 
security updates. Security updates would be applied to dev/stg immediately and 
after a few jobs were successfully executed, it would be applied to production. 
Would this approach work?

I guess the approach with security updates would be the same, no matter whether 
it's Fedora or RHEL. So the only difference in the volume and speed of standard 
updates.

This doesn't mean I'm in favor of running Fedora, I think you have much more 
experienced view on this. I'm just thinking aloud about some of the details.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: CI for Taskotron and Related Projects

[Kamil Paral]

 I've just re-started the CI on qadevel (the future one, not cloud) that
 mkrizek set up a while back. It was destroyed when I started working on
 replacing qadevel.cloud but never finished it.
 
 The link is:
 
 https://qadevel.fedoraproject.org/buildmaster/waterfall

Great. Just to refresh my memory, will we receive some kind of a notification 
when some project fails to build?

 
 Note that there will be ssl cert warnings since it's self signed. Once
 the thing gets rebuilt for real, those ssl warnings will go away.

Not sure if it is intentional, but plain http does not work, the connection is 
refused. Not an issue, just saying.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: new project: taskotron-vmbuilder

[Kamil Paral]

 On Wed, 2015-04-29 at 11:21 -0400, Kamil Paral wrote:
  
  I still haven't created project and repo in Phab, but if you approve
  this as a way forward, I will.
 
 Just one thought - this could probably be rather useful for openQA
 too. We already have one case where we want to begin a test with a
 specific existing image (the fedup test that's currently under
 review). We could use a system like this to do other tests that don't
 involve actually running an installation (I'd quite like to do stuff
 like the Base and desktop validation tests and maybe some package
 management tests every day, for e.g.)

If taskotron-vmbuilder works for you as well, that's great. I can add some 
actions specific for openQA needs, just tell me (or file some tickets). 

Of course, you can also use virt-builder directly, if that's easier for you. 
Currently taskotron-vmbuilder can't do anything extra that virt-builder can't. 
I just consider using the templates a bit cleaner than creating custom shell 
scripts for each image type. But it in the future, it might be extended with 
some extra functionality, like e.g. booting the image once and performing some 
first boot adjustments that can't be done when chrooted during image creation.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: new project: taskotron-vmbuilder

[Kamil Paral]

  I still haven't created project and repo in Phab, but if you approve
  this as a way forward, I will.
 
 I vote for creating the project and repo in phabricator.

Done.

PS: I also noticed that our projects were editable by anyone (all users), 
which didn't seem to me as a great idea, so I changed it to administrators 
for all of them.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: 22 Final TC1 live fail: timeout due to SMP performance bug?

[Kamil Paral]

  https://bugzilla.redhat.com/show_bug.cgi?id=1210857 , but I see
  you've found it already.
  
  For the time being, it seems that there's no benefit in giving the
  VMs more than one CPU?
 
 Yeah, I think it'd make sense to change the config to one CPU. That's
 what I use on openqa.happyassassin.net, FWIW.

In case you or Jan made some changes, a (supposedly) fixed kernel has just been 
pushed stable. So multicore systems should be fine once again.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: Freeze Break Request: Increase execution timeouts in taskotron's buildbot

[Kamil Paral]

 I'd argue that we do need to get the increased timeout working but that
 doesn't negate your point about python-yamlish.
 
 However, there are two problems:
 1. there's no newer python-yamlish package to update to - that's
something that I wrote the specfile for and isn't in the fedora
repos, as far as I know.
 
 2. we can't deploy a newer libtaskotron in production yet - that'll
bring in stuff that I don't want to push into production yet.
 
 That being said, I've started a build of python-yamlish-0.18 and will
 get that pushed out to dev/stg later today.

Yeah, I was referring to the particular issues we were seeing in the past - 
those should be resolved by a newer yamlish. Thanks for the update. There are 
still some other issues for which the increased timeout will be useful, until 
we have a better way to deal with them. I have reviewed the 2-hour-long 
upgradepath runs yesterday - the main check including all Koji communication 
takes 10 minutes, the rest is communication with Bodhi. 2 hours of querying 
Bodhi! It's absolutely insane. Of course we don't usually have 550 builds to 
check, but we will need to do something about it, otherwise we'll have issues 
every time there is a freeze. And we might be even partly responsible for Bodhi 
being so overloaded lately.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Fwd: Freeze Break Request: Increase execution timeouts in taskotron's buildbot

[Kamil Paral]

Just a note - we don't need an increased timeout, if we deploy newer 
python-yamlish. That could have been done a long time ago, but we forgot to 
update the spec file, so nobody found out. That's why I tried to fix it in 
https://phab.qadevel.cloud.fedoraproject.org/D337 . Once that is accepted, the 
next libtaskotron build will require the newer python-yamlish.

- Forwarded Message -
From: Tim Flink tfl...@redhat.com
To: infrastruct...@lists.fedoraproject.org
Sent: Thursday, April 9, 2015 11:42:43 PM
Subject: Freeze Break Request: Increase execution timeouts in taskotron's   
buildbot

As the set of packages in f22 has grown with freeze, some of the tasks
(most often depcheck) are not completing before hitting the default
timeout of 20 minutes for execution in buildbot.

I want to double the timeout for task execution from 20 to 40 minutes.
I've made the change in dev and stg already and the change works - long
tasks are no longer being killed prior to completion. This freeze break
is for production.

+1s?

Tim

diff --git 
a/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2 
b/roles/taskotron/bu
index d7a698f..1a63b0e 100644
--- a/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2
+++ b/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2
@@ -175,9 +175,7 @@ factory.addStep(ShellCommand(command=[runtask,
 Interpolate('%(prop:taskname)s.yml')],
  descriptionDone=[Interpolate('%(prop:taskname)s 
on %(prop:item)s')],
  name='runtask',
-{% if deployment_type in ['dev', 'stg'] %}
  timeout=2400,
-{% endif %}
  logfiles={'taskotron.log': {'filename': 
'/var/log/taskotron/taskotron.log',

___
infrastructure mailing list
infrastruct...@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: Could we disable AutoQA until it can produce usable results?

[Kamil Paral]

 
 Witness (a):
 https://admin.fedoraproject.org/updates/libguestfs-1.29.31-1.fc22
 
 The log file in this case is 3452 lines long.
 
 Witness (b):
 https://admin.fedoraproject.org/updates/ocaml-camlp4-4.02.0-0.9.git87c6a6b0.fc22
 
 The log file in this case is 3452 lines long (not coincidentally - it
 is exactly the same file as above).
 
 And in both cases the log files are a morass of randomness.

Sorry about this. Adam already noted how to search in those logs. Recently, we 
have implemented support for task artifacts, and split upgradepath results per 
build and per update, so you'll be able to see only the output relevant to you:
http://taskotron-dev.fedoraproject.org/artifacts/20150324/6a4b8934-d24c-11e4-8b08-52540073dd95/task_output/

We still don't have any way to present it to you easily (you need to know how 
to construct these arcane URLs), but we're working on it. And we still haven't 
implemented result splitting for depcheck yet.

In your exact case, this inferior architecture error message is a known 
irregularly occurring issue and it's a bug somewhere in depcheck or dependency 
solving stack, but we haven't been able to pinpoint it and fix it yet. Recently 
we have received some guidance from libsolv maintainer and we're trying to 
reproduce the problem with some debugging info. Unfortunately, it's quite 
difficult to reproduce.

Disabling the tests until they work flawlessly and we have a nice way of 
representing the results is on the table, if many maintainers think it's better 
than having at least some logs in some form. However, having this kind of 
feedback helps us to prioritize the tasks we work on.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: Exit status for runtask should be 1 if task outcome is FAILED?

[Kamil Paral]

 That's exactly why I thought about runtask. To be more precise, we're
 currently working on rpmgrill. It ships it's own fetch-build script
 which is currently tied to Fedoras Koji.

Just a note, IIRC you don't need just all the rpms, but also the build log for 
certain subtests. That's currently not provided by our koji directive, and I'm 
not sure if we'll want to add it or not. In such cases, you might need to 
implement some bits on your own (e.g. fetching the build log).

 
 IMHO rpmgrill shouldn't be concerned about where the builds come from.
 If I can use 'runtask' to fetch the builds, let rpmgrill handle the
 analysis and report back to bodhi or any other similar system that would
 be very beneficial. Furthermore that can run easily in a Jenkins and the
 exit status would provide a way for the CI to figure if the package is
 not shipping with nasty problems.

I see, so you want to use the exit status as a super-simple way to 
distinguishing pass/fail in Jenkins, instead of providing some more advanced 
structure that the CI understands (junit, etc). I think that's a valid request.

  1. We have other outcomes defined at the moment, like NEEDS_INSPECTION:
  https://docs.qadevel.cloud.fedoraproject.org/libtaskotron/latest/library.html#libtaskotron.check.CheckDetail
  What should happen in that case?
 Theoretically it'll be an exit code 1 as well. What I don't know
 tho is, if it's practical to change it to 1. Perhaps it would mean that
 currently all packages which are under test would basically 'fail'. You
 might have a better insight here what the consequences of that change
 would be.

I think NEEDS_INSPECTION would also be considered as failed, and the exit code 
would be a predefined value (e.g. 100), so that we can still distinguish 
execution errors from a task failure.

 
 
  2. Some tasks do not check just a single item. For example, depcheck and
  upgradepath check everything that is currently proposed for stable or
  testing updates. They can return a dozen test results. What should happen
  when there are mixed results?
 I'd assume to default to the worst result. So if any one of them has a
 a FAILED state it means exit with 1. What I'm not sure about is if it's
 currently easy to accumulate the worst result case with the current
 code. Might be easier thought than done.

We had a long discussion with Josef about this, and our conclusion is that it 
should be the script author who is in business of deciding what the overall 
result is. The easiest way to implement this seems to be to let the author 
define one additional result in TAP (the very last result in TAP), according to 
any logic he or she needs, and we will consider this last result to be the 
decision of what exit code to return.

So, if you have just a single result in TAP, it's the one that is used for exit 
code decision. If you have multiple of them, the last one is used.

This means you will *have to* return at least a basic TAP from rpmgrill 
(there's a perl library for that), or from the wrapper around rpmgrill (e.g. 
converting its json to TAP). But TAP itself is extremely simple, so it 
shouldn't be a problem doing it even manually, i.e. in tests written in bash 
(we don't support this yet, but plan it in the future). This is the simplest 
TAP we can consume (I just discovered a bug here, but I'll fix it):

TAP version 13
1..1
not ok

And this is the simplest version that makes sense if you want to report to 
resultsdb:

TAP version 13
1..1
not ok
  ---
  item: htop-1.0.3-4.fc21
  outcome: FAILED
  type: koji_build
  ...


 
 
  3. At the moment, it's technically possible to leave out the 'resultsdb'
  directive the from task recipe. I cannot imagine a use case for this,
  because in development mode, this only reports to stdout, but it's
  possible if you need it. If we wanted to inspect task outcomes, we need to
  do it from TAP, therefore in the 'resultsdb' directive. It might be
  confusing to people that if you commented out that directive, runtask
  would stop returning correct return codes.
 IMHO that's the beauty of Taskotron being decoupled from the execution
 and the result reporting. My main idea was just using libtaskotron for
 task execution and perhaps in the future to report to other aggregation
 systems inside Red Hat (the equivalent to Bodhi).
 
 So I thought from a developer point of view, it allows to run build
 analysers against your package in your CI using Taskotron without the
 need to report the results back into a results database.
 
 I hope I'm not twisting the intention of the whole Taskotron idea.

I was always thinking either about running tests in in the full taskotron 
environment (with resultsdb etc), or locally without any special needs (mainly 
for test development). But if you want to use libtaskotron as a runner in a 
different CI environment and we can satisfy your needs, I think that's great. 
And it will make sharing tests between Taskotron and internal Red Hat CI 
systems 

Re: Exit status for runtask should be 1 if task outcome is FAILED?

[Kamil Paral]

 Hi,
 
 I'd like to enquire if it's desired to let runtask exit with '1' if the
 task in question has a 'FAILED' outcome.
 
 Maybe there was a conscious decision of not providing this feature and
 I'd like to hear about it.
 
 I've joined Taskotron on phabricator. If there are no obvious
 resentments against such an idea, I'm happy to create a task and
 implement it.
 
 Kind Regards,
 --
 Róman Joost

Hey Róman,

thanks for joining the project and welcome :)

We have never really thought about the feature you're talking about. We 
currently return non-zero in case the execution fails - ctrl+c, an exception 
thrown. If we wanted to extend it to the FAILED task result, I see the 
following issues:

1. We have other outcomes defined at the moment, like NEEDS_INSPECTION:
https://docs.qadevel.cloud.fedoraproject.org/libtaskotron/latest/library.html#libtaskotron.check.CheckDetail
What should happen in that case?

2. Some tasks do not check just a single item. For example, depcheck and 
upgradepath check everything that is currently proposed for stable or testing 
updates. They can return a dozen test results. What should happen when there 
are mixed results?

3. At the moment, it's technically possible to leave out the 'resultsdb' 
directive the from task recipe. I cannot imagine a use case for this, because 
in development mode, this only reports to stdout, but it's possible if you need 
it. If we wanted to inspect task outcomes, we need to do it from TAP, therefore 
in the 'resultsdb' directive. It might be confusing to people that if you 
commented out that directive, runtask would stop returning correct return 
codes.

One further thought, instead of implementing this by default, we can also 
create a new directive or variable which will allow task creators to set the 
exit code, according to TAP or any other logic they want.

Why exactly do you need this? For local testing? runtask is just an execution 
wrapper, so you should be able to run your `./script some args` the same way as 
`runtask -i item -t type script.yml`. Except for some set up steps (like 'koji' 
or 'bodhi' directive), so if you use some of these, I can see a reason why you 
would want to run it through runtask.

Kamil
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

libtaskotron: /var/cache/taskotron dir added

[Kamil Paral]

Hello,

those of you who use libtaskotron from git, please note that libtaskotron now 
requires one more directory (/var/cache/taskotron) to exist [1]. From now on, 
you should be notified if you're missing any of the important directories right 
at 'runtask' execution start (the execution will exit in that case).

I have also added this directory into conf/tmpfiles.d/taskotron.conf, so if you 
use automatic pruning of taskotron directories, be sure to update this file in 
your system.

Cheers,
Kamil

[1] https://phab.qadevel.cloud.fedoraproject.org/D266
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: testdays is down again

[Kamil Paral]

  ...
  it's down :(
 
 Seems to be working for me. Could you describe down a bit more? :)

I've found the issue. It's this:
http://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security

testdays.qa.fedoraproject.org runs only on http, not https. But if you have 
visited https://qa.fedoraproject.org/ (e.g. 
https://qa.fedoraproject.org/blockerbugs/ ) recently, it sets this header for 
you:

  Strict-Transport-Security: max-age=15768000; includeSubDomains; preload

and then firefox automatically forces it for all subdomains, including 
testdays.qa.fedoraproject.org (for the next 6 months).

I guess the solution is to do some of these:
* set up testdays with https
* move it to a different domain
* don't include subdomains in the HSTS header

I have created a ticket in Phab:
https://phab.qadevel.cloud.fedoraproject.org/T434
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

[dev machines] script for cleaning up temporary taskotron files

[Kamil Paral]

Hello,

if you use libtaskotron in a developer environment (running some taskotron 
tasks locally from time to time), there is a new script that will make sure 
taskotron temporary files will get cleaned up regularly. Look into 
conf/tmpfiles.d/ in libtaskotron checkout.

This will probably get handy with the recent addition for task artifacts - you 
probably don't want to keep them around permanently on a developer machine.

Cheers,
Kamil
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: Coconut failures: use dl.fp.o not download.fp.o?

[Kamil Paral]

  Please note, though, that we think that the mirror issue is a
  legitimate bug, either caused by dnf stack, or anaconda. Jan
  reported it yesterday here:
  https://bugzilla.redhat.com/show_bug.cgi?id=1196164
  
  If this turns out to be true, it would also mean that OpenQA helped
  us uncover quite an interesting bug, which would be much harder to
  spot without the massive number of installations executed regularly.
  So there is also some value in using closest mirror as installation
  source.
 
 Yep, that's an interesting one for sure, and sorry, I should've been
 clearer: I was talking only about the tests where the actual test
 involves specifying a particular repository URL (via the GUI or a
 kickstart or on the cmdline or whatever), and we're currently using
 'download.fedoraproject.org' in those URLs.

OK, in that case, it completely makes sense. Moreover, I believe that using 
inst.repo=download.fp.o is explicitly forbidden and unsupported by anaconda (I 
would have to dig into past bug reports to find it). The problem is that 
anaconda does not resolve that once and then provide it to yum/dnf, but it uses 
it verbatim. Therefore that URL resolves itself on every request, potentially 
leading to different mirrors being used for metadata and packages, which can 
utterly break everything.

At least that's what I remember from the past, maybe now with dnf it might work 
a bit differently. In any case, we should avoid using inst.repo=download.fp.o 
by any means. It's potentially very error-prone.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: Coconut failure for TC5

[Kamil Paral]

 Hey folks! Anyone paying attention to the Project Coconut box logs
 might've noticed it failed when it tried to run against Alpha TC5. The
 problem is to do with the download URLs. I wrote fedfind to always use
 https://download.fedoraproject.org/ URLs for mirror HTTP downloads,
 but this turns out not to work great for TCs/RCs because of mirror
 lag, people (and Coconut) want the images immediately but mirrors
 don't pick them up for a few hours after the TC/RC is cut. 

I think the problem is different. (Almost) nobody mirrors TCs/RCs. There are 
about 3 mirrors in the whole world which follow stage/ directory:
http://mirrors.fedoraproject.org/mirrorlist?path=/pub/alt/stage

Also, you would need to ask infrastructure folks, but I don't think 
download.fp.o can be used for this. I think it redirects to a random close 
mirror without actually checking the path you are requesting. If you wanted to 
be sure that the file is present, you would need to make calls like these in 
advance:
http://mirrors.fedoraproject.org/mirrorlist?path=/pub/alt/stage/22_Alpha_TC4/Workstation/i386/iso/Fedora-Workstation-netinst-i386-22_Alpha_TC4.iso

Those calls are quite slow and probably expensive. And they change in time, you 
can't use them immediately after TC/RC is published. So, for anything under 
stage/, I believe using dl.fp.o is the best (and only) choice.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel