As I've been working on the email reduction feature, I've been rather frustrated with how difficult it is to test AutoQA as a framework. I've written scripts to make testing depcheck (minus commenting and build gathering) easier but I have yet to find a reasonable way to exercise the whole AutoQA stack other than running jobs on my dev system with comments disabled and hope that they finish before production does.
Before I go farther, I realize that what I'm discussing would be a non-trivial amount of work and I'm kind of thinking that it would be a little too much for 0.5.0. Then again, more testing could help improve the release. Thoughts on that part would be welcome. I can think of two methods that we could use to make testing easier and start down the road of retiring our mascot: - Refactor bodhi_utils and koji_utils so that they can be stubbed out in testing - Find an alternative to the production instances of koji and bodhi Either way, setting up the conditions for self-tests (making mock rpms, setting up the metadata, creating repos) is going to be a decent amount of work and will likely end up with fragile, high maintenance tests if we go beyond basic smoke tests. There were concerns over the refactoring I proposed for koji_utils a while back so I assume that isn't the way we want to proceed for now. I looked into setting up test instances of bodhi and koji but I'm rather intimidated by the amount of maintenance and hacking that would be required to induce the conditions that we would need for more complete testing on a regular basis. With this in mind, I started hacking at a mockup for replacing bodhi and koji for testing purposes that does nothing more than implement the parts of those interfaces that we're using in AutoQA. It took a little while to reverse engineer the interfaces to bodhi and koji but I do have some code that is able to fool the koji client with hard-coded results (haven't tried it with AutoQA yet nor have I gotten very far on mocking up bodhi). I can send out code if anyone wants to see it (kind of ugly ATM, though). While finishing the mockup would also be non-trivial, I think that it will be more manageable and flexible in the long run in addition to being better suited to our needs since we can add in testability features as needed without having to bug other teams about features that other people may never use. Anyhow, thoughts on the concepts? On the timing (try to have some for 0.5.0 or not)? Anything that I didn't consider? Tim
signature.asc
Description: OpenPGP digital signature
_______________________________________________ autoqa-devel mailing list [email protected] https://fedorahosted.org/mailman/listinfo/autoqa-devel
