On Mon, May 15, 2023 at 10:23 AM Roy Smith <r...@panix.com> wrote: > > The past few days, I've worked on T336630 and T336624, both of which describe > failures I was seeing which other people weren't. In both cases, it turned > out to be caused by something in my user-config.py. Many of these tests also > rely on external resources (i.e. live testing against testwiki or metawiki). > I come from a shop where our test environment was hermetic. That eliminated > this kind of flakiness due to environmental differences. It's a great way to > work, but I recognize that's not what we've got here, and not easy to get to > that point.
I want to point out that there should remain a difference between unit tests and integration tests. For integration tests, I believe it's a great idea to actually spawn a mediawiki instance and check how a page has been affected by various pywikibot API calls. However, that's a massive overkill for unit tests. Here's how I would write such tests for pywikibot API: mock the internal API that calls mediawiki API, and assert on requests to have certain parameters and provide hardcoded responses to these requests. This way, if a unit test fails, you know it's Pywikibot's code that is to blame and not something like a MediaWiki change or the environment setup. Though, having tests depend on user-supplied user-config.py is just ... broken. Unless the goal is to be able to test the user-supplied user-config.py, that is. YiFei Zhu > Given all that, any suggestions on how I should set up a dev environment on > my local machine to minimize this kind of problem in the future? > > Also, once I figured out what was going on, I closed T336630 as "declined". > Was that the right state for "I'm not going to do any more work on this > because it turned out to be a problem with the test environment". Should I > have used "invalid"? Something else? > _______________________________________________ > pywikibot mailing list -- pywikibot@lists.wikimedia.org > Public archives at > https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/message/ABX4WN5W6YBTAJXBNFX7H3VFRZ6ECXRH/ > To unsubscribe send an email to pywikibot-le...@lists.wikimedia.org _______________________________________________ pywikibot mailing list -- pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/message/KYSAZCFYRB7NGXG4USODZOBEXPRYOIIG/ To unsubscribe send an email to pywikibot-le...@lists.wikimedia.org
