AbdealiJK added a comment.
I ran the tests using pytest-attrib as mentioned in the gerrit comments, and found the number of tests being run on the nosetests and pytest were different, where the pytest builds always found more tests than the nose one. PyTest build - https://travis-ci.org/AbdealiJK/pywikibot-core/builds/131373171 Nose build - https://travis-ci.org/AbdealiJK/pywikibot-core/builds/131404927 To be able to compare which tests were being not collected, I used the `nose-ignore-docstring` plugin to give the name of the function rather than the docstring. I wrote the script http://pastebin.com/FnMQThYK which uses regex to find all the tests that were collected and run in the pytest and nose travis logs. When doing this, I found a lot of tests which inherit from `DefaultSiteTestCase` but were not being run by the SITE_ONLY=1 jobs in nose. This seems like a bug in nose, as the `language` and `code` attributes are set correctly, and only some of the subclasses of `DefaultSiteTestCase` are not collected, a lot of them are collected correctly. Examples of subclasses of `DefaultSiteTestCase` which are collected by pytest but not by nose: - pagegenerators_tests:TestDayPageGenerator - pagegenerators_tests:LiveRCPageGeneratorTestCase - pagegenerators_tests:PageGeneratorIntersectTestCase - pagegenerators_tests:TestDequePreloadingGenerator - pagegenerators_tests:TestFactoryGenerator - pagegenerators_tests:TestLogeventsFactoryGenerator - patrolbot_tests:TestPatrolBot Other than this, there were also some tests that were collected by nosetest, but not by pytest: - site_tests:TestPagePreloading - script_tests:TestScriptSimulate (This is only skipped in Job 18, but is found correctly in other jobs) - site_tests:TestSiteGenerators.test_allpages_langlinks_enabled I've created appropriate issues at https://github.com/AbdealiJK/pytest-attrib/issues/2 and https://github.com/AbdealiJK/pytest-attrib/issues/1 TASK DETAIL https://phabricator.wikimedia.org/T135659 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AbdealiJK Cc: gerritbot, AbdealiJK, Aklapper, Zppix, jayvdb, pywikibot-bugs-list, Lewizho99, Maathavan, Jay8g, Krenair _______________________________________________ pywikibot-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
