Hi folks,
I need help debugging a strange test failure in pylint-django that started happening after the upgrade to pylint/astroid 2.0.

For all I can tell we've already adjusted the code and this failure is happening randomly. pylint-django's test used uses the test_functional.LintModuleTest class from pylint (I know it isn't public interface) and executes against input files comparing the linter results with expected.

The error that we get is:

E           Failed: Wrong results for file "func_noerror_foreignkeys":
E
E           Unexpected in testdata:
E             27: no-member

The strange thing is that on one test job in Travis this passed with Python 3.4 and Django 2.0 (see job #447.3):
https://travis-ci.org/PyCQA/pylint-django/builds/407129036

and on the next commit this failed on Python 3.4/Django 2.0 (see #448.3):
https://travis-ci.org/PyCQA/pylint-django/builds/407800899

however it passed on 3.5 + Django 2.0.

The two test jobs use the same code (minor differences in commits due to squash/conflicts).

When the linter is executed manually against the offending file it produces a 10/10 score.


If I modify the test suite like so:

--- a/pylint_django/tests/test_func.py
+++ b/pylint_django/tests/test_func.py
@@ -43,6 +43,8 @@ def get_tests(input_dir='input', sort=False):

     suite = []
     for fname in os.listdir(input_dir):
+        if fname.find('func_noerror_foreign') == -1:
+            continue
         if fname != '__init__.py' and fname.endswith('.py'):
             suite.append(test_functional.FunctionalTestFile(input_dir, fname))

then I get only a subset of the tests executed (including the offending one) and this time it passes.


Anyone seen such erratic behaviour ? I think this is a flaky test but I have no idea how to approach debugging that.


For the record this is blocking us from releasing a new version of pylint-django so any help is appreciated.

--
Alex

_______________________________________________
code-quality mailing list
code-quality@python.org
https://mail.python.org/mailman/listinfo/code-quality

Reply via email to