perdasilva commented on issue #13103: Flaky test test_gluon_rnn.test_layer_bidirectional URL: https://github.com/apache/incubator-mxnet/issues/13103#issuecomment-482495472 @haojin2 no good: ``` ====================================================================== FAIL: test_gluon_rnn.test_layer_bidirectional ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/work/mxnet/tests/python/unittest/common.py", line 110, in test_new orig_test(*args, **kwargs) File "/work/mxnet/tests/python/unittest/common.py", line 177, in test_new orig_test(*args, **kwargs) File "/work/mxnet/tests/python/unittest/test_gluon_rnn.py", line 283, in test_layer_bidirectional assert_allclose(net(data).asnumpy(), ref_net(data).asnumpy(), rtol=2e-7) File "/usr/local/lib/python3.5/dist-packages/numpy/testing/_private/utils.py", line 1452, in assert_allclose verbose=verbose, header=header, equal_nan=equal_nan) File "/usr/local/lib/python3.5/dist-packages/numpy/testing/_private/utils.py", line 789, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=2e-07, atol=0 (mismatch 0.06493506493507084%) x: array([0.424288, 0.560531, 0.600333, ..., 0.402131, 0.560952, 0.505039], dtype=float32) y: array([0.424288, 0.560531, 0.600333, ..., 0.402131, 0.560952, 0.505039], dtype=float32) -------------------- >> begin captured logging << -------------------- tests.python.unittest.common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1305130208 to reproduce. --------------------- >> end captured logging << --------------------- ---------------------------------------------------------------------- Ran 1 test in 0.030s ``` What I did to the test code: ``` # Added import statement from tests.python.unittest.common import with_seed # Added with_seed decorator to test function @with_seed() def test_layer_bidirectional(): # Update rtol as suggested to the assertion statement assert_allclose(net(data).asnumpy(), ref_net(data).asnumpy(), rtol=2e-7) ``` To test the changes, I have a g3.8xlarge instance with nvidia drivers 418 and nvidia-docker: ``` # On host $ docker run -ti -v `pwd`:/work/mxnet mxnetcd/build.ubuntu_cpu_static /bin/bash # Within container $ source tools/staticbuild/build.sh cu92mkl pip $ exit # On host $ docker run -ti --runtime=nvidia -v `pwd`:/work/mxnet mxnetcd/build.ubuntu_gpu_cu92 /bin/bash $ export PYTHONPATH=./python/ $ MXNET_TEST_COUNT=10000 nosetests --logging-level=DEBUG --verbose -s tests/python/unittest/test_gluon_rnn.py:test_layer_bidirectional ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
