Hello, I'm currently working on auto scaling and encountering a consistent test failure on CPU. At the moment, I'm not really sure what's causing this, considering the setup should be identical.
http://jenkins.mxnet-ci-dev.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/ci-master/557/pipeline/694 ====================================================================== FAIL: test_sparse_operator.test_sparse_mathematical_core ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/work/mxnet/tests/python/unittest/common.py", line 157, in test_new orig_test(*args, **kwargs) File "/work/mxnet/tests/python/unittest/test_sparse_operator.py", line 1084, in test_sparse_mathematical_core density=density, ograd_density=ograd_density) File "/work/mxnet/tests/python/unittest/test_sparse_operator.py", line 1056, in check_mathematical_core density=density, ograd_density=ograd_density) File "/work/mxnet/tests/python/unittest/test_sparse_operator.py", line 698, in check_sparse_mathematical_core assert_almost_equal(arr_grad, input_grad, equal_nan=True) File "/work/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal raise AssertionError(msg) AssertionError: Items are not equal: Error nan exceeds tolerance rtol=0.000010, atol=0.000000. Location of maximum error:(0, 0), a=inf, b=-inf a: array([[inf], [inf], [inf],... b: array([[-inf], [-inf], [-inf],... -------------------- >> begin captured stdout << --------------------- pass 0 0.0, 0.0, False --------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=2103230797 to reproduce. --------------------- >> end captured logging << --------------------- Does this ring any bells? Thanks in advance! -Marco
