access2rohit opened a new issue #17397: log_softmax for large tensor is flaky URL: https://github.com/apache/incubator-mxnet/issues/17397 ## Description log_softmax fails randomly when using very large tensors. ## Occurrences *** Maximum errors for vector of size 5000000000: rtol=0.0001, atol=1e-20 1: Error 228.818179 Location of error: (1, 69287122), EXPECTED_log_softmax0_output=-17.01618748, FORWARD_log_softmax0_output=-16.63553619 2: Error 228.818174 Location of error: (1, 66902942), EXPECTED_log_softmax0_output=-17.01618358, FORWARD_log_softmax0_output=-16.63553238 3: Error 228.818163 Location of error: (1, 42527178), EXPECTED_log_softmax0_output=-17.01618356, FORWARD_log_softmax0_output=-16.63553238 4: Error 228.818160 Location of error: (1, 36123476), EXPECTED_log_softmax0_output=-17.01618356, FORWARD_log_softmax0_output=-16.63553238 5: Error 228.818143 Location of error: (1, 52855011), EXPECTED_log_softmax0_output=-17.01618353, FORWARD_log_softmax0_output=-16.63553238 6: Error 228.818141 Location of error: (1, 16069295), EXPECTED_log_softmax0_output=-17.01618353, FORWARD_log_softmax0_output=-16.63553238 7: Error 228.818126 Location of error: (1, 3578168), EXPECTED_log_softmax0_output=-17.01618546, FORWARD_log_softmax0_output=-16.63553429 8: Error 228.818122 Location of error: (1, 37536126), EXPECTED_log_softmax0_output=-17.01618349, FORWARD_log_softmax0_output=-16.63553238 9: Error 228.818113 Location of error: (1, 61821668), EXPECTED_log_softmax0_output=-17.01618737, FORWARD_log_softmax0_output=-16.63553619 10: Error 228.818111 Location of error: (1, 42215697), EXPECTED_log_softmax0_output=-17.01618737, FORWARD_log_softmax0_output=-16.63553619 FAIL ====================================================================== FAIL: test_large_array.test_nn ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/ubuntu/incubator-mxnet/tests/nightly/test_large_array.py", line 447, in test_nn check_log_softmax() File "/home/ubuntu/incubator-mxnet/tests/nightly/test_large_array.py", line 138, in check_log_softmax check_symbolic_forward(sym, [data], [np.log(np_softmax(data, axis=axis)+1e-20)]) File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1202, in check_symbolic_forward equal_nan=equal_nan) File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 627, in assert_almost_equal raise AssertionError(msg) AssertionError: Items are not equal: Error 228.818179 exceeds tolerance rtol=1.000000e-04, atol=1.000000e-20 (mismatch at least 0.000000%). Location of maximum error: (1, 69287122), EXPECTED_log_softmax0_output=-17.01618748, FORWARD_log_softmax0_output=-16.63553619 ACTUAL: array([[-17.9350638 , -17.96627981, -19.72406303, ..., -19.82472448, -17.56042908, -18.59646628], [-20.59984509, -18.51675027, -19.10637481, ..., -19.50312141,... DESIRED: array([[-17.554623, -17.585838, -19.343622, ..., -19.444284, -17.179989, -18.216026], [-20.219194, -18.1361 , -18.725725, ..., -19.12247 , -19.1559 ,... ---------------------------------------------------------------------- Ran 1 test in 3267.809s
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
