haojin2 commented on issue #14902: flaky test: test_bilinear_resize_op URL: https://github.com/apache/incubator-mxnet/issues/14902#issuecomment-494174441 Another failure: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-14959/12/pipeline Seems like the error margin is pretty small with a very small atol: ``` ====================================================================== FAIL: test_operator_gpu.test_bilinear_resize_op ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\nose\util.py", line 620, in newfunc return func(*arg, **kw) File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\gpu\../unittest\common.py", line 177, in test_new orig_test(*args, **kwargs) File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\gpu\../unittest\test_operator.py", line 7498, in test_bilinear_resize_op check_bilinear_resize_modes_op(shape_0, shape_1=shape_1, mode='like') File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\gpu\../unittest\test_operator.py", line 7472, in check_bilinear_resize_modes_op check_numeric_gradient(resize_sym, [data_np, date_np_like]) File "C:\jenkins_slave\workspace\ut-python-gpu\windows_package\python\mxnet\test_utils.py", line 980, in check_numeric_gradient ("NUMERICAL_%s"%name, "BACKWARD_%s"%name)) File "C:\jenkins_slave\workspace\ut-python-gpu\windows_package\python\mxnet\test_utils.py", line 503, in assert_almost_equal raise AssertionError(msg) AssertionError: Items are not equal: Error 1.004549 exceeds tolerance rtol=0.010000, atol=0.000000. Location of maximum error:(1, 0, 12, 12), a=0.004321, b=0.004278 NUMERICAL_data: array([[[[0.84230304, 0. , 0.57831407, ..., 0.1641959 , 0. , 0.331074 ], [0. , 0. , 0. , ..., 0. ,... BACKWARD_data: array([[[[0.84230524, 0. , 0.5783209 , ..., 0.16419744, 0. , 0.33107594], [0. , 0. , 0. , ..., 0. ,... -------------------- >> begin captured logging << -------------------- common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1608036320 to reproduce. --------------------- >> end captured logging << --------------------- ``` Experimenting with bumping up the atol to a reasonable level now, please don't hurry to disable the test for now.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
