[GitHub] zheng-da commented on issue #9820: Flaky test_gluon_model_zoo_gpu.test_training @ Python3: MKLDNN-GPU
zheng-da commented on issue #9820: Flaky test_gluon_model_zoo_gpu.test_training @ Python3: MKLDNN-GPU URL: https://github.com/apache/incubator-mxnet/issues/9820#issuecomment-367791248 After one day of testing, I think I know where the problem is now. Basically, the current implementation sometimes needs to convert data format (from the MKLDNN format to the default format) inside an NDArray. In the threaded execution engine, while an NDArray is being converted in a thread, the array can also be read by another thread. In this case, the other thread can read wrong data. I changed the code to avoid data layout conversion inside an NDArray. It seems to work fine now. I'll have more tests. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zheng-da commented on issue #9820: Flaky test_gluon_model_zoo_gpu.test_training @ Python3: MKLDNN-GPU
zheng-da commented on issue #9820: Flaky test_gluon_model_zoo_gpu.test_training @ Python3: MKLDNN-GPU URL: https://github.com/apache/incubator-mxnet/issues/9820#issuecomment-367488040 Most likely, what happens is that an operation needs to wait for some variables before proceeding to the next operator, but somehow the implementation doesn't wait for the right variables or misses some variables. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zheng-da commented on issue #9820: Flaky test_gluon_model_zoo_gpu.test_training @ Python3: MKLDNN-GPU
zheng-da commented on issue #9820: Flaky test_gluon_model_zoo_gpu.test_training @ Python3: MKLDNN-GPU URL: https://github.com/apache/incubator-mxnet/issues/9820#issuecomment-367486176 I run more tests. The bug occurs when MKLDNN is compiled into MXNet and we use the threaded engine, which is the default executing engine. If we use the naive executing engine, the problem doesn't show up. It means that the bug isn't in the operator implementation. It should be in the executor engine or in the NDArray. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zheng-da commented on issue #9820: Flaky test_gluon_model_zoo_gpu.test_training @ Python3: MKLDNN-GPU
zheng-da commented on issue #9820: Flaky test_gluon_model_zoo_gpu.test_training @ Python3: MKLDNN-GPU URL: https://github.com/apache/incubator-mxnet/issues/9820#issuecomment-367414057 @marcoabreu I was testing it yesterday. It seems the problem exists when MKLDNN is compiled into MXNet, so it should be a bug caused by the code inside USE_MKLDNN. I'll look into the problem this week and try to fix it as soon as possible. I agree with @pengzhao-intel . This problem might be related to the other flaky failures. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
