reminisce commented on issue #10341: Deadlock during ThreadedEnginePerDevice 
destructor after CuDNNConvolutionOp<float>::SelectAlgo called.
URL: 
https://github.com/apache/incubator-mxnet/issues/10341#issuecomment-378504188
 
 
   Added some prints. It seems the `on_complete()` can never finish while 
`WaitForVar()` keeps waiting. Since this piece of code is running in a worker 
thread for gpu context, we don't need to push those operations into engine for 
execution.
   ```
   ubuntu@ip-172-31-34-217:~/unison/mxnet/gpu/tests/python/gpu$ 
MXNET_ENGINE_INFO=1 nosetests --verbose 
test_operator_gpu:test_op_output_names_monitor
   /home/ubuntu/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:34: 
FutureWarning: Conversion of the second argument of issubdtype from `float` to 
`np.floating` is deprecated. In future, it will be treated as `np.float64 == 
np.dtype(float).type`.
     from ._conv import register_converters as _register_converters
   
/home/ubuntu/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py:46:
 DeprecationWarning: OpenSSL.rand is deprecated - you should use os.urandom 
instead
     import OpenSSL.SSL
   /home/ubuntu/anaconda3/lib/python3.6/site-packages/nose/util.py:453: 
DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() 
or inspect.getfullargspec()
     inspect.getargspec(func)
   [INFO] Setting module np/mx/python random seeds, use 
MXNET_MODULE_SEED=532507193 to reproduce.
   [06:57:48] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc68271b8 thread id = 140626778085120
   [06:57:48] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc6823000 thread id = 140626778085120
   [06:57:48] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827000 thread id = 140626778085120
   [06:57:48] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc6823028 thread id = 140626778085120
   [06:57:48] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc68271b8 thread id = 140626778085120
   test_operator_gpu.test_op_output_names_monitor ... [06:57:48] 
src/engine/threaded_engine.cc:440: npending = 2 thread id = 140625260009216
   [06:57:48] src/engine/threaded_engine.cc:440: npending = 1 thread id = 
140625260009216
   [06:57:48] src/engine/threaded_engine.cc:440: npending = 0 thread id = 
140625260009216
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827210 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc6823050 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine_perdevice.cc:186: Pushing opr_block to 
execute: 0x561dc6823050 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:440: npending = 0 thread id = 
140624745461504
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827268 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc6823050 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine_perdevice.cc:186: Pushing opr_block to 
execute: 0x561dc6823050 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:440: npending = 0 thread id = 
140625089599232
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc68272c0 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc6823050 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine_perdevice.cc:186: Pushing opr_block to 
execute: 0x561dc6823050 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:440: npending = 0 thread id = 
140624745461504
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827210 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827268 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc68272c0 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827318 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827370 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc6823050 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:440: npending = 0 thread id = 
140625260009216
   [06:57:49] src/engine/threaded_engine.cc:295: Adding read dependency: 
0x561dc6827210 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:295: Adding read dependency: 
0x561dc6827268 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:295: Adding read dependency: 
0x561dc68272c0 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827318 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827370 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc6823050 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine_perdevice.cc:186: Pushing opr_block to 
execute: 0x561dc6823050 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827210 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827268 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc68272c0 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827318 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827370 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827210 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827268 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc68272c0 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827318 thread id = 140626778085120
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827370 thread id = 140626778085120
   ok
   
   ----------------------------------------------------------------------
   Ran 1 test in 1.348s
   
   OK
   [06:57:49] src/operator/nn/./cudnn/cudnn_convolution-inl.h:528: CUDNN conv 
new var: 0x561dc6827528 thread id = 140625089599232
   [06:57:49] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827528 thread id = 140625089599232
   [06:57:49] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc6823078 thread id = 140625089599232
   [06:57:49] src/engine/threaded_engine_perdevice.cc:186: Pushing opr_block to 
execute: 0x561dc6823078 thread id = 140625089599232
   [06:57:49] src/engine/threaded_engine.cc:364: Wait for 0x561dc6827528 thread 
id = 140625089599232
   [06:57:49] src/engine/threaded_engine.cc:295: Adding read dependency: 
0x561dc6827528 thread id = 140625089599232
   [06:57:49] src/engine/threaded_engine.cc:415: Complete write dep for 
0x561dc6827528 thread id = 140624745461504
   [06:57:49] src/engine/threaded_engine.cc:420: PushToExecute 0x561dc68230a0 
thread id = 140624745461504
   [06:57:49] src/engine/threaded_engine.cc:425[06:57:49] 
src/engine/./threaded_engine.h:351: ExecuteOprBlock 
0x561dc68230a0shutdown_phase=: Fin PushToExecute 0x561dc68230a0 thread id = 
140624745461504
   1
   [06:57:49] src/engine/threaded_engine.cc:440: npending = 4 thread id = 
140624745461504
   [06:57:49] src/engine/threaded_engine.cc:440: npending = 3 thread id = 
140625260009216
   [06:57:50] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827000 thread id = 140626778085120
   [06:57:50] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc68230a0 thread id = 140626778085120
   [06:57:50] src/engine/./threaded_engine.h:351: ExecuteOprBlock 
0x561dc68230a0shutdown_phase=1
   [06:57:50] src/engine/threaded_engine.cc:440: npending = 3 thread id = 
140626778085120
   [06:57:50] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827058 thread id = 140626778085120
   [06:57:50] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc68230a0 thread id = 140626778085120
   [06:57:50] src/engine/./threaded_engine.h:351: ExecuteOprBlock 
0x561dc68230a0shutdown_phase=1
   [06:57:50] src/engine/threaded_engine.cc:440: npending = 3 thread id = 
140626778085120
   [06:57:50] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc68270b0 thread id = 140626778085120
   [06:57:50] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc68230a0 thread id = 140626778085120
   [06:57:50] src/engine/./threaded_engine.h:351: ExecuteOprBlock 
0x561dc68230a0shutdown_phase=1
   [06:57:50] src/engine/threaded_engine.cc:440: npending = 3 thread id = 
140626778085120
   [06:57:50] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827108 thread id = 140626778085120
   [06:57:50] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc68230a0 thread id = 140626778085120
   [06:57:50] src/engine/./threaded_engine.h:351: ExecuteOprBlock 
0x561dc68230a0shutdown_phase=1
   [06:57:50] src/engine/threaded_engine.cc:440: npending = 3 thread id = 
140626778085120
   [06:57:50] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827160 thread id = 140626778085120
   [06:57:50] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc68230a0 thread id = 140626778085120
   [06:57:50] src/engine/./threaded_engine.h:351: ExecuteOprBlock 
0x561dc68230a0shutdown_phase=1
   [06:57:50] src/engine/threaded_engine.cc:440: npending = 3 thread id = 
140626778085120
   [06:57:50] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc68271b8 thread id = 140626778085120
   [06:57:50] src/engine/threaded_engine.cc:304: Pusing opr_block to execute: 
0x561dc68230a0 thread id = 140626778085120
   [06:57:50] src/engine/./threaded_engine.h:351: ExecuteOprBlock 
0x561dc68230a0shutdown_phase=1
   [06:57:50] src/engine/threaded_engine.cc:440: npending = 3 thread id = 
140626778085120
   [06:57:50] src/engine/threaded_engine.cc:300: Adding write dependency: 
0x561dc6827318 thread id = 140626778085120
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Reply via email to