gkyh0805 opened a new issue #10519: Can too many connection for the symbol make 
unintended computation?
URL: https://github.com/apache/incubator-mxnet/issues/10519
 
 
   
   when too many connection for the single symbol occur, can the some grads for 
relu_new_1 be skipped due to the MXNET_EXEC_INPLACE_GRAD_SUM_CAP?
   
   I saw "graph_executor.cc" and there is a comment
   [
           // Note: For a symbol like the following:
           // data = mx.sym.Variable('data')
           // sym = data + data + data + data + data + data + data
           // the node entries v passed in here are of the same node of
           // op _identity_with_attr_like_rhs. We should skip adding a node
           // to its own control_deps.
   ]
   
   I cannot understand the command perfectly, however, relu_new_1 in the 
following code is duplicated a lots and may have a dependency each other.
   
   Then, can the gradient be different according to 
MXNET_EXEC_INPLACE_GRAD_SUM_CAP?
   (the small connection accept all duplicate gradient and the many connection 
choose only one gradient among them?)
   
   In my case, the usage of the identity symbol for reducing the direct 
connection show the quite different training loss even though the seed is fixed.
   
   How does this code execute in mxnet through the graph executor?
   
   ```python
   conv_new_1 = mx.sym.Convolution(data=relu1, kernel=(1, 1), num_filter=1024, 
name="conv_new_1", lr_mult=3.0)
   relu_new_1 = mx.sym.Activation(data=conv_new_1, act_type='relu', 
name='relu_new_1')
   
   sym1 = mx.sym.Convolution(data=relu_new_1, kernel=(1, 1), num_filter=1024, 
name="sym1")
   sym2 = mx.sym.Convolution(data=relu_new_1, kernel=(1, 1), num_filter=1024, 
name="sym2")
   ...
   symn = mx.sym.Convolution(data=relu_new_1, kernel=(1, 1), num_filter=1024, 
name="symn")
   
   syma1 = relu_new_1 + sym1
   syma2 = relu_new_1 + sym2
   ...
   syman = relu_new_1 + symn
   
   final_out = syma1 + syma2 + ... + syman
   ```
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Reply via email to