Hello fellow MXNetters
We've seen that the subgraph execution mechanism that is used to run things
like the foreach operator causes MXExecutorForward to block, instead of just
issuing the ops in the normal asynchronous way
(https://github.com/apache/incubator-mxnet/blob/212364b0cba28aeda989378f6e630f7a61749bf3/src/executor/graph_executor.cc#L1352).
On its own this is a surprising fact that can lead to some issues if you're
not expecting it, like your time being spent in MXExecutorForward instead of
WaitAll / WaitRead . Is there a reason that this process isn't just
automatically done on a separate thread for you? Is it to ensure that
subsequent ops on the original thread are correctly serialized wrt the ops
produced by the foreach?
More importantly, this has the unfortunate implication that if you are using
multi-device parallelism with foreach, by just looping over your executors and
calling Forward on them, you will inadvertently serialize much of the
computation: you can't call Forward on the second executor until Forward on the
first executor has returned, and the foreach causes that first Forward call to
block until the forward pass is (mostly) done!
So it kills multi-device parallelism unless one starts making thread pools so
that the one can 'unblock' Forward (and probably the subsequent Backward) and
have each device's Forward being run in a separate thread.
Is this intended? Are we missing something about how you are supposed to use
subgraphs in conjunction with multi-device parallelism? It seems like a
weakness in the current design of subgraph execution. It also appears that the
python API doesn't have any strategy to deal with this issue, as you can see on
https://github.com/apache/incubator-mxnet/blob/2276bb0e30b1fe601eb288cb4f1b673484892d4b/python/mxnet/executor_manager.py#L281,
it's not making separate threads or anything there.
Thanks!
Tali + Sebastian
