ptrendx commented on issue #18734: URL: https://github.com/apache/incubator-mxnet/issues/18734#issuecomment-703261686
It's hard to answer exactly why you see this without knowing your training script (or at least the part before you start other processes). That said, the fact that you get the error during NDArrayFree suggests that before you fork you created some NDArrays on the GPU. Maybe the issue is intermittent because it only tries to get rid of those particular NDArrays during garbage collection in Python? The error itself happens in the child process and I don't believe it should happen in the parent process so as long as you do not need to do anything cuda related in the child processes you should be ok I think. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
