jpsalada opened a new issue #20143: URL: https://github.com/apache/incubator-mxnet/issues/20143
## Description Getting stack overflow in the file cudnn_cnn_infer64_8.dll when running inference on Windows 10 with MXNET 6.0+MKL+CUDA 10.1 cpp package, using GPU and IIS. The error will NOT occur if I run the same code and input from a console application. MXNET was compiled locally (CMakeCache.txt [here](https://justpaste.it/8y59u)). I am trying to run RetinaFace Res50 model. ### Error Message Unhandled exception at 0x00007FFD8F563D88 (cudnn_cnn_infer64_8.dll) in w3wp.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x0000002C7BFC3000). ## To Reproduce //load model and configure ctx with gpu mxnet::cpp::NDArray data(mxnet::cpp::Shape(batch_size, num_channels, height, width), ctx, false); data.SyncCopyFromCPU(img_data, batch_size * num_channels * height * width); data.WaitToRead(); args["data"] = data; Executor *exec = sym_net.SimpleBind(*Ctx, args, map<string, NDArray>(), map<string, OpReqType>(), aux); exec->Forward(false); vector<uint> cls_shape = exec->outputs[idx].GetShape(); uint sz = 1; int i = 0; while (i < cls_shape.size()) { sz *= cls_shape[i++]; } vector<float> cls_data(sz); //will crash in the next instruction exec->outputs[idx].SyncCopyToCPU((float*)&(cls_data[0]), static_cast<mx_uint>( sz)); `` ### Steps to reproduce 1. I have created a c++ dll which contains the above code. 2. The dll is called from a C# WebAPI deployed on Windows IIS. 3. The application will always crash in the same instruction as exemplified previously. ## What have you tried to solve it? 1. Since when I run the same code pipeline using a console application it does not crash, I checked with ProcessExplorer that both are using the same CUDA files 2. I also tried to replace my code with used in the example image_classification_predict.cc, and the same behavior occured when getting the output from the model 3. Make sure the image input arriving is okey and also there are not errors when loading the model. ## Environment <details> <summary>Environment Information</summary> ----------System Info---------- Platform : Windows-10-10.0.19041-SP0 system : Windows node : RM-UK-DT-0165 release : 10 version : 10.0.19041 ----------Hardware Info---------- machine : AMD64 processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel Name Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz Microsoft Remote Display Adapter NVIDIA GeForce RTX 2080 </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
