rayjs opened a new issue #12894: Training crash SSD with LeakyReLU(rrelu) URL: https://github.com/apache/incubator-mxnet/issues/12894 ## Description Training SSD networks with LeakyReLU (rrelu) activation causes the training to crash. I have tried different networks and vgg16_reduced.py as well. It always crashes ## Environment info (Required) ``` ----------Python Info---------- ('Version :', '2.7.12') ('Compiler :', 'GCC 5.4.0 20160609') ('Build :', ('default', 'Dec 4 2017 14:50:18')) ('Arch :', ('64bit', 'ELF')) ------------Pip Info----------- ('Version :', '18.1') ('Directory :', '/usr/local/lib/python2.7/dist-packages/pip') ----------MXNet Info----------- ('Version :', '1.3.1') ('Directory :', '/home/xx/Documents/extern_libs/incubator-mxnet/python/mxnet') Hashtag not found. Not installed from pre-built package. ----------System Info---------- ('Platform :', 'Linux-4.4.0-137-generic-x86_64-with-Ubuntu-16.04-xenial') ('system :', 'Linux') ('node :', 'et2') ('release :', '4.4.0-137-generic') ('version :', '#163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018') ----------Hardware Info---------- ('machine :', 'x86_64') ('processor :', 'x86_64') Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 94 Model name: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz Stepping: 3 CPU MHz: 3377.625 CPU max MHz: 3600.0000 CPU min MHz: 800.0000 BogoMIPS: 6383.90 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 6144K NUMA node0 CPU(s): 0-3 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch invpcid_single intel_pt ssbd ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp flush_l1d ----------Network Test---------- Setting timeout: 10 Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.1568 sec, LOAD: 1.4836 sec. Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.1446 sec, LOAD: 2.0502 sec. Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.1938 sec,LOAD: 1.3499 sec. Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.1528 sec, LOAD: 0.1908 sec. Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.2012 sec, LOAD: 0.0508 sec. Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.3913 sec, LOAD: 0.8876 sec. ``` Package used (Python/R/Scala/Julia): Python ## Build info (Required if built from source) Compiler (gcc/clang/mingw/visual studio): gcc MXNet commit hash: 74638105f5480349cf57cda40a37475d626dbf41 Build config: make -j4 USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 ## Error Message: ``` [16:24:25] src/io/iter_image_det_recordio.cc:283: ImageDetRecordIOParser: /home/sj/Documents/extern_libs/incubator-mxnet/example/ssd/data/train.rec, use 3 threads for decoding.. [16:24:26] src/io/iter_image_det_recordio.cc:340: ImageDetRecordIOParser: /home/sj/Documents/extern_libs/incubator-mxnet/example/ssd/data/train.rec, label padding width: 350 [16:24:26] src/io/iter_image_det_recordio.cc:283: ImageDetRecordIOParser: /home/sj/Documents/extern_libs/incubator-mxnet/example/ssd/data/val.rec, use 3 threads for decoding.. [16:24:26] src/io/iter_image_det_recordio.cc:340: ImageDetRecordIOParser: /home/sj/Documents/extern_libs/incubator-mxnet/example/ssd/data/val.rec, label padding width: 350 [<Symbol relu4_3>, <Symbol relu7>, <Symbol multi_feat_2_conv_3x3_relu>, <Symbol multi_feat_3_conv_3x3_relu>, <Symbol multi_feat_4_conv_3x3_relu>, <Symbol multi_feat_5_conv_3x3_relu>] <Symbol relu4_3> <Symbol relu7> <Symbol multi_feat_2_conv_3x3_relu> <Symbol multi_feat_3_conv_3x3_relu> <Symbol multi_feat_4_conv_3x3_relu> <Symbol multi_feat_5_conv_3x3_relu> INFO:root:Experimental: start training from scratch with (gpu(0),gpu(1)) [16:24:32] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:109: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable) Traceback (most recent call last): File "train.py", line 156, in <module> voc07_metric=args.use_voc07_metric) File "/home/xx/Documents/extern_libs/incubator-mxnet/example/ssd/train/train_net.py", line 301, in train_net monitor=monitor) File "/home/xx/Documents/extern_libs/incubator-mxnet/python/mxnet/module/base_module.py", line 539, in fit self.update_metric(eval_metric, data_batch.label) File "/home/xx/Documents/extern_libs/incubator-mxnet/python/mxnet/module/module.py", line 773, in update_metric self._exec_group.update_metric(eval_metric, labels, pre_sliced) File "/home/xx/Documents/extern_libs/incubator-mxnet/python/mxnet/module/executor_group.py", line 639, in update_metric eval_metric.update_dict(labels_, preds) File "/home/xx/Documents/extern_libs/incubator-mxnet/python/mxnet/metric.py", line 132, in update_dict self.update(label, pred) File "/home/xx/Documents/extern_libs/incubator-mxnet/example/ssd/train/metric.py", line 48, in update cls_prob = preds[0].asnumpy() File "/home/xx/Documents/extern_libs/incubator-mxnet/python/mxnet/ndarray/ndarray.py", line 1980, in asnumpy ctypes.c_size_t(data.size))) File "/home/xx/Documents/extern_libs/incubator-mxnet/python/mxnet/base.py", line 252, in check_call raise MXNetError(py_str(_LIB.MXGetLastError())) mxnet.base.MXNetError: [16:24:32] include/mxnet/././resource.h:155: Check failed: req.type == ResourceRequest::kTempSpace (42955292 vs. 1) Stack trace returned 10 entries: [bt] (0) /home/xx/Documents/extern_libs/incubator-mxnet//lib/libmxnet.so(dmlc::StackTrace[abi:cxx11]()+0x5b) [0x7f98868b8fdb] [bt] (1) /home/xx/Documents/extern_libs/incubator-mxnet//lib/libmxnet.so(mshadow::Tensor<mshadow::gpu, 1, unsigned int> mxnet::Resource::get_space_typed<mshadow::gpu, 1, unsigned int>(mshadow::Shape<1>, mshadow::Stream<mshadow::gpu>*) const+0x6c5) [0x7f9889b02f35] [bt] (2) /home/xx/Documents/extern_libs/incubator-mxnet//lib/libmxnet.so(mxnet::op::LeakyReLUOp<mshadow::gpu, float>::Forward(mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)+0x5bc) [0x7f988b936e0c] [bt] (3) /home/xx/Documents/extern_libs/incubator-mxnet//lib/libmxnet.so(mxnet::op::OperatorState::Forward(mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)+0x363) [0x7f98892bdbc3] [bt] (4) /home/xx/Documents/extern_libs/incubator-mxnet//lib/libmxnet.so(mxnet::exec::StatefulComputeExecutor::Run(mxnet::RunContext, bool)+0x59) [0x7f9889a0c829] [bt] (5) /home/xx/Documents/extern_libs/incubator-mxnet//lib/libmxnet.so(+0x3e29526) [0x7f98899d7526] [bt] (6) /home/xx/Documents/extern_libs/incubator-mxnet//lib/libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0x8f5) [0x7f9889926045] [bt] (7) /home/xx/Documents/extern_libs/incubator-mxnet//lib/libmxnet.so(void mxnet::engine::ThreadedEnginePerDevice::GPUWorker<(dmlc::ConcurrentQueueType)0>(mxnet::Context, bool, mxnet::engine::ThreadedEnginePerDevice::ThreadWorkerBlock<(dmlc::ConcurrentQueueType)0>*, std::shared_ptr<dmlc::ManualEvent> const&)+0xeb) [0x7f988993c78b] [bt] (8) /home/sj/Documents/extern_libs/incubator-mxnet//lib/libmxnet.so(std::_Function_handler<void (std::shared_ptr<dmlc::ManualEvent>), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*, bool)::{lambda()#4}::operator()() const::{lambda(std::shared_ptr<dmlc::ManualEvent>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<dmlc::ManualEvent>&&)+0x4e) [0x7f988993c9fe] [bt] (9) /home/sj/Documents/extern_libs/incubator-mxnet//lib/libmxnet.so(std::thread::_Impl<std::_Bind_simple<std::function<void (std::shared_ptr<dmlc::ManualEvent>)> (std::shared_ptr<dmlc::ManualEvent>)> >::_M_run()+0x4a) [0x7f988992563a] ``` ## Minimum reproducible example In vgg16_reduced.py in example/ssd/symbol, make the following changes as shown below. ``` relu1_1 = mx.symbol.LeakyReLU(data=conv1_1, act_type="rrelu", name="relu1_1")``` Replacing LeakyReLU with activations at other positions also causes the training to crash ## Steps to reproduce python train.py --gpus 0,1 --batch-size 32 --pretrained '' ## What have you tried to solve it? I have had to replace LeakyReLU with other activations to get around this issue.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
