matteosal opened a new issue #19841:
URL: https://github.com/apache/incubator-mxnet/issues/19841
## Description
Some specific symbol evaluation fail with a CUDA error. I have reproduced
this problem on:
* mxnet 1.6 + CUDA 10.1
* mxnet 1.6 + CUDA 11.2
* mxnet 1.7 + CUDA 11.2
(didn't try mxnet 1.7 + CUDA 10.1). My GPU is a GTX 1650 Mobile.
### Error Message
```
mxnet.base.MXNetError: Traceback (most recent call last):
[bt] (11) /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fcfe137d293]
[bt] (10) /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fcfe1241609]
[bt] (9) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6d84) [0x7fcfcfe04d84]
[bt] (8)
/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::function<void
(std::shared_ptr<dmlc::ManualEvent>)>, std::shared_ptr<dmlc::ManualEvent> > >
>::_M_run()+0x4a) [0x7fcf900d826a]
[bt] (7)
/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(std::_Function_handler<void
(std::shared_ptr<dmlc::ManualEvent>),
mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*,
bool)::{lambda()#4}::operator()()
const::{lambda(std::shared_ptr<dmlc::ManualEvent>)#1}>::_M_invoke(std::_Any_data
const&, std::shared_ptr<dmlc::ManualEvent>&&)+0x4e) [0x7fcf900dcb9e]
[bt] (6)
/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(void
mxnet::engine::ThreadedEnginePerDevice::GPUWorker<(dmlc::ConcurrentQueueType)0>(mxnet::Context,
bool,
mxnet::engine::ThreadedEnginePerDevice::ThreadWorkerBlock<(dmlc::ConcurrentQueueType)0>*,
std::shared_ptr<dmlc::ManualEvent> const&)+0x11d) [0x7fcf900dc8fd]
[bt] (5)
/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext,
mxnet::engine::OprBlock*)+0x11f) [0x7fcf900d91ff]
[bt] (4)
/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(+0x18ae22a)
[0x7fcf900ff22a]
[bt] (3)
/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::exec::FComputeExecutor::Run(mxnet::RunContext,
bool)+0x6d) [0x7fcf900f143d]
[bt] (2)
/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(void
mxnet::FusedOp::Forward<mshadow::gpu>(nnvm::NodeAttrs const&, mxnet::OpContext
const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&,
std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&,
std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)+0x9e5)
[0x7fcf938e6585]
[bt] (1)
/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::FusedOp::CompileCode(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, int)+0x1228) [0x7fcf938de8c8]
[bt] (0)
/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x72)
[0x7fcf8ff79e92]
File
"/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/src/operator/fusion/fused_op.cu",
line 672
MXNetError: CUDA Driver: an illegal memory access was encountered
```
## To Reproduce
Download this json file then run the script:
[sym.zip](https://github.com/apache/incubator-mxnet/files/5926995/sym.zip)
```
import mxnet as mx
ctx = mx.gpu()
sym = mx.symbol.load('path/to/sym.json')
ex = sym.bind(ctx, args={"input": mx.nd.array([[1, 2, 3, 4]], ctx=ctx)})
outputs = ex.forward()
print(outputs)
```
## What have you tried to solve it?
I'm building mxnet from source with cmake, and I also happen to be in
possession of a 1.6 + CUDA 10.1 build (which is one of the combinations I've
tried) which **doesn't** have this problem (but I don't have all the details on
the settings/environment which produced it). This suggests that the problem can
be fixed by some build settings, however I have spent significant time
rebuilding with various settings and never managed to get anything. These are
my cmake settings:
```
cmake \
`# GENERAL FLAGS` \
-DCMAKE_INSTALL_PREFIX=$output_dir \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_SKIP_BUILD_RPATH=On \
-DUSE_OPENCV=OFF \
-DUSE_F16C=Off `# float16 support`\
-DUSE_INT64_TENSOR_SIZE=On \
-DCMAKE_C_FLAGS_RELEASE="-DNDEBUG" \
-DCMAKE_CXX_FLAGS_RELEASE="-DNDEBUG --std=c++14" \
`# MATH BACKENDS` \
-DBLAS=MKL \
-DUSE_LAPACK=OFF \
-DUSE_MKLDNN=OFF \
-DUSE_MKL_IF_AVAILABLE=ON \
-DMKL_USE_ILP64=ON \
-DMKL_USE_SINGLE_DYNAMIC_LIBRARY=OFF \
-DMKL_ROOT=$mkl_dir \
-DMKL_INCLUDE_DIR=$mkl_dir \
-DMKL_RT_LIBRARY="$mkl_dir/libmkl_def.so;$mkl_dir/libmkl_intel_ilp64.so;$mkl_dir/libmkl_core.so;$mkl_dir/libmkl_intel_thread.so;$mkl_dir/libiomp5.so"
\
`# OPENMP` \
-DUSE_OPENMP=ON \
-DOpenMP_C_FLAGS="-I$mkl_dir" \
-DOpenMP_C_LIB_NAMES="libiomp5" \
-DOpenMP_CXX_FLAGS="-I$mkl_dir" \
-DOpenMP_CXX_LIB_NAMES="libiomp5" \
-DOpenMP_libiomp5_LIBRARY="$mkl_dir/libiomp5.so" \
`# CUDA` \
-DUSE_CUDA=ON \
-DUSE_CUDNN=ON \
-DCUDNN_LIBRARY=$cudnn_dir/lib64/libcudnn.so.8 \
-DCUDNN_INCLUDE=$cudnn_dir/include \
-DUSE_NCCL=OFF \
-DCUDNN_ROOT:PATH=$cudnn_dir \
-DMXNET_CUDA_ARCH="7.5+PTX" \
-DCUDAToolkit_ROOT=$cuda_dir \
-DCMAKE_CUDA_COMPILER:PATH=$cuda_dir/bin/nvcc \
```
## Environment
<details>
<summary>Environment Information</summary>
```
----------Python Info----------
Version : 3.8.5
Compiler : GCC 9.3.0
Build : ('default', 'Jul 28 2020 12:59:40')
Arch : ('64bit', 'ELF')
------------Pip Info-----------
Version : 20.0.2
Directory : /usr/lib/python3/dist-packages/pip
----------MXNet Info-----------
Version : 1.7.0
Directory :
/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet
Commit hash file
"/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/COMMIT_HASH"
not found. Not installed from pre-built package or built from source.
Library :
['/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so']
Build features:
✔ CUDA
✔ CUDNN
✖ NCCL
✔ CUDA_RTC
✖ TENSORRT
✔ CPU_SSE
✔ CPU_SSE2
✔ CPU_SSE3
✖ CPU_SSE4_1
✖ CPU_SSE4_2
✖ CPU_SSE4A
✖ CPU_AVX
✖ CPU_AVX2
✔ OPENMP
✖ SSE
✖ F16C
✖ JEMALLOC
✖ BLAS_OPEN
✖ BLAS_ATLAS
✔ BLAS_MKL
✖ BLAS_APPLE
✖ LAPACK
✖ MKLDNN
✖ OPENCV
✖ CAFFE
✖ PROFILER
✖ DIST_KVSTORE
✖ CXX14
✔ INT64_TENSOR_SIZE
✔ SIGNAL_HANDLER
✖ DEBUG
✖ TVM_OP
----------System Info----------
Platform : Linux-5.8.0-41-generic-x86_64-with-glibc2.29
system : Linux
node : pajarulo
release : 5.8.0-41-generic
version : #46~20.04.1-Ubuntu SMP Mon Jan 18 17:52:23 UTC 2021
----------Hardware Info----------
machine : x86_64
processor : x86_64
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 158
Model name: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Stepping: 13
CPU MHz: 4161.311
CPU max MHz: 5000,0000
CPU min MHz: 800,0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 256 KiB
L1i cache: 256 KiB
L2 cache: 2 MiB
L3 cache: 16 MiB
NUMA node0 CPU(s): 0-15
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass
disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and
__user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB
conditional, RSB filling
Vulnerability Srbds: Mitigation; TSX disabled
Vulnerability Tsx async abort: Mitigation; TSX disabled
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep
mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
syscall nx pdpe1gb rdtscp lm constant_tsc art arch
_perfmon pebs bts rep_good nopl xtopology
nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2
ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2
apic movbe popcnt tsc_deadline_timer aes
xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single
ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi
flexpriority ept vpid ept_ad fsgsbase
tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt
intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat
pln pts hwp hwp_notify hwp_act_window
hwp_epp md_clear flush_l1d arch_capabilities
```
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
