hetong007 opened a new issue #11960: float16 at small size breaks nn.Conv2D if padding > 0 URL: https://github.com/apache/incubator-mxnet/issues/11960 ## Description When training with input image size at 128x128, `Conv2D` with `padding` > 0 will crash. It is 100% reproducible on my machine, with cuda 9.0, cudnn 7.1.4 ## Environment info (Required) ``` $ python diagnose.py ----------Python Info---------- ('Version :', '2.7.12') ('Compiler :', 'GCC 5.4.0 20160609') ('Build :', ('default', 'Dec 4 2017 14:50:18')) ('Arch :', ('64bit', 'ELF')) ------------Pip Info----------- ('Version :', '10.0.1') ('Directory :', '/usr/local/lib/python2.7/dist-packages/pip') ----------MXNet Info----------- ('Version :', '1.3.0') ('Directory :', '/usr/local/lib/python2.7/dist-packages/mxnet') ('Commit Hash :', 'f5b95b090815e879b57dca233604dcb3f1df967a') ----------System Info---------- ('Platform :', 'Linux-4.4.0-1061-aws-x86_64-with-Ubuntu-16.04-xenial') ('system :', 'Linux') ('node :', 'ip-172-31-7-0') ('release :', '4.4.0-1061-aws') ('version :', '#70-Ubuntu SMP Fri May 25 21:47:34 UTC 2018') ----------Hardware Info---------- ('machine :', 'x86_64') ('processor :', 'x86_64') Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz Stepping: 1 CPU MHz: 2673.300 CPU max MHz: 3000.0000 CPU min MHz: 1200.0000 BogoMIPS: 4600.19 Hypervisor vendor: Xen Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 46080K NUMA node0 CPU(s): 0-15,32-47 NUMA node1 CPU(s): 16-31,48-63 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq monitor est ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt ida ----------Network Test---------- Setting timeout: 10 Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0097 sec, LOAD: 0.4656 sec. Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0026 sec, LOAD: 0.3205 sec. Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0234 sec, LOAD: 0.1151 sec. Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0177 sec, LOAD: 0.2072 sec. Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.1112 sec, LOAD: 0.1922 sec. Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.2604 sec, LOAD: 0.1222 sec. ``` Package used (Python/R/Scala/Julia): Python ## Minimum reproducible example ```python import mxnet as mx from mxnet.gluon import nn # float32 works img_fp32 = mx.nd.ones((1, 256, 32, 32), ctx=mx.gpu(0)) net_fp32 = nn.HybridSequential() net_fp32.add(nn.Conv2D(channels=256, kernel_size=3, strides=2, padding=1, use_bias=False)) net_fp32.initialize(ctx=mx.gpu(0)) p = net_fp32(img_fp32) print(p[0][0][0][0]) # float16 without padding works img_fp16 = mx.nd.ones((1, 256, 32, 32), ctx=mx.gpu(0)).astype('float16') net_fp16_1 = nn.HybridSequential() net_fp16_1.add(nn.Conv2D(channels=256, kernel_size=3, strides=2, use_bias=False)) net_fp16_1.initialize(ctx=mx.gpu(0)) net_fp16_1.cast('float16') p = net_fp16_1(img_fp16) print(p[0][0][0][0]) # float16 with padding on large input works img_fp16_large = mx.nd.ones((1, 256, 56, 56), ctx=mx.gpu(0)).astype('float16') net_fp16_2 = nn.HybridSequential() net_fp16_2.add(nn.Conv2D(channels=256, kernel_size=3, strides=2, padding=1, use_bias=False)) net_fp16_2.initialize(ctx=mx.gpu(0)) net_fp16_2.cast('float16') p = net_fp16_2(img_fp16_large) print(p[0][0][0][0]) # float16 with padding on small input fails! p = net_fp16_2(img_fp16) print(p[0][0][0][0]) ``` The above script outputs something like this: ``` $ python test.py [00:04:03] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable) [-1.5991344] <NDArray 1 @gpu(0)> [-0.9443] <NDArray 1 @gpu(0)> [0.679] <NDArray 1 @gpu(0)> Floating point exception (core dumped) ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
