leezu opened a new issue #13100: Hybridization generates superfluous backward ops that may induce storage fallbacks URL: https://github.com/apache/incubator-mxnet/issues/13100 ## Description Hybridizing creates superfluous backward ops in some conditions. This is bad, as these ops may consequently trigger a storage fallback. In the example below, no gradient for `csr` is requested. Still `operator = _backward_square` is generated and consequently a storage fallback occurs. Note that this example works without storage fallback in the imperative and symbolic API (for imperative, just remove hybridize. For symbolic, see https://github.com/apache/incubator-mxnet/blob/master/example/sparse/factorization_machine/model.py#L39 ) ## Environment info (Required) mxnet ``` ----------Python Info---------- Version : 3.6.6 Compiler : GCC 8.2.0 Build : ('default', 'Oct 13 2018 05:47:55') Arch : ('64bit', 'ELF') ------------Pip Info----------- Version : 10.0.0.dev0 Directory : /home/leonard/software/pip/src/pip ----------MXNet Info----------- /home/leonard/.local/lib64/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`. from ._conv import register_converters as _register_converters Version : 1.3.1 Directory : /home/leonard/.local/lib64/python3.6/site-packages/mxnet Commit Hash : 0bea50ec201d19cf393a2dce37d9a6b1625be144 ----------System Info---------- Platform : Linux-4.19.0-gentoo-x86_64-Intel-R-_Core-TM-_i7-7500U_CPU_@_2.70GHz-with-gentoo-2.4.1 system : Linux node : leonard-xps13 release : 4.19.0-gentoo version : #4 SMP Sun Oct 28 09:21:38 UTC 2018 ----------Hardware Info---------- machine : x86_64 processor : Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 2 Core(s) per socket: 2 Socket(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 142 Model name: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz Stepping: 9 CPU MHz: 3499.226 CPU max MHz: 3500.0000 CPU min MHz: 400.0000 BogoMIPS: 5808.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 4096K Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp flush_l1d ----------Network Test---------- Setting timeout: 10 Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0133 sec, LOAD: 1.3410 sec. Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.1139 sec, LOAD: 0.2640 sec. Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.2299 sec, LOAD: 0.8850 sec. Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0157 sec, LOAD: 1.0251 sec. Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0164 sec, LOAD: 1.1068 sec. Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0497 sec, LOAD: 2.2623 sec. ``` Package used: Python ## Error Message: ``` [07:31:15] src/operator/tensor/././../../common/utils.h:450: Storage type fallback detected: operator = _backward_square input storage types = [default, csr, ] output storage types = [default, ] params = {} context.dev_mask = cpu The operator with default storage type will be dispatched for execution. You're seeing this warning message because the operator above is unable to process the given ndarrays with specified storage types, context and parameter. Temporary dense ndarrays are generated in order to execute the operator. This does not affect the correctness of the programme. You can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning. [[0. 0.] [0. 0.]] ``` ## Minimum reproducible example ``` import mxnet as mx dns = mx.nd.array([[0, 0], [1, 2], [0, 0], [3, 4], [0, 0]]) rsp = dns.tostype('row_sparse') csr = mx.nd.sparse.csr_matrix(mx.nd.zeros(shape=(2, 5))) class Net(mx.gluon.HybridBlock): def hybrid_forward(self, F, csr, rsp): csr = csr.square() return F.dot(csr.square(), rsp) rsp.attach_grad() net = Net() net.hybridize() with mx.autograd.record(): l = net(csr, rsp) l.backward() print(l.asnumpy()) ``` ## Steps to reproduce (Paste the commands you ran that produced the error.) 1. Run above code example. Note the storage fallback that occurs. 2. Remove the `.hybridize()` line. Note that the storage fallback disappears.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
