(It looks like the previous email didn’t go through. Resending it)
Hi everyone, I’ve been working on sparse tensor support in MXNet. I’d like to share a bit regarding what I worked on and gather some inputs/feature requests from the community. Recently sparse tensor CPU support has been merged to MXNet master with: - Two sparse data formats: Compressed Sparse Row <https://mxnet.incubator.apache.org/versions/master/api/python/ndarray/sparse.html#mxnet.ndarray.sparse.CSRNDArray>(CSR, for sparse inputs) and Row Sparse <https://mxnet.incubator.apache.org/versions/master/api/python/ndarray/sparse.html#mxnet.ndarray.sparse.RowSparseNDArray> (for sparse gradients) - Two data iterators for sparse data input: NDArrayIter <https://mxnet.incubator.apache.org/versions/master/api/python/io/io.html#mxnet.io.NDArrayIter> and LibSVMIter <https://mxnet.incubator.apache.org/versions/master/api/python/io/io.html#mxnet.io.LibSVMIter> - Three optimizers for sparse gradient updates: Ftrl <https://mxnet.incubator.apache.org/versions/master/api/python/optimization/optimization.html#mxnet.optimizer.Ftrl> (@CNevd), SGD <https://mxnet.incubator.apache.org/versions/master/api/python/optimization/optimization.html#mxnet.optimizer.SGD> and Adam <https://mxnet.incubator.apache.org/versions/master/api/python/optimization/optimization.html#mxnet.optimizer.Adam> - Sparse storage conversion <https://mxnet.incubator.apache.org/versions/master/api/python/ndarray/sparse.html#mxnet.ndarray.sparse.cast_storage> , matrix-matrix product <https://mxnet.incubator.apache.org/versions/master/api/python/ndarray/sparse.html#mxnet.ndarray.sparse.dot> , matrix-vector product <https://mxnet.incubator.apache.org/versions/master/api/python/ndarray/sparse.html#mxnet.ndarray.sparse.dot>, and sparse gradient aggregation <https://mxnet.incubator.apache.org/versions/master/api/python/ndarray/sparse.html#mxnet.ndarray.sparse.add_n> operators (CPU @reminisce, GPU @stefanhenneking) - Many sparse element-wise CPU operators including: arithmetic (e.g. elemwise_add), rounding, trigonometric, hyperbolic, exponents, logarithms, and power operators (mainly implemented for Row Sparse but not yet for CSR @cjolivier01). - Distributed kv-store with sparse push <https://mxnet.incubator.apache.org/versions/master/api/python/kvstore/kvstore.html#mxnet.kvstore.KVStore.push> /pull <https://mxnet.incubator.apache.org/versions/master/api/python/kvstore/kvstore.html#mxnet.kvstore.KVStore.row_sparse_pull> (CPU only, 64-bit hashed keys not supported for distributed training) - Distributed linear regression <https://github.com/apache/incubator-mxnet/tree/master/example/sparse> example with sparse data There’re also some ongoing benchmarking efforts for matrix multiplication, memory usage and distributed training within MXNet (@anirudh2290) and tutorials <https://github.com/apache/incubator-mxnet/pull/7921> regarding basic sparse operations (work in progress, comments are welcome). The future work I have in mind includes: - Update document to reflect available sparse operators and benchmark results - Sparse embedding operator - Adagrad optimizer for sparse gradient updates - Reduce sum operator for CSR - Gluon interface support - Factorization machine example - Noise contrastive estimation example *What sparse related features and operator support would you need and what do you want to use it for? Do you want any item in the list of future work to become available sooner? Any feedback is welcome. Thanks a lot.* Best, Haibin
