drmasud opened a new issue #16960: large numpy array to mxnet ndarray conversion URL: https://github.com/apache/incubator-mxnet/issues/16960 ## Description Conversion fails on large matrices. ### Error Message Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/utils.py", line 146, in array return _array(source_array, ctx=ctx, dtype=dtype) File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 2505, in array arr[:] = source_array File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 449, in __setitem__ self._set_nd_basic_indexing(key, value) File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 715, in _set_nd_basic_indexing self._sync_copyfrom(value) File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 881, in _sync_copyfrom ctypes.c_size_t(source_array.size))) File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/base.py", line 253, in check_call raise MXNetError(py_str(_LIB.MXGetLastError())) mxnet.base.MXNetError: [19:09:04] src/ndarray/ndarray_function.cc:51: Check failed: size == to->Size() (-294967296 vs. 4000000000) : copying size mismatch, from: 18446744072529682432 bytes, to: 16000000000 bytes. Stack trace: [bt] (0) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2795cb) [0x7efff4d2c5cb] [bt] (1) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x259399b) [0x7efff704699b] [bt] (2) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::NDArray::SyncCopyFromCPU(void const*, unsigned long) const+0x284) [0x7efff6fe5934] [bt] (3) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(MXNDArraySyncCopyFromCPU+0x2b) [0x7efff6d8ebcb] [bt] (4) /usr/lib64/libffi.so.6(ffi_call_unix64+0x4c) [0x7f0059bbacec] [bt] (5) /usr/lib64/libffi.so.6(ffi_call+0x1f5) [0x7f0059bba615] [bt] (6) /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2a0) [0x7f0059dcd290] [bt] (7) /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(+0x9586) [0x7f0059dc6586] [bt] (8) /usr/lib64/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x90) [0x7f0061bce7e0] ### Steps to reproduce T= nd.array(np.random.randn(5000000,800)) (Paste the commands you ran that produced the error.) ## What have you tried to solve it? 1. workaround: If you convert in smaller chunks and concatenate the ndarray to create the final mxnet
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
