haehn commented on issue #3030: Does mxnet support .npy format file? URL: https://github.com/apache/incubator-mxnet/issues/3030#issuecomment-321934681 the same error is thrown if the array is large.. the underlying exception is different tho: ``` print X_train.shape, Y_train.shape, X_train.nbytes print X_val.shape, Y_val.shape, X_val.nbytes print X_test.shape, Y_test.shape, X_test.nbytes (212700, 6, 119, 119) (212700,) 36144536400 (70900, 6, 119, 119) (70900,) 12048178800 (70900, 6, 119, 119) (70900,) 12048178800 # # # t0 = time.time() batch_size = 100 train_iter = mx.io.NDArrayIter(data=X_train, label=Y_train, batch_size=batch_size) val_iter = mx.io.NDArrayIter(data=X_val, label=Y_val, batch_size=batch_size) test_iter = mx.io.NDArrayIter(data=X_test, label=Y_test, batch_size=batch_size) print 'iterators configured', time.time()-t0, 'seconds' --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-6-713633301713> in <module>() 4 t0 = time.time() 5 batch_size = 100 ----> 6 train_iter = mx.io.NDArrayIter(data=X_train, label=Y_train, batch_size=batch_size) 7 val_iter = mx.io.NDArrayIter(data=X_val, label=Y_val, batch_size=batch_size) 8 test_iter = mx.io.NDArrayIter(data=X_test, label=Y_test, batch_size=batch_size) /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/io.pyc in __init__(self, data, label, batch_size, shuffle, last_batch_handle, data_name, label_name) 577 super(NDArrayIter, self).__init__(batch_size) 578 --> 579 self.data = _init_data(data, allow_empty=False, default_name=data_name) 580 self.label = _init_data(label, allow_empty=True, default_name=label_name) 581 /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/io.pyc in _init_data(data, allow_empty, default_name) 485 except: 486 raise TypeError(("Invalid type '%s' for %s, " % (type(v), k)) + \ --> 487 "should be NDArray or numpy.ndarray") 488 489 return list(data.items()) TypeError: Invalid type '<type 'numpy.ndarray'>' for data, should be NDArray or numpy.ndarray ``` but the real problem is ``` a = mx.nd.array(X_train[0:50000]) # no problem a = mx.nd.array(X_train[0:60000]) # fails MXNetError: [16:37:29] include/mxnet/././tensor_blob.h:247: Check failed: this->shape_.Size() == shape.Size() (5097960000 vs. 802992704) TBlob.get_with_shape: new and old shape do not match total elements Stack trace returned 10 entries: [bt] (0) /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x18b0dc) [0x7f655a9510dc] [bt] (1) /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x27d680) [0x7f655aa43680] [bt] (2) /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x27db65) [0x7f655aa43b65] [bt] (3) /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/libmxnet.so(+0xc8047d) [0x7f655b44647d] [bt] (4) /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/libmxnet.so(+0xc5d29b) [0x7f655b42329b] [bt] (5) /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/libmxnet.so(MXNDArraySyncCopyFromCPU+0xa) [0x7f655b2f0a1a] [bt] (6) /lib64/libffi.so.6(ffi_call_unix64+0x4c) [0x7f66065e3dcc] [bt] (7) /lib64/libffi.so.6(ffi_call+0x1f5) [0x7f66065e36f5] [bt] (8) /home/dhaehn/D1/lib64/python2.7/lib-dynload/_ctypes.so(_ctypes_callproc+0x30b) [0x7f66067f6c8b] [bt] (9) /home/dhaehn/D1/lib64/python2.7/lib-dynload/_ctypes.so(+0xaa85) [0x7f66067f0a85] # but.. a = mx.nd.array(X_train[50000:80000]) # no problem ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
With regards, Apache Git Services
