zheng-da commented on issue #13418: [MXNET-1185] Support large array in several operators (part 1) URL: https://github.com/apache/incubator-mxnet/pull/13418#issuecomment-442068921 thanks for the great work. your code modifies a lot of code and the test doesn't cover all of them. could you please add more unit tests to make sure the modified code is covered? Here I tried a bunch of tests. Most of them work. Unfortunately, broadcast div fails. Could you please double check the problem in broadcast div? ```python import mxnet as mx import numpy as np long_dim = 50000000 arr = mx.nd.ones(shape=(long_dim, 200)) deg = mx.nd.ones(shape=(long_dim,1)) * 2 # Test broadcast div res = arr/deg assert np.sum(arr[-1].asnumpy() == 0.5) == arr.shape[1] # Test element-wise arr2 = mx.nd.ones(shape=(long_dim, 200)) res = arr + arr2 assert np.sum(res[-1].asnumpy() == 2) == arr.shape[1] res = arr + 1 assert np.sum(res[-1].asnumpy() == 2) == arr.shape[1] res = mx.nd.sqrt(arr + 3) assert np.sum(res[-1].asnumpy() == 2) == arr.shape[1] # Test reduce assert mx.nd.sum(arr).asnumpy() == arr.shape[0] * arr.shape[1] # Test dot weight = mx.nd.ones(shape=(200, 100)) res = mx.nd.dot(arr, weight) assert np.sum(res[-1].asnumpy() == 200) == weight.shape[1] # Test FullyConnected res = mx.nd.FullyConnected(arr, weight, num_hidden=weight.shape[1], no_bias=True) assert np.sum(res[-1].asnumpy() == 200) == weight.shape[1] # Test broadcast range = mx.nd.arange(0, long_dim).reshape(long_dim, 1) res = mx.nd.broadcast_to(range, shape=(range.shape[0], 200)) assert np.sum(res[-1].asnumpy() == long_dim) == res.shape[1] res = mx.nd.broadcast_like(range, arr) assert np.sum(res[-1].asnumpy() == long_dim) == arr.shape[1] # Test clip data = res res = mx.nd.clip(data, a_min=100, a_max=1000) assert np.sum(res[-1].asnumpy() == 1000) == arr.shape[1] # Test take idx = mx.nd.arange(long_dim-1000, long_dim) res = mx.nd.take(arr, idx) assert np.sum(res[-1].asnumpy() == 1) == res.shape[1] # Test slice res = mx.nd.slice(arr, begin=(long_dim-1000, 1), end=(long_dim, 100)) assert np.sum(res[-1].asnumpy() == 1) == res.shape[1] # Test slice assign res = arr.copy() res[long_dim-1:long_dim] = 1000 assert np.sum(res[-1].asnumpy() == 1000) == arr.shape[1] # Test expand_dims res = mx.nd.expand_dims(arr, axis=1) assert res.shape == (arr.shape[0], 1, arr.shape[1]) # Test squeeze data = res res = mx.nd.squeeze(data) assert sum(res.shape == arr.shape) == 2 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
