[GitHub] dabraude commented on a change in pull request #9883: added function for loading content of nd_array files from a buffer
dabraude commented on a change in pull request #9883: added function for loading content of nd_array files from a buffer URL: https://github.com/apache/incubator-mxnet/pull/9883#discussion_r170708924 ## File path: src/c_api/c_api.cc ## @@ -322,6 +322,38 @@ int MXNDArrayLoad(const char* fname, API_END(); } +int MXNDArrayLoadFileContent(const void *nd_file, Review comment: Yup, for some reason this comment is linked to the old code. The header and c_api have been changed to MXNDArrayLoadFromBuffer This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dabraude commented on a change in pull request #9883: added function for loading content of nd_array files from a buffer
dabraude commented on a change in pull request #9883: added function for loading content of nd_array files from a buffer URL: https://github.com/apache/incubator-mxnet/pull/9883#discussion_r170687334 ## File path: tests/python/unittest/test_ndarray.py ## @@ -291,6 +293,59 @@ def test_ndarray_legacy_load(): assert same(data[i].asnumpy(), legacy_data[i].asnumpy()) +@with_seed() +def test_buffer_load(): +nrepeat = 10 +for repeat in range(nrepeat): +# test load_buffer as list +data = [] +for i in range(10): +data.append(random_ndarray(np.random.randint(1, 5))) +with tempfile.NamedTemporaryFile(mode='wb', delete = False) as tmp: +fname = tmp.name +tmp.close() # close file so this works in Windows Review comment: I just wrote a simple class in the end :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dabraude commented on a change in pull request #9883: added function for loading content of nd_array files from a buffer
dabraude commented on a change in pull request #9883: added function for loading content of nd_array files from a buffer URL: https://github.com/apache/incubator-mxnet/pull/9883#discussion_r170681457 ## File path: tests/python/unittest/test_ndarray.py ## @@ -291,6 +293,59 @@ def test_ndarray_legacy_load(): assert same(data[i].asnumpy(), legacy_data[i].asnumpy()) +@with_seed() +def test_buffer_load(): +nrepeat = 10 +for repeat in range(nrepeat): +# test load_buffer as list +data = [] +for i in range(10): +data.append(random_ndarray(np.random.randint(1, 5))) +with tempfile.NamedTemporaryFile(mode='wb', delete = False) as tmp: +fname = tmp.name +tmp.close() # close file so this works in Windows Review comment: TemporaryDirectory() doesn't exists in python 2 tempfile.mkdtemp exists in 2 & 3 but doesn't have a with, still that is probably the easiest so I can go with that? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dabraude commented on a change in pull request #9883: added function for loading content of nd_array files from a buffer
dabraude commented on a change in pull request #9883: added function for loading content of nd_array files from a buffer URL: https://github.com/apache/incubator-mxnet/pull/9883#discussion_r170630806 ## File path: tests/python/unittest/test_ndarray.py ## @@ -291,6 +293,48 @@ def test_ndarray_legacy_load(): assert same(data[i].asnumpy(), legacy_data[i].asnumpy()) +@with_seed() +def test_buffer_load(): +nrepeat = 10 +tmp = tempfile.NamedTemporaryFile(mode='wb', dir=os.path.join(os.getcwd())) Review comment: I had that at first but it was also giving me permission issues, I have pushed a change without the directory specified. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dabraude commented on a change in pull request #9883: added function for loading content of nd_array files from a buffer
dabraude commented on a change in pull request #9883: added function for loading content of nd_array files from a buffer URL: https://github.com/apache/incubator-mxnet/pull/9883#discussion_r170584847 ## File path: tests/python/unittest/test_ndarray.py ## @@ -291,6 +291,48 @@ def test_ndarray_legacy_load(): assert same(data[i].asnumpy(), legacy_data[i].asnumpy()) +@with_seed() +def test_buffer_load(): +nrepeat = 10 +fname = 'tmp_list.bin' Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dabraude commented on a change in pull request #9883: added function for loading content of nd_array files
dabraude commented on a change in pull request #9883: added function for loading content of nd_array files URL: https://github.com/apache/incubator-mxnet/pull/9883#discussion_r170463783 ## File path: src/c_api/c_api.cc ## @@ -322,6 +322,38 @@ int MXNDArrayLoad(const char* fname, API_END(); } +int MXNDArrayLoadFileContent(const void *nd_file, Review comment: changed to suggested name, which is better This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dabraude commented on a change in pull request #9883: added function for loading content of nd_array files
dabraude commented on a change in pull request #9883: added function for
loading content of nd_array files
URL: https://github.com/apache/incubator-mxnet/pull/9883#discussion_r170463743
##
File path: src/c_api/c_api.cc
##
@@ -322,6 +322,38 @@ int MXNDArrayLoad(const char* fname,
API_END();
}
+int MXNDArrayLoadFileContent(const void *nd_file,
+size_t size,
+mx_uint *out_size,
+NDArrayHandle** out_arr,
+mx_uint *out_name_size,
+const char*** out_names) {
+ MXAPIThreadLocalEntry *ret = MXAPIThreadLocalStore::Get();
+ ret->ret_vec_str.clear();
+ API_BEGIN();
+ std::vector data;
+ std::vector = ret->ret_vec_str;
+ {
+std::unique_ptr fi(new
dmlc::MemoryFixedSizeStream((void*)nd_file, size)); // NOLINT(*)
+mxnet::NDArray::Load(fi.get(), , );
Review comment:
Just checked multiple versions of this, it will return -1 with the last
error message being:
`Check failed: header == kMXAPINDArrayListMagic Invalid NDArray file format`
Python can check for that pretty easily.
Just added a check against NULL pointers which I didn't think of originally.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
