[GitHub] ThomasDelteil commented on issue #10528: [MXNET-316] Remove empty buckets causing index errors

[GitBox]

ThomasDelteil commented on issue #10528: [MXNET-316] Remove empty buckets 
causing index errors
URL: https://github.com/apache/incubator-mxnet/pull/10528#issuecomment-380875511
 
 
   Thanks @harusametime ! Could we get a test to make sure it works and we 
don't introduce it this bug in the future?
   - [x] Could you please create a Jira ticket and add it to the title of your 
PR? Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ThomasDelteil commented on issue #10490: [MXNET-313] Python: Resolve some of the undefined names

[GitBox]

ThomasDelteil commented on issue #10490: [MXNET-313] Python: Resolve some of 
the undefined names
URL: https://github.com/apache/incubator-mxnet/pull/10490#issuecomment-380877338
 
 
   ```
   C: 28, 0: third party import "import numpy as np" should be placed before 
"import mxnet as mx" (wrong-import-order)
   ```
   
   - [x] could you please create a JIRA ticket for your PR? Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #10538: [MXNET-318] Allow dot for fp16 on GPU

[GitBox]

rahul003 commented on issue #10538: [MXNET-318] Allow dot for fp16 on GPU
URL: https://github.com/apache/incubator-mxnet/pull/10538#issuecomment-381037280
 
 
   Yes, I'll update the PR. Thanks!
   
   On Thu, Apr 12, 2018, 7:52 PM Anirudh Subramanian 
   wrote:
   
   > *@anirudh2290* commented on this pull request.
   > --
   >
   > In src/operator/tensor/dot-inl.h
   > 

   > :
   >
   > > @@ -69,9 +69,10 @@ void DotForward_(const nnvm::NodeAttrs& attrs,
   ><< "Binary function only support input/output with the same type";
   >CHECK_EQ(outputs[0].type_flag_, inputs[1].type_flag_)
   ><< "Binary function only support input/output with the same type";
   > -  CHECK(outputs[0].type_flag_ == kFloat32 || outputs[0].type_flag_ == 
kFloat64)
   > -  << "dot only supports float32 and float64";
   > -  MSHADOW_SGL_DBL_TYPE_SWITCH(outputs[0].type_flag_, DType, {
   > +  CHECK(outputs[0].type_flag_ == kFloat32 || outputs[0].type_flag_ == 
kFloat64 ||
   >
   > don't we need to do the same for BatchDotForward?
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #10435: [MXNET-289] Update example to resize data iterator to fix hang in dist sync for last epoch

[GitBox]

rahul003 commented on issue #10435: [MXNET-289] Update example to resize data 
iterator to fix hang in dist sync for last epoch
URL: https://github.com/apache/incubator-mxnet/pull/10435#issuecomment-381037235
 
 
   Yes, there's no need to do it in that case.
   
   On Thu, Apr 12, 2018, 8:46 PM Haibin Lin  wrote:
   
   > *@eric-haibin-lin* commented on this pull request.
   > --
   >
   > In example/image-classification/common/fit.py
   > 

   > :
   >
   > >
   >  # data iterators
   >  (train, val) = data_loader(args, kv)
   > +if 'dist' in args.kv_store and 'sync' in args.kv_store:
   >
   > so there's no need to resize for async right?
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ThomasDelteil commented on a change in pull request #10537: [MX-307] Add .md tutorials to .ipynb for CI integration

[GitBox]

ThomasDelteil commented on a change in pull request #10537: [MX-307] Add .md 
tutorials to .ipynb for CI integration
URL: https://github.com/apache/incubator-mxnet/pull/10537#discussion_r181295076
 
 

 ##
 File path: tests/nightly/test_tutorial_config.txt
 ##
 @@ -1,20 +1,31 @@
 basic/ndarray
+basic/ndarray_indexing
 basic/symbol
 basic/module
 basic/data
-python/linear-regression
-python/mnist
-python/predict_image
-onnx/super_resolution
-onnx/fine_tuning_gluon
-onnx/inference_on_onnx_model
-basic/ndarray_indexing
-python/matrix_factorization
+gluon/customop
+gluon/data_augmentation
+gluon/datasets
 gluon/ndarray
 gluon/mnist
 gluon/autograd
 gluon/gluon
 gluon/hybrid
+nlp/cnn
+onnx/super_resolution
+onnx/fine_tuning_gluon
+onnx/inference_on_onnx_model
+python/matrix_factorization
+python/linear-regression
+python/mnist
+python/predict_image
+python/data_augmentation
+python/data_augmentation_with_masks
+python/kvstore
+python/types_of_data_augmentation
 sparse/row_sparse
 sparse/csr
-sparse/train
 
 Review comment:
   Indeed, going forward, there will be a one individual test per tutorial, to 
allow the use of annotation like `@highCpu`, `@highMemory`, `@gpu`. And there 
will be an integration test that will check that each notebook has been added 
to the test suite. 
   
   This will be part of my next PR, as part of this work of integrating 
tutorials to the CI 😃 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ThomasDelteil commented on a change in pull request #10537: [MX-307] Add .md tutorials to .ipynb for CI integration

[GitBox]

ThomasDelteil commented on a change in pull request #10537: [MX-307] Add .md 
tutorials to .ipynb for CI integration
URL: https://github.com/apache/incubator-mxnet/pull/10537#discussion_r181295076
 
 

 ##
 File path: tests/nightly/test_tutorial_config.txt
 ##
 @@ -1,20 +1,31 @@
 basic/ndarray
+basic/ndarray_indexing
 basic/symbol
 basic/module
 basic/data
-python/linear-regression
-python/mnist
-python/predict_image
-onnx/super_resolution
-onnx/fine_tuning_gluon
-onnx/inference_on_onnx_model
-basic/ndarray_indexing
-python/matrix_factorization
+gluon/customop
+gluon/data_augmentation
+gluon/datasets
 gluon/ndarray
 gluon/mnist
 gluon/autograd
 gluon/gluon
 gluon/hybrid
+nlp/cnn
+onnx/super_resolution
+onnx/fine_tuning_gluon
+onnx/inference_on_onnx_model
+python/matrix_factorization
+python/linear-regression
+python/mnist
+python/predict_image
+python/data_augmentation
+python/data_augmentation_with_masks
+python/kvstore
+python/types_of_data_augmentation
 sparse/row_sparse
 sparse/csr
-sparse/train
 
 Review comment:
   Indeed, going forward, there will be a one individual test per tutorial, to 
allow the use of annotation like `@highCpu`, `@hihMemory`. And there will be an 
integration test that will check that each notebook has been added to the test 
suite. 
   
   This will be part of my next PR, as part of this work of integrating 
tutorials to the CI 😃 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 closed pull request #10538: [MXNET-318] Allow dot for fp16 on GPU

[GitBox]

rahul003 closed pull request #10538: [MXNET-318] Allow dot for fp16 on GPU
URL: https://github.com/apache/incubator-mxnet/pull/10538
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/operator/tensor/dot-inl.h b/src/operator/tensor/dot-inl.h
index c5f278e78a4..a6833c288cc 100644
--- a/src/operator/tensor/dot-inl.h
+++ b/src/operator/tensor/dot-inl.h
@@ -69,9 +69,10 @@ void DotForward_(const nnvm::NodeAttrs& attrs,
   << "Binary function only support input/output with the same type";
   CHECK_EQ(outputs[0].type_flag_, inputs[1].type_flag_)
   << "Binary function only support input/output with the same type";
-  CHECK(outputs[0].type_flag_ == kFloat32 || outputs[0].type_flag_ == kFloat64)
-  << "dot only supports float32 and float64";
-  MSHADOW_SGL_DBL_TYPE_SWITCH(outputs[0].type_flag_, DType, {
+  CHECK(outputs[0].type_flag_ == kFloat32 || outputs[0].type_flag_ == kFloat64 
|| 
+ctx.run_ctx.ctx.dev_mask() == mshadow::gpu::kDevMask)
+  << "dot only supports float32/float64 for CPU, and 
float16/float32/float64 for GPU";
+  MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, DType, {
 if (inputs[0].ndim() == 1 && inputs[1].ndim() == 1) {
   CHECK_NE(req[0], kAddTo) << "AddTo not yet supported";
   Tensor out = outputs[0].get(s);
@@ -129,7 +130,7 @@ void DotBackward_(const nnvm::NodeAttrs& attrs,
   Stream *s = ctx.get_stream();
   CHECK_NE(req[0], kWriteInplace);
   CHECK_NE(req[1], kWriteInplace);
-  MSHADOW_SGL_DBL_TYPE_SWITCH(outputs[0].type_flag_, DType, {
+  MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, DType, {
 if (inputs[1].ndim() == 1 && inputs[2].ndim() == 1) {
   Tensor mout_grad = inputs[0].get(s);
   Tensor mlhs_data = inputs[1].get(s);
diff --git a/tests/python/unittest/test_operator.py 
b/tests/python/unittest/test_operator.py
index 5d382220a7a..78fd84145ec 100644
--- a/tests/python/unittest/test_operator.py
+++ b/tests/python/unittest/test_operator.py
@@ -2070,7 +2070,7 @@ def test_stn():
 @with_seed(1234)
 def test_dot():
 ctx=default_context()
-dtypes = ['float32', 'float64']
+dtypes = ['float16', 'float32', 'float64']
 
 # Test normal dot.
 for data_type in dtypes:
@@ -2094,10 +2094,16 @@ def test_dot():
 c = mx.sym.dot(a, b)
 exe = c.simple_bind(ctx=ctx, a=a_npy.shape, b=b_npy.shape)
 outputs = exe.forward(is_train=True, a=a_npy, b=b_npy)
-assert_almost_equal(outputs[0].asnumpy(), c_npy, rtol=1e-3)
-exe.backward(out_grads=[mx.nd.array(ograd_npy, mx.cpu())])
-assert_almost_equal(exe.grad_dict['a'].asnumpy(), 
agrad_npy, rtol=1e-3)
-assert_almost_equal(exe.grad_dict['b'].asnumpy(), 
bgrad_npy, rtol=1e-3)
+assert_almost_equal(outputs[0].asnumpy(), c_npy, 
+rtol=1e-2 if data_type == 'float16' 
else 1e-3,
+atol=1e-2 if data_type == 'float16' 
else 1e-3)
+exe.backward(out_grads=[mx.nd.array(ograd_npy, 
mx.cpu()).astype(data_type)])
+assert_almost_equal(exe.grad_dict['a'].asnumpy(), 
agrad_npy, 
+rtol=1e-2 if data_type == 'float16' 
else 1e-3,
+atol=1e-2 if data_type == 'float16' 
else 1e-3)
+assert_almost_equal(exe.grad_dict['b'].asnumpy(), 
bgrad_npy, 
+rtol=1e-2 if data_type == 'float16' 
else 1e-3,
+atol=1e-2 if data_type == 'float16' 
else 1e-3)
 
 # Test dot with transpose flag using gradient checker.
 def dot_sym(data_type):


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on a change in pull request #10511: add naming tutorial

[GitBox]

szha commented on a change in pull request #10511: add naming tutorial
URL: https://github.com/apache/incubator-mxnet/pull/10511#discussion_r181286454
 
 

 ##
 File path: python/mxnet/gluon/block.py
 ##
 @@ -187,17 +187,17 @@ def __setattr__(self, name, value):
 existing = getattr(self, name)
 if isinstance(existing, (Parameter, Block)) and not 
isinstance(value, type(existing)):
 raise TypeError('Changing attribute type for {name} from 
{type1} to {type2}' \
-'is not allowed.'.format(name=name,
- type1=type(existing),
- type2=type(value)))
-if isinstance(existing, Block):
-for i, c in enumerate(self._children):
-if c is existing:
-self._children[i] = value
-elif isinstance(value, Block):
-self.register_child(value)
-elif isinstance(value, Block):
-self.register_child(value)
+'is not allowed.'.format(
+name=name, type1=type(existing), 
type2=type(value)))
+
+if isinstance(value, Block):
+self.register_child(value, name)
 
 Review comment:
   Just noticed. This means it's finally possible to print the model `repr` in 
order.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on issue #10525: Fix NNPACK header file position error

[GitBox]

piiswrong commented on issue #10525: Fix NNPACK header file position error
URL: https://github.com/apache/incubator-mxnet/pull/10525#issuecomment-381024773
 
 
   It's a major work item. We'll need to plan it


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on a change in pull request #10511: add naming tutorial

[GitBox]

piiswrong commented on a change in pull request #10511: add naming tutorial
URL: https://github.com/apache/incubator-mxnet/pull/10511#discussion_r181286250
 
 

 ##
 File path: python/mxnet/gluon/block.py
 ##
 @@ -187,17 +187,17 @@ def __setattr__(self, name, value):
 existing = getattr(self, name)
 if isinstance(existing, (Parameter, Block)) and not 
isinstance(value, type(existing)):
 raise TypeError('Changing attribute type for {name} from 
{type1} to {type2}' \
-'is not allowed.'.format(name=name,
- type1=type(existing),
- type2=type(value)))
-if isinstance(existing, Block):
-for i, c in enumerate(self._children):
-if c is existing:
-self._children[i] = value
-elif isinstance(value, Block):
-self.register_child(value)
-elif isinstance(value, Block):
-self.register_child(value)
+'is not allowed.'.format(
+name=name, type1=type(existing), 
type2=type(value)))
+
+if isinstance(value, Block):
+self.register_child(value, name)
 
 Review comment:
   _children is OrderedDict now


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 commented on a change in pull request #10371: [MXNET-263] Support for dot(dns, csr) = dns and dot(dns, csr.T) = dns on GPU

[GitBox]

haojin2 commented on a change in pull request #10371: [MXNET-263] Support for 
dot(dns, csr) = dns and dot(dns, csr.T) = dns on GPU
URL: https://github.com/apache/incubator-mxnet/pull/10371#discussion_r181283102
 
 

 ##
 File path: src/operator/tensor/dot-inl.cuh
 ##
 @@ -442,6 +444,99 @@ struct DotCsrRspDnsScalarKernel {
   }
 };
 
+/*!
+ * \brief GPU Kernel to re-arrange nnz elements to csc order
+ * Parallelization by output elements: 1 thread/row of csr
+ */
+struct CscDataIndicesKernel {
+  template
+  __device__ __forceinline__ static void Map(int tid,
+ const DType* csr_data,
+ const IType* csr_indices,
+ const CType* csr_indptr,
+ DType* csc_data,
+ AtomicIType* csc_indices,
+ AtomicIType* csc_indptr,
+ AtomicIType* col_counters,
+ const nnvm::dim_t num_rows,
+ const nnvm::dim_t num_cols) {
+if (tid < num_rows) {
+  for (CType i = csr_indptr[tid]; i < csr_indptr[tid + 1]; ++i) {
+// target column
+const IType target_col = csr_indices[i];
+const int target_offset = atomicAdd(&col_counters[target_col], 1);
 
 Review comment:
   Well...maybe u r right about this, but I guess then we'll have to add a sort 
after this to ensure the order.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 commented on a change in pull request #10371: [MXNET-263] Support for dot(dns, csr) = dns and dot(dns, csr.T) = dns on GPU

[GitBox]

haojin2 commented on a change in pull request #10371: [MXNET-263] Support for 
dot(dns, csr) = dns and dot(dns, csr.T) = dns on GPU
URL: https://github.com/apache/incubator-mxnet/pull/10371#discussion_r181282944
 
 

 ##
 File path: src/operator/tensor/dot-inl.cuh
 ##
 @@ -442,6 +444,99 @@ struct DotCsrRspDnsScalarKernel {
   }
 };
 
+/*!
+ * \brief GPU Kernel to re-arrange nnz elements to csc order
+ * Parallelization by output elements: 1 thread/row of csr
+ */
+struct CscDataIndicesKernel {
+  template
+  __device__ __forceinline__ static void Map(int tid,
+ const DType* csr_data,
+ const IType* csr_indices,
+ const CType* csr_indptr,
+ DType* csc_data,
+ AtomicIType* csc_indices,
+ AtomicIType* csc_indptr,
+ AtomicIType* col_counters,
+ const nnvm::dim_t num_rows,
+ const nnvm::dim_t num_cols) {
+if (tid < num_rows) {
+  for (CType i = csr_indptr[tid]; i < csr_indptr[tid + 1]; ++i) {
+// target column
+const IType target_col = csr_indices[i];
+const int target_offset = atomicAdd(&col_counters[target_col], 1);
 
 Review comment:
   No, this only affects the order of data within the column, when we are doing 
the final multiplication-accumulation the result should still be the same.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on a change in pull request #10511: add naming tutorial

[GitBox]

szha commented on a change in pull request #10511: add naming tutorial
URL: https://github.com/apache/incubator-mxnet/pull/10511#discussion_r181282688
 
 

 ##
 File path: python/mxnet/gluon/block.py
 ##
 @@ -187,17 +187,17 @@ def __setattr__(self, name, value):
 existing = getattr(self, name)
 if isinstance(existing, (Parameter, Block)) and not 
isinstance(value, type(existing)):
 raise TypeError('Changing attribute type for {name} from 
{type1} to {type2}' \
-'is not allowed.'.format(name=name,
- type1=type(existing),
- type2=type(value)))
-if isinstance(existing, Block):
-for i, c in enumerate(self._children):
-if c is existing:
-self._children[i] = value
-elif isinstance(value, Block):
-self.register_child(value)
-elif isinstance(value, Block):
-self.register_child(value)
+'is not allowed.'.format(
+name=name, type1=type(existing), 
type2=type(value)))
+
+if isinstance(value, Block):
+self.register_child(value, name)
 
 Review comment:
   Removing the block update logic can cause having stale blocks in `_children`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin commented on a change in pull request #10371: [MXNET-263] Support for dot(dns, csr) = dns and dot(dns, csr.T) = dns on GPU

[GitBox]

eric-haibin-lin commented on a change in pull request #10371: [MXNET-263] 
Support for dot(dns, csr) = dns and dot(dns, csr.T) = dns on GPU
URL: https://github.com/apache/incubator-mxnet/pull/10371#discussion_r181282408
 
 

 ##
 File path: tests/python/unittest/test_sparse_operator.py
 ##
 @@ -1286,10 +1309,18 @@ def test_sparse_dot_zero_output(lhs_shape, trans_lhs, 
rhs_num_cols):
 test_dot_csr(lhs_shape, (lhs_shape[1], rnd.randint(5, 10)), 'default', 
False, lhs_d, rhs_d)  # test gpu SpMM
 test_dot_csr(lhs_shape, (lhs_shape[0], rnd.randint(5, 10)), 'default', 
True, lhs_d, rhs_d)  # (scalar kernel)
 test_dot_dns_csr(lhs_shape, (lhs_shape[1], rnd.randint(50, 200)), 
lhs_d, lhs_d)
+test_dot_dns_csr(lhs_shape, (rnd.randint(50, 200), lhs_shape[1]), 
lhs_d, lhs_d, trans_rhs=True)
 for rhs_d in density:
 test_dot_csr(lhs_shape, (lhs_shape[1], rnd.randint(1, 10)), 
'row_sparse', False, lhs_d, rhs_d)
 test_dot_csr(lhs_shape, (lhs_shape[0], rnd.randint(1, 10)), 
'row_sparse', True, lhs_d, rhs_d)
-
+test_infer_forward_stype(lhs_shape, (lhs_shape[1], rnd.randint(10, 
20)),
+ lhs_d, rhs_d, False, False)
+test_infer_forward_stype(lhs_shape, (rnd.randint(10, 20), 
lhs_shape[1]),
+ lhs_d, rhs_d, False, True)
+test_infer_forward_stype(lhs_shape, (lhs_shape[0], rnd.randint(10, 
20)),
+ lhs_d, rhs_d, True, False)
+test_infer_forward_stype(lhs_shape, (rnd.randint(10, 20), 
lhs_shape[0]),
+ lhs_d, rhs_d, True, True)
 
 Review comment:
   We should think of a test to check if the result is determinist


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin commented on a change in pull request #10371: [MXNET-263] Support for dot(dns, csr) = dns and dot(dns, csr.T) = dns on GPU

[GitBox]

eric-haibin-lin commented on a change in pull request #10371: [MXNET-263] 
Support for dot(dns, csr) = dns and dot(dns, csr.T) = dns on GPU
URL: https://github.com/apache/incubator-mxnet/pull/10371#discussion_r181282253
 
 

 ##
 File path: src/operator/tensor/dot-inl.cuh
 ##
 @@ -442,6 +444,99 @@ struct DotCsrRspDnsScalarKernel {
   }
 };
 
+/*!
+ * \brief GPU Kernel to re-arrange nnz elements to csc order
+ * Parallelization by output elements: 1 thread/row of csr
+ */
+struct CscDataIndicesKernel {
+  template
+  __device__ __forceinline__ static void Map(int tid,
+ const DType* csr_data,
+ const IType* csr_indices,
+ const CType* csr_indptr,
+ DType* csc_data,
+ AtomicIType* csc_indices,
+ AtomicIType* csc_indptr,
+ AtomicIType* col_counters,
+ const nnvm::dim_t num_rows,
+ const nnvm::dim_t num_cols) {
+if (tid < num_rows) {
+  for (CType i = csr_indptr[tid]; i < csr_indptr[tid + 1]; ++i) {
+// target column
+const IType target_col = csr_indices[i];
+const int target_offset = atomicAdd(&col_counters[target_col], 1);
 
 Review comment:
   I don't think this provides deterministic result.. The order of accumulation 
could be different across runs


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin closed pull request #10535: render fix

[GitBox]

eric-haibin-lin closed pull request #10535: render fix
URL: https://github.com/apache/incubator-mxnet/pull/10535
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/docs/tutorials/scala/mxnet_scala_on_intellij.md 
b/docs/tutorials/scala/mxnet_scala_on_intellij.md
index 676ee664cb1..c93d99c4504 100644
--- a/docs/tutorials/scala/mxnet_scala_on_intellij.md
+++ b/docs/tutorials/scala/mxnet_scala_on_intellij.md
@@ -1,7 +1,5 @@
 # Run MXNet Scala Examples Using the IntelliJ IDE
 
-
-
 This tutorial guides you through setting up a Scala project in the IntelliJ 
IDE and shows how to use an MXNet package from your application.
 
 ## Prerequisites:
@@ -17,18 +15,18 @@ To use this tutorial you need the following items, however 
after this list, inst
 
 For other operating systems, visit each Prerequisite's website and follow 
their installations instructions. For macOS, you're in luck:
 
-1. Install brew:
+**Step 1.** Install brew:
 ```
 /usr/bin/ruby -e "$(curl -fsSL 
https://raw.githubusercontent.com/Homebrew/install/master/install)"
 ```
 
-2. Install Java 8 JDK:
+**Step 2.** Install Java 8 JDK:
 ```
 brew tap caskroom/versions
 brew cask install java8
 ```
 
-3. Install maven:
+**Step 3.** Install maven:
 ```
 brew update
 brew install maven
@@ -58,13 +56,13 @@ make scalainstall
 
 Now that you've installed your prerequisites, you are ready to setup IntelliJ 
and your first MXNet-Scala project!
 
-1. Install and setup IntelliJ:
+**Step 1.** Install and setup IntelliJ:
 - When prompted for what to features to enable during IntelliJ's first 
startup, make sure you select Scala.
 
 - Install the plugin for IntelliJ IDE by following these steps:
On **Menu**, choose **Preferences**, choose **Plugins**, type **Scala**, 
and then choose **Install**. For further plugin help and instructions, refer to 
[Scala plugin setup for IDE](https://www.jetbrains.com/help/idea/scala.html).
 
-2. Create a new project:
+**Step 2.** Create a new project:
 
 ![intellij 
welcome](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/scala/intellij-welcome.png)
 
@@ -133,8 +131,8 @@ Set the project's location. The rest of the settings can be 
left as their defaul
 After clicking Finish, you will be presented with the project's first view.
 The project's `pom.xml` will be open for editing.
 
-3. Setup project properties:
-  - Specify project properties in `pom.xml` by pasting the following content 
in the `` tag. You will be overwriting the  tag in 
the process, upgrading from `2.11.5` to `2.11.8`.
+**Step 3.** Setup project properties:
+  - Specify project properties in `pom.xml` by pasting the following content 
in the `` tag. You will be overwriting the `` tag in 
the process, upgrading from `2.11.5` to `2.11.8`.
 
 ```xml
 
@@ -143,8 +141,8 @@ The project's `pom.xml` will be open for editing.
 
 ```
 
-4. Setup project profiles and platforms:
-
+**Step 4.** Setup project profiles and platforms:
+
   - Specify project profiles and platforms in `pom.xml` by pasting the 
following content below the `` tag:
 
 ```xml
@@ -170,7 +168,7 @@ The project's `pom.xml` will be open for editing.
 
 ```
 
-5. Setup project dependencies:
+**Step 5.** Setup project dependencies:
 
   - Specify project dependencies in `pom.xml` adding the dependencies listed 
below. Place them inside the `` tag:
 
@@ -237,7 +235,7 @@ The project's `pom.xml` will be open for editing.
 
 Note the `` tag and update it to match the file path to the jar 
file that was created when you built the MXNet-Scala package. It can be found 
in the `mxnet-incubator/scala-package/assembly/{platform}/target` directory, 
and is named with the pattern 
`mxnet-full_${scala.binary.version}-${platform}-{version-SNAPSHOT}.jar`.
 
-5. Import dependencies with Maven:
+**Step 6.** Import dependencies with Maven:
 
   - Note the prompt in the lower right corner that states "Maven projects need 
to be imported".
 
@@ -245,12 +243,12 @@ Note the `` tag and update it to match the 
file path to the jar file
 
 Click "Import Changes" in this prompt.
 
-6. Build the project:
+**Step 7.** Build the project:
 - To build the project, from the menu choose Build, and then choose Build 
Project.
 
 **Note**: During the build you may experience `[ERROR] scalac error: bad 
option: '-make:transitive'`. You can fix this by deleting or commenting this 
out in your `pom.xml`. This line in question is: `-make:transitive`.
 
-7. Run the Hello World App:
+**Step 8.** Run the Hello World App:
 
 ![hello world 
app](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/scala/intellij-project-hello-world-app.png)
 
@@ -260,7 +258,7 @@ Navigate to the App included with the project.
 
 Run the App by clicking the green arrow, and verify the Hello World output
 
-

[incubator-mxnet] branch master updated: render fix (#10535)

[haibin]

This is an automated email from the ASF dual-hosted git repository.

haibin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new fb50257  render fix (#10535)
fb50257 is described below

commit fb50257aeb3281d7fc90abc38162edbbbe4a5cb2
Author: Aaron Markham 
AuthorDate: Thu Apr 12 20:49:28 2018 -0700

render fix (#10535)

* render fix

* more render bugs workarounds
---
 docs/tutorials/scala/mxnet_scala_on_intellij.md | 30 -
 1 file changed, 14 insertions(+), 16 deletions(-)

diff --git a/docs/tutorials/scala/mxnet_scala_on_intellij.md 
b/docs/tutorials/scala/mxnet_scala_on_intellij.md
index 676ee66..c93d99c 100644
--- a/docs/tutorials/scala/mxnet_scala_on_intellij.md
+++ b/docs/tutorials/scala/mxnet_scala_on_intellij.md
@@ -1,7 +1,5 @@
 # Run MXNet Scala Examples Using the IntelliJ IDE
 
-
-
 This tutorial guides you through setting up a Scala project in the IntelliJ 
IDE and shows how to use an MXNet package from your application.
 
 ## Prerequisites:
@@ -17,18 +15,18 @@ To use this tutorial you need the following items, however 
after this list, inst
 
 For other operating systems, visit each Prerequisite's website and follow 
their installations instructions. For macOS, you're in luck:
 
-1. Install brew:
+**Step 1.** Install brew:
 ```
 /usr/bin/ruby -e "$(curl -fsSL 
https://raw.githubusercontent.com/Homebrew/install/master/install)"
 ```
 
-2. Install Java 8 JDK:
+**Step 2.** Install Java 8 JDK:
 ```
 brew tap caskroom/versions
 brew cask install java8
 ```
 
-3. Install maven:
+**Step 3.** Install maven:
 ```
 brew update
 brew install maven
@@ -58,13 +56,13 @@ make scalainstall
 
 Now that you've installed your prerequisites, you are ready to setup IntelliJ 
and your first MXNet-Scala project!
 
-1. Install and setup IntelliJ:
+**Step 1.** Install and setup IntelliJ:
 - When prompted for what to features to enable during IntelliJ's first 
startup, make sure you select Scala.
 
 - Install the plugin for IntelliJ IDE by following these steps:
On **Menu**, choose **Preferences**, choose **Plugins**, type **Scala**, 
and then choose **Install**. For further plugin help and instructions, refer to 
[Scala plugin setup for IDE](https://www.jetbrains.com/help/idea/scala.html).
 
-2. Create a new project:
+**Step 2.** Create a new project:
 
 ![intellij 
welcome](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/scala/intellij-welcome.png)
 
@@ -133,8 +131,8 @@ Set the project's location. The rest of the settings can be 
left as their defaul
 After clicking Finish, you will be presented with the project's first view.
 The project's `pom.xml` will be open for editing.
 
-3. Setup project properties:
-  - Specify project properties in `pom.xml` by pasting the following content 
in the `` tag. You will be overwriting the  tag in 
the process, upgrading from `2.11.5` to `2.11.8`.
+**Step 3.** Setup project properties:
+  - Specify project properties in `pom.xml` by pasting the following content 
in the `` tag. You will be overwriting the `` tag in 
the process, upgrading from `2.11.5` to `2.11.8`.
 
 ```xml
 
@@ -143,8 +141,8 @@ The project's `pom.xml` will be open for editing.
 
 ```
 
-4. Setup project profiles and platforms:
-
+**Step 4.** Setup project profiles and platforms:
+
   - Specify project profiles and platforms in `pom.xml` by pasting the 
following content below the `` tag:
 
 ```xml
@@ -170,7 +168,7 @@ The project's `pom.xml` will be open for editing.
 
 ```
 
-5. Setup project dependencies:
+**Step 5.** Setup project dependencies:
 
   - Specify project dependencies in `pom.xml` adding the dependencies listed 
below. Place them inside the `` tag:
 
@@ -237,7 +235,7 @@ The project's `pom.xml` will be open for editing.
 
 Note the `` tag and update it to match the file path to the jar 
file that was created when you built the MXNet-Scala package. It can be found 
in the `mxnet-incubator/scala-package/assembly/{platform}/target` directory, 
and is named with the pattern 
`mxnet-full_${scala.binary.version}-${platform}-{version-SNAPSHOT}.jar`.
 
-5. Import dependencies with Maven:
+**Step 6.** Import dependencies with Maven:
 
   - Note the prompt in the lower right corner that states "Maven projects need 
to be imported".
 
@@ -245,12 +243,12 @@ Note the `` tag and update it to match the 
file path to the jar file
 
 Click "Import Changes" in this prompt.
 
-6. Build the project:
+**Step 7.** Build the project:
 - To build the project, from the menu choose Build, and then choose Build 
Project.
 
 **Note**: During the build you may experience `[ERROR] scalac error: bad 
option: '-make:transitive'`. You can fix this by deleting or commenting this 
out in your `pom.xml`. This line in question is: `-make:transitive`.
 
-7. Run the Hello World App:
+**Step 8.** Run the Hello World App:
 
 ![hello world 
app]

[GitHub] piiswrong commented on a change in pull request #10511: add naming tutorial

[GitBox]

piiswrong commented on a change in pull request #10511: add naming tutorial
URL: https://github.com/apache/incubator-mxnet/pull/10511#discussion_r181280172
 
 

 ##
 File path: docs/tutorials/gluon/naming.md
 ##
 @@ -0,0 +1,236 @@
+
+# Naming of Gluon Parameter and Blocks
+
+In gluon, each Parameter or Block has a name (and prefix). Parameter names are 
specified by users and Block names can be either specified by users or 
automatically created.
+
+In this tutorial we talk about the best practices on naming. First, let's 
import MXNet and Gluon:
+
+
+```python
+from __future__ import print_function
+import mxnet as mx
+from mxnet import gluon
+```
+
+## Naming Blocks
+
+When creating a block, you can assign a prefix to it:
+
+
+```python
+mydense = gluon.nn.Dense(100, prefix='mydense_')
+print(mydense.prefix)
+```
+
+mydense_
+
+
+When no prefix is given, Gluon will automatically generate one:
+
+
+```python
+dense0 = gluon.nn.Dense(100)
+print(dense0.prefix)
+```
+
+dense0_
+
+
+When you create more Blocks of the same kind, they will be named differently 
to avoid collision:
+
+
+```python
+dense1 = gluon.nn.Dense(100)
+print(dense1.prefix)
+```
+
+dense1_
+
+
+## Naming Parameters
+
+Parameters within a Block will be named by prepending the prefix of the Block 
to the name of the Parameter:
+
+
+```python
+print(dense0.collect_params())
+```
+
+dense0_ (
+  Parameter dense0_weight (shape=(100, 0), dtype=)
+  Parameter dense0_bias (shape=(100,), dtype=)
+)
+
+
+## Name scopes
+
+To manage the names of nested Blocks, each Block has a `name_scope` attached 
to it. All Blocks created within a name scope will have its parent Block's 
prefix prepended to its name.
+
+Let's demonstrate this by first define a simple neural net:
+
+
+```python
+class Model(gluon.Block):
+def __init__(self, **kwargs):
+super(Model, self).__init__(**kwargs)
+with self.name_scope():
+self.dense0 = gluon.nn.Dense(20)
+self.dense1 = gluon.nn.Dense(20)
+self.mydense = gluon.nn.Dense(20, prefix='mydense_')
+
+def forward(self, x):
+x = mx.nd.relu(self.dense0(x))
+x = mx.nd.relu(self.dense1(x))
+return mx.nd.relu(self.mydense(x))
+```
+
+Now let's instantiate our neural net.
+
+- Note that `model0.dense0` is named as `model0_dense0_` instead of `dense0_`.
+
+- Also note that although we specified `mydense_` as prefix for 
`model.mydense`, its parent's prefix is automatically prepended to generate the 
prefix `model0_mydense_`.
+
+
+```python
+model0 = Model()
+model0.initialize()
+model0(mx.nd.zeros((1, 20)))
+print(model0.prefix, model0.dense0.prefix, model0.dense1.prefix, 
model0.mydense.prefix)
+```
+
+model0_ model0_dense0_ model0_dense1_ model0_mydense_
+
+
+If we instantiate `Model` again, it will be given a different name like shown 
before for `Dense`.
+
+- Note that `model1.dense0` is still named as `dense0_` instead of `dense2_`, 
following dense layers in previously created `model0`. This is because each 
instance of model's name scope is independent of each other.
+
+
+```python
+model1 = Model()
+print(model1.prefix, model1.dense0.prefix, model1.dense1.prefix, 
model1.mydense.prefix)
+```
+
+model1_ model1_dense0_ model1_dense1_ model1_mydense_
+
+
+**It is recommended that you manually specify prefix for the top level Block 
(i.e. `model = Model(prefix='mymodel_')`) to avoid potential confusions in 
naming**
+
+The same principle also applies to container blocks like Sequantial. 
`name_scope` can be used inside `__init__` as well as out side of `__init__`:
+
+
+```python
+net = gluon.nn.Sequential()
+with net.name_scope():
+net.add(gluon.nn.Dense(20))
+net.add(gluon.nn.Dense(20))
+print(net.prefix, net[0].prefix, net[1].prefix)
+```
+
+sequential0_ sequential0_dense0_ sequential0_dense1_
+
+
+`gluon.model_zoo` also behaves similarly:
+
+
+```python
+net = gluon.nn.Sequential()
+with net.name_scope():
+net.add(gluon.model_zoo.vision.alexnet(pretrained=True))
+net.add(gluon.model_zoo.vision.alexnet(pretrained=True))
+print(net.prefix, net[0].prefix, net[1].prefix)
+```
+
+sequential1_ sequential1_alexnet0_ sequential1_alexnet1_
+
+
+## Saving and loading
+
+Because model0 and model1 have different prefixes, their Parameters also have 
different names:
+
+
+```python
+print(model0.collect_params(), '\n')
+print(model1.collect_params())
+```
+
+model0_ (
+  Parameter model0_dense0_weight (shape=(20L, 20L), dtype=)
+  Parameter model0_dense0_bias (shape=(20L,), dtype=)
+  Parameter model0_dense1_weight (shape=(20L, 20L), dtype=)
+  Parameter model0_dense1_bias (shape=(20L,), dtype=)
+  Parameter model0_mydense_weight (shape=(20L, 20L), dtype=)
+  Parameter model0_mydense_bias (shape=(20L,), dtype=)
+) 
+
+model1_ (
+  Parameter model1_dense0_weight (shape=(20, 0), dtype=)
+  Parameter model1_dense0_bias (shape=(20,), dtype=

[GitHub] piiswrong commented on a change in pull request #10511: add naming tutorial

[GitBox]

piiswrong commented on a change in pull request #10511: add naming tutorial
URL: https://github.com/apache/incubator-mxnet/pull/10511#discussion_r181280006
 
 

 ##
 File path: docs/tutorials/gluon/naming.md
 ##
 @@ -0,0 +1,236 @@
+
+# Naming of Gluon Parameter and Blocks
+
+In gluon, each Parameter or Block has a name (and prefix). Parameter names are 
specified by users and Block names can be either specified by users or 
automatically created.
+
+In this tutorial we talk about the best practices on naming. First, let's 
import MXNet and Gluon:
+
+
+```python
+from __future__ import print_function
+import mxnet as mx
+from mxnet import gluon
+```
+
+## Naming Blocks
+
+When creating a block, you can assign a prefix to it:
+
+
+```python
+mydense = gluon.nn.Dense(100, prefix='mydense_')
+print(mydense.prefix)
+```
+
+mydense_
+
+
+When no prefix is given, Gluon will automatically generate one:
+
+
+```python
+dense0 = gluon.nn.Dense(100)
+print(dense0.prefix)
+```
+
+dense0_
+
+
+When you create more Blocks of the same kind, they will be named differently 
to avoid collision:
+
+
+```python
+dense1 = gluon.nn.Dense(100)
+print(dense1.prefix)
+```
+
+dense1_
+
+
+## Naming Parameters
+
+Parameters within a Block will be named by prepending the prefix of the Block 
to the name of the Parameter:
+
+
+```python
+print(dense0.collect_params())
+```
+
+dense0_ (
+  Parameter dense0_weight (shape=(100, 0), dtype=)
+  Parameter dense0_bias (shape=(100,), dtype=)
+)
+
+
+## Name scopes
+
+To manage the names of nested Blocks, each Block has a `name_scope` attached 
to it. All Blocks created within a name scope will have its parent Block's 
prefix prepended to its name.
+
+Let's demonstrate this by first define a simple neural net:
+
+
+```python
+class Model(gluon.Block):
+def __init__(self, **kwargs):
+super(Model, self).__init__(**kwargs)
+with self.name_scope():
+self.dense0 = gluon.nn.Dense(20)
+self.dense1 = gluon.nn.Dense(20)
+self.mydense = gluon.nn.Dense(20, prefix='mydense_')
+
+def forward(self, x):
+x = mx.nd.relu(self.dense0(x))
+x = mx.nd.relu(self.dense1(x))
+return mx.nd.relu(self.mydense(x))
+```
+
+Now let's instantiate our neural net.
+
+- Note that `model0.dense0` is named as `model0_dense0_` instead of `dense0_`.
+
+- Also note that although we specified `mydense_` as prefix for 
`model.mydense`, its parent's prefix is automatically prepended to generate the 
prefix `model0_mydense_`.
+
+
+```python
+model0 = Model()
+model0.initialize()
+model0(mx.nd.zeros((1, 20)))
+print(model0.prefix, model0.dense0.prefix, model0.dense1.prefix, 
model0.mydense.prefix)
+```
+
+model0_ model0_dense0_ model0_dense1_ model0_mydense_
+
+
+If we instantiate `Model` again, it will be given a different name like shown 
before for `Dense`.
+
+- Note that `model1.dense0` is still named as `dense0_` instead of `dense2_`, 
following dense layers in previously created `model0`. This is because each 
instance of model's name scope is independent of each other.
+
+
+```python
+model1 = Model()
+print(model1.prefix, model1.dense0.prefix, model1.dense1.prefix, 
model1.mydense.prefix)
+```
+
+model1_ model1_dense0_ model1_dense1_ model1_mydense_
+
+
+**It is recommended that you manually specify prefix for the top level Block 
(i.e. `model = Model(prefix='mymodel_')`) to avoid potential confusions in 
naming**
+
+The same principle also applies to container blocks like Sequantial. 
`name_scope` can be used inside `__init__` as well as out side of `__init__`:
+
+
+```python
+net = gluon.nn.Sequential()
+with net.name_scope():
+net.add(gluon.nn.Dense(20))
+net.add(gluon.nn.Dense(20))
+print(net.prefix, net[0].prefix, net[1].prefix)
 
 Review comment:
   to make it look nicer


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin commented on a change in pull request #10435: [MXNET-289] Update example to resize data iterator to fix hang in dist sync for last epoch

[GitBox]

eric-haibin-lin commented on a change in pull request #10435: [MXNET-289] 
Update example to resize data iterator to fix hang in dist sync for last epoch
URL: https://github.com/apache/incubator-mxnet/pull/10435#discussion_r181279912
 
 

 ##
 File path: example/image-classification/common/fit.py
 ##
 @@ -155,9 +159,16 @@ def fit(args, network, data_loader, **kwargs):
 head = '%(asctime)-15s Node[' + str(kv.rank) + '] %(message)s'
 logging.basicConfig(level=logging.DEBUG, format=head)
 logging.info('start with arguments %s', args)
+
+epoch_size = get_epoch_size(args, kv)
 
 # data iterators
 (train, val) = data_loader(args, kv)
+if 'dist' in args.kv_store and 'sync' in args.kv_store:
 
 Review comment:
   so there's no need to resize for async right?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kpmurali commented on issue #10449: [MXNET-293] Navigation updates

[GitBox]

kpmurali commented on issue #10449: [MXNET-293] Navigation updates
URL: https://github.com/apache/incubator-mxnet/pull/10449#issuecomment-381008978
 
 
   So can this be merged now?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kpmurali commented on a change in pull request #10534: [MXNET-315] Fix the version select in the navbar to be independent of order

[GitBox]

kpmurali commented on a change in pull request #10534: [MXNET-315] Fix the 
version select in the navbar to be independent of order
URL: https://github.com/apache/incubator-mxnet/pull/10534#discussion_r181276577
 
 

 ##
 File path: docs/build_version_doc/update_all_version.sh
 ##
 @@ -93,7 +93,7 @@ function update_mxnet_css {
 for tag in $tag_list; do
 # This Python script is expecting the tag_list.txt and it will use that as 
the entries to populate
 
-python AddVersion.py --root_url "$root_url" --file_path 
"$built/versions/$tag" --current_version "$tag" || exit 1
+python AddVersion.py --root_url "$root_url" --file_path 
"$built/versions/$tag" --current_version "$tag" --default_tag "$tag_default" || 
exit 1
 
 Review comment:
   Changed the variable in AddVersion.py to be tag_default


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on issue #10535: render fix

[GitBox]

szha commented on issue #10535: render fix
URL: https://github.com/apache/incubator-mxnet/pull/10535#issuecomment-381007837
 
 
   
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-10535/2/tutorials/scala/mxnet_scala_on_intellij.html


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kpmurali commented on issue #10415: [MXNET-284] Update release version text to 1.2.0

[GitBox]

kpmurali commented on issue #10415: [MXNET-284] Update release version text to 
1.2.0
URL: https://github.com/apache/incubator-mxnet/pull/10415#issuecomment-381007230
 
 
   This PR is no longer WIP.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudh2290 commented on a change in pull request #10538: [MXNET-318] Allow dot for fp16 on GPU

[GitBox]

anirudh2290 commented on a change in pull request #10538: [MXNET-318] Allow dot 
for fp16 on GPU
URL: https://github.com/apache/incubator-mxnet/pull/10538#discussion_r181274419
 
 

 ##
 File path: src/operator/tensor/dot-inl.h
 ##
 @@ -69,9 +69,10 @@ void DotForward_(const nnvm::NodeAttrs& attrs,
   << "Binary function only support input/output with the same type";
   CHECK_EQ(outputs[0].type_flag_, inputs[1].type_flag_)
   << "Binary function only support input/output with the same type";
-  CHECK(outputs[0].type_flag_ == kFloat32 || outputs[0].type_flag_ == kFloat64)
-  << "dot only supports float32 and float64";
-  MSHADOW_SGL_DBL_TYPE_SWITCH(outputs[0].type_flag_, DType, {
+  CHECK(outputs[0].type_flag_ == kFloat32 || outputs[0].type_flag_ == kFloat64 
|| 
 
 Review comment:
   don't we need to do the same for BatchDotForward?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on a change in pull request #10435: [MXNET-289] Update example to resize data iterator to fix hang in dist sync for last epoch

[GitBox]

rahul003 commented on a change in pull request #10435: [MXNET-289] Update 
example to resize data iterator to fix hang in dist sync for last epoch
URL: https://github.com/apache/incubator-mxnet/pull/10435#discussion_r181272232
 
 

 ##
 File path: example/image-classification/common/fit.py
 ##
 @@ -155,9 +159,16 @@ def fit(args, network, data_loader, **kwargs):
 head = '%(asctime)-15s Node[' + str(kv.rank) + '] %(message)s'
 logging.basicConfig(level=logging.DEBUG, format=head)
 logging.info('start with arguments %s', args)
+
+epoch_size = get_epoch_size(args, kv)
 
 # data iterators
 (train, val) = data_loader(args, kv)
+if 'dist' in args.kv_store and 'sync' in args.kv_store:
 
 Review comment:
   async should be good because no machine waits for other machines to pull 
gradients. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mengyuest commented on issue #9533: Cannot find engine branch for minpy package

[GitBox]

mengyuest commented on issue #9533: Cannot find engine branch for minpy package
URL: 
https://github.com/apache/incubator-mxnet/issues/9533#issuecomment-380999710
 
 
   I used this and it worked well (though the function is very simple).
   
   ```python
   import mxnet as mx
   from mxnet import nd
   import numpy as np
   import time
   
   t0=time.time()
   mx.random.seed(1)
   x=nd.array([[1,2,3,4,5,6],[3,4,56,7,8,9]],ctx=mx.gpu(0))
   y1=nd.dot(x,x.T)
   print(y1)
   t1=time.time()
   x=np.array([[1,2,3,4,5,6],[3,4,56,7,8,9]])
   y2=np.dot(x,x.T)
   print(y2)
   t2=time.time()
   print(t1-t0,t2-t1)
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] harusametime commented on issue #10528: [MXNET-316] Remove empty buckets causing index errors

[GitBox]

harusametime commented on issue #10528: [MXNET-316] Remove empty buckets 
causing index errors
URL: https://github.com/apache/incubator-mxnet/pull/10528#issuecomment-380967400
 
 
   @ThomasDelteil  I created a ticket and added the label [MXNET-316] to the 
title of my PR. Do you mean you need my test code? I am not familiar with 
sharing a test for now. But if you want, give me a bit of time for that.



This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #10531: Float16 Support for dot

[GitBox]

rahul003 commented on issue #10531: Float16 Support for dot
URL: 
https://github.com/apache/incubator-mxnet/issues/10531#issuecomment-380997060
 
 
   Dot for fp16 on GPU was supported internally but a (probably older) CHECK 
was blocking it. Created a PR to fix this.
   
   https://github.com/apache/incubator-mxnet/pull/10538


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 opened a new pull request #10538: [MXNET-318] Allow dot for fp16 on GPU

[GitBox]

rahul003 opened a new pull request #10538: [MXNET-318] Allow dot for fp16 on GPU
URL: https://github.com/apache/incubator-mxnet/pull/10538
 
 
   ## Description ##
   Dot for fp16 on GPU is supported internally by the BLASEngine , but a check 
was blocking that. 
   
   
https://github.com/dmlc/mshadow/blob/0b4cedd7015cc69191f8338a8feaacda90697758/mshadow/dot_engine-inl.h#L417
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to 
the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues) 
created (except PRs with tiny changes)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing 
distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a 
new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments 
are documented. 
   - For new examples, README.md is added to explain the what the example does, 
the source of the dataset, expected performance on test set and reference to 
the original paper if applicable
   - Check the API doc at 
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
   - [ ] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [ ] Changed CHECK and type switches 
   - [ ] Added test for fp16
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] harusametime commented on issue #10528: [MXNET-316] Remove empty buckets causing index errors

[GitBox]

harusametime commented on issue #10528: [MXNET-316] Remove empty buckets 
causing index errors
URL: https://github.com/apache/incubator-mxnet/pull/10528#issuecomment-380967400
 
 
   @ThomasDelteil  I created a ticket and added the label [MXNET-316] to the 
title of my PR. Sorry, I am not familiar with sharing a test for now. Give me a 
bit of time.



This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhanghang1989 commented on issue #10536: [MXNET-317] Add Data Parallel

[GitBox]

zhanghang1989 commented on issue #10536: [MXNET-317] Add Data Parallel
URL: https://github.com/apache/incubator-mxnet/pull/10536#issuecomment-380994417
 
 
   @pengzhao-intel This PR is mainly for Synchronized Cross GPU Batch Norm 
https://github.com/zhanghang1989/MXNet-Gluon-SyncBN


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] pengzhao-intel commented on issue #10536: [MXNET-317] Add Data Parallel

[GitBox]

pengzhao-intel commented on issue #10536: [MXNET-317] Add Data Parallel
URL: https://github.com/apache/incubator-mxnet/pull/10536#issuecomment-380991886
 
 
   @zhanghang1989 I am interested in this PR.  
   Could you elaborate what this PR is for?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] spidyDev commented on issue #10425: [MXNET-175] Raise user warning on mismatch between module data_names and data iter names

[GitBox]

spidyDev commented on issue #10425: [MXNET-175] Raise user warning on mismatch 
between module data_names and data iter names
URL: https://github.com/apache/incubator-mxnet/pull/10425#issuecomment-380990561
 
 
   @piiswrong  Could you review ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sugi1229 commented on issue #10516: How to run model trained with python on Scala

[GitBox]

sugi1229 commented on issue #10516: How to run model trained with python on 
Scala
URL: 
https://github.com/apache/incubator-mxnet/issues/10516#issuecomment-380990318
 
 
   @BenLag2906 
   Thanks, I changed it, it worked!!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on a change in pull request #10537: [MX-307] Add .md tutorials to .ipynb for CI integration

[GitBox]

rahul003 commented on a change in pull request #10537: [MX-307] Add .md 
tutorials to .ipynb for CI integration
URL: https://github.com/apache/incubator-mxnet/pull/10537#discussion_r181262665
 
 

 ##
 File path: tests/nightly/test_tutorial_config.txt
 ##
 @@ -1,20 +1,31 @@
 basic/ndarray
+basic/ndarray_indexing
 basic/symbol
 basic/module
 basic/data
-python/linear-regression
-python/mnist
-python/predict_image
-onnx/super_resolution
-onnx/fine_tuning_gluon
-onnx/inference_on_onnx_model
-basic/ndarray_indexing
-python/matrix_factorization
+gluon/customop
+gluon/data_augmentation
+gluon/datasets
 gluon/ndarray
 gluon/mnist
 gluon/autograd
 gluon/gluon
 gluon/hybrid
+nlp/cnn
+onnx/super_resolution
+onnx/fine_tuning_gluon
+onnx/inference_on_onnx_model
+python/matrix_factorization
+python/linear-regression
+python/mnist
+python/predict_image
+python/data_augmentation
+python/data_augmentation_with_masks
+python/kvstore
+python/types_of_data_augmentation
 sparse/row_sparse
 sparse/csr
-sparse/train
 
 Review comment:
   Can we pick them automatically so we don't need to keep updating this file?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ThomasDelteil opened a new pull request #10537: [MX-307] Add .md tutorials to .ipynb for CI integration

[GitBox]

ThomasDelteil opened a new pull request #10537: [MX-307] Add .md tutorials to 
.ipynb for CI integration
URL: https://github.com/apache/incubator-mxnet/pull/10537
 
 
   ## Description ##
   - **+11** tutorials downloadable as jupyter notebooks and ready to integrate 
in the CI
   - Added  where necessary to beautify the notebooks
   - updated notebooks to remove `!bash`-like commands so they can run more 
easily on Windows
   - updated paths to be cross-platform
   - tested against master using the tutorial test runner (see log) on python 2 
and python 3 (integration in CI coming up)
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to 
the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues) 
created (except PRs with tiny changes)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing 
distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a 
new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments 
are documented. 
   - For new examples, README.md is added to explain the what the example does, 
the source of the dataset, expected performance on test set and reference to 
the original paper if applicable
   - Check the API doc at 
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
   - [ ] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change
   
   ```
   Test Summary Start
   31 tutorials tested:
   basic/ndarray
   basic/ndarray_indexing
   basic/symbol
   basic/module
   basic/data
   gluon/customop
   gluon/data_augmentation
   gluon/datasets
   gluon/ndarray
   gluon/mnist
   gluon/autograd
   gluon/gluon
   gluon/hybrid
   nlp/cnn
   onnx/super_resolution
   onnx/fine_tuning_gluon
   onnx/inference_on_onnx_model
   python/matrix_factorization
   python/linear-regression
   python/mnist
   python/predict_image
   python/data_augmentation
   python/data_augmentation_with_masks
   python/kvstore
   python/types_of_data_augmentation
   sparse/row_sparse
   sparse/csr
   sparse/train
   speech_recognition/ctc
   unsupervised_learning/gan
   vision/large_scale_classification
   
   0 tests failed:
   Test Summary End
   Stats start
   [Passed: 31 of 31]
   Total time: 514.86s
   Stats end
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ThomasDelteil commented on issue #10537: [MX-307] Add .md tutorials to .ipynb for CI integration

[GitBox]

ThomasDelteil commented on issue #10537: [MX-307] Add .md tutorials to .ipynb 
for CI integration
URL: https://github.com/apache/incubator-mxnet/pull/10537#issuecomment-380988861
 
 
   @thomelane if you have time for a review 😄 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 commented on a change in pull request #10371: [MXNET-263] Support for dot(dns, csr) = dns and dot(dns, csr.T) = dns on GPU

[GitBox]

haojin2 commented on a change in pull request #10371: [MXNET-263] Support for 
dot(dns, csr) = dns and dot(dns, csr.T) = dns on GPU
URL: https://github.com/apache/incubator-mxnet/pull/10371#discussion_r181261640
 
 

 ##
 File path: src/operator/tensor/dot-inl.cuh
 ##
 @@ -895,6 +988,131 @@ inline void DotCsrRspDnsImpl(const OpContext& ctx,
   });
 }
 
+/*
+ * \brief GPU Impl of dot(dns, csr) = csr
+ */
+template
+inline void DotDnsCsrCsrImpl(const OpContext& ctx,
+ const TBlob& lhs, const NDArray& rhs,
+ const OpReqType req, NDArray* ret) {
+  LOG(FATAL) << "dot(dense, csr) = csr is not implemented on GPU";
+}
+
+/*
+ * \brief GPU Impl of dot(dns, csr) = dns and dot(dns, csr.T) = dns
+ */
+template
+inline void DotDnsCsrDnsImpl(const OpContext& ctx,
+ const TBlob& dns, const NDArray& rhs,
+ const OpReqType req, NDArray* ret,
+ const bool transpose_b) {
+  CHECK_EQ(req, kWriteTo);
+  CHECK_EQ(rhs.storage_type(), kCSRStorage);
+
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using nnvm::dim_t;
+
+  /* Initialize data structures */
+  mshadow::Stream* s = ctx.get_stream();
+  TBlob csr_data = rhs.data();
+  TBlob csr_indices = rhs.aux_data(csr::kIdx);
+  TBlob csr_indptr = rhs.aux_data(csr::kIndPtr);
+  if (!rhs.storage_initialized()) {
+FillZerosCsrImpl(s, *ret);
+return;
+  }
+
+  // if dot(dense, csr) = dns, transform to csc first
+  if (!transpose_b) {
+// LOG(FATAL) << "dot(dns, csr) = dns not implemented yet";
+const nnvm::dim_t csr_rows = rhs.shape()[0];
+const nnvm::dim_t csr_cols = rhs.shape()[1];
+const nnvm::dim_t dns_rows = dns.shape_[0];
+const nnvm::dim_t nnz = rhs.storage_shape().Size();
+
+MSHADOW_SGL_DBL_TYPE_SWITCH(csr_data.type_flag_, DType, {
+  MSHADOW_IDX_TYPE_SWITCH(csr_indices.type_flag_, IType, {
+MSHADOW_IDX_TYPE_SWITCH(csr_indptr.type_flag_, CType, {
+  DType* csc_data_ptr = NULL;
+  unsigned long long* csc_indices_ptr = NULL;
+  unsigned long long* csc_indptr_ptr = NULL;
+  unsigned long long* col_counters = NULL;
+  size_t ull_mem_size = sizeof(unsigned long long);
+  void* temp_storage = NULL;
+  size_t temp_storage_bytes = 0;
+  CType out_num_rows = ret->shape()[0];
+  CType out_num_cols = ret->shape()[1];
+  // Get necessary temporary storage amount
+  cub::DeviceScan::ExclusiveSum(NULL,
+temp_storage_bytes,
+csc_indices_ptr,
+csc_indices_ptr,
+csr_cols+1,
+Stream::GetStream(s));
+  temp_storage_bytes += (ull_mem_size - (temp_storage_bytes % 
ull_mem_size));
+  Tensor workspace =
+ctx.requested[0].get_space_typed(
+  Shape1(nnz*sizeof(DType) + nnz*ull_mem_size +
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham commented on issue #10535: render fix

[GitBox]

aaronmarkham commented on issue #10535: render fix
URL: https://github.com/apache/incubator-mxnet/pull/10535#issuecomment-380986807
 
 
   Old view: 
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-10488/2/tutorials/scala/mxnet_scala_on_intellij.html
   
   New view: http://107.21.90.138/tutorials/scala/mxnet_scala_on_intellij.html
   
   You can search for pom.xml to spot the difference. Also, I see that the 
markdown converter is making the numbering all 1's. Will force a fix in a 
second.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhanghang1989 commented on a change in pull request #10536: [MXNET-317] Add Data Parallel

[GitBox]

zhanghang1989 commented on a change in pull request #10536: [MXNET-317] Add 
Data Parallel
URL: https://github.com/apache/incubator-mxnet/pull/10536#discussion_r181260210
 
 

 ##
 File path: python/mxnet/gluon/contrib/parallel.py
 ##
 @@ -0,0 +1,241 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=broad-except, redefined-builtin
+"""Synchronized DataParallel"""
+import threading
+from ... import autograd
+from ...ndarray import NDArray
+from ..utils import split_and_load
+
+__all__ = ['DataParallel', 'Barrier', 'parallel_apply', 'split_kwargs']
+
+
+class Barrier(object):
+"""Shared NDArray for cross device operation.
+
+A cross device operation that allows synchronized push and pull. It can be 
used in
+Cross-gpu Sycnhronized Batch Normalization and Sparse Blocks.
+
+Parameters
+--
+counter : int
+Number of deivces.
+operation : callable
+The cross device operation is applying (e.g. AllReduce).
+"""
+def __init__(self, counter, operation):
+self.mutex = threading.Lock()
+self.all_tasks_done = threading.Condition(self.mutex)
+self.counter = counter
+self.op = operation
+self._clear()
+
+def push(self, x):
+"""Push a NDArray from one of the device.
+Input:
+x (NDArray)
+
+Output:
+idx (int), the output index
+"""
+with self.mutex:
+if self.push_tasks == 0:
+self._clear()
+self.list.append(x)
+idx = len(self.list) - 1
+self.push_tasks -= 1
+
+with self.all_tasks_done:
+if self.push_tasks == 0:
+self.all_tasks_done.notify_all()
+while self.push_tasks:
+self.all_tasks_done.wait()
+
+self._sync_op()
+return idx
+
+def pull(self, idx):
+"""Pull the output to each device
+Input:
+idx (int)
+
+Output:
+out (NDArray)
+"""
+return self.out[idx]
+
+def get(self):
 
 Review comment:
   you mean removing get() function?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin commented on a change in pull request #10536: [MXNET-317] Add Data Parallel

[GitBox]

eric-haibin-lin commented on a change in pull request #10536: [MXNET-317] Add 
Data Parallel
URL: https://github.com/apache/incubator-mxnet/pull/10536#discussion_r181259393
 
 

 ##
 File path: python/mxnet/gluon/contrib/parallel.py
 ##
 @@ -0,0 +1,241 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=broad-except, redefined-builtin
+"""Synchronized DataParallel"""
+import threading
+from ... import autograd
+from ...ndarray import NDArray
+from ..utils import split_and_load
+
+__all__ = ['DataParallel', 'Barrier', 'parallel_apply', 'split_kwargs']
+
+
+class Barrier(object):
+"""Shared NDArray for cross device operation.
+
+A cross device operation that allows synchronized push and pull. It can be 
used in
+Cross-gpu Sycnhronized Batch Normalization and Sparse Blocks.
+
+Parameters
+--
+counter : int
+Number of deivces.
+operation : callable
+The cross device operation is applying (e.g. AllReduce).
+"""
+def __init__(self, counter, operation):
+self.mutex = threading.Lock()
+self.all_tasks_done = threading.Condition(self.mutex)
+self.counter = counter
+self.op = operation
+self._clear()
+
+def push(self, x):
+"""Push a NDArray from one of the device.
+Input:
+x (NDArray)
+
+Output:
+idx (int), the output index
+"""
+with self.mutex:
+if self.push_tasks == 0:
+self._clear()
+self.list.append(x)
+idx = len(self.list) - 1
+self.push_tasks -= 1
+
+with self.all_tasks_done:
+if self.push_tasks == 0:
+self.all_tasks_done.notify_all()
+while self.push_tasks:
+self.all_tasks_done.wait()
+
+self._sync_op()
+return idx
+
+def pull(self, idx):
+"""Pull the output to each device
+Input:
+idx (int)
+
+Output:
+out (NDArray)
+"""
+return self.out[idx]
+
+def get(self):
 
 Review comment:
   I think Sparse can reuse the pull() interface 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 commented on a change in pull request #10533: [MXNET-314] [WIP] Support Integer Type parsing in CSVIter

[GitBox]

haojin2 commented on a change in pull request #10533: [MXNET-314] [WIP] Support 
Integer Type parsing in CSVIter
URL: https://github.com/apache/incubator-mxnet/pull/10533#discussion_r181259524
 
 

 ##
 File path: src/io/iter_csv.cc
 ##
 @@ -88,6 +154,34 @@ class CSVIter: public IIterator {
 end_ = false;
   }
 
+ private:
+  inline TBlob AsTBlob(const dmlc::Row& row, const TShape& 
shape) {
+CHECK_EQ(row.length, shape.Size())
+<< "The data size in CSV do not match size of shape: "
+<< "specified shape=" << shape << ", the csv row-length=" << 
row.length;
+const int* ptr = row.value;
+return TBlob((int*)ptr, shape, cpu::kDevMask, 0);  // NOLINT(*)
+  }
+  // dummy label
+  mshadow::TensorContainer dummy_label;
+  std::unique_ptr > label_parser_;
+  std::unique_ptr > data_parser_;
+};
+
+class CSVIterFloat: public CSVIterBase {
+ public:
+  virtual ~CSVIterFloat() {}
+  virtual void Init(const std::vector >& 
kwargs) {
+param_.InitAllowUnknown(kwargs);
+data_parser_.reset(dmlc::Parser::Create(param_.data_csv.c_str(), 
0, 1, "csv"));
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin commented on a change in pull request #10435: [MXNET-289] Update example to resize data iterator to fix hang in dist sync for last epoch

[GitBox]

eric-haibin-lin commented on a change in pull request #10435: [MXNET-289] 
Update example to resize data iterator to fix hang in dist sync for last epoch
URL: https://github.com/apache/incubator-mxnet/pull/10435#discussion_r181258687
 
 

 ##
 File path: example/image-classification/common/fit.py
 ##
 @@ -155,9 +159,16 @@ def fit(args, network, data_loader, **kwargs):
 head = '%(asctime)-15s Node[' + str(kv.rank) + '] %(message)s'
 logging.basicConfig(level=logging.DEBUG, format=head)
 logging.info('start with arguments %s', args)
+
+epoch_size = get_epoch_size(args, kv)
 
 # data iterators
 (train, val) = data_loader(args, kv)
+if 'dist' in args.kv_store and 'sync' in args.kv_store:
 
 Review comment:
   Is this also required for "async"?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhanghang1989 commented on a change in pull request #10536: [MXNET-317] Add Data Parallel

[GitBox]

zhanghang1989 commented on a change in pull request #10536: [MXNET-317] Add 
Data Parallel
URL: https://github.com/apache/incubator-mxnet/pull/10536#discussion_r181257729
 
 

 ##
 File path: python/mxnet/gluon/contrib/parallel.py
 ##
 @@ -0,0 +1,245 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint : disable=broad-except
+"""Synchronized DataParallel"""
+import threading
+from ... import autograd
+from ...ndarray import NDArray
+from ..utils import split_and_load
+
+__all__ = ['DataParallel', 'Barrier', 'parallel_apply', 'split_kwargs']
+
+
+class Barrier(object):
+"""Shared NDArray for cross device operation.
+
+A cross device operation that allows synchronized push and pull. It can be 
used in
+Cross-gpu Sycnhronized Batch Normalization and Sparse Blocks.
+
+Parameters
+--
+counter : int
+Number of deivces.
+operation : callable
+The cross device operation is applying (e.g. AllReduce).
+"""
+def __init__(self, counter, operation):
+self.mutex = threading.Lock()
+self.all_tasks_done = threading.Condition(self.mutex)
+self.counter = counter
+self.op = operation
+self._clear()
+
+def push(self, x):
+"""Push a NDArray from one of the device.
+Input:
+x (NDArray)
+
+Output:
+idx (int), the output index
+"""
+with self.mutex:
+if self.push_tasks == 0:
+self._clear()
+self.list.append(x)
+idx = len(self.list) - 1
+self.push_tasks -= 1
+
+with self.all_tasks_done:
+if self.push_tasks == 0:
+self.all_tasks_done.notify_all()
+while self.push_tasks:
+self.all_tasks_done.wait()
+
+self._sync_op()
+return idx
+
+def pull(self, F, idx):
+"""Pull the output to each device
+Input:
+idx (int)
+
+Output:
+out (NDArray)
+"""
+return self.out[idx]
+
+def get(self):
+"""Get the operation output
+Output:
+out (NDArray)
+"""
+return self.out
+
+def _sync_op(self):
+with self.mutex:
+if self.reduce_tasks == 1:
+assert(len(self.list) == self.counter)
+self.out = self.op(*self.list)
+if isinstance(self.out, list):
+for xi in self.out:
+xi.wait_to_read()
+else:
+self.out.wait_to_read()
+self.reduce_tasks -= 1
+else:
+self.reduce_tasks -= 1
+
+with self.all_tasks_done:
+if self.reduce_tasks == 0:
+self.all_tasks_done.notify_all()
+while self.reduce_tasks:
+self.all_tasks_done.wait()
+
+def _clear(self):
+self.list = []
+self.push_tasks = self.counter
+self.reduce_tasks = self.counter
+
+def __len__(self):
+return len(self.list)
+
+def __repr__(self):
+return 'ParallelState'
+
+
+class DataParallel(object):
+"""Data parallelism
+
+Hide the difference of single/multiple GPUs to the user.
+Inputs and outputs are both list of NDArrays in different contexts.
+In the forward pass, the module is replicated on each device,
+and each replica handles a portion of the input. During the backwards
+pass, gradients from each replica are summed into the original module.
+
+Parameters
+--
+module : object
+Network to be parallelized.
+ctx_list : list
+A list of contexts
+sync : bool
+enable synchronization (default: False).
+
+
+Inputs:
+- **inputs**: list of input (NDArrays)
+
+Outputs:
+- **outputs**: list of output (NDArrays)
+
+
+Example::
+>>> ctx = [mx.gpu(0), mx.gpu(1)]
+>>> net = DataParallel(model, ctx_list=ctx)
+>>> y = net(x)
+"""
+def __init__(self, module, ctx_list=None, sync=False):
+module.collect_params().reset_ctx(ctx=ctx_list)
+self.ctx_list = ctx_list
+self.module = module
+

[GitHub] eric-haibin-lin commented on a change in pull request #10536: [MXNET-317] Add Data Parallel

[GitBox]

eric-haibin-lin commented on a change in pull request #10536: [MXNET-317] Add 
Data Parallel
URL: https://github.com/apache/incubator-mxnet/pull/10536#discussion_r181257318
 
 

 ##
 File path: python/mxnet/gluon/contrib/parallel.py
 ##
 @@ -0,0 +1,245 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint : disable=broad-except
+"""Synchronized DataParallel"""
+import threading
+from ... import autograd
+from ...ndarray import NDArray
+from ..utils import split_and_load
+
+__all__ = ['DataParallel', 'Barrier', 'parallel_apply', 'split_kwargs']
+
+
+class Barrier(object):
+"""Shared NDArray for cross device operation.
+
+A cross device operation that allows synchronized push and pull. It can be 
used in
+Cross-gpu Sycnhronized Batch Normalization and Sparse Blocks.
+
+Parameters
+--
+counter : int
+Number of deivces.
+operation : callable
+The cross device operation is applying (e.g. AllReduce).
+"""
+def __init__(self, counter, operation):
+self.mutex = threading.Lock()
+self.all_tasks_done = threading.Condition(self.mutex)
+self.counter = counter
+self.op = operation
+self._clear()
+
+def push(self, x):
+"""Push a NDArray from one of the device.
+Input:
+x (NDArray)
+
+Output:
+idx (int), the output index
+"""
+with self.mutex:
+if self.push_tasks == 0:
+self._clear()
+self.list.append(x)
+idx = len(self.list) - 1
+self.push_tasks -= 1
+
+with self.all_tasks_done:
+if self.push_tasks == 0:
+self.all_tasks_done.notify_all()
+while self.push_tasks:
+self.all_tasks_done.wait()
+
+self._sync_op()
+return idx
+
+def pull(self, F, idx):
+"""Pull the output to each device
+Input:
+idx (int)
+
+Output:
+out (NDArray)
+"""
+return self.out[idx]
+
+def get(self):
+"""Get the operation output
+Output:
+out (NDArray)
+"""
+return self.out
+
+def _sync_op(self):
+with self.mutex:
+if self.reduce_tasks == 1:
+assert(len(self.list) == self.counter)
+self.out = self.op(*self.list)
+if isinstance(self.out, list):
+for xi in self.out:
+xi.wait_to_read()
+else:
+self.out.wait_to_read()
+self.reduce_tasks -= 1
+else:
+self.reduce_tasks -= 1
+
+with self.all_tasks_done:
+if self.reduce_tasks == 0:
+self.all_tasks_done.notify_all()
+while self.reduce_tasks:
+self.all_tasks_done.wait()
+
+def _clear(self):
+self.list = []
+self.push_tasks = self.counter
+self.reduce_tasks = self.counter
+
+def __len__(self):
+return len(self.list)
+
+def __repr__(self):
+return 'ParallelState'
+
+
+class DataParallel(object):
+"""Data parallelism
+
+Hide the difference of single/multiple GPUs to the user.
+Inputs and outputs are both list of NDArrays in different contexts.
+In the forward pass, the module is replicated on each device,
+and each replica handles a portion of the input. During the backwards
+pass, gradients from each replica are summed into the original module.
+
+Parameters
+--
+module : object
+Network to be parallelized.
+ctx_list : list
+A list of contexts
+sync : bool
+enable synchronization (default: False).
+
+
+Inputs:
+- **inputs**: list of input (NDArrays)
+
+Outputs:
+- **outputs**: list of output (NDArrays)
+
+
+Example::
+>>> ctx = [mx.gpu(0), mx.gpu(1)]
+>>> net = DataParallel(model, ctx_list=ctx)
+>>> y = net(x)
+"""
+def __init__(self, module, ctx_list=None, sync=False):
+module.collect_params().reset_ctx(ctx=ctx_list)
+self.ctx_list = ctx_list
+self.module = module
+  

[GitHub] eric-haibin-lin commented on a change in pull request #10536: [MXNET-317] Add Data Parallel

[GitBox]

eric-haibin-lin commented on a change in pull request #10536: [MXNET-317] Add 
Data Parallel
URL: https://github.com/apache/incubator-mxnet/pull/10536#discussion_r181257221
 
 

 ##
 File path: python/mxnet/gluon/contrib/parallel.py
 ##
 @@ -0,0 +1,245 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint : disable=broad-except
+"""Synchronized DataParallel"""
+import threading
+from ... import autograd
+from ...ndarray import NDArray
+from ..utils import split_and_load
+
+__all__ = ['DataParallel', 'Barrier', 'parallel_apply', 'split_kwargs']
 
 Review comment:
   Maybe we don't want to include "Barrier" in `__all__` and display it on the 
API doc.  For sparse I'm considering adding & exposing a wrapper around Barrier 
so that users only need to pass in indices and get weights. Normally I don't 
think ppl need to see Barrier. SyncBN can still access it by parallel.Barrier


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhanghang1989 commented on issue #10536: [MXNET-317] Add Data Parallel

[GitBox]

zhanghang1989 commented on issue #10536: [MXNET-317] Add Data Parallel
URL: https://github.com/apache/incubator-mxnet/pull/10536#issuecomment-380982207
 
 
   Hi @eric-haibin-lin @piiswrong , DataParallel and Barrier (ParallelState) 
are included as discussed.
   Barrier is renamed => ParallelState, as suggested by @szha 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin commented on issue #10531: Float16 Support for dot

[GitBox]

eric-haibin-lin commented on issue #10531: Float16 Support for dot
URL: 
https://github.com/apache/incubator-mxnet/issues/10531#issuecomment-380981624
 
 
   @rahul003 @haojin2 are you guys working on that?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #10531: Float16 Support for dot

[GitBox]

rahul003 commented on issue #10531: Float16 Support for dot
URL: 
https://github.com/apache/incubator-mxnet/issues/10531#issuecomment-380977138
 
 
   Removing my earlier comment.
   
   Actually float16 dot does seem supported for GPU. What error did you 
encounter?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #10531: Float16 Support for dot

[GitBox]

rahul003 commented on issue #10531: Float16 Support for dot
URL: 
https://github.com/apache/incubator-mxnet/issues/10531#issuecomment-380977138
 
 
   Removing my earlier comment.
   
   Actually float16 dot does seem supported for GPU. What error did you 
encounter?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #10531: Float16 Support for dot

[GitBox]

rahul003 commented on issue #10531: Float16 Support for dot
URL: 
https://github.com/apache/incubator-mxnet/issues/10531#issuecomment-380956997
 
 
   @haojin2 Looks like we missed this. I went through dot-inl.h. Looks like 
that's disabled for no reason the GPU through the check. Could you take a look? 
There's BlasEngine defined. This might be an easy fix.
   
   The concern might be that there's no CPU support for BLASEngine with half_t, 
but that's fine. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhanghang1989 opened a new pull request #10536: Add Data Parallel

[GitBox]

zhanghang1989 opened a new pull request #10536: Add Data Parallel
URL: https://github.com/apache/incubator-mxnet/pull/10536
 
 
   ## Description ##
   (Brief description on what this PR is about)
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to 
the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues) 
created (except PRs with tiny changes)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing 
distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a 
new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments 
are documented. 
   - For new examples, README.md is added to explain the what the example does, 
the source of the dataset, expected performance on test set and reference to 
the original paper if applicable
   - Check the API doc at 
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
   - [ ] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [ ] Feature1, tests, (and when applicable, API doc)
   - [ ] Feature2, tests, (and when applicable, API doc)
   
   ## Comments ##
   - If this change is a backward incompatible change, why must this change be 
made.
   - Interesting edge cases to note here
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on a change in pull request #10533: [MXNET-314] [WIP] Support Integer Type parsing in CSVIter

[GitBox]

rahul003 commented on a change in pull request #10533: [MXNET-314] [WIP] 
Support Integer Type parsing in CSVIter
URL: https://github.com/apache/incubator-mxnet/pull/10533#discussion_r181238273
 
 

 ##
 File path: src/io/iter_csv.cc
 ##
 @@ -128,23 +229,45 @@ class CSVIter: public IIterator {
 const real_t* ptr = row.value;
 return TBlob((real_t*)ptr, shape, cpu::kDevMask, 0);  // NOLINT(*)
   }
-
-  CSVIterParam param_;
-  // output instance
-  DataInst out_;
-  // internal instance counter
-  unsigned inst_counter_{0};
-  // at end
-  bool end_{false};
   // dummy label
   mshadow::TensorContainer dummy_label;
-  // label parser
-  size_t label_ptr_{0}, label_size_{0};
-  size_t data_ptr_{0}, data_size_{0};
   std::unique_ptr > label_parser_;
   std::unique_ptr > data_parser_;
 };
 
+class CSVIter: public IIterator {
+ public:
+  CSVIter() {}
+  virtual ~CSVIter() {}
+
+  // intialize iterator loads data in
+  virtual void Init(const std::vector >& 
kwargs) {
+param_.InitAllowUnknown(kwargs);
+if (param_.dtype.has_value() && param_.dtype.value() == mshadow::kInt32) {
+  iterator_.reset(reinterpret_cast(new CSVIterInt()));
+} else if (!param_.dtype.has_value() || param_.dtype.value() == 
mshadow::kFloat32) {
 
 Review comment:
   Is this value syntax how optional works?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham commented on a change in pull request #10534: [MXNET-315] Fix the version select in the navbar to be independent of order

[GitBox]

aaronmarkham commented on a change in pull request #10534: [MXNET-315] Fix the 
version select in the navbar to be independent of order
URL: https://github.com/apache/incubator-mxnet/pull/10534#discussion_r181246453
 
 

 ##
 File path: docs/build_version_doc/update_all_version.sh
 ##
 @@ -93,7 +93,7 @@ function update_mxnet_css {
 for tag in $tag_list; do
 # This Python script is expecting the tag_list.txt and it will use that as 
the entries to populate
 
-python AddVersion.py --root_url "$root_url" --file_path 
"$built/versions/$tag" --current_version "$tag" || exit 1
+python AddVersion.py --root_url "$root_url" --file_path 
"$built/versions/$tag" --current_version "$tag" --default_tag "$tag_default" || 
exit 1
 
 Review comment:
   Did you need to flip the argument or could you be consistent?
   default_tag vs tag_default?
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham commented on issue #10535: render fix

[GitBox]

aaronmarkham commented on issue #10535: render fix
URL: https://github.com/apache/incubator-mxnet/pull/10535#issuecomment-380968048
 
 
   @eric-haibin-lin - please review... should get this in for RC0...


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudh2290 commented on issue #10431: Elementwise multiplication of two ndarrays returns wrong results

[GitBox]

anirudh2290 commented on issue #10431: Elementwise multiplication of two 
ndarrays returns wrong results
URL: 
https://github.com/apache/incubator-mxnet/issues/10431#issuecomment-380967924
 
 
   You have to add fix  in the PR for it to automatically close.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham opened a new pull request #10535: render fix

[GitBox]

aaronmarkham opened a new pull request #10535: render fix
URL: https://github.com/apache/incubator-mxnet/pull/10535
 
 
   ## Description ##
   
   Display patch fix for a render issue. 
   
   For some reason the markdown converter is messing up on a comment tag 
(ridiculous):
   
   ```
   
   ```
   
   After that comment was hit, backticks didn't render and tags like 
`` were being turned into html, among other shenanigans:
   
   ```
   - Specify project profiles and platforms in `pom.xml` by pasting the 
following content below the `` tag:
   ```
   
   Root cause TBD.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] harusametime commented on issue #10528: [MXNET-316] Remove empty buckets causing index errors

[GitBox]

harusametime commented on issue #10528: [MXNET-316] Remove empty buckets 
causing index errors
URL: https://github.com/apache/incubator-mxnet/pull/10528#issuecomment-380967400
 
 
   @ThomasDelteil  Thank you for your testing. I created a ticket and added the 
label [MXNET-316] to the title of my PR.



This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudh2290 closed issue #10431: Elementwise multiplication of two ndarrays returns wrong results

[GitBox]

anirudh2290 closed issue #10431: Elementwise multiplication of two ndarrays 
returns wrong results
URL: https://github.com/apache/incubator-mxnet/issues/10431
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] harusametime commented on issue #10528: [MXNET-316] Remove empty buckets causing index errors

[GitBox]

harusametime commented on issue #10528: [MXNET-316] Remove empty buckets 
causing index errors
URL: https://github.com/apache/incubator-mxnet/pull/10528#issuecomment-380967400
 
 
   @ThomasDelteil  Thank you for your comment. I created a ticket and added the 
label [MXNET-316] to the title of my PR.



This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on a change in pull request #10533: [MXNET-314] [WIP] Support Integer Type parsing in CSVIter

[GitBox]

rahul003 commented on a change in pull request #10533: [MXNET-314] [WIP] 
Support Integer Type parsing in CSVIter
URL: https://github.com/apache/incubator-mxnet/pull/10533#discussion_r181237709
 
 

 ##
 File path: src/io/iter_csv.cc
 ##
 @@ -54,27 +57,90 @@ struct CSVIterParam : public dmlc::Parameter 
{
 index_t shape1[] = {1};
 DMLC_DECLARE_FIELD(label_shape).set_default(TShape(shape1, shape1 + 1))
 .describe("The shape of one label.");
+DMLC_DECLARE_FIELD(dtype)
+  .add_enum("float32", mshadow::kFloat32)
+  .add_enum("int32", mshadow::kInt32)
+  .set_default(dmlc::optional())
 
 Review comment:
   --


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] roywei commented on issue #10429: [MXNET-210]give warning for variables with same name in graph visualization

[GitBox]

roywei commented on issue #10429: [MXNET-210]give warning for variables with 
same name in graph visualization
URL: https://github.com/apache/incubator-mxnet/pull/10429#issuecomment-380963420
 
 
   @marcoabreu I am adding a unit test which require graphviz, may I know where 
shall I configure that in CI for Windows? Thanks


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on a change in pull request #10533: [MXNET-314] [WIP] Support Integer Type parsing in CSVIter

[GitBox]

rahul003 commented on a change in pull request #10533: [MXNET-314] [WIP] 
Support Integer Type parsing in CSVIter
URL: https://github.com/apache/incubator-mxnet/pull/10533#discussion_r181237908
 
 

 ##
 File path: src/io/iter_csv.cc
 ##
 @@ -88,6 +154,34 @@ class CSVIter: public IIterator {
 end_ = false;
   }
 
+ private:
+  inline TBlob AsTBlob(const dmlc::Row& row, const TShape& 
shape) {
+CHECK_EQ(row.length, shape.Size())
+<< "The data size in CSV do not match size of shape: "
+<< "specified shape=" << shape << ", the csv row-length=" << 
row.length;
+const int* ptr = row.value;
+return TBlob((int*)ptr, shape, cpu::kDevMask, 0);  // NOLINT(*)
+  }
+  // dummy label
+  mshadow::TensorContainer dummy_label;
+  std::unique_ptr > label_parser_;
+  std::unique_ptr > data_parser_;
+};
+
+class CSVIterFloat: public CSVIterBase {
+ public:
+  virtual ~CSVIterFloat() {}
+  virtual void Init(const std::vector >& 
kwargs) {
+param_.InitAllowUnknown(kwargs);
+data_parser_.reset(dmlc::Parser::Create(param_.data_csv.c_str(), 
0, 1, "csv"));
 
 Review comment:
   Just for better readability, maybe explicitly specify the dtype for data as 
well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on a change in pull request #10533: [MXNET-314] [WIP] Support Integer Type parsing in CSVIter

[GitBox]

rahul003 commented on a change in pull request #10533: [MXNET-314] [WIP] 
Support Integer Type parsing in CSVIter
URL: https://github.com/apache/incubator-mxnet/pull/10533#discussion_r181238273
 
 

 ##
 File path: src/io/iter_csv.cc
 ##
 @@ -128,23 +229,45 @@ class CSVIter: public IIterator {
 const real_t* ptr = row.value;
 return TBlob((real_t*)ptr, shape, cpu::kDevMask, 0);  // NOLINT(*)
   }
-
-  CSVIterParam param_;
-  // output instance
-  DataInst out_;
-  // internal instance counter
-  unsigned inst_counter_{0};
-  // at end
-  bool end_{false};
   // dummy label
   mshadow::TensorContainer dummy_label;
-  // label parser
-  size_t label_ptr_{0}, label_size_{0};
-  size_t data_ptr_{0}, data_size_{0};
   std::unique_ptr > label_parser_;
   std::unique_ptr > data_parser_;
 };
 
+class CSVIter: public IIterator {
+ public:
+  CSVIter() {}
+  virtual ~CSVIter() {}
+
+  // intialize iterator loads data in
+  virtual void Init(const std::vector >& 
kwargs) {
+param_.InitAllowUnknown(kwargs);
+if (param_.dtype.has_value() && param_.dtype.value() == mshadow::kInt32) {
+  iterator_.reset(reinterpret_cast(new CSVIterInt()));
+} else if (!param_.dtype.has_value() || param_.dtype.value() == 
mshadow::kFloat32) {
 
 Review comment:
   Is this value syntax how optional works?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on a change in pull request #10533: [MXNET-314] [WIP] Support Integer Type parsing in CSVIter

[GitBox]

rahul003 commented on a change in pull request #10533: [MXNET-314] [WIP] 
Support Integer Type parsing in CSVIter
URL: https://github.com/apache/incubator-mxnet/pull/10533#discussion_r181237709
 
 

 ##
 File path: src/io/iter_csv.cc
 ##
 @@ -54,27 +57,90 @@ struct CSVIterParam : public dmlc::Parameter 
{
 index_t shape1[] = {1};
 DMLC_DECLARE_FIELD(label_shape).set_default(TShape(shape1, shape1 + 1))
 .describe("The shape of one label.");
+DMLC_DECLARE_FIELD(dtype)
+  .add_enum("float32", mshadow::kFloat32)
+  .add_enum("int32", mshadow::kInt32)
+  .set_default(dmlc::optional())
 
 Review comment:
   Could you default to float32?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #10431: Elementwise multiplication of two ndarrays returns wrong results

[GitBox]

rahul003 commented on issue #10431: Elementwise multiplication of two ndarrays 
returns wrong results
URL: 
https://github.com/apache/incubator-mxnet/issues/10431#issuecomment-380957781
 
 
   @anirudh2290 Could you close this? I thought that would be done 
automatically by linking PR to an issue?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #10431: Elementwise multiplication of two ndarrays returns wrong results

[GitBox]

rahul003 commented on issue #10431: Elementwise multiplication of two ndarrays 
returns wrong results
URL: 
https://github.com/apache/incubator-mxnet/issues/10431#issuecomment-380957781
 
 
   @anirudh2290 Could you close this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #10531: Float16 Support for dot

[GitBox]

rahul003 commented on issue #10531: Float16 Support for dot
URL: 
https://github.com/apache/incubator-mxnet/issues/10531#issuecomment-380956997
 
 
   @haojin2 Looks like we missed this. I went through dot-inl.h. Looks like 
that's disabled for no reason the GPU through the check. Could you take a look? 
There's BlasEngine defined. This might be an easy fix.
   
   The concern might be that there's no CPU support for BLASEngine with half_t, 
but that's fine. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kpmurali opened a new pull request #10534: [MXNET-315] Fix the version select in the navbar to be independent of order

[GitBox]

kpmurali opened a new pull request #10534: [MXNET-315] Fix the version select 
in the navbar to be independent of order
URL: https://github.com/apache/incubator-mxnet/pull/10534
 
 
   ## Description ##
   Fix the version select in the navbar to be independent of the order versions 
listed. Currently, the default version is taken to be the version at the top of 
the navbar. Changed the AddVersion script to take it as a parameter.
   
   ### Changes ###
   - [  x ] Updated update_all_version.sh to pass the default tag as a 
parameter to AddVersion
   - [  x ] Use the default tag parameter to decide the root url
   
   Tested manually by Krishnan and Aaron at http://54.210.6.225/ .
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 opened a new pull request #10533: [MXNET-314] [WIP] Support Integer Type parsing in CSVIter

[GitBox]

haojin2 opened a new pull request #10533: [MXNET-314] [WIP] Support Integer 
Type parsing in CSVIter
URL: https://github.com/apache/incubator-mxnet/pull/10533
 
 
   ## Description ##
   Support parsing of int32 type in CSVIter
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [x] The PR title starts with [MXNET-314]
   - [x] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing 
distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a 
new build option with NCCL)
   - [x] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments 
are documented. 
   - For new examples, README.md is added to explain the what the example does, 
the source of the dataset, expected performance on test set and reference to 
the original paper if applicable
   - Check the API doc at 
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
   - [x] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [x] Support integer data type in CSVIter
   - [ ] Corresponding tests for the new CSVIter
   
   ## Comments ##
   In the past all parsers default to parse the files in real_t type, but when 
the files contain large integers we lose precision. e.g. 2001 actually will 
have the same float32 representation as 2000. To provide a way to 
accurately read the data for users who have such needs, we need to let the 
iterators be able to support parsing the input texts as integers.
   This PR depends on dmlc/dmlc-core#388 to build work properly.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dabraude commented on a change in pull request #10524: [MXNET-312] Added Matthew's Correlation Coefficient to metrics

[GitBox]

dabraude commented on a change in pull request #10524: [MXNET-312] Added 
Matthew's Correlation Coefficient to metrics
URL: https://github.com/apache/incubator-mxnet/pull/10524#discussion_r181226180
 
 

 ##
 File path: python/mxnet/metric.py
 ##
 @@ -661,6 +682,107 @@ def reset(self):
 self.metrics.reset_stats()
 
 
+@register
+class MCC(EvalMetric):
+"""Computes the Matthews Correlation Coefficient of a binary 
classification problem.
+
+While slower to compute the MCC can give insight that F1 or Accuracy 
cannot.
+For instance, if the network always predicts the same result
+then the MCC will immeadiately show this. The MCC is also symetric with 
respect
+to positive and negative catagorisation, however, there needs to be both
 
 Review comment:
   Thanks fixed (and changed to US spelling)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dabraude commented on a change in pull request #10524: [MXNET-312] Added Matthew's Correlation Coefficient to metrics

[GitBox]

dabraude commented on a change in pull request #10524: [MXNET-312] Added 
Matthew's Correlation Coefficient to metrics
URL: https://github.com/apache/incubator-mxnet/pull/10524#discussion_r181226099
 
 

 ##
 File path: python/mxnet/metric.py
 ##
 @@ -661,6 +682,107 @@ def reset(self):
 self.metrics.reset_stats()
 
 
+@register
+class MCC(EvalMetric):
+"""Computes the Matthews Correlation Coefficient of a binary 
classification problem.
+
+While slower to compute the MCC can give insight that F1 or Accuracy 
cannot.
+For instance, if the network always predicts the same result
+then the MCC will immeadiately show this. The MCC is also symetric with 
respect
+to positive and negative catagorisation, however, there needs to be both
+positive and negative examples in the labels or it will always return 0.
+MCC of 0 is uncorrelated, 1 is completely correlated, and -1 is negatively 
correlated.
+
+.. math::
+\\text{MCC} = \\frac{ TP \\times TN - FP \\times FN }
+{\\sqrt{ (TP + FP) ( TP + FN ) ( TN + FP ) ( TN + FN ) } }
+
+where 0 terms in the denominator are replaced by 1.
+
+.. note::
+
+This MCC only supports binary classification.
+
+Parameters
+--
+name : str
+Name of this metric instance for display.
+output_names : list of str, or None
+Name of predictions that should be used when updating with update_dict.
+By default include all predictions.
+label_names : list of str, or None
+Name of labels that should be used when updating with update_dict.
+By default include all labels.
+average : str, default 'macro'
+Strategy to be used for aggregating across mini-batches.
+"macro": average the MCC for each batch.
+"micro": compute a single MCC across all batches.
+
+Examples
+
+In this example the network almost always predicts positive
+>>> false_positives = 1000
+>>> false_negatives = 1
+>>> true_positives = 1
+>>> true_negatives = 1
+>>> predicts = [mx.nd.array(
+[[.3, .7]]*false_positives +
+[[.7, .3]]*true_negatives +
+[[.7, .3]]*false_negatives +
+[[.3, .7]]*true_positives
+)]
+>>> labels  = [mx.nd.array(
+[0.]*(false_positives + true_negatives) +
+[1.]*(false_negatives + true_positives)
+)]
+>>> f1 = mx.metric.F1()
+>>> f1.update(preds = predicts, labels = labels)
+>>> mcc = mx.metric.MCC()
+>>> mcc.update(preds = predicts, labels = labels)
+>>> print f1.get()
+('f1', 0.95233560306652054)
+>>> print mcc.get()
+('mcc', 0.01917751877733392)
+"""
+
+def __init__(self, name='mcc',
+ output_names=None, label_names=None, average="macro"):
+self.average = average
+self.metrics = _BinaryClassificationMetrics()
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] KaiserSozo opened a new issue #10532: NDArray failed to allocate CPU memory

[GitBox]

KaiserSozo opened a new issue #10532: NDArray failed to allocate CPU memory
URL: https://github.com/apache/incubator-mxnet/issues/10532
 
 
   Traceback (most recent call last):
 File "SOM.py", line 109, in 
   test_som_with_color_data()
 File "SOM.py", line 97, in test_som_with_color_data
   img2 = nd.reshape(weights.data(),shape=(som_dim,som_dim,-1)).asnumpy()
 File 
"C:\Users\Vitaly\Anaconda2\lib\site-packages\mxnet\ndarray\ndarray.py", line 
1868, in asnumpy
   ctypes.c_size_t(data.size)))
 File "C:\Users\Vitaly\Anaconda2\lib\site-packages\mxnet\base.py", line 
149, in check_call
   raise MXNetError(py_str(_LIB.MXGetLastError()))
   mxnet.base.MXNetError: [15:44:16] d:\program files 
(x86)\jenkins\workspace\mxnet\mxnet\src\storage\./cpu_device_storage.h:70: 
Failed to allocate CPU Memory
   
   code:
   
  som_dim = 100
   net = SOMNetwork(input_dim=3, dim=som_dim, sigma=3)
   net.collect_params().initialize(ctx=ctx)
   net.hybridize()
   test_data = nd.random.uniform(0, 1, (500, 3))
   weights = net.params.get('weight')
   img1 = nd.reshape(weights.data(), shape=(som_dim,som_dim,-1)).asnumpy()
   plt.figure(1)
   plt.subplot(121)
   plt.imshow(img1)
   start = time.time()
   for i, (data) in enumerate(test_data):
   net.n = i
   data = data.as_in_context(ctx)
   output = net(data)
   weights.set_data(weights.data() + output)
   end = time.time()
   print(end - start)
   img2 = nd.reshape(weights.data(),shape=(som_dim,som_dim,-1)).asnumpy()
   plt.subplot(122)
   plt.imshow(img2)
   plt.show()
   
   As I understand fails asnumpy function. But as for img1 and for img2 shape 
of NDArray is 100x100x3.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] marcoabreu commented on issue #10525: Fix NNPACK header file position error

[GitBox]

marcoabreu commented on issue #10525: Fix NNPACK header file position error
URL: https://github.com/apache/incubator-mxnet/pull/10525#issuecomment-380936468
 
 
   Would you mind taking care of that? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ThomasDelteil commented on a change in pull request #10524: [MXNET-312] Added Matthew's Correlation Coefficient to metrics

[GitBox]

ThomasDelteil commented on a change in pull request #10524: [MXNET-312] Added 
Matthew's Correlation Coefficient to metrics
URL: https://github.com/apache/incubator-mxnet/pull/10524#discussion_r181213305
 
 

 ##
 File path: python/mxnet/metric.py
 ##
 @@ -661,6 +682,107 @@ def reset(self):
 self.metrics.reset_stats()
 
 
+@register
+class MCC(EvalMetric):
+"""Computes the Matthews Correlation Coefficient of a binary 
classification problem.
+
+While slower to compute the MCC can give insight that F1 or Accuracy 
cannot.
+For instance, if the network always predicts the same result
+then the MCC will immeadiately show this. The MCC is also symetric with 
respect
+to positive and negative catagorisation, however, there needs to be both
 
 Review comment:
   categorisation*


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong closed pull request #10410: Fix output names of nn operators.

[GitBox]

piiswrong closed pull request #10410: Fix output names of nn operators.
URL: https://github.com/apache/incubator-mxnet/pull/10410
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/executor/graph_executor.cc b/src/executor/graph_executor.cc
index 9108bae1732..4d24f5501ef 100644
--- a/src/executor/graph_executor.cc
+++ b/src/executor/graph_executor.cc
@@ -1457,6 +1457,7 @@ void GraphExecutor::ExecuteMonCallback(size_t nid) {
   output_names.emplace_back(std::to_string(i));
 }
   }
+  CHECK_EQ(opnode.exec->out_array.size(), output_names.size());
   for (index_t i = 0; i < opnode.exec->out_array.size(); ++i) {
 NDArray *cpy = new NDArray(opnode.exec->out_array[i]);
 std::string name = inode.source->attrs.name + "_" + output_names[i];
diff --git a/src/operator/nn/activation.cc b/src/operator/nn/activation.cc
index 08028265c48..382efeb1447 100644
--- a/src/operator/nn/activation.cc
+++ b/src/operator/nn/activation.cc
@@ -150,6 +150,10 @@ The following activation functions are supported:
 )code" ADD_FILELINE)
 .set_attr_parser(ParamParser)
 .set_attr("FInferStorageType", ActivationStorageType)
+.set_attr("FListOutputNames",
+[](const NodeAttrs& attrs) {
+return std::vector{"output"};
+})
 .set_attr("FCompute", ActivationCompute)
 #if MXNET_USE_MKLDNN == 1
 .set_attr("FComputeEx", ActivationComputeExCPU)
diff --git a/src/operator/nn/concat.cc b/src/operator/nn/concat.cc
index 81dc95f1a5a..a7fcb1c8817 100644
--- a/src/operator/nn/concat.cc
+++ b/src/operator/nn/concat.cc
@@ -248,6 +248,10 @@ Example::
   }
   return ret;
 })
+.set_attr("FListOutputNames",
+[](const NodeAttrs& attrs) {
+return std::vector{"output"};
+})
 #if MXNET_USE_MKLDNN == 1
 .set_attr("FResourceRequest", [](const NodeAttrs& n) {
   return std::vector{ResourceRequest::kTempSpace};
diff --git a/src/operator/nn/pooling.cc b/src/operator/nn/pooling.cc
index f719e0753e0..9843cb7f4f6 100644
--- a/src/operator/nn/pooling.cc
+++ b/src/operator/nn/pooling.cc
@@ -368,7 +368,11 @@ height, width)*.
 })
 .set_attr("FListOutputNames",
 [](const NodeAttrs& attrs) {
-  return std::vector{"output"};
+  const PoolingParam ¶m = nnvm::get(attrs.parsed);
+  if (GetNumOutputs(param) == 2)
+return std::vector{"output", "workspace"};
+  else
+return std::vector{"output"};
 })
 .set_attr_parser(PoolingParamParser)
 .set_attr("FInferStorageType", PoolingStorageType)
diff --git a/src/operator/nn/softmax.cc b/src/operator/nn/softmax.cc
index 0f559475d1c..f8cc6fee9a2 100644
--- a/src/operator/nn/softmax.cc
+++ b/src/operator/nn/softmax.cc
@@ -96,6 +96,10 @@ Example::
 
 )code" ADD_FILELINE)
 .set_attr_parser(ParamParser)
+.set_attr("FListOutputNames",
+[](const NodeAttrs& attrs) {
+return std::vector{"output"};
+})
 .set_attr("FCompute", SoftmaxCompute)
 #if MXNET_USE_MKLDNN == 1
 .set_attr("FComputeEx", SoftmaxComputeExCPU)
diff --git a/src/operator/nn/softmax_activation.cc 
b/src/operator/nn/softmax_activation.cc
index bdfd8b065de..8a28243dfce 100644
--- a/src/operator/nn/softmax_activation.cc
+++ b/src/operator/nn/softmax_activation.cc
@@ -58,6 +58,10 @@ Example::
 
 )code" ADD_FILELINE)
 .set_attr_parser(ParamParser)
+.set_attr("FListOutputNames",
+[](const NodeAttrs& attrs) {
+return std::vector{"output"};
+})
 .set_attr("FCompute", SoftmaxActivationCompute)
 .set_attr("FGradient", 
ElemwiseGradUseOut{"_backward_SoftmaxActivation"})
 .add_arguments(SoftmaxActivationParam::__FIELDS__());
diff --git a/src/operator/nn/upsampling.cc b/src/operator/nn/upsampling.cc
index 44b619ac951..5aa111e26f7 100644
--- a/src/operator/nn/upsampling.cc
+++ b/src/operator/nn/upsampling.cc
@@ -132,6 +132,10 @@ NNVM_REGISTER_OP(UpSampling)
 [](const NodeAttrs& attrs) {
   return ListArguments(nnvm::get(attrs.parsed));
 })
+.set_attr("FListOutputNames",
+[](const NodeAttrs& attrs) {
+return std::vector{"output"};
+})
 .set_attr("FInferShape", UpSamplingShape)
 .set_attr("FInferType", UpSamplingType)
 .set_attr("FResourceRequest", [](const NodeAttrs& n) {
diff --git a/tests/python/unittest/test_operator.py 
b/tests/python/unittest/test_operator.py
index a0ae4807e38..4451aa25ba0 100644
--- a/tests/python/unittest/test_operator.py
+++ b/tests/python/unittest/test_operator.py
@@ -5444,6 +5444,25 @@ def get_output_names_callback(name, arr):
 lrn_sym = mx.sym.LRN(data, nsize=1, name='lrn')
 check_name(lrn_sym, ['lrn_output', 'lrn_tmp_norm'])
 
+act_sym = mx.sym.Activation(data, act_type='relu', name='act')
+check_name(act_sym, ['act_output'])
+
+cc_sym = mx.sym.concat(data, data, dim=0, name='concat')
+check_name(cc_sym, ['concat_output'])
+
+sm_sym = mx.sym.softmax(data, name='softmax')
+check_name(sm_sym, ['softmax_output'])
+
+sa_sym = mx.sym.Softm

[GitHub] piiswrong closed pull request #10515: Prevent partially update of ParameterDict

[GitBox]

piiswrong closed pull request #10515: Prevent partially update of ParameterDict
URL: https://github.com/apache/incubator-mxnet/pull/10515
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/python/mxnet/gluon/parameter.py b/python/mxnet/gluon/parameter.py
index ab511ab6e42..8d0c5ba7af4 100644
--- a/python/mxnet/gluon/parameter.py
+++ b/python/mxnet/gluon/parameter.py
@@ -661,8 +661,9 @@ def update(self, other):
 assert self._params[k] is v, \
 "Cannot update self with other because they have different 
" \
 "Parameters with the same name '%s'"%k
-else:
-self._params[k] = v
+
+for k, v in other.items():
+self._params[k] = v
 
 def initialize(self, init=initializer.Uniform(), ctx=None, verbose=False,
force_reinit=False):


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-mxnet] branch master updated: Prevent partially update of ParameterDict (#10515)

[jxie]

This is an automated email from the ASF dual-hosted git repository.

jxie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 4852a99  Prevent partially update of ParameterDict (#10515)
4852a99 is described below

commit 4852a99d096682b83d8e2093240e8a9022f4699e
Author: Kenta Murata 
AuthorDate: Fri Apr 13 05:01:37 2018 +0900

Prevent partially update of ParameterDict (#10515)

* Prevent partially update of ParameterDict if keys are conflicted

* Update parameter.py
---
 python/mxnet/gluon/parameter.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/python/mxnet/gluon/parameter.py b/python/mxnet/gluon/parameter.py
index ab511ab..8d0c5ba 100644
--- a/python/mxnet/gluon/parameter.py
+++ b/python/mxnet/gluon/parameter.py
@@ -661,8 +661,9 @@ class ParameterDict(object):
 assert self._params[k] is v, \
 "Cannot update self with other because they have different 
" \
 "Parameters with the same name '%s'"%k
-else:
-self._params[k] = v
+
+for k, v in other.items():
+self._params[k] = v
 
 def initialize(self, init=initializer.Uniform(), ctx=None, verbose=False,
force_reinit=False):

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[incubator-mxnet] branch master updated: Fix output names of nn operators. (#10410)

[jxie]

This is an automated email from the ASF dual-hosted git repository.

jxie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new b96affb  Fix output names of nn operators. (#10410)
b96affb is described below

commit b96affb251a72f4f10895c8adced1acbca19edb0
Author: Da Zheng 
AuthorDate: Thu Apr 12 13:01:57 2018 -0700

Fix output names of nn operators. (#10410)

* Fix output names of nn operators.

* Fix typos.

* add tests.

* add pooling output names.

* add checks.

* Fix tests.

* Print tests.

* Fix a bug in the test.

* add test for pooling outputs.

* Update test_operator.py
---
 src/executor/graph_executor.cc |  1 +
 src/operator/nn/activation.cc  |  4 
 src/operator/nn/concat.cc  |  4 
 src/operator/nn/pooling.cc |  6 +-
 src/operator/nn/softmax.cc |  4 
 src/operator/nn/softmax_activation.cc  |  4 
 src/operator/nn/upsampling.cc  |  4 
 tests/python/unittest/test_operator.py | 19 +++
 8 files changed, 45 insertions(+), 1 deletion(-)

diff --git a/src/executor/graph_executor.cc b/src/executor/graph_executor.cc
index 9108bae..4d24f55 100644
--- a/src/executor/graph_executor.cc
+++ b/src/executor/graph_executor.cc
@@ -1457,6 +1457,7 @@ void GraphExecutor::ExecuteMonCallback(size_t nid) {
   output_names.emplace_back(std::to_string(i));
 }
   }
+  CHECK_EQ(opnode.exec->out_array.size(), output_names.size());
   for (index_t i = 0; i < opnode.exec->out_array.size(); ++i) {
 NDArray *cpy = new NDArray(opnode.exec->out_array[i]);
 std::string name = inode.source->attrs.name + "_" + output_names[i];
diff --git a/src/operator/nn/activation.cc b/src/operator/nn/activation.cc
index 0802826..382efeb 100644
--- a/src/operator/nn/activation.cc
+++ b/src/operator/nn/activation.cc
@@ -150,6 +150,10 @@ The following activation functions are supported:
 )code" ADD_FILELINE)
 .set_attr_parser(ParamParser)
 .set_attr("FInferStorageType", ActivationStorageType)
+.set_attr("FListOutputNames",
+[](const NodeAttrs& attrs) {
+return std::vector{"output"};
+})
 .set_attr("FCompute", ActivationCompute)
 #if MXNET_USE_MKLDNN == 1
 .set_attr("FComputeEx", ActivationComputeExCPU)
diff --git a/src/operator/nn/concat.cc b/src/operator/nn/concat.cc
index 81dc95f..a7fcb1c 100644
--- a/src/operator/nn/concat.cc
+++ b/src/operator/nn/concat.cc
@@ -248,6 +248,10 @@ Example::
   }
   return ret;
 })
+.set_attr("FListOutputNames",
+[](const NodeAttrs& attrs) {
+return std::vector{"output"};
+})
 #if MXNET_USE_MKLDNN == 1
 .set_attr("FResourceRequest", [](const NodeAttrs& n) {
   return std::vector{ResourceRequest::kTempSpace};
diff --git a/src/operator/nn/pooling.cc b/src/operator/nn/pooling.cc
index 7ee655f..9e4063e 100644
--- a/src/operator/nn/pooling.cc
+++ b/src/operator/nn/pooling.cc
@@ -365,7 +365,11 @@ height, width)*.
 })
 .set_attr("FListOutputNames",
 [](const NodeAttrs& attrs) {
-  return std::vector{"output"};
+  const PoolingParam ¶m = nnvm::get(attrs.parsed);
+  if (GetNumOutputs(param) == 2)
+return std::vector{"output", "workspace"};
+  else
+return std::vector{"output"};
 })
 .set_attr_parser(PoolingParamParser)
 .set_attr("FInferStorageType", PoolingStorageType)
diff --git a/src/operator/nn/softmax.cc b/src/operator/nn/softmax.cc
index 0f55947..f8cc6fe 100644
--- a/src/operator/nn/softmax.cc
+++ b/src/operator/nn/softmax.cc
@@ -96,6 +96,10 @@ Example::
 
 )code" ADD_FILELINE)
 .set_attr_parser(ParamParser)
+.set_attr("FListOutputNames",
+[](const NodeAttrs& attrs) {
+return std::vector{"output"};
+})
 .set_attr("FCompute", SoftmaxCompute)
 #if MXNET_USE_MKLDNN == 1
 .set_attr("FComputeEx", SoftmaxComputeExCPU)
diff --git a/src/operator/nn/softmax_activation.cc 
b/src/operator/nn/softmax_activation.cc
index bdfd8b0..8a28243 100644
--- a/src/operator/nn/softmax_activation.cc
+++ b/src/operator/nn/softmax_activation.cc
@@ -58,6 +58,10 @@ Example::
 
 )code" ADD_FILELINE)
 .set_attr_parser(ParamParser)
+.set_attr("FListOutputNames",
+[](const NodeAttrs& attrs) {
+return std::vector{"output"};
+})
 .set_attr("FCompute", SoftmaxActivationCompute)
 .set_attr("FGradient", 
ElemwiseGradUseOut{"_backward_SoftmaxActivation"})
 .add_arguments(SoftmaxActivationParam::__FIELDS__());
diff --git a/src/operator/nn/upsampling.cc b/src/operator/nn/upsampling.cc
index 44b619a..5aa111e 100644
--- a/src/operator/nn/upsampling.cc
+++ b/src/operator/nn/upsampling.cc
@@ -132,6 +132,10 @@ NNVM_REGISTER_OP(UpSampling)
 [](const NodeAttrs& attrs) {
   return ListArguments(nnvm::get(attrs.parsed));
 })
+.set_attr("FListOutputNames",
+[](const NodeAttrs& attrs) {
+return std::vector{"output"};
+})
 .set_attr("FInferShape", UpSamplingShape)
 .set_attr("FInferType", UpSamp

[GitHub] piiswrong commented on issue #10425: [MXNET-175] NdArrayiterator dict fix

[GitBox]

piiswrong commented on issue #10425: [MXNET-175] NdArrayiterator dict fix
URL: https://github.com/apache/incubator-mxnet/pull/10425#issuecomment-380925451
 
 
   That's incorrect usage. We shouldn't address it by special casing 
ndarrayiter in module.
   
   One thing we can do is to check data_iter.provide_data against 
module(data_names) and raise a warning if they don't match


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong closed pull request #10424: [MXNET-185] Improved error message

[GitBox]

piiswrong closed pull request #10424: [MXNET-185] Improved error message
URL: https://github.com/apache/incubator-mxnet/pull/10424
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/python/mxnet/gluon/block.py b/python/mxnet/gluon/block.py
index fd95641274d..64003585dba 100644
--- a/python/mxnet/gluon/block.py
+++ b/python/mxnet/gluon/block.py
@@ -458,7 +458,13 @@ def _build_cache(self, *args):
 for name in out.list_inputs()]
 
 def _finish_deferred_init(self, hybrid, *args):
-self.infer_shape(*args)
+try:
+self.infer_shape(*args)
+except Exception as e:
+error_msg = "Deferred initialization failed because shape"\
+" cannot be inferred \n {}".format(e)
+raise ValueError(error_msg)
+
 if hybrid:
 for is_arg, i in self._cached_op_args:
 if not is_arg:
@@ -509,11 +515,14 @@ def _infer_attrs(self, infer_fn, attr, *args):
 """Generic infer attributes."""
 inputs, out = self._get_graph(*args)
 args, _ = _flatten(args)
-arg_attrs, _, aux_attrs = getattr(out, infer_fn)(
-**{i.name: getattr(j, attr) for i, j in zip(inputs, args)})
+with warnings.catch_warnings(record=True) as w:
+arg_attrs, _, aux_attrs = getattr(out, infer_fn)(
+**{i.name: getattr(j, attr) for i, j in zip(inputs, args)})
+if arg_attrs is None:
+raise ValueError(w[0].message)
 sdict = {i: j for i, j in zip(out.list_arguments(), arg_attrs)}
 sdict.update({name : attr for name, attr in \
-  zip(out.list_auxiliary_states(), aux_attrs)})
+ zip(out.list_auxiliary_states(), aux_attrs)})
 for i in self.collect_params().values():
 setattr(i, attr, sdict[i.name])
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-mxnet] branch master updated: [MXNET-185] Improved error message (#10424)

[jxie]

This is an automated email from the ASF dual-hosted git repository.

jxie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 4623c6d  [MXNET-185] Improved error message (#10424)
4623c6d is described below

commit 4623c6d7c473f3f7a8ac5ca441f2c042576623d5
Author: Ankit Khedia <36249596+ankkhe...@users.noreply.github.com>
AuthorDate: Thu Apr 12 12:52:51 2018 -0700

[MXNET-185] Improved error message (#10424)

* Improved error message

* fix pylint errors

* addressed PR comments

* fixed lint issues

* lint error fix

* append  warning message

* lint errors

* removed redundant message

* fixed warning

* Update block.py
---
 python/mxnet/gluon/block.py | 17 +
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/python/mxnet/gluon/block.py b/python/mxnet/gluon/block.py
index fd95641..6400358 100644
--- a/python/mxnet/gluon/block.py
+++ b/python/mxnet/gluon/block.py
@@ -458,7 +458,13 @@ class HybridBlock(Block):
 for name in out.list_inputs()]
 
 def _finish_deferred_init(self, hybrid, *args):
-self.infer_shape(*args)
+try:
+self.infer_shape(*args)
+except Exception as e:
+error_msg = "Deferred initialization failed because shape"\
+" cannot be inferred \n {}".format(e)
+raise ValueError(error_msg)
+
 if hybrid:
 for is_arg, i in self._cached_op_args:
 if not is_arg:
@@ -509,11 +515,14 @@ class HybridBlock(Block):
 """Generic infer attributes."""
 inputs, out = self._get_graph(*args)
 args, _ = _flatten(args)
-arg_attrs, _, aux_attrs = getattr(out, infer_fn)(
-**{i.name: getattr(j, attr) for i, j in zip(inputs, args)})
+with warnings.catch_warnings(record=True) as w:
+arg_attrs, _, aux_attrs = getattr(out, infer_fn)(
+**{i.name: getattr(j, attr) for i, j in zip(inputs, args)})
+if arg_attrs is None:
+raise ValueError(w[0].message)
 sdict = {i: j for i, j in zip(out.list_arguments(), arg_attrs)}
 sdict.update({name : attr for name, attr in \
-  zip(out.list_auxiliary_states(), aux_attrs)})
+ zip(out.list_auxiliary_states(), aux_attrs)})
 for i in self.collect_params().values():
 setattr(i, attr, sdict[i.name])
 

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[GitHub] piiswrong commented on a change in pull request #10424: [MXNET-185] Improved error message

[GitBox]

piiswrong commented on a change in pull request #10424: [MXNET-185] Improved 
error message
URL: https://github.com/apache/incubator-mxnet/pull/10424#discussion_r181202015
 
 

 ##
 File path: python/mxnet/gluon/block.py
 ##
 @@ -520,11 +520,11 @@ def _infer_attrs(self, infer_fn, attr, *args):
 **{i.name: getattr(j, attr) for i, j in zip(inputs, args)})
 if arg_attrs is None:
 raise ValueError(w[0].message)
-sdict = {i: j for i, j in zip(out.list_arguments(), arg_attrs)}
-sdict.update({name : attr for name, attr in \
- zip(out.list_auxiliary_states(), aux_attrs)})
-for i in self.collect_params().values():
-setattr(i, attr, sdict[i.name])
+sdict = {i: j for i, j in zip(out.list_arguments(), arg_attrs)}
 
 Review comment:
   no. that's not how python variable scope works


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sbodenstein opened a new issue #10531: Float16 Support for dot

[GitBox]

sbodenstein opened a new issue #10531: Float16 Support for dot
URL: https://github.com/apache/incubator-mxnet/issues/10531
 
 
   Is there any work underway to add Float16 support for the dot operator? I 
know there is a ticket for the general job of adding Float16 support 
(https://issues.apache.org/jira/browse/MXNET-90), but wanted to know what the 
timeframe was for support to dot, which is a very core operator that also 
benefits greatly from the V100 TensorCores.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on issue #10525: Fix NNPACK header file position error

[GitBox]

piiswrong commented on issue #10525: Fix NNPACK header file position error
URL: https://github.com/apache/incubator-mxnet/pull/10525#issuecomment-380922761
 
 
   It's likely not working properly. I think there was an issue on updating 
nnpack support to include latest changes from nnpack upstream. We should do 
that first.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] yzhliu commented on a change in pull request #10462: [MXNET-62] add test against spark integration

[GitBox]

yzhliu commented on a change in pull request #10462: [MXNET-62] add test 
against spark integration
URL: https://github.com/apache/incubator-mxnet/pull/10462#discussion_r181194921
 
 

 ##
 File path: 
scala-package/spark/src/test/scala/org/apache/mxnet/spark/MXNetGeneralSuite.scala
 ##
 @@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mxnet.spark
+
+import org.apache.spark.SparkContext
+import org.apache.spark.mllib.linalg.Vectors
+import org.apache.spark.mllib.regression.LabeledPoint
+import org.apache.spark.rdd.RDD
+
+class MXNetGeneralSuite extends SharedSparkContext {
+
+  private def parseRawData(sc: SparkContext, path: String): RDD[LabeledPoint] 
= {
+val raw = sc.textFile(path)
+raw.map { s =>
+  val parts = s.split(' ')
+  val label = java.lang.Double.parseDouble(parts(0))
+  val features = 
Vectors.dense(parts(1).trim().split(',').map(java.lang.Double.parseDouble))
+  LabeledPoint(label, features)
+}
+  }
+
+  test("run spark with MLP") {
+val trainData = parseRawData(sc,
+  "/Users/nanzhu/code/mxnet/scala-package/spark/train.txt")
 
 Review comment:
   @szha Any proper places can we put these testing files?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] yzhliu commented on a change in pull request #10462: [MXNET-62] add test against spark integration

[GitBox]

yzhliu commented on a change in pull request #10462: [MXNET-62] add test 
against spark integration
URL: https://github.com/apache/incubator-mxnet/pull/10462#discussion_r181193145
 
 

 ##
 File path: scala-package/spark/bin/run-mnist-example.sh
 ##
 @@ -72,8 +71,8 @@ fi
 
 HOST=`hostname`
 
-$SPARK_HOME/bin/spark-submit --master spark://$HOST:7077 \
-  --class org.apache.mxnet.spark.example.ClassificationExample \
+$SPARK_HOME/bin/spark-submit --master local[*] \
+  --class ml.dmlc.mxnet.spark.example.ClassificationExample \
 
 Review comment:
   why changing it back?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] spidyDev commented on issue #10425: [MXNET-175] NdArrayiterator dict fix

[GitBox]

spidyDev commented on issue #10425: [MXNET-175] NdArrayiterator dict fix
URL: https://github.com/apache/incubator-mxnet/pull/10425#issuecomment-380904371
 
 
   @piiswrong  I agree. But user can create a NDIterator with multiple 
key-value but bind only few of them. For example,
   ```
   train_iter = mx.io.NDArrayIter({
   'i2': mx.nd.ones((5, 4, 3, 3)),
   'i0': mx.nd.zeros(5, ),
   'i1': mx.nd.ones(5, ),
   'i3': mx.nd.ones((5, 4, 3, 3))
   }, batch_size=5)
   
   net = mx.sym.add_n(mx.sym.var('i2'), mx.sym.var('i3'))
   mod = mx.module.Module(net, ['i2', 'i3'], None)
   mod.bind([('i2', (5, 4, 3, 3)), ('i3', (5, 4, 3, 3))])
   mod.init_params()
   mod.predict(train_iter)
   assert_almost_equal(mod.get_outputs()[0].asnumpy(), 2*np.ones(shape=(5, 
4, 3, 3)))
   ```
   
   This might not be ideal , but valid ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 commented on issue #10376: Flaky test_gluon.test_lambda

[GitBox]

haojin2 commented on issue #10376: Flaky test_gluon.test_lambda
URL: 
https://github.com/apache/incubator-mxnet/issues/10376#issuecomment-380903253
 
 
   @reminisce Sure np!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] marcoabreu commented on issue #10525: Fix NNPACK header file position error

[GitBox]

marcoabreu commented on issue #10525: Fix NNPACK header file position error
URL: https://github.com/apache/incubator-mxnet/pull/10525#issuecomment-380902775
 
 
   Should we enable it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] reminisce commented on issue #10376: Flaky test_gluon.test_lambda

[GitBox]

reminisce commented on issue #10376: Flaky test_gluon.test_lambda
URL: 
https://github.com/apache/incubator-mxnet/issues/10376#issuecomment-380902567
 
 
   That's the problem of using too small `atol` for comparing two values close 
to zero. @haojin2 Can you change `atol` to `1e-3` along with your PR?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on a change in pull request #10524: [MXNET-312] Added Matthew's Correlation Coefficient to metrics

[GitBox]

piiswrong commented on a change in pull request #10524: [MXNET-312] Added 
Matthew's Correlation Coefficient to metrics
URL: https://github.com/apache/incubator-mxnet/pull/10524#discussion_r181179544
 
 

 ##
 File path: python/mxnet/metric.py
 ##
 @@ -661,6 +682,107 @@ def reset(self):
 self.metrics.reset_stats()
 
 
+@register
+class MCC(EvalMetric):
+"""Computes the Matthews Correlation Coefficient of a binary 
classification problem.
+
+While slower to compute the MCC can give insight that F1 or Accuracy 
cannot.
+For instance, if the network always predicts the same result
+then the MCC will immeadiately show this. The MCC is also symetric with 
respect
+to positive and negative catagorisation, however, there needs to be both
+positive and negative examples in the labels or it will always return 0.
+MCC of 0 is uncorrelated, 1 is completely correlated, and -1 is negatively 
correlated.
+
+.. math::
+\\text{MCC} = \\frac{ TP \\times TN - FP \\times FN }
+{\\sqrt{ (TP + FP) ( TP + FN ) ( TN + FP ) ( TN + FN ) } }
+
+where 0 terms in the denominator are replaced by 1.
+
+.. note::
+
+This MCC only supports binary classification.
+
+Parameters
+--
+name : str
+Name of this metric instance for display.
+output_names : list of str, or None
+Name of predictions that should be used when updating with update_dict.
+By default include all predictions.
+label_names : list of str, or None
+Name of labels that should be used when updating with update_dict.
+By default include all labels.
+average : str, default 'macro'
+Strategy to be used for aggregating across mini-batches.
+"macro": average the MCC for each batch.
+"micro": compute a single MCC across all batches.
+
+Examples
+
+In this example the network almost always predicts positive
+>>> false_positives = 1000
+>>> false_negatives = 1
+>>> true_positives = 1
+>>> true_negatives = 1
+>>> predicts = [mx.nd.array(
+[[.3, .7]]*false_positives +
+[[.7, .3]]*true_negatives +
+[[.7, .3]]*false_negatives +
+[[.3, .7]]*true_positives
+)]
+>>> labels  = [mx.nd.array(
+[0.]*(false_positives + true_negatives) +
+[1.]*(false_negatives + true_positives)
+)]
+>>> f1 = mx.metric.F1()
+>>> f1.update(preds = predicts, labels = labels)
+>>> mcc = mx.metric.MCC()
+>>> mcc.update(preds = predicts, labels = labels)
+>>> print f1.get()
+('f1', 0.95233560306652054)
+>>> print mcc.get()
+('mcc', 0.01917751877733392)
+"""
+
+def __init__(self, name='mcc',
+ output_names=None, label_names=None, average="macro"):
+self.average = average
+self.metrics = _BinaryClassificationMetrics()
 
 Review comment:
   _average and _metrics


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on issue #10525: Fix NNPACK header file position error

[GitBox]

piiswrong commented on issue #10525: Fix NNPACK header file position error
URL: https://github.com/apache/incubator-mxnet/pull/10525#issuecomment-380901021
 
 
   I think nnpack should be moved into nn instead.
   
   @marcoabreu nnpack is not tested in the CI. I was added a long time ago and 
I don't think it was officially supported


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on issue #10525: Fix NNPACK header file position error

[GitBox]

piiswrong commented on issue #10525: Fix NNPACK header file position error
URL: https://github.com/apache/incubator-mxnet/pull/10525#issuecomment-380901021
 
 
   I think nnpack should be moved into nn instead.
   
   @marcoabreu nnpack is not tested in the CI. It was added a long time ago and 
I don't think it was officially supported


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on issue #10525: Fix NNPACK header file position error

[GitBox]

piiswrong commented on issue #10525: Fix NNPACK header file position error
URL: https://github.com/apache/incubator-mxnet/pull/10525#issuecomment-380901021
 
 
   I think nnpack should be moved into nn instead.
   
   @marcoabreu nnpack is not tested in the CI


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on issue #10425: [MXNET-175] NdArrayiterator dict fix

[GitBox]

piiswrong commented on issue #10425: [MXNET-175] NdArrayiterator dict fix
URL: https://github.com/apache/incubator-mxnet/pull/10425#issuecomment-380900106
 
 
   I don't think this is the right fix for this issue. The names on ndarrayiter 
doesn't automatically get's passed to module. The correct usage is to bind with 
data_iter.provide_data


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on issue #10425: [MXNET-175] NdArrayiterator dict fix

[GitBox]

piiswrong commented on issue #10425: [MXNET-175] NdArrayiterator dict fix
URL: https://github.com/apache/incubator-mxnet/pull/10425#issuecomment-380900106
 
 
   I don't think this is the right fix for this issue. The names in ndarrayiter 
doesn't automatically get's passed to module. The correct usage is to bind with 
data_iter.provide_data


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 commented on issue #10376: Flaky test_gluon.test_lambda

[GitBox]

haojin2 commented on issue #10376: Flaky test_gluon.test_lambda
URL: 
https://github.com/apache/incubator-mxnet/issues/10376#issuecomment-380900183
 
 
   I met the same failure in: 
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-10371/6/pipeline.
 I guess maybe the atol is too strict?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 commented on a change in pull request #10371: [MXNET-263] Support for dot(dns, csr) = dns and dot(dns, csr.T) = dns on GPU

[GitBox]

haojin2 commented on a change in pull request #10371: [MXNET-263] Support for 
dot(dns, csr) = dns and dot(dns, csr.T) = dns on GPU
URL: https://github.com/apache/incubator-mxnet/pull/10371#discussion_r181177330
 
 

 ##
 File path: src/operator/tensor/dot-inl.cuh
 ##
 @@ -895,6 +988,131 @@ inline void DotCsrRspDnsImpl(const OpContext& ctx,
   });
 }
 
+/*
+ * \brief GPU Impl of dot(dns, csr) = csr
+ */
+template
+inline void DotDnsCsrCsrImpl(const OpContext& ctx,
+ const TBlob& lhs, const NDArray& rhs,
+ const OpReqType req, NDArray* ret) {
+  LOG(FATAL) << "dot(dense, csr) = csr is not implemented on GPU";
+}
+
+/*
+ * \brief GPU Impl of dot(dns, csr) = dns and dot(dns, csr.T) = dns
+ */
+template
+inline void DotDnsCsrDnsImpl(const OpContext& ctx,
+ const TBlob& dns, const NDArray& rhs,
+ const OpReqType req, NDArray* ret,
+ const bool transpose_b) {
+  CHECK_EQ(req, kWriteTo);
+  CHECK_EQ(rhs.storage_type(), kCSRStorage);
+
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using nnvm::dim_t;
+
+  /* Initialize data structures */
+  mshadow::Stream* s = ctx.get_stream();
+  TBlob csr_data = rhs.data();
+  TBlob csr_indices = rhs.aux_data(csr::kIdx);
+  TBlob csr_indptr = rhs.aux_data(csr::kIndPtr);
+  if (!rhs.storage_initialized()) {
+FillZerosCsrImpl(s, *ret);
+return;
+  }
+
+  // if dot(dense, csr) = dns, transform to csc first
+  if (!transpose_b) {
+// LOG(FATAL) << "dot(dns, csr) = dns not implemented yet";
+const nnvm::dim_t csr_rows = rhs.shape()[0];
+const nnvm::dim_t csr_cols = rhs.shape()[1];
+const nnvm::dim_t dns_rows = dns.shape_[0];
+const nnvm::dim_t nnz = rhs.storage_shape().Size();
+
+MSHADOW_SGL_DBL_TYPE_SWITCH(csr_data.type_flag_, DType, {
+  MSHADOW_IDX_TYPE_SWITCH(csr_indices.type_flag_, IType, {
+MSHADOW_IDX_TYPE_SWITCH(csr_indptr.type_flag_, CType, {
+  DType* csc_data_ptr = NULL;
+  unsigned long long* csc_indices_ptr = NULL;
+  unsigned long long* csc_indptr_ptr = NULL;
+  unsigned long long* col_counters = NULL;
+  size_t ull_mem_size = sizeof(unsigned long long);
+  void* temp_storage = NULL;
+  size_t temp_storage_bytes = 0;
+  CType out_num_rows = ret->shape()[0];
+  CType out_num_cols = ret->shape()[1];
+  // Get necessary temporary storage amount
+  cub::DeviceScan::ExclusiveSum(NULL,
+temp_storage_bytes,
+csc_indices_ptr,
+csc_indices_ptr,
+csr_cols+1,
+Stream::GetStream(s));
+  temp_storage_bytes += (ull_mem_size - (temp_storage_bytes % 
ull_mem_size));
+  Tensor workspace =
+ctx.requested[0].get_space_typed(
+  Shape1(nnz*sizeof(DType) + nnz*ull_mem_size +
+ 2*(csr_cols + 1)*ull_mem_size +
+ temp_storage_bytes),
+  s);
+  csc_indices_ptr = reinterpret_cast(workspace.dptr_);
+  csc_indptr_ptr = reinterpret_cast(
+ workspace.dptr_ + nnz*ull_mem_size);
+  col_counters = reinterpret_cast(
+   workspace.dptr_ + nnz*ull_mem_size + 
(csr_cols+1)*ull_mem_size);
+  csc_data_ptr = reinterpret_cast(workspace.dptr_ + 
nnz*ull_mem_size +
+  2*(csr_cols+1)*ull_mem_size);
+  temp_storage = reinterpret_cast(workspace.dptr_ + 
nnz*sizeof(DType) +
+ nnz*ull_mem_size + 
2*(csr_cols+1)*ull_mem_size);
+  mxnet_op::Kernel::Launch(
+s, dns_rows*csr_cols, ret->data().dptr());
+  // Reset values for indptr, ready for histogramming
+  mxnet_op::Kernel::Launch(
+s, csr_cols + 1, csc_indptr_ptr);
+  // Histogramming on col id
+  mxnet_op::Kernel::Launch(
+s, nnz, csr_indices.dptr(), csc_indptr_ptr, nnz);
+  cub::DeviceScan::ExclusiveSum(temp_storage,
+temp_storage_bytes,
+csc_indptr_ptr,
+csc_indptr_ptr,
+csr_cols+1,
+Stream::GetStream(s));
+  // Reset values for col_counter, ready for the final transform
+  mxnet_op::Kernel::Launch(
+s, csr_cols+1, col_counters);
+  // Transform to CSC
+  mxnet_op::Kernel::Launch(
+s, csr_rows, csr_data.dptr(), csr_indices.dptr(),
+csr_indptr.dptr(), csc_data_ptr, csc_indices_ptr,
+csc_indptr_ptr, col_counters, csr_rows, csr_cols);
+