[GitHub] uniquezhengjie commented on issue #12105: Floating point exception (core dumped)

[GitBox]

uniquezhengjie commented on issue #12105: Floating point exception (core dumped)
URL: 
https://github.com/apache/incubator-mxnet/issues/12105#issuecomment-411983303
 
 
   @szha  if i run run mxnet-mkl  version with 1.0.0, i can import ,but when i 
get output layer "net_out = self.model.get_outputs()", final come out "Floating 
point exception".
   i can upgrade to latest version (mxnet-mkl 1.2.1.post1) from 1.0.0 with "pip 
install --upgrade mxnet-mkl",  but when i import mxnet get that error .
   i don't know how to do next step ...


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ZhennanQin commented on issue #12104: [DO NOT REVIEW] Subgraph API

[GitBox]

ZhennanQin commented on issue #12104: [DO NOT REVIEW] Subgraph API
URL: https://github.com/apache/incubator-mxnet/pull/12104#issuecomment-411983046
 
 
   @reminisce Then we need to call PartitionGraph() inside CreateSubgraphNode() 
to traverse nodes, otherwise we have to implement a separate traverse method. 
If that meet your expectation, then I'm OK with this solution.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ZhennanQin commented on a change in pull request #12104: [DO NOT REVIEW] Subgraph API

[GitBox]

ZhennanQin commented on a change in pull request #12104: [DO NOT REVIEW] 
Subgraph API
URL: https://github.com/apache/incubator-mxnet/pull/12104#discussion_r209150407
 
 

 ##
 File path: src/executor/graph_executor.cc
 ##
 @@ -1718,6 +1860,11 @@ Executor *Executor::SimpleBind(nnvm::Symbol symbol,
std::unordered_map* 
shared_buffer,
Executor* shared_exec) {
   auto exec = new exec::GraphExecutor();
+  if (!exec->subgraph_property().empty()) {
+symbol = exec::PartitionGraph(symbol, exec->subgraph_property(), 
arg_shape_map, arg_dtype_map,
 
 Review comment:
   Yes, that's also my question. We indeed have some fusion passes only for 
inference, we need a way to disable it on training. If we can't describe this 
in subgraph property, then we need to find another way to indicate that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] reminisce commented on a change in pull request #12104: [DO NOT REVIEW] Subgraph API

[GitBox]

reminisce commented on a change in pull request #12104: [DO NOT REVIEW] 
Subgraph API
URL: https://github.com/apache/incubator-mxnet/pull/12104#discussion_r209150302
 
 

 ##
 File path: src/executor/graph_executor.cc
 ##
 @@ -1699,6 +1701,146 @@ GraphExecutor::CachedSegOpr 
GraphExecutor::CreateCachedSegOpr(size_t topo_start,
 iter->c_str());
   return ret;
 }
+
+// Infer shapes, dtypes, stypes, contexts for the forward graph
+static nnvm::Graph InferForwardAttrs(nnvm::Graph g,
+ nnvm::ShapeVector arg_shapes,
+ nnvm::DTypeVector arg_dtypes,
+ StorageTypeVector arg_stypes,
+ const Context& default_ctx,
+ const std::map& 
ctx_map,
+ const std::vector& in_arg_ctxes,
+ const std::vector& 
aux_state_ctxes) {
+  const auto& indexed_graph = g.indexed_graph();
+  const auto num_forward_inputs = indexed_graph.input_nodes().size();
+  g = AssignContext(g, default_ctx, ctx_map, in_arg_ctxes, {},
+   aux_state_ctxes, {}, num_forward_inputs, g.outputs.size());
+  g = InferShape(std::move(g), std::move(arg_shapes), "__shape__");
+  if (g.GetAttr("shape_num_unknown_nodes") != 0U) {
+HandleInferShapeError(num_forward_inputs, indexed_graph,
+  g.GetAttr("shape"));
+  }
+  g = InferType(std::move(g), std::move(arg_dtypes), "__dtype__");
+  if (g.GetAttr("dtype_num_unknown_nodes") != 0U) {
+HandleInferTypeError(num_forward_inputs, indexed_graph,
+ g.GetAttr("dtype"));
+  }
+  g = InferStorageType(std::move(g), std::move(arg_stypes), 
"__storage_type__");
+  if (g.GetAttr("storage_type_num_unknown_nodes") != 0U) {
+HandleInferStorageTypeError(num_forward_inputs, indexed_graph,
+g.GetAttr("storage_type"));
+  }
+  return g;
+}
+
+// Given input attr arrays, partition the graph using the backend name equal 
to prop_name.
+// This is a common function for bind and simple_bind flows.
+static nnvm::Symbol PartitionGraph(const nnvm::Symbol& src,
+   const std::string& prop_name,
+   const nnvm::ShapeVector& arg_shapes,
+   const nnvm::DTypeVector& arg_dtypes,
+   const StorageTypeVector arg_stypes,
+   const Context& default_ctx,
+   const std::map& 
ctx_map,
+   const std::vector& in_arg_ctxes,
+   const std::vector& 
aux_state_ctxes) {
+  auto subgraph_prop = 
op::SubgraphPropertyRegistry::Get()->CreateSubgraphProperty(prop_name);
+  nnvm::Symbol ret = src.Copy();
+  nnvm::Graph g;
+  g.outputs = ret.outputs;
+  g = InferForwardAttrs(g, arg_shapes, arg_dtypes, arg_stypes, default_ctx,
+ctx_map, in_arg_ctxes, aux_state_ctxes);
+  subgraph_prop->SetAttr("graph", g);
 
 Review comment:
   I think this can be done in `CreateSubgraphSelector()`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] reminisce commented on issue #12104: [DO NOT REVIEW] Subgraph API

[GitBox]

reminisce commented on issue #12104: [DO NOT REVIEW] Subgraph API
URL: https://github.com/apache/incubator-mxnet/pull/12104#issuecomment-411981764
 
 
   @ZhennanQin CachedOp will be submitted through a another PR. We want to make 
this PR as small as possible.
   
   I think what you described about fusion can be done in graph partitioning as 
well. In `CreateSubgraphNode()`, you can replace the subgraph symbol with a new 
subgraph symbol consisting of fused operators. Does this make sense?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] reminisce commented on a change in pull request #12104: [DO NOT REVIEW] Subgraph API

[GitBox]

reminisce commented on a change in pull request #12104: [DO NOT REVIEW] 
Subgraph API
URL: https://github.com/apache/incubator-mxnet/pull/12104#discussion_r209149128
 
 

 ##
 File path: src/executor/graph_executor.cc
 ##
 @@ -1718,6 +1860,11 @@ Executor *Executor::SimpleBind(nnvm::Symbol symbol,
std::unordered_map* 
shared_buffer,
Executor* shared_exec) {
   auto exec = new exec::GraphExecutor();
+  if (!exec->subgraph_property().empty()) {
+symbol = exec::PartitionGraph(symbol, exec->subgraph_property(), 
arg_shape_map, arg_dtype_map,
 
 Review comment:
   Is it necessary to indicate whether a subgraph property is for inference 
only or not?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack Transformation API

[GitBox]

tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack 
Transformation API
URL: https://github.com/apache/incubator-mxnet/pull/12047#discussion_r209143790
 
 

 ##
 File path: src/c_api/c_api.cc
 ##
 @@ -494,6 +494,57 @@ int MXNDArrayGetData(NDArrayHandle handle,
   API_END();
 }
 
+int MXNDArrayToDLPack(NDArrayHandle handle,
+  DLManagedTensorHandle *out_dlpack) {
+  API_BEGIN();
+  NDArray *arr = static_cast(handle);
+  *out_dlpack = arr->ToDLPack();
+  API_END();
+}
+
+int MXNDArrayFromDLPack(DLManagedTensorHandle dlpack,
+NDArrayHandle *out_handle) {
+  API_BEGIN();
+  NDArray *pdata = new NDArray();
+  *pdata = NDArray::FromDLPack(
+   static_cast(dlpack));
+  *out_handle = pdata;
+  API_END();
+}
+
+int MXNDArrayCallDLPackDeleter(DLManagedTensorHandle dlpack) {
+  API_BEGIN();
+  if (dlpack) {
+DLManagedTensor *p_dlpack = static_cast(dlpack);
+p_dlpack->deleter(p_dlpack);
+  }
+  API_END();
+}
+
+
+typedef struct {
+char py_object[16];
 
 Review comment:
   This seems to be dangerous unless pycapsule is already standardized and 
won't change


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack Transformation API

[GitBox]

tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack 
Transformation API
URL: https://github.com/apache/incubator-mxnet/pull/12047#discussion_r209143649
 
 

 ##
 File path: python/mxnet/ndarray/ndarray.py
 ##
 @@ -3851,3 +3898,117 @@ def histogram(a, bins=10, range=None):
 return _internal._histogram(data=a, bin_cnt=bins, range=range)
 raise ValueError("bins argument should be either an integer or an NDArray")
 # pylint: enable= no-member, protected-access, redefined-builtin
+
+pycapsule_dlpack_deleter = ctypes.CFUNCTYPE(None, ctypes.c_void_p)(
+_LIB.MXNDArrayCallDLPackCapsuleDeleter)
+
+def to_dlpack_for_read(data):
+"""Returns a reference view of NDArray that represents as DLManagedTensor 
until
+   all previous write operations on the current array are finished.
+
+Parameters
+--
+data: NDArray
+input data.
+
+Returns
+---
+PyCapsule (the pointer of DLManagedTensor)
+a reference view of NDArray that represents as DLManagedTensor.
+
+Examples
+
+>>> x = mx.nd.ones((2,3))
+>>> y = mx.nd.to_dlpack_for_read(x)
+>>> type(y)
+
+>>> z = mx.nd.from_dlpack(y)
+>>> z
+[[1. 1. 1.]
+ [1. 1. 1.]]
+
+"""
+data.wait_to_read()
+dlpack = DLPackHandle()
+check_call(_LIB.MXNDArrayToDLPack(data.handle, ctypes.byref(dlpack)))
+return ctypes.pythonapi.PyCapsule_New(dlpack, b'dltensor', 
pycapsule_dlpack_deleter)
+
+def to_dlpack_for_write(data):
+"""Returns a reference view of NDArray that represents as DLManagedTensor 
until
+   all previous read/write operations on the current array are finished.
+
+Parameters
+--
+data: NDArray
+input data.
+
+Returns
+---
+PyCapsule (the pointer of DLManagedTensor)
+a reference view of NDArray that represents as DLManagedTensor.
+
+Examples
+
+>>> x = mx.nd.ones((2,3))
+>>> w = mx.nd.to_dlpack_for_write(x)
+>>> type(w)
+
+>>> u = mx.nd.from_dlpack(w)
+>>> u += 1
+>>> x
+[[2. 2. 2.]
+ [2. 2. 2.]]
+
+"""
+check_call(_LIB.MXNDArrayWaitToWrite(data.handle))
+dlpack = DLPackHandle()
+check_call(_LIB.MXNDArrayToDLPack(data.handle, ctypes.byref(dlpack)))
+return ctypes.pythonapi.PyCapsule_New(dlpack, b'dltensor', 
pycapsule_dlpack_deleter)
 
 Review comment:
   cannot directly pass in string, instead use a global string object, 
PyCapsule requires the string to outlive the capsule


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack Transformation API

[GitBox]

tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack 
Transformation API
URL: https://github.com/apache/incubator-mxnet/pull/12047#discussion_r209143480
 
 

 ##
 File path: include/mxnet/c_api.h
 ##
 @@ -737,6 +741,57 @@ MXNET_DLL int MXNDArrayGetShape(NDArrayHandle handle,
  */
 MXNET_DLL int MXNDArrayGetData(NDArrayHandle handle,
void **out_pdata);
+/*!
+* \brief Create a reference view of NDArray that
+*  represents as DLManagedTensor until
+*  all the pending writes with respect NDArray are finished.
+* \param handle the handle to the ndarray
+* \param out_dlpack pointer holder to get pointer of DLManagedTensor
+* \return 0 when success, -1 when failure happens
+*/
+MXNET_DLL int MXNDArrayToDLPackForRead(NDArrayHandle handle,
+   DLManagedTensorHandle *out_dlpack);
+
+/*!
+* \brief Create a reference view of NDArray that
+*  represents as DLManagedTensor until
+*  all the pending reads/writes with respect NDArray are finished.
+* \param handle the handle to the ndarray
+* \param out_dlpack pointer holder to get pointer of DLManagedTensor
+* \return 0 when success, -1 when failure happens
+*/
+MXNET_DLL int MXNDArrayToDLPackForWrite(NDArrayHandle handle,
+DLManagedTensorHandle *out_dlpack);
+
+/*!
+* \brief Create a NDArray backed by a dlpack tensor.
+*
+* This allows us to create a NDArray using the memory
+* allocated by an external deep learning framework
+* that is DLPack compatible.
+*
+* The memory is retained until the NDArray went out of scope.
+*
+* \param dlpack the pointer of the input DLManagedTensor
+* \param out_handle pointer holder to get pointer of NDArray
+* \return 0 when success, -1 when failure happens
+*/
+MXNET_DLL int MXNDArrayFromDLPack(DLManagedTensorHandle dlpack,
+  NDArrayHandle *out_handle);
+/*!
+ * \brief Delete a dlpack tensor
+ * \param dlpack the pointer of the input DLManagedTensor
+ * \return 0 when success, -1 when failure happens
+ */
+MXNET_DLL int MXNDArrayCallDLPackDeleter(DLManagedTensorHandle dlpack);
+
+/*!
+ * \brief Delete a dlpack tensor
+ * \param dlpack_capsule the pointer of a PyCapsule storing DLManagedTensor
+ * \return 0 when success, -1 when failure happens
+ */
+MXNET_DLL void MXNDArrayCallDLPackCapsuleDeleter(PyObjectHandle 
dlpack_capsule);
 
 Review comment:
   There is some subtlty here but they can never-the-less be implemented


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack Transformation API

[GitBox]

tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack 
Transformation API
URL: https://github.com/apache/incubator-mxnet/pull/12047#discussion_r209143434
 
 

 ##
 File path: include/mxnet/c_api.h
 ##
 @@ -737,6 +741,57 @@ MXNET_DLL int MXNDArrayGetShape(NDArrayHandle handle,
  */
 MXNET_DLL int MXNDArrayGetData(NDArrayHandle handle,
void **out_pdata);
+/*!
+* \brief Create a reference view of NDArray that
+*  represents as DLManagedTensor until
+*  all the pending writes with respect NDArray are finished.
+* \param handle the handle to the ndarray
+* \param out_dlpack pointer holder to get pointer of DLManagedTensor
+* \return 0 when success, -1 when failure happens
+*/
+MXNET_DLL int MXNDArrayToDLPackForRead(NDArrayHandle handle,
+   DLManagedTensorHandle *out_dlpack);
+
+/*!
+* \brief Create a reference view of NDArray that
+*  represents as DLManagedTensor until
+*  all the pending reads/writes with respect NDArray are finished.
+* \param handle the handle to the ndarray
+* \param out_dlpack pointer holder to get pointer of DLManagedTensor
+* \return 0 when success, -1 when failure happens
+*/
+MXNET_DLL int MXNDArrayToDLPackForWrite(NDArrayHandle handle,
+DLManagedTensorHandle *out_dlpack);
+
+/*!
+* \brief Create a NDArray backed by a dlpack tensor.
+*
+* This allows us to create a NDArray using the memory
+* allocated by an external deep learning framework
+* that is DLPack compatible.
+*
+* The memory is retained until the NDArray went out of scope.
+*
+* \param dlpack the pointer of the input DLManagedTensor
+* \param out_handle pointer holder to get pointer of NDArray
+* \return 0 when success, -1 when failure happens
+*/
+MXNET_DLL int MXNDArrayFromDLPack(DLManagedTensorHandle dlpack,
+  NDArrayHandle *out_handle);
+/*!
+ * \brief Delete a dlpack tensor
+ * \param dlpack the pointer of the input DLManagedTensor
+ * \return 0 when success, -1 when failure happens
+ */
+MXNET_DLL int MXNDArrayCallDLPackDeleter(DLManagedTensorHandle dlpack);
+
+/*!
+ * \brief Delete a dlpack tensor
+ * \param dlpack_capsule the pointer of a PyCapsule storing DLManagedTensor
+ * \return 0 when success, -1 when failure happens
+ */
+MXNET_DLL void MXNDArrayCallDLPackCapsuleDeleter(PyObjectHandle 
dlpack_capsule);
 
 Review comment:
   Take a look at destructor at https://github.com/dmlc/tvm/pull/1573


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack Transformation API

[GitBox]

tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack 
Transformation API
URL: https://github.com/apache/incubator-mxnet/pull/12047#discussion_r209143369
 
 

 ##
 File path: python/mxnet/_ctypes/ndarray.py
 ##
 @@ -31,21 +31,24 @@
 
 class NDArrayBase(object):
 """Base data structure for ndarray"""
-__slots__ = ["handle", "writable"]
+__slots__ = ["handle", "writable", "dlpack"]
 # pylint: disable= no-member
 
-def __init__(self, handle, writable=True):
+def __init__(self, handle, writable=True, dlpack=None):
 
 Review comment:
   Actually, your implementation of ToDLPack is fine and the Manager class is 
OK. Since there is already the Manager class and the deleter will get called, 
you do not need to keep reference to the dlpack in the NDArray


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack Transformation API

[GitBox]

tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack 
Transformation API
URL: https://github.com/apache/incubator-mxnet/pull/12047#discussion_r209142905
 
 

 ##
 File path: python/mxnet/_ctypes/ndarray.py
 ##
 @@ -31,21 +31,24 @@
 
 class NDArrayBase(object):
 """Base data structure for ndarray"""
-__slots__ = ["handle", "writable"]
+__slots__ = ["handle", "writable", "dlpack"]
 # pylint: disable= no-member
 
-def __init__(self, handle, writable=True):
+def __init__(self, handle, writable=True, dlpack=None):
 
 Review comment:
   You just need to create new NDArray that copies the original NDArray, and 
create a Deleter function that deletes the NDArray in the context. There is no 
need for python NDArray to keep reference of the dlpack object. As the 
DLManagedTensor manages itself


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on issue #12105: Floating point exception (core dumped)

[GitBox]

szha commented on issue #12105: Floating point exception (core dumped)
URL: 
https://github.com/apache/incubator-mxnet/issues/12105#issuecomment-411972084
 
 
   1.0.0 was directly on mklml and not on mkldnn, so the bug likely have 
already been addressed with mkldnn integration. @uniquezhengjie is there 
blocker that prevents you from upgrading to the latest version?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] junrushao1994 commented on a change in pull request #12078: [MXNET-749] Correct usages of `CutSubgraph` in 3 control flow operators

[GitBox]

junrushao1994 commented on a change in pull request #12078: [MXNET-749] Correct 
usages of `CutSubgraph` in 3 control flow operators
URL: https://github.com/apache/incubator-mxnet/pull/12078#discussion_r209141352
 
 

 ##
 File path: src/operator/control_flow.cc
 ##
 @@ -1225,6 +1225,9 @@ static bool BackwardCondStorageType(const 
nnvm::NodeAttrs& attrs,
 CHECK(sync_in_in(input_locs, out_attrs, _out_attrs, is_udf));
 return ret;
   };
+  for (const dim_t _in : params.cond_input_locs) {
+(*out_attrs)[cond_in] = 0;
+  }
 
 Review comment:
   Okay I updated "0 => kUndefinedStorage"


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] junrushao1994 commented on a change in pull request #12078: [MXNET-749] Correct usages of `CutSubgraph` in 3 control flow operators

[GitBox]

junrushao1994 commented on a change in pull request #12078: [MXNET-749] Correct 
usages of `CutSubgraph` in 3 control flow operators
URL: https://github.com/apache/incubator-mxnet/pull/12078#discussion_r209139550
 
 

 ##
 File path: src/operator/control_flow.cc
 ##
 @@ -1225,6 +1225,9 @@ static bool BackwardCondStorageType(const 
nnvm::NodeAttrs& attrs,
 CHECK(sync_in_in(input_locs, out_attrs, _out_attrs, is_udf));
 return ret;
   };
+  for (const dim_t _in : params.cond_input_locs) {
+(*out_attrs)[cond_in] = 0;
+  }
 
 Review comment:
   @eric-haibin-lin In this case, anything other than `kUndefinedStorage` will 
work because we don't bp through `cond`. Do you suggest that I use 
`kDefaultStorage` to make it clearer?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] junrushao1994 commented on a change in pull request #12078: [MXNET-749] Correct usages of `CutSubgraph` in 3 control flow operators

[GitBox]

junrushao1994 commented on a change in pull request #12078: [MXNET-749] Correct 
usages of `CutSubgraph` in 3 control flow operators
URL: https://github.com/apache/incubator-mxnet/pull/12078#discussion_r209139550
 
 

 ##
 File path: src/operator/control_flow.cc
 ##
 @@ -1225,6 +1225,9 @@ static bool BackwardCondStorageType(const 
nnvm::NodeAttrs& attrs,
 CHECK(sync_in_in(input_locs, out_attrs, _out_attrs, is_udf));
 return ret;
   };
+  for (const dim_t _in : params.cond_input_locs) {
+(*out_attrs)[cond_in] = 0;
+  }
 
 Review comment:
   @eric-haibin-lin In this case, anything other than `kBadStorageID` will work 
because we don't bp through `cond`. Do you suggest that I use `kDefaultStorage` 
to make it clearer?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin commented on a change in pull request #12078: [MXNET-749] Correct usages of `CutSubgraph` in 3 control flow operators

[GitBox]

eric-haibin-lin commented on a change in pull request #12078: [MXNET-749] 
Correct usages of `CutSubgraph` in 3 control flow operators
URL: https://github.com/apache/incubator-mxnet/pull/12078#discussion_r209138871
 
 

 ##
 File path: src/operator/control_flow.cc
 ##
 @@ -1225,6 +1225,9 @@ static bool BackwardCondStorageType(const 
nnvm::NodeAttrs& attrs,
 CHECK(sync_in_in(input_locs, out_attrs, _out_attrs, is_udf));
 return ret;
   };
+  for (const dim_t _in : params.cond_input_locs) {
+(*out_attrs)[cond_in] = 0;
+  }
 
 Review comment:
   Is 0 a magic number?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] iblis17 commented on issue #10149: WIP: import Julia binding

[GitBox]

iblis17 commented on issue #10149: WIP: import Julia binding
URL: https://github.com/apache/incubator-mxnet/pull/10149#issuecomment-411966671
 
 
   @eric-haibin-lin well, no.
   This PR is still for julia 0.6.
   If we want to support 1.0, we need to upgrade to 0.7 first.
   Julia 0.7 and 1.0 are almost the same codebase, but without deprecation 
warning.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin commented on issue #10149: WIP: import Julia binding

[GitBox]

eric-haibin-lin commented on issue #10149: WIP: import Julia binding
URL: https://github.com/apache/incubator-mxnet/pull/10149#issuecomment-411964473
 
 
   julia 1.0 is launched! That’s the supported julia version for this PR? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] wkcn commented on a change in pull request #12047: [MXNET-779]Add DLPack Transformation API

[GitBox]

wkcn commented on a change in pull request #12047: [MXNET-779]Add DLPack 
Transformation API
URL: https://github.com/apache/incubator-mxnet/pull/12047#discussion_r209134860
 
 

 ##
 File path: python/mxnet/_ctypes/ndarray.py
 ##
 @@ -31,21 +31,24 @@
 
 class NDArrayBase(object):
 """Base data structure for ndarray"""
-__slots__ = ["handle", "writable"]
+__slots__ = ["handle", "writable", "dlpack"]
 # pylint: disable= no-member
 
-def __init__(self, handle, writable=True):
+def __init__(self, handle, writable=True, dlpack=None):
 
 Review comment:
   I use NDArrayDLManager whose member is a copy of NDArray, which increases 
refcount.
   The deleter of DLPack is to delete NDArrayDLManager, which decreases NDArray 
refcount.
   
   If a new NDArray() created by DLPack, the deleter of DLPack will be called 
by the new NDArray. Similarly, If a PyTorch Tensor createed by DLPack (MXNet), 
the deleter will be called by the new PyTorch Tensor.
   
   NDArray (CPP and Python class) doesn't have deleter API, so I set the DLPack 
as the member of NDArray (Python). When NDArray generated by DLPack releases, 
the DLPack object in the NDArray will release, then the deleter function in 
DLPack will be called.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] wkcn commented on a change in pull request #12047: [MXNET-779]Add DLPack Transformation API

[GitBox]

wkcn commented on a change in pull request #12047: [MXNET-779]Add DLPack 
Transformation API
URL: https://github.com/apache/incubator-mxnet/pull/12047#discussion_r209134860
 
 

 ##
 File path: python/mxnet/_ctypes/ndarray.py
 ##
 @@ -31,21 +31,24 @@
 
 class NDArrayBase(object):
 """Base data structure for ndarray"""
-__slots__ = ["handle", "writable"]
+__slots__ = ["handle", "writable", "dlpack"]
 # pylint: disable= no-member
 
-def __init__(self, handle, writable=True):
+def __init__(self, handle, writable=True, dlpack=None):
 
 Review comment:
   I use NDArrayDLManager whose member is a copy of NDArray, which increases 
refcount.
   The deleter of DLPack is to delete NDArrayDLManager, which decreases 
refcount.
   
   If a new NDArray() created by DLPack, the deleter of DLPack will be called 
by the new NDArray. Similarly, If a PyTorch Tensor createed by DLPack (MXNet), 
the deleter will be called by the new PyTorch Tensor.
   
   NDArray (CPP and Python class) doesn't have deleter API, so I set the DLPack 
as the member of NDArray (Python). When NDArray generated by DLPack releases, 
the DLPack object in the NDArray will release, then the deleter function in 
DLPack will be called.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ZhennanQin commented on issue #12104: [DO NOT REVIEW] Subgraph API

[GitBox]

ZhennanQin commented on issue #12104: [DO NOT REVIEW] Subgraph API
URL: https://github.com/apache/incubator-mxnet/pull/12104#issuecomment-411962054
 
 
   Another question: what's the sequence should we execute below 2 kinds of 
subgraph pass:
   1. the pass to cover all operators that backends support.(just like 
default_subgraph_op)
   2. Fusion pass to combine several ops into a new ops.
   
   I'm asking this is because, if we want to apply 2 after 1, then we need to 
have the ability to traverse into subgraph node to do fusion. Do we support 
this?  


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] TS0713 opened a new issue #12119: mxnet package in r (3.5.1) - It doesn't have lstm and gluon

[GitBox]

TS0713 opened a new issue #12119: mxnet package in r (3.5.1) - It doesn't have 
lstm and gluon 
URL: https://github.com/apache/incubator-mxnet/issues/12119
 
 
   After I have installed mxnet package in RStudio (R-3.5.1) - I didn't find 
lstm and few other functions disappeared from mxnet package. Please provide 
other link where I can download mxnet containing complete functions as 
mentioned in mxnet documentations. 
   
   Have installed using below lines as mentioned on mxnet page 
   cran <- getOption("repos")
   cran["dmlc"] <- 
"https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/R/CRAN/;
   options(repos = cran)
   install.packages("mxnet")
   
   **library(mxnet)
   mx.lstm()
   # Error in mx.lstm() : could not find function "mx.lstm"
   mx.gluon.nn.Sequential 
   # Error: object 'mx.gluon.nn.Sequential' not found**


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ZhennanQin commented on issue #12104: [DO NOT REVIEW] Subgraph API

[GitBox]

ZhennanQin commented on issue #12104: [DO NOT REVIEW] Subgraph API
URL: https://github.com/apache/incubator-mxnet/pull/12104#issuecomment-411959295
 
 
   Can this change cover hybridize gluon model? I don't see the change in 
cached_op.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ZhennanQin commented on a change in pull request #12104: [DO NOT REVIEW] Subgraph API

[GitBox]

ZhennanQin commented on a change in pull request #12104: [DO NOT REVIEW] 
Subgraph API
URL: https://github.com/apache/incubator-mxnet/pull/12104#discussion_r209132005
 
 

 ##
 File path: src/executor/graph_executor.cc
 ##
 @@ -1718,6 +1860,11 @@ Executor *Executor::SimpleBind(nnvm::Symbol symbol,
std::unordered_map* 
shared_buffer,
Executor* shared_exec) {
   auto exec = new exec::GraphExecutor();
+  if (!exec->subgraph_property().empty()) {
+symbol = exec::PartitionGraph(symbol, exec->subgraph_property(), 
arg_shape_map, arg_dtype_map,
 
 Review comment:
   Many subgraph pass can only be used on inference, eg. default_subgraph_op 
doesn't have FGradient attribute and can't support backward computation. Maybe 
default_subgraph_op will support backward in future, but some mkldnn fusion 
pass is designed to be used in inference only. Shall we allow subgraph_property 
to describe if it's inference only and apply it as it requires?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] liangxi627 commented on issue #1486: cannot find reference 'FullyConnected' in 'symbol.py'

[GitBox]

liangxi627 commented on issue #1486: cannot find reference 'FullyConnected' in 
'symbol.py'   
URL: 
https://github.com/apache/incubator-mxnet/issues/1486#issuecomment-411957906
 
 
   @deluopo How do you solve the problem? I met the same error.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] liangxi627 commented on issue #10936: aws-ec2 make -j4 and throw the error

[GitBox]

liangxi627 commented on issue #10936: aws-ec2 make -j4 and throw the error
URL: 
https://github.com/apache/incubator-mxnet/issues/10936#issuecomment-411957695
 
 
   @liuzx32 How do you solve the problem? I met the same error, and my opencv 
version is 3.1.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] reminisce commented on a change in pull request #11325: [MXNET-703] TensorRT runtime integration

[GitBox]

reminisce commented on a change in pull request #11325: [MXNET-703] TensorRT 
runtime integration
URL: https://github.com/apache/incubator-mxnet/pull/11325#discussion_r209128767
 
 

 ##
 File path: python/mxnet/contrib/tensorrt.py
 ##
 @@ -0,0 +1,110 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+""" Module to enable the use of TensorRT optimized graphs."""
+
+import ctypes
+import logging
+import os
+
+from mxnet.symbol import Symbol
 
 Review comment:
   Should here be relative import?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack Transformation API

[GitBox]

tqchen commented on a change in pull request #12047: [MXNET-779]Add DLPack 
Transformation API
URL: https://github.com/apache/incubator-mxnet/pull/12047#discussion_r209127635
 
 

 ##
 File path: python/mxnet/_ctypes/ndarray.py
 ##
 @@ -31,21 +31,24 @@
 
 class NDArrayBase(object):
 """Base data structure for ndarray"""
-__slots__ = ["handle", "writable"]
+__slots__ = ["handle", "writable", "dlpack"]
 # pylint: disable= no-member
 
-def __init__(self, handle, writable=True):
+def __init__(self, handle, writable=True, dlpack=None):
 
 Review comment:
   you can create a new NDArray() that copies the original NDArray(which 
increases refcount) and put that as a context 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] wkcn commented on a change in pull request #12047: [MXNET-779]Add DLPack Transformation API

[GitBox]

wkcn commented on a change in pull request #12047: [MXNET-779]Add DLPack 
Transformation API
URL: https://github.com/apache/incubator-mxnet/pull/12047#discussion_r209124075
 
 

 ##
 File path: python/mxnet/_ctypes/ndarray.py
 ##
 @@ -31,21 +31,24 @@
 
 class NDArrayBase(object):
 """Base data structure for ndarray"""
-__slots__ = ["handle", "writable"]
+__slots__ = ["handle", "writable", "dlpack"]
 # pylint: disable= no-member
 
-def __init__(self, handle, writable=True):
+def __init__(self, handle, writable=True, dlpack=None):
 
 Review comment:
   NDArray in MXNet and TVM are different. NDArray in TVM has the function 
`IncRef` and `DecRef` to change the reference count, however that in MXNet uses 
`NDArray::ptr_` (std::shared_ptr) to manage the reference count. 
`NDArray::ptr_` is a private member of `NDArray`.
   The PR is similar to the PyTorch DLPack implementation 
[Code](https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/DLConvertor.cpp#L128)
   I add a `NDArrayDLManager` to manage the reference count of NDArray. 
[Code](https://github.com/apache/incubator-mxnet/pull/12047/files#diff-67d2910557c85e38570cd7c4f940cdbdR315)
 line 315 in src/ndarray/ndarray.cc
   
   Setting `dlpack` as the NDArray(Python Class) member is to avoid the release 
of the store of data, e.g. When the original NDArray and the PyCapsule (DLPack) 
are release, the new NDArray (generated by from_dlpack) still exists. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] KellenSunderland opened a new pull request #12115: WIP

[GitBox]

KellenSunderland opened a new pull request #12115: WIP
URL: https://github.com/apache/incubator-mxnet/pull/12115
 
 
   WIP - please do not review.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] wkcn commented on a change in pull request #12047: [MXNET-779]Add DLPack Transformation API

[GitBox]

wkcn commented on a change in pull request #12047: [MXNET-779]Add DLPack 
Transformation API
URL: https://github.com/apache/incubator-mxnet/pull/12047#discussion_r209124075
 
 

 ##
 File path: python/mxnet/_ctypes/ndarray.py
 ##
 @@ -31,21 +31,24 @@
 
 class NDArrayBase(object):
 """Base data structure for ndarray"""
-__slots__ = ["handle", "writable"]
+__slots__ = ["handle", "writable", "dlpack"]
 # pylint: disable= no-member
 
-def __init__(self, handle, writable=True):
+def __init__(self, handle, writable=True, dlpack=None):
 
 Review comment:
   NDArray in MXNet and TVM are different. NDArray in TVM has the function 
`IncRef` and `DecRef` to change the reference count, however that in MXNet uses 
`NDArray::ptr_` (std::shared_ptr) to manage the reference count. 
`NDArray::ptr_` is a private member of `NDArray`.
   The PR is similar to the PyTorch DLPack implementation 
[Code](https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/DLConvertor.cpp#L128)
   I add a `NDArrayDLManager` to manage the reference count of NDArray. 
[Code](https://github.com/apache/incubator-mxnet/pull/12047/files#diff-67d2910557c85e38570cd7c4f940cdbdR315)
 line 315 in src/ndarray/ndarray.cc
   
   Setting `dlpack` as the NDArray(Python Class) member is to avoid the release 
of NDArrayDLManager, e.g. When the original NDArray and the PyCapsule (DLPack) 
are release, the new NDArray (generated by from_dlpack) still exists. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] uniquezhengjie edited a comment on issue #12105: Floating point exception (core dumped)

[GitBox]

uniquezhengjie edited a comment on issue #12105: Floating point exception (core 
dumped)
URL: 
https://github.com/apache/incubator-mxnet/issues/12105#issuecomment-411947874
 
 
   @lanking520 this is my system info, i test Python 3.6.4 and Python 2.7.12 on 
ubuntu 16.04 and 14.04. 
   root@5360f6cad7ec:/# cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c
 8  Intel(R) Xeon(R) CPU E5-26xx v4
   root@5360f6cad7ec:/# getconf LONG_BIT
   64
   root@5360f6cad7ec:/# uname -a
   Linux 5360f6cad7ec 4.4.0-91-generic #114-Ubuntu SMP Tue Aug 8 11:56:56 UTC 
2017 x86_64 x86_64 x86_64 GNU/Linux
   
   
   i found that when i run pip install mxnet (not mxnet-mkl)or  build form 
source(https://mxnet.incubator.apache.org/install/index.html?platform=Linux=Python=CPU)
 i can import mxnet normally ,but when i inference my model  ,it cost mort 
time(almost 4~ 5 times than mkl in my normal computers(which  can use 
mxnet-mkl)). 
   my cpu(Intel(R) Xeon(R) CPU E5-26xx v4) can't support mkl?  


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] uniquezhengjie commented on issue #12105: Floating point exception (core dumped)

[GitBox]

uniquezhengjie commented on issue #12105: Floating point exception (core dumped)
URL: 
https://github.com/apache/incubator-mxnet/issues/12105#issuecomment-411947874
 
 
   @lanking520 this is my system info, i test Python 3.6.4 and Python 2.7.12 on 
ubuntu 16.04 and 14.04. 
   root@5360f6cad7ec:/# cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c
 8  Intel(R) Xeon(R) CPU E5-26xx v4
   root@5360f6cad7ec:/# getconf LONG_BIT
   64
   root@5360f6cad7ec:/# uname -a
   Linux 5360f6cad7ec 4.4.0-91-generic #114-Ubuntu SMP Tue Aug 8 11:56:56 UTC 
2017 x86_64 x86_64 x86_64 GNU/Linux
   
   
   i run pip install mxnet (not mxnet-mkl)or  build form 
source(https://mxnet.incubator.apache.org/install/index.html?platform=Linux=Python=CPU)
 i can import mxnet normally ,but when i inference my model  ,it cost mort 
time(almost 4~ 5 times than mkl in my normal computers(which  can use 
mxnet-mkl)). 
   my cpu(Intel(R) Xeon(R) CPU E5-26xx v4) can't support mkl?  


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhreshold commented on issue #12118: fix potential floating number overflow, enable float16

[GitBox]

zhreshold commented on issue #12118: fix potential floating number overflow, 
enable float16
URL: https://github.com/apache/incubator-mxnet/pull/12118#issuecomment-411946601
 
 
   @ijkguo 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhreshold opened a new pull request #12118: fix potential floating number overflow, enable float16

[GitBox]

zhreshold opened a new pull request #12118: fix potential floating number 
overflow, enable float16
URL: https://github.com/apache/incubator-mxnet/pull/12118
 
 
   ## Description ##
   contrib.box_nms and contrib.bipartite_matchin use floating number array to 
store indices which can be extremely large. 
   These indices may not be simply replaced by integer arrays so in order to 
avoid overflow when float32/float16 is casted to integer, this PR force use 
float64 in indices arrays. (the new limit is 2^53 + 1)
   
   - Test updated for various dtypes
   - no functionality change
   - no API change
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [x] Changes are complete (i.e. I finished coding on this PR)
   - [x] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing 
distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a 
new build option with NCCL)
   - [x] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments 
are documented. 
   - For new examples, README.md is added to explain the what the example does, 
the source of the dataset, expected performance on test set and reference to 
the original paper if applicable
   - Check the API doc at 
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
   - [x] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] safrooze commented on issue #12116: Excessive memory allocation without static_alloc

[GitBox]

safrooze commented on issue #12116: Excessive memory allocation without 
static_alloc
URL: 
https://github.com/apache/incubator-mxnet/issues/12116#issuecomment-411945004
 
 
   @zheng-da @szha 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] safrooze opened a new issue #12116: Excessive memory allocation without static_alloc

[GitBox]

safrooze opened a new issue #12116: Excessive memory allocation without 
static_alloc
URL: https://github.com/apache/incubator-mxnet/issues/12116
 
 
   ## Description
   The change in #11951 that fixes nested call on CachedOp causes excessive 
memory allocation when hybridize() is called with default `static_alloc=False`. 
In my specific case, the memory allocation grows from 1.5GB to over 10GB. 
   
   ## Environment info (Required)
   ```
   --Python Info--
   Version  : 3.4.5
   Compiler : GCC 4.4.7 20120313 (Red Hat 4.4.7-1)
   Build: ('default', 'Jul  2 2016 17:47:47')
   Arch : ('64bit', 'ELF')
   Pip Info---
   Version  : 18.0
   Directory: 
/home/ec2-user/anaconda3/envs/mxnet_p34/lib/python3.4/site-packages/pip
   --MXNet Info---
   Version  : 1.3.0
   Directory: /home/ec2-user/src/mxnet/python/mxnet
   Hashtag not found. Not installed from pre-built package.
   --System Info--
   Platform : Linux-4.9.93-41.60.amzn1.x86_64-x86_64-with-glibc2.2.5
   system   : Linux
   node : ip-172-31-73-235
   release  : 4.9.93-41.60.amzn1.x86_64
   version  : #1 SMP Fri Apr 13 21:58:27 UTC 2018
   --Hardware Info--
   machine  : x86_64
   processor: x86_64
   Architecture:  x86_64
   CPU op-mode(s):32-bit, 64-bit
   Byte Order:Little Endian
   CPU(s):8
   On-line CPU(s) list:   0-7
   Thread(s) per core:2
   Core(s) per socket:4
   Socket(s): 1
   NUMA node(s):  1
   Vendor ID: GenuineIntel
   CPU family:6
   Model: 79
   Model name:Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
   Stepping:  1
   CPU MHz:   2699.945
   BogoMIPS:  4600.11
   Hypervisor vendor: Xen
   Virtualization type:   full
   L1d cache: 32K
   L1i cache: 32K
   L2 cache:  256K
   L3 cache:  46080K
   NUMA node0 CPU(s): 0-7
   --Network Test--
   Setting timeout: 10
   Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0023 sec, LOAD: 
0.0982 sec.
   Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.1248 sec, LOAD: 
0.4074 sec.
   Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0031 sec, 
LOAD: 0.1043 sec.
   Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.1631 sec, LOAD: 
0.4245 sec.
   Timing for FashionMNIST: 
https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz,
 DNS: 0.0184 sec, LOAD: 0.5672 sec.
   Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0014 
sec, LOAD: 0.4493 sec.
   ```
   
   I'm using Python package
   
   ## Build info (Required if built from source)
   
   This is reproducible through package from PyPI: `pip install --pre -U 
mxnet-cu90mkl==1.3.0b20180801`. Installing the previous version 
(`1.3.0b20180726`) doesn't have this problem.
   
   I also confirmed that the exact commit in #11951 resulted in this regression 
by building from source before and at this node.
   
   Compiler: gcc
   MXNet commit hash - BROKEN: ed203047573db7681feb5631901a609e6e074971
   MXNet commit hash - GOOD: 98a41af52a2c7cb9e30a6cf81796d431212f217d
   
   Build config:
   (Paste the content of config.mk, or the build command.)
   
   ## Minimum reproducible example
   Don't have one yet. Might be able to come up with one if it's necessary.
   
   ## Steps to reproduce
   1. build hybrid network.
   2. call `hybridize()`
   3. check memory usage using nvidida-smi
   
   ## What have you tried to solve it?
   Nothing yet.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudhacharya opened a new pull request #12117: [MXNET-782] Fix Custom Metric Creation in R tutorial

[GitBox]

anirudhacharya opened a new pull request #12117: [MXNET-782] Fix Custom Metric 
Creation in R tutorial
URL: https://github.com/apache/incubator-mxnet/pull/12117
 
 
   ## Description ##
   Fix custom metric creation in tutorial
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to 
the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues) 
created (except PRs with tiny changes)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing 
distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a 
new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments 
are documented. 
   - For new examples, README.md is added to explain the what the example does, 
the source of the dataset, expected performance on test set and reference to 
the original paper if applicable
   - Check the API doc at 
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
   - [ ] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change
   
   
   ## Comments ##
   @hetong007 @ankkhedia 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-mxnet-site] branch asf-site updated: Bump the publish timestamp.

[zhasheng]

This is an automated email from the ASF dual-hosted git repository.

zhasheng pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-site.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 4512a17  Bump the publish timestamp.
4512a17 is described below

commit 4512a17fcebb5ed9a5ae54ce5b1674a09b4380ef
Author: mxnet-ci 
AuthorDate: Fri Aug 10 00:46:10 2018 +

Bump the publish timestamp.
---
 date.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/date.txt b/date.txt
new file mode 100644
index 000..873c255
--- /dev/null
+++ b/date.txt
@@ -0,0 +1 @@
+Fri Aug 10 00:46:10 UTC 2018

[GitHub] junrushao1994 commented on issue #12078: [MXNET-749] Correct usages of `CutSubgraph` in 3 control flow operators

[GitBox]

junrushao1994 commented on issue #12078: [MXNET-749] Correct usages of 
`CutSubgraph` in 3 control flow operators
URL: https://github.com/apache/incubator-mxnet/pull/12078#issuecomment-411940622
 
 
   @zheng-da Hi, I add unittests for `foreach`, `cond` and `while_loop` 
according to the code you provided offline. Thank you so much for providing the 
testcases!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] KellenSunderland opened a new pull request #12115: WIP

[GitBox]

KellenSunderland opened a new pull request #12115: WIP
URL: https://github.com/apache/incubator-mxnet/pull/12115
 
 
   WIP - please do not review.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] lihaofd commented on a change in pull request #12099: Fix precision issue of test case test_rnnrelu_bidirectional and test_rnnrelu_sym

[GitBox]

lihaofd commented on a change in pull request #12099: Fix precision issue of 
test case test_rnnrelu_bidirectional and test_rnnrelu_sym
URL: https://github.com/apache/incubator-mxnet/pull/12099#discussion_r209115582
 
 

 ##
 File path: tests/python/unittest/test_operator.py
 ##
 @@ -191,13 +191,12 @@ def test_rnnrelu_sym():
 stack.add(mx.rnn.RNNCell(H, activation='relu', prefix='l1_'))
 stack.add(mx.rnn.RNNCell(H, activation='relu', prefix='l2_'))
 
-check_rnn_consistency(fused, stack, T, N, I, H, 'write')
-check_rnn_consistency(fused, stack, T, N, I, H, 'add')
-check_rnn_consistency(fused, stack, T, N, I, H, 'null')
-
+check_rnn_consistency(fused, stack, T, N, I, H, 'write', rtol=1e-2, 
atol=1e-2)
+check_rnn_consistency(fused, stack, T, N, I, H, 'add', rtol=1e-2, 
atol=1e-2)
+check_rnn_consistency(fused, stack, T, N, I, H, 'null', rtol=1e-2, 
atol=1e-2)
 
 
 Review comment:
   just change back tolerance for test_rnnrelu_sym . thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] lanking520 commented on issue #12114: Do we have pyspark support in mxnet.

[GitBox]

lanking520 commented on issue #12114: Do we have pyspark support in mxnet.
URL: 
https://github.com/apache/incubator-mxnet/issues/12114#issuecomment-411934957
 
 
   Hi @shravankumar147 , currently we do not have `pyspark` available, however, 
we do have Spark on Scala, you can see some code and example 
[here](https://github.com/apache/incubator-mxnet/tree/master/scala-package/spark).
   
   @mxnet-label-bot could you please add [feature, python] in here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ctcyang commented on issue #12031: [WIP] Fix CPUPinned unexpected behaviour

[GitBox]

ctcyang commented on issue #12031: [WIP] Fix CPUPinned unexpected behaviour
URL: https://github.com/apache/incubator-mxnet/pull/12031#issuecomment-411933083
 
 
   Added [WIP] tag due to bug with `--kv-store dist_sync_device`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] lanking520 commented on issue #12112: flaky test: test_contrib_control_flow.test_while_loop_for_foreach

[GitBox]

lanking520 commented on issue #12112: flaky test: 
test_contrib_control_flow.test_while_loop_for_foreach
URL: 
https://github.com/apache/incubator-mxnet/issues/12112#issuecomment-411932497
 
 
   @mxnet-label-bot please add [Flaky, test] to this issue


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 commented on issue #12073: get name from symbol cpp

[GitBox]

haojin2 commented on issue #12073: get name from symbol cpp
URL: 
https://github.com/apache/incubator-mxnet/issues/12073#issuecomment-411931655
 
 
   @timprepscius Fix is merged, can you try that out and let me know if you 
found and problems? Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudhacharya commented on issue #12066: Unable to library the package "mxnet" in R 3.5.1 on Mac

[GitBox]

anirudhacharya commented on issue #12066: Unable to library the package "mxnet" 
in R 3.5.1 on Mac
URL: 
https://github.com/apache/incubator-mxnet/issues/12066#issuecomment-411931271
 
 
   @Ruoyu-Qian if you notice, the difference is not just the path of the 
installation but the package itself. Your system has ``libBLAS.dylib``, whereas 
you need ``libRblas.0.dylib``
   
   My guess is the R installation on your MacOS was not right because the 
latest R package when installed on Mac automatically configures ``openblas`` 
for better performance with array operations.
   
   You could try linking your existing ``openblas`` to the R installation as 
follows -
   ```
   cd /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/
   ln -sf  
/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
 libRblas.dylib
   ```
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 commented on issue #12112: flaky test: test_contrib_control_flow.test_while_loop_for_foreach

[GitBox]

haojin2 commented on issue #12112: flaky test: 
test_contrib_control_flow.test_while_loop_for_foreach
URL: 
https://github.com/apache/incubator-mxnet/issues/12112#issuecomment-411930310
 
 
   @junrushao1994 @zheng-da This seems like a numerical tolerance issue, can 
you guys take a look?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 commented on issue #12112: flaky test: test_contrib_control_flow.test_while_loop_for_foreach

[GitBox]

haojin2 commented on issue #12112: flaky test: 
test_contrib_control_flow.test_while_loop_for_foreach
URL: 
https://github.com/apache/incubator-mxnet/issues/12112#issuecomment-411929960
 
 
   @mxnet-label-bot [Flaky, Test].


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy closed issue #7490: How to train with input layer dropout?

[GitBox]

nswamy closed issue #7490: How to train with input layer dropout?
URL: https://github.com/apache/incubator-mxnet/issues/7490
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudhacharya commented on issue #7490: How to train with input layer dropout?

[GitBox]

anirudhacharya commented on issue #7490: How to train with input layer dropout?
URL: 
https://github.com/apache/incubator-mxnet/issues/7490#issuecomment-411929585
 
 
   @nswamy please close this issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #12074: Document MXNET_LIBRARY_PATH environment variable which was not docume…

[GitBox]

nswamy commented on a change in pull request #12074: Document 
MXNET_LIBRARY_PATH environment variable which was not docume…
URL: https://github.com/apache/incubator-mxnet/pull/12074#discussion_r209108279
 
 

 ##
 File path: docs/faq/env_var.md
 ##
 @@ -8,6 +8,18 @@ For example, you can set these environment variables in Linux 
or macOS as follow
 export MXNET_GPU_WORKER_NTHREADS=3
 ```
 
+Or in powershell:
+```
+$env:MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
+```
+
+## Variables controling the execution environment
 
 Review comment:
   _controlling_


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #12074: Document MXNET_LIBRARY_PATH environment variable which was not docume…

[GitBox]

nswamy commented on a change in pull request #12074: Document 
MXNET_LIBRARY_PATH environment variable which was not docume…
URL: https://github.com/apache/incubator-mxnet/pull/12074#discussion_r209108277
 
 

 ##
 File path: docs/faq/env_var.md
 ##
 @@ -8,6 +8,18 @@ For example, you can set these environment variables in Linux 
or macOS as follow
 export MXNET_GPU_WORKER_NTHREADS=3
 ```
 
+Or in powershell:
+```
+$env:MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
+```
+
+## Variables controling the execution environment
+
+* MXNET_LIBRARY_PATH
 
 Review comment:
   where is this used? is this only for windows. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy closed pull request #11562: [MXNET-244] Update RaspberryPI instructions

[GitBox]

nswamy closed pull request #11562: [MXNET-244] Update RaspberryPI instructions
URL: https://github.com/apache/incubator-mxnet/pull/11562
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/docs/install/index.md b/docs/install/index.md
index 57c50eb9bb0..833bedf08af 100644
--- a/docs/install/index.md
+++ b/docs/install/index.md
@@ -2,18 +2,18 @@
 
 Indicate your preferred configuration. Then, follow the customized commands to 
install *MXNet*.
 
-  
-v1.2.1
-
-
-  v1.2.1
-  v1.1.0
-  v1.0.0
-  v0.12.1
-  v0.11.0
-  master
-
-  
+
+  v1.2.1
+  
+  
+v1.2.1
+v1.1.0
+v1.0.0
+v0.12.1
+v0.11.0
+master
+  
+
 
 
 
@@ -52,7 +52,7 @@ Indicate your preferred configuration. Then, follow the 
customized commands to i
 
 
   Raspberry 
Pi
-  NVIDIA Jetson 
TX2
+  NVIDIA Jetson
 
 
 
@@ -1846,6 +1846,7 @@ You can also run distributed deep learning with *MXNet* 
on AWS using [Cloudforma
  
 
 
+
 
   
 
@@ -1853,9 +1854,43 @@ MXNet supports the Debian based Raspbian ARM based 
operating system so you can r
 
 These instructions will walk through how to build MXNet for the Raspberry Pi 
and install the Python bindings for the library.
 
+You can do a dockerized cross compilation build on your local machine or a 
native build on-device.
+
 The complete MXNet library and its requirements can take almost 200MB of RAM, 
and loading large models with the library can take over 1GB of RAM. Because of 
this, we recommend running MXNet on the Raspberry Pi 3 or an equivalent device 
that has more than 1 GB of RAM and a Secure Digital (SD) card that has at least 
4 GB of free memory.
 
-**Install MXNet**
+**Cross compilation build (Experimental)**
+
+## Docker installation
+**Step 1**  Install Docker on your machine by following the [docker 
installation 
instructions](https://docs.docker.com/engine/installation/linux/ubuntu/#install-using-the-repository).
+
+*Note* - You can install Community Edition (CE)
+
+**Step 2** [Optional] Post installation steps to manage Docker as a non-root 
user.
+
+Follow the four steps in this [docker 
documentation](https://docs.docker.com/engine/installation/linux/linux-postinstall/#manage-docker-as-a-non-root-user)
 to allow managing docker containers without *sudo*.
+
+## Build
+
+The following command will build a container with dependencies and tools and 
then compile MXNet for
+ARMv7. The resulting artifact will be located in 
`build/mxnet-x.x.x-py2.py3-none-any.whl`, copy this
+file to your Raspberry Pi.
+
+```bash
+ci/build.py -p armv7
+```
+
+## Install
+
+Create a virtualenv and install the package we created previously.
+
+```bash
+virtualenv -p `which python3` mxnet_py3
+source mxnet_py3/bin/activate
+pip install mxnet-x.x.x-py2.py3-none-any.whl
+```
+
+
+**Native Build**
 
 Installing MXNet is a two-step process:
 
@@ -1874,35 +1909,46 @@ On Raspbian versions Wheezy and later, you need the 
following dependencies:
 
 - A C++ compiler that supports C++ 11. The C++ compiler compiles and builds 
MXNet source code. Supported compilers include the following:
 
-- [G++ (4.8 or later)](https://gcc.gnu.org/gcc-4.8/)
+- [G++ (4.8 or later)](https://gcc.gnu.org/gcc-4.8/). Make sure to use gcc 4 
and not 5 or 6 as there
+  are known bugs with these compilers.
 
 Install these dependencies using the following commands in any directory:
 
 ```bash
 sudo apt-get update
-sudo apt-get -y install git cmake build-essential g++-4.8 c++-4.8 
liblapack* libblas* libopencv*
+sudo apt-get -y install git cmake ninja-build build-essential g++-4.9 
c++-4.9 liblapack* libblas* libopencv* libopenblas* python3-dev virtualenv
 ```
 
-Clone the MXNet source code repository using the following ```git``` command 
in your home directory:
+Clone the MXNet source code repository using the following `git` command in 
your home directory:
 ```bash
 git clone https://github.com/apache/incubator-mxnet.git --recursive
 cd incubator-mxnet
 ```
 
-If you aren't processing images with MXNet on the Raspberry Pi, you can 
minimize the size of the compiled library by building MXNet without the Open 
Source Computer Vision (OpenCV) library with the following commands:
+Build:
 ```bash
-export USE_OPENCV = 0
-make
+mkdir -p build && cd build
+cmake \
+-DUSE_SSE=OFF \
+-DUSE_CUDA=OFF \
+-DUSE_OPENCV=ON \
+-DUSE_OPENMP=ON \
+-DUSE_MKL_IF_AVAILABLE=OFF \
+-DUSE_SIGNAL_HANDLER=ON \
+-DCMAKE_BUILD_TYPE=Release \
+-GNinja ..
+ninja -j1
 ```
+Some compilation units require memory close to 1GB, so it's recommended that 
you enable swap as
+explained below and be cautious about increasing the number of jobs when 
building (-j)
 
-Otherwise, you 

[incubator-mxnet] branch master updated: [MXNET-244] Update RaspberryPI instructions (#11562)

[nswamy]

This is an automated email from the ASF dual-hosted git repository.

nswamy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new bb0f8a6  [MXNET-244] Update RaspberryPI instructions (#11562)
bb0f8a6 is described below

commit bb0f8a6d7738ad06ee7523a8be41a41139ebcaf6
Author: Pedro Larroy <928489+lar...@users.noreply.github.com>
AuthorDate: Fri Aug 10 01:24:39 2018 +0200

[MXNET-244] Update RaspberryPI instructions (#11562)

* Update RaspberryPI instructions
---
 docs/install/index.md | 227 ++
 1 file changed, 136 insertions(+), 91 deletions(-)

diff --git a/docs/install/index.md b/docs/install/index.md
index 57c50eb..833bedf 100644
--- a/docs/install/index.md
+++ b/docs/install/index.md
@@ -2,18 +2,18 @@
 
 Indicate your preferred configuration. Then, follow the customized commands to 
install *MXNet*.
 
-  
-v1.2.1
-
-
-  v1.2.1
-  v1.1.0
-  v1.0.0
-  v0.12.1
-  v0.11.0
-  master
-
-  
+
+  v1.2.1
+  
+  
+v1.2.1
+v1.1.0
+v1.0.0
+v0.12.1
+v0.11.0
+master
+  
+
 
 
 
@@ -52,7 +52,7 @@ Indicate your preferred configuration. Then, follow the 
customized commands to i
 
 
   Raspberry 
Pi
-  NVIDIA Jetson 
TX2
+  NVIDIA Jetson
 
 
 
@@ -1846,6 +1846,7 @@ You can also run distributed deep learning with *MXNet* 
on AWS using [Cloudforma
  
 
 
+
 
   
 
@@ -1853,9 +1854,43 @@ MXNet supports the Debian based Raspbian ARM based 
operating system so you can r
 
 These instructions will walk through how to build MXNet for the Raspberry Pi 
and install the Python bindings for the library.
 
+You can do a dockerized cross compilation build on your local machine or a 
native build on-device.
+
 The complete MXNet library and its requirements can take almost 200MB of RAM, 
and loading large models with the library can take over 1GB of RAM. Because of 
this, we recommend running MXNet on the Raspberry Pi 3 or an equivalent device 
that has more than 1 GB of RAM and a Secure Digital (SD) card that has at least 
4 GB of free memory.
 
-**Install MXNet**
+**Cross compilation build (Experimental)**
+
+## Docker installation
+**Step 1**  Install Docker on your machine by following the [docker 
installation 
instructions](https://docs.docker.com/engine/installation/linux/ubuntu/#install-using-the-repository).
+
+*Note* - You can install Community Edition (CE)
+
+**Step 2** [Optional] Post installation steps to manage Docker as a non-root 
user.
+
+Follow the four steps in this [docker 
documentation](https://docs.docker.com/engine/installation/linux/linux-postinstall/#manage-docker-as-a-non-root-user)
 to allow managing docker containers without *sudo*.
+
+## Build
+
+The following command will build a container with dependencies and tools and 
then compile MXNet for
+ARMv7. The resulting artifact will be located in 
`build/mxnet-x.x.x-py2.py3-none-any.whl`, copy this
+file to your Raspberry Pi.
+
+```bash
+ci/build.py -p armv7
+```
+
+## Install
+
+Create a virtualenv and install the package we created previously.
+
+```bash
+virtualenv -p `which python3` mxnet_py3
+source mxnet_py3/bin/activate
+pip install mxnet-x.x.x-py2.py3-none-any.whl
+```
+
+
+**Native Build**
 
 Installing MXNet is a two-step process:
 
@@ -1874,35 +1909,46 @@ On Raspbian versions Wheezy and later, you need the 
following dependencies:
 
 - A C++ compiler that supports C++ 11. The C++ compiler compiles and builds 
MXNet source code. Supported compilers include the following:
 
-- [G++ (4.8 or later)](https://gcc.gnu.org/gcc-4.8/)
+- [G++ (4.8 or later)](https://gcc.gnu.org/gcc-4.8/). Make sure to use gcc 4 
and not 5 or 6 as there
+  are known bugs with these compilers.
 
 Install these dependencies using the following commands in any directory:
 
 ```bash
 sudo apt-get update
-sudo apt-get -y install git cmake build-essential g++-4.8 c++-4.8 
liblapack* libblas* libopencv*
+sudo apt-get -y install git cmake ninja-build build-essential g++-4.9 
c++-4.9 liblapack* libblas* libopencv* libopenblas* python3-dev virtualenv
 ```
 
-Clone the MXNet source code repository using the following ```git``` command 
in your home directory:
+Clone the MXNet source code repository using the following `git` command in 
your home directory:
 ```bash
 git clone https://github.com/apache/incubator-mxnet.git --recursive
 cd incubator-mxnet
 ```
 
-If you aren't processing images with MXNet on the Raspberry Pi, you can 
minimize the size of the compiled library by building MXNet without the Open 
Source Computer Vision (OpenCV) library with the following commands:
+Build:
 ```bash
-export USE_OPENCV = 0
-make
+mkdir -p build && cd build
+cmake \
+-DUSE_SSE=OFF \
+-DUSE_CUDA=OFF \
+-DUSE_OPENCV=ON \
+-DUSE_OPENMP=ON \
+-DUSE_MKL_IF_AVAILABLE=OFF \

[GitHub] sandeep-krishnamurthy closed issue #8921: windows7::can not stop mxnet process

[GitBox]

sandeep-krishnamurthy closed issue #8921: windows7::can not stop mxnet process 
URL: https://github.com/apache/incubator-mxnet/issues/8921
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sandeep-krishnamurthy closed issue #8638: allow_extra parameter in Line 652 in incubator-mxnet/python/mxnet/module/base_module.py needs to be removed to make things work

[GitBox]

sandeep-krishnamurthy closed issue #8638: allow_extra parameter in Line 652 in 
incubator-mxnet/python/mxnet/module/base_module.py needs to be removed to make 
things work
URL: https://github.com/apache/incubator-mxnet/issues/8638
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ifeherva commented on issue #11984: Generalized broadcast_like operator

[GitBox]

ifeherva commented on issue #11984: Generalized broadcast_like operator
URL: https://github.com/apache/incubator-mxnet/pull/11984#issuecomment-411928046
 
 
   Good point, see last commit.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on issue #10792: [MXNET-400] support string type for kvstore key in cpp-package

[GitBox]

nswamy commented on issue #10792: [MXNET-400] support string type for kvstore 
key in cpp-package
URL: https://github.com/apache/incubator-mxnet/pull/10792#issuecomment-411926670
 
 
   @mxnet-label-bot please add [kvstore, pr-awaiting-review]


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] KellenSunderland commented on a change in pull request #11325: [MXNET-703] TensorRT runtime integration

[GitBox]

KellenSunderland commented on a change in pull request #11325: [MXNET-703] 
TensorRT runtime integration
URL: https://github.com/apache/incubator-mxnet/pull/11325#discussion_r209104790
 
 

 ##
 File path: tests/python/tensorrt/test_tensorrt_lenet5.py
 ##
 @@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import numpy as np
+import mxnet as mx
+from common import *
+from lenet5_common import get_iters
+
+
+def run_inference(sym, arg_params, aux_params, mnist, all_test_labels, 
batch_size, use_tensorrt):
+"""Run inference with either MXNet or TensorRT"""
+mx.contrib.tensorrt.set_use_tensorrt(use_tensorrt)
+
+shared_buffer = merge_dicts(arg_params, aux_params)
+if not use_tensorrt:
+shared_buffer = dict([(k, v.as_in_context(mx.gpu(0))) for k, v in 
shared_buffer.items()])
+
+executor = sym.simple_bind(ctx=mx.gpu(0),
 
 Review comment:
   Yeah I've got a version of an implementation that does it.  Shouldn't take 
me too long to add it.  I wasn't 100% sure this was what was being asked for 
offline.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] shravankumar147 opened a new issue #12114: Do we have pyspark support in mxnet.

[GitBox]

shravankumar147 opened a new issue #12114: Do we have pyspark support in mxnet.
URL: https://github.com/apache/incubator-mxnet/issues/12114
 
 
   Note: Providing complete information in the most concise form is the best 
way to get help. This issue template serves as the checklist for essential 
information to most of the technical issues and bug reports. For non-technical 
issues and feature requests, feel free to present the information in what you 
believe is the best form.
   
   For Q & A and discussion, please start a discussion thread at 
https://discuss.mxnet.io 
   
   ## Description
   (Brief description of the problem in no more than 2 sentences.)
   
   ## Environment info (Required)
   
   ```
   What to do:
   1. Download the diagnosis script from 
https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
   2. Run the script using `python diagnose.py` and paste its output here.
   
   ```
   
   Package used (Python/R/Scala/Julia):
   (I'm using ...)
   
   For Scala user, please provide:
   1. Java version: (`java -version`)
   2. Maven version: (`mvn -version`)
   3. Scala runtime if applicable: (`scala -version`)
   
   For R user, please provide R `sessionInfo()`:
   
   ## Build info (Required if built from source)
   
   Compiler (gcc/clang/mingw/visual studio):
   
   MXNet commit hash:
   (Paste the output of `git rev-parse HEAD` here.)
   
   Build config:
   (Paste the content of config.mk, or the build command.)
   
   ## Error Message:
   (Paste the complete error message, including stack trace.)
   
   ## Minimum reproducible example
   (If you are using your own code, please provide a short script that 
reproduces the error. Otherwise, please provide link to the existing example.)
   
   ## Steps to reproduce
   (Paste the commands you ran that produced the error.)
   
   1.
   2.
   
   ## What have you tried to solve it?
   
   1. I have tried TensorflowOnSpark from Yahoo.
   
   I would like to know, do we have pyspark support on mxnet, with yarn as 
cluster manager. In that case can we train a model with >10 billion records. 
   
   Curious to know your thought on this.
   
   Thanks
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin opened a new pull request #12113: [WIP] zipfian random sampler without replacement

[GitBox]

eric-haibin-lin opened a new pull request #12113: [WIP] zipfian random sampler 
without replacement 
URL: https://github.com/apache/incubator-mxnet/pull/12113
 
 
   ## Description ##
   https://github.com/dmlc/gluon-nlp/issues/262 
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to 
the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues) 
created (except PRs with tiny changes)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing 
distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a 
new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments 
are documented. 
   - For new examples, README.md is added to explain the what the example does, 
the source of the dataset, expected performance on test set and reference to 
the original paper if applicable
   - Check the API doc at 
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
   - [ ] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [ ] Feature1, tests, (and when applicable, API doc)
   - [ ] Feature2, tests, (and when applicable, API doc)
   
   ## Comments ##
   - If this change is a backward incompatible change, why must this change be 
made.
   - Interesting edge cases to note here
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-mxnet] branch master updated: Update fully_connected.cc documentation (#12097)

[haibin]

This is an automated email from the ASF dual-hosted git repository.

haibin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 470c3dc  Update fully_connected.cc documentation (#12097)
470c3dc is described below

commit 470c3dc22895823c6cd7255bb85b510cf6abf6d8
Author: Haibin Lin 
AuthorDate: Thu Aug 9 15:54:52 2018 -0700

Update fully_connected.cc documentation (#12097)
---
 src/operator/nn/fully_connected.cc | 12 +---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/src/operator/nn/fully_connected.cc 
b/src/operator/nn/fully_connected.cc
index d1d84e9..f720a10 100644
--- a/src/operator/nn/fully_connected.cc
+++ b/src/operator/nn/fully_connected.cc
@@ -250,9 +250,15 @@ The learnable parameters include both ``weight`` and 
``bias``.
 
 If ``no_bias`` is set to be true, then the ``bias`` term is ignored.
 
-Note that the operator also supports forward computation with `row_sparse` 
weight and bias,
-where the length of `weight.indices` and `bias.indices` must be equal to 
`num_hidden`.
-This could be used for model inference with `row_sparse` weights trained with 
`SparseEmbedding`.
+.. Note::
+
+The sparse support for FullyConnected is limited to forward evaluation 
with `row_sparse`
+weight and bias, where the length of `weight.indices` and `bias.indices` 
must be equal
+to `num_hidden`. This could be useful for model inference with 
`row_sparse` weights
+trained with importance sampling or noise contrastive estimation.
+
+To compute linear transformation with 'csr' sparse data, sparse.dot is 
recommended instead
+of sparse.FullyConnected.
 
 )code" ADD_FILELINE)
 .set_num_inputs([](const NodeAttrs& attrs) {

[GitHub] eric-haibin-lin closed pull request #12097: Update fully_connected.cc documentation

[GitBox]

eric-haibin-lin closed pull request #12097: Update fully_connected.cc 
documentation
URL: https://github.com/apache/incubator-mxnet/pull/12097
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/operator/nn/fully_connected.cc 
b/src/operator/nn/fully_connected.cc
index d1d84e97529..f720a10c002 100644
--- a/src/operator/nn/fully_connected.cc
+++ b/src/operator/nn/fully_connected.cc
@@ -250,9 +250,15 @@ The learnable parameters include both ``weight`` and 
``bias``.
 
 If ``no_bias`` is set to be true, then the ``bias`` term is ignored.
 
-Note that the operator also supports forward computation with `row_sparse` 
weight and bias,
-where the length of `weight.indices` and `bias.indices` must be equal to 
`num_hidden`.
-This could be used for model inference with `row_sparse` weights trained with 
`SparseEmbedding`.
+.. Note::
+
+The sparse support for FullyConnected is limited to forward evaluation 
with `row_sparse`
+weight and bias, where the length of `weight.indices` and `bias.indices` 
must be equal
+to `num_hidden`. This could be useful for model inference with 
`row_sparse` weights
+trained with importance sampling or noise contrastive estimation.
+
+To compute linear transformation with 'csr' sparse data, sparse.dot is 
recommended instead
+of sparse.FullyConnected.
 
 )code" ADD_FILELINE)
 .set_num_inputs([](const NodeAttrs& attrs) {


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-mxnet] branch master updated: Add unique number of parameters to summary output in Gluon Block (#12077)

[zhasheng]

This is an automated email from the ASF dual-hosted git repository.

zhasheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 23b8f89  Add unique number of parameters to summary output in Gluon 
Block (#12077)
23b8f89 is described below

commit 23b8f8977174b85cb57fd625a9ccef30b4a47c0c
Author: Shuai Zheng 
AuthorDate: Thu Aug 9 15:44:21 2018 -0700

Add unique number of parameters to summary output in Gluon Block (#12077)

* add unique parameters in summary output

* rebuild
---
 python/mxnet/gluon/block.py | 10 ++
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/python/mxnet/gluon/block.py b/python/mxnet/gluon/block.py
index f58b00d..af54c66 100644
--- a/python/mxnet/gluon/block.py
+++ b/python/mxnet/gluon/block.py
@@ -656,10 +656,12 @@ class Block(object):
 trainable_params += summary[layer]['trainable']
 shared_params += summary[layer]['shared']
 print('='*80)
-print('Total params: ' + str(total_params))
-print('Trainable params: ' + str(trainable_params))
-print('Non-trainable params: ' + str(total_params - 
trainable_params))
-print('Shared params: ' + str(shared_params))
+print('Parameters in forward computation graph, duplicate 
included')
+print('   Total params: ' + str(total_params))
+print('   Trainable params: ' + str(trainable_params))
+print('   Non-trainable params: ' + str(total_params - 
trainable_params))
+print('Shared params in forward computation graph: ' + 
str(shared_params))
+print('Unique parameters in model: ' + str(total_params - 
shared_params))
 print('-'*80)
 finally:
 for h in hooks:

[GitHub] szha closed pull request #12077: Add unique number of parameters to summary output in Gluon Block

[GitBox]

szha closed pull request #12077: Add unique number of parameters to summary 
output in Gluon Block
URL: https://github.com/apache/incubator-mxnet/pull/12077
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/python/mxnet/gluon/block.py b/python/mxnet/gluon/block.py
index f58b00dded2..af54c66c42d 100644
--- a/python/mxnet/gluon/block.py
+++ b/python/mxnet/gluon/block.py
@@ -656,10 +656,12 @@ def _summary_hook(block, _, outputs):
 trainable_params += summary[layer]['trainable']
 shared_params += summary[layer]['shared']
 print('='*80)
-print('Total params: ' + str(total_params))
-print('Trainable params: ' + str(trainable_params))
-print('Non-trainable params: ' + str(total_params - 
trainable_params))
-print('Shared params: ' + str(shared_params))
+print('Parameters in forward computation graph, duplicate 
included')
+print('   Total params: ' + str(total_params))
+print('   Trainable params: ' + str(trainable_params))
+print('   Non-trainable params: ' + str(total_params - 
trainable_params))
+print('Shared params in forward computation graph: ' + 
str(shared_params))
+print('Unique parameters in model: ' + str(total_params - 
shared_params))
 print('-'*80)
 finally:
 for h in hooks:


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209101051
 
 

 ##
 File path: scala-package/core/src/main/scala/org/apache/mxnet/IO.scala
 ##
 @@ -136,11 +142,29 @@ class DataBatch(val data: IndexedSeq[NDArray],
 val pad: Int,
 // the key for the bucket that should be used for this batch,
 // for bucketing io only
-val bucketKey: AnyRef = null,
+val bucketKey: AnyRef,
 // use ListMap to indicate the order of data/label loading
 // (must match the order of input data/label)
-private val providedData: ListMap[String, Shape] = null,
-private val providedLabel: ListMap[String, Shape] = null) {
+private val providedData: ListMap[String, Shape],
+private val providedLabel: ListMap[String, Shape],
+val dataDType: DType,
+val labelDType: DType,
+val dataLayout: String,
+val labelLayout: String) {
 
 Review comment:
   can you elaborate a bit? DataDesc already has `name`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] reminisce commented on a change in pull request #11325: [MXNET-703] TensorRT runtime integration

[GitBox]

reminisce commented on a change in pull request #11325: [MXNET-703] TensorRT 
runtime integration
URL: https://github.com/apache/incubator-mxnet/pull/11325#discussion_r209100986
 
 

 ##
 File path: tests/python/tensorrt/test_tensorrt_lenet5.py
 ##
 @@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import numpy as np
+import mxnet as mx
+from common import *
+from lenet5_common import get_iters
+
+
+def run_inference(sym, arg_params, aux_params, mnist, all_test_labels, 
batch_size, use_tensorrt):
+"""Run inference with either MXNet or TensorRT"""
+mx.contrib.tensorrt.set_use_tensorrt(use_tensorrt)
+
+shared_buffer = merge_dicts(arg_params, aux_params)
+if not use_tensorrt:
+shared_buffer = dict([(k, v.as_in_context(mx.gpu(0))) for k, v in 
shared_buffer.items()])
+
+executor = sym.simple_bind(ctx=mx.gpu(0),
 
 Review comment:
   I overlooked this part. Thank @piiswrong for pointing it out. 
@KellenSunderland I believe we all agreed that symbol binding API for TRT 
should be also in the contrib. Could you make the change? Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vandanavk commented on issue #12099: Fix precision issue of test case test_rnnrelu_bidirectional and test_rnnrelu_sym

[GitBox]

vandanavk commented on issue #12099: Fix precision issue of test case 
test_rnnrelu_bidirectional and test_rnnrelu_sym
URL: https://github.com/apache/incubator-mxnet/pull/12099#issuecomment-411920211
 
 
   @eric-haibin-lin please change the tag to pr-awating-review. thanks


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on a change in pull request #11325: [MXNET-703] TensorRT runtime integration

[GitBox]

piiswrong commented on a change in pull request #11325: [MXNET-703] TensorRT 
runtime integration
URL: https://github.com/apache/incubator-mxnet/pull/11325#discussion_r209099570
 
 

 ##
 File path: tests/python/tensorrt/test_tensorrt_lenet5.py
 ##
 @@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import numpy as np
+import mxnet as mx
+from common import *
+from lenet5_common import get_iters
+
+
+def run_inference(sym, arg_params, aux_params, mnist, all_test_labels, 
batch_size, use_tensorrt):
+"""Run inference with either MXNet or TensorRT"""
+mx.contrib.tensorrt.set_use_tensorrt(use_tensorrt)
+
+shared_buffer = merge_dicts(arg_params, aux_params)
+if not use_tensorrt:
+shared_buffer = dict([(k, v.as_in_context(mx.gpu(0))) for k, v in 
shared_buffer.items()])
+
+executor = sym.simple_bind(ctx=mx.gpu(0),
 
 Review comment:
   This is still using shared_buffer to pass in weights.
   
   Per our last discussion could you make a new API. For example, 
mx.sym.contrib.tensorrt_simple_bind.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] yzhliu commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

yzhliu commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209099209
 
 

 ##
 File path: 
scala-package/core/src/main/scala/org/apache/mxnet/io/NDArrayIter.scala
 ##
 @@ -101,10 +115,16 @@ class NDArrayIter(data: IndexedSeq[(String, NDArray)],
   private var cursor = -dataBatchSize
 
   private val (_provideData: ListMap[String, Shape],
-   _provideLabel: ListMap[String, Shape]) = {
+   _provideLabel: ListMap[String, Shape],
 
 Review comment:
   My major point was no need to fetch the data twice


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 opened a new issue #12112: flaky test: test_contrib_control_flow.test_while_loop_for_foreach

[GitBox]

haojin2 opened a new issue #12112: flaky test: 
test_contrib_control_flow.test_while_loop_for_foreach
URL: https://github.com/apache/incubator-mxnet/issues/12112
 
 
   
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1374/pipeline
   ==
   
   FAIL: test_contrib_control_flow.test_while_loop_for_foreach
   
   --
   
   Traceback (most recent call last):
   
 File "C:\Anaconda3\envs\py3\lib\site-packages\nose\case.py", line 197, in 
runTest
   
   self.test(*self.arg)
   
 File 
"C:\jenkins_slave\workspace\ut-python-gpu\tests\python\unittest\common.py", 
line 172, in test_new
   
   orig_test(*args, **kwargs)
   
 File 
"C:\jenkins_slave\workspace\ut-python-gpu\tests\python\unittest\test_contrib_control_flow.py",
 line 727, in test_while_loop_for_foreach
   
   n_steps=5,
   
 File 
"C:\jenkins_slave\workspace\ut-python-gpu\tests\python\unittest\test_contrib_control_flow.py",
 line 480, in case_4
   
   **params
   
 File 
"C:\jenkins_slave\workspace\ut-python-gpu\tests\python\unittest\test_contrib_control_flow.py",
 line 244, in _verify_while_loop
   
   assert_almost_equal(imp_grad, sym_grad, rtol=1e-4, atol=1e-4)
   
 File 
"C:\jenkins_slave\workspace\ut-python-gpu\windows_package\python\mxnet\test_utils.py",
 line 493, in assert_almost_equal
   
   raise AssertionError(msg)
   
   AssertionError: 
   
   Items are not equal:
   
   Error 2.176908 exceeds tolerance rtol=0.000100, atol=0.000100.  Location of 
maximum error:(1, 1, 4), a=-1.308214, b=-1.308716
   
a: array([[[ 0.,  0.,  0., ...,  0.,
   
 0.,  0.],
   
   [ 0.,  0.,  0., ...,  0.,...
   
b: array([[[ 0.,  0.,  0., ...,  0.,
   
 0.,  0.],
   
   [ 0.,  0.,  0., ...,  0.,...
   
    >> begin captured logging << 
   
   common: INFO: Setting test np/mx/python random seeds, use 
MXNET_TEST_SEED=448101308 to reproduce.
   
   - >> end captured logging << -
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] yzhliu commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

yzhliu commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209098637
 
 

 ##
 File path: scala-package/core/src/main/scala/org/apache/mxnet/IO.scala
 ##
 @@ -136,11 +142,29 @@ class DataBatch(val data: IndexedSeq[NDArray],
 val pad: Int,
 // the key for the bucket that should be used for this batch,
 // for bucketing io only
-val bucketKey: AnyRef = null,
+val bucketKey: AnyRef,
 // use ListMap to indicate the order of data/label loading
 // (must match the order of input data/label)
-private val providedData: ListMap[String, Shape] = null,
-private val providedLabel: ListMap[String, Shape] = null) {
+private val providedData: ListMap[String, Shape],
+private val providedLabel: ListMap[String, Shape],
+val dataDType: DType,
+val labelDType: DType,
+val dataLayout: String,
+val labelLayout: String) {
 
 Review comment:
   @nswamy maybe need to keep the `Map`, I think some usage queries by name.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] yzhliu commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

yzhliu commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209097993
 
 

 ##
 File path: scala-package/core/src/main/scala/org/apache/mxnet/IO.scala
 ##
 @@ -136,11 +142,29 @@ class DataBatch(val data: IndexedSeq[NDArray],
 val pad: Int,
 // the key for the bucket that should be used for this batch,
 // for bucketing io only
-val bucketKey: AnyRef = null,
+val bucketKey: AnyRef,
 // use ListMap to indicate the order of data/label loading
 // (must match the order of input data/label)
-private val providedData: ListMap[String, Shape] = null,
-private val providedLabel: ListMap[String, Shape] = null) {
+private val providedData: ListMap[String, Shape],
+private val providedLabel: ListMap[String, Shape],
+val dataDType: DType,
+val labelDType: DType,
+val dataLayout: String,
+val labelLayout: String) {
 
 Review comment:
   how about replace `providedData` by `ListMap[String, DataDesc]`, and 
providedLabel respectively. (can create a new constructor to keep 
backward-compatibility.
   Actually there's no guarantee that all inputs share the same layout and 
dtype.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] andrewfayres commented on a change in pull request #12110: [MXNET-730][WIP] Scala test in nightly

[GitBox]

andrewfayres commented on a change in pull request #12110: [MXNET-730][WIP] 
Scala test in nightly
URL: https://github.com/apache/incubator-mxnet/pull/12110#discussion_r209094090
 
 

 ##
 File path: 
scala-package/examples/src/main/scala/org/apache/mxnetexamples/Util.scala
 ##
 @@ -42,4 +48,30 @@ object Util {
 }
if (!success) throw new Exception(s"$url Download failed!")
   }
+
+  /**
+* This Util is designed to manage the tests in CI
+* @param name the name of the test
+* @return runTest and number of epoch
+*/
+  def testManager(name: String) : (Boolean, Int) = {
 
 Review comment:
   Agreeing with Naveen here. Convention is that we control this type of 
behavior through something like environment variables, build parameters, or 
config files.
   
   One other thing is that we should try to avoid having code which is unique 
to our CI testing inside the src folder.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] lanking520 commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

lanking520 commented on a change in pull request #11844: [MXNET-689] add 
DataDesc type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209093471
 
 

 ##
 File path: scala-package/core/src/main/scala/org/apache/mxnet/IO.scala
 ##
 @@ -216,6 +244,30 @@ object DataBatch {
   this
 }
 
+/**
+  * Set the dtype.
+  * @param dataDType The dtype of the data, default is Float32
+  * @param labelDType The dtype of the label, default is Int32
+  * @return this
+  */
+def setDType(dataDType: DType, labelDType: DType): Builder = {
 
 Review comment:
   But since user always define these field as null, it will introduce crashes 
when we fetch the information from here, I didn't even know why we need 
provideData and provideLabel here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209093042
 
 

 ##
 File path: scala-package/core/src/main/scala/org/apache/mxnet/IO.scala
 ##
 @@ -216,6 +244,30 @@ object DataBatch {
   this
 }
 
+/**
+  * Set the dtype.
+  * @param dataDType The dtype of the data, default is Float32
+  * @param labelDType The dtype of the label, default is Int32
+  * @return this
+  */
+def setDType(dataDType: DType, labelDType: DType): Builder = {
 
 Review comment:
   Look at the constructor you are getting all 4 fields of DataDesc. 
   ```
   private val providedData: ListMap[String, Shape],
   private val providedLabel: ListMap[String, Shape],
   val dataDType: DType,
   val labelDType: DType,
   val dataLayout: String,
   val labelLayout: String)
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] lanking520 commented on issue #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

lanking520 commented on issue #11844: [MXNET-689] add DataDesc type for the 
Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#issuecomment-411910896
 
 
   @nswamy How to bypass the `Shape` field in DataDesc? That's the block I have 
to make this change.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy edited a comment on issue #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy edited a comment on issue #11844: [MXNET-689] add DataDesc type for the 
Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#issuecomment-411910486
 
 
   Why do think `setDType` and `setLayout` is needed when you get it through 
the constructor as a DataDesc ?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on issue #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy commented on issue #11844: [MXNET-689] add DataDesc type for the Scala 
Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#issuecomment-411910486
 
 
   Why do think `setDType` and `setLayout` is need when you get it through the 
constructor as a DataDesc ?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] lanking520 commented on issue #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

lanking520 commented on issue #11844: [MXNET-689] add DataDesc type for the 
Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#issuecomment-411909617
 
 
   Thanks @nswamy @yzhliu @gigasquid for CR and testing. Next Commit will cover:
   - spacing problem fixes
   - Introduce DataDesc somewhere to reduce the amount of args
   - `setDType` and `setLayout`. I think these fields should be available to be 
defined in databatches.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] lanking520 commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

lanking520 commented on a change in pull request #11844: [MXNET-689] add 
DataDesc type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209089051
 
 

 ##
 File path: 
scala-package/examples/src/main/scala/org/apache/mxnetexamples/rnn/BucketIo.scala
 ##
 @@ -94,8 +96,22 @@ object BucketIo {
   class BucketSentenceIter(
   path: String, vocab: Map[String, Int], var buckets: IndexedSeq[Int],
   _batchSize: Int, private val initStates: IndexedSeq[(String, (Int, 
Int))],
-  seperateChar: String = "  ", text2Id: Text2Id = defaultText2Id,
-  readContent: ReadContent = defaultReadContent) extends DataIter {
+  seperateChar: String, text2Id: Text2Id,
+  readContent: ReadContent,
+  dataLayout: String,
+  labelLayout: String,
+  dataDType : DType,
+  labelDType: DType) extends DataIter {
+
+// scalastyle:off
 
 Review comment:
   Same reason explained above


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] lanking520 commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

lanking520 commented on a change in pull request #11844: [MXNET-689] add 
DataDesc type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209088975
 
 

 ##
 File path: scala-package/pom.xml
 ##
 @@ -231,6 +231,7 @@
   ${skipTests}
   
${project.build.directory}/surefire-reports
   .
+  F
 
 Review comment:
   It brings the whole stacktraces for Scalatest. So yes, needed and good for 
all developers to debug.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] lanking520 commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

lanking520 commented on a change in pull request #11844: [MXNET-689] add 
DataDesc type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209088740
 
 

 ##
 File path: 
scala-package/core/src/main/scala/org/apache/mxnet/io/NDArrayIter.scala
 ##
 @@ -101,10 +115,16 @@ class NDArrayIter(data: IndexedSeq[(String, NDArray)],
   private var cursor = -dataBatchSize
 
   private val (_provideData: ListMap[String, Shape],
-   _provideLabel: ListMap[String, Shape]) = {
+   _provideLabel: ListMap[String, Shape],
 
 Review comment:
   @yzhliu recommend do this several days ago in previous CR. Which one works 
better?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] lanking520 commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

lanking520 commented on a change in pull request #11844: [MXNET-689] add 
DataDesc type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209088431
 
 

 ##
 File path: 
scala-package/core/src/main/scala/org/apache/mxnet/io/NDArrayIter.scala
 ##
 @@ -42,27 +43,40 @@ import scala.collection.immutable.ListMap
 class NDArrayIter(data: IndexedSeq[(String, NDArray)],
   label: IndexedSeq[(String, NDArray)],
   private val dataBatchSize: Int, shuffle: Boolean,
-  lastBatchHandle: String) extends DataIter {
-
+  lastBatchHandle: String,
+  dataDType: DType, labelDType: DType,
+  dataLayout: String, labelLayout: String) extends DataIter {
+
+  // scalastyle:off
 
 Review comment:
   I finally figure out the scala style issue. Scala does not allow user to 
define a function with more than a certain numbers of args...


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209076297
 
 

 ##
 File path: scala-package/core/src/main/scala/org/apache/mxnet/Layout.scala
 ##
 @@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mxnet
+
+object Layout {
 
 Review comment:
   add doc string to this, it will be confusing to users and may assume these 
are the only types supported


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209081679
 
 

 ##
 File path: 
scala-package/core/src/main/scala/org/apache/mxnet/io/NDArrayIter.scala
 ##
 @@ -101,10 +115,16 @@ class NDArrayIter(data: IndexedSeq[(String, NDArray)],
   private var cursor = -dataBatchSize
 
   private val (_provideData: ListMap[String, Shape],
-   _provideLabel: ListMap[String, Shape]) = {
+   _provideLabel: ListMap[String, Shape],
 
 Review comment:
   can you separate the _provideDataDesc and _provideLabelDesc into separate 
statement


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209074345
 
 

 ##
 File path: scala-package/core/src/main/scala/org/apache/mxnet/IO.scala
 ##
 @@ -170,6 +194,10 @@ object DataBatch {
 private var label: IndexedSeq[NDArray] = null
 private var index: IndexedSeq[Long] = null
 private var pad: Int = 0
+private var dataLayout: String = Layout.UNDEFINED
 
 Review comment:
   where are these layout and dtype used? if it is in construction you can 
declare the parameters in constructor as private


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209083445
 
 

 ##
 File path: scala-package/core/src/test/scala/org/apache/mxnet/IOSuite.scala
 ##
 @@ -99,7 +101,9 @@ class IOSuite extends FunSuite with BeforeAndAfterAll {
   "data_shape" -> "(3,28,28)",
   "batch_size" -> "100",
   "preprocess_threads" -> "4",
-  "prefetch_buffer" -> "1"
+  "prefetch_buffer" -> "1",
 
 Review comment:
   ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209087658
 
 

 ##
 File path: scala-package/pom.xml
 ##
 @@ -231,6 +231,7 @@
   ${skipTests}
   
${project.build.directory}/surefire-reports
   .
+  F
 
 Review comment:
   is this needed? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209074637
 
 

 ##
 File path: scala-package/core/src/main/scala/org/apache/mxnet/IO.scala
 ##
 @@ -216,6 +244,30 @@ object DataBatch {
   this
 }
 
+/**
+  * Set the dtype.
+  * @param dataDType The dtype of the data, default is Float32
+  * @param labelDType The dtype of the label, default is Int32
+  * @return this
+  */
+def setDType(dataDType: DType, labelDType: DType): Builder = {
 
 Review comment:
   don't see why we need this since the constructor already takes it. This is 
not going to be changed between the iterator invokations


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209086544
 
 

 ##
 File path: 
scala-package/examples/src/main/scala/org/apache/mxnetexamples/rnn/BucketIo.scala
 ##
 @@ -94,8 +96,22 @@ object BucketIo {
   class BucketSentenceIter(
   path: String, vocab: Map[String, Int], var buckets: IndexedSeq[Int],
   _batchSize: Int, private val initStates: IndexedSeq[(String, (Int, 
Int))],
-  seperateChar: String = "  ", text2Id: Text2Id = defaultText2Id,
-  readContent: ReadContent = defaultReadContent) extends DataIter {
+  seperateChar: String, text2Id: Text2Id,
+  readContent: ReadContent,
+  dataLayout: String,
+  labelLayout: String,
+  dataDType : DType,
+  labelDType: DType) extends DataIter {
+
+// scalastyle:off
 
 Review comment:
   why are turning off scalastyle, i saw in a few other places.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Roshrini commented on issue #8638: allow_extra parameter in Line 652 in incubator-mxnet/python/mxnet/module/base_module.py needs to be removed to make things work

[GitBox]

Roshrini commented on issue #8638: allow_extra parameter in Line 652 in 
incubator-mxnet/python/mxnet/module/base_module.py needs to be removed to make 
things work
URL: 
https://github.com/apache/incubator-mxnet/issues/8638#issuecomment-411906466
 
 
   Thanks Vatsal for replying. I tried to reproduce the issue with setting 
parameters creating a module and trying to run the training. I wasn't able to 
reproduce it.
   
   I will be closing this issue for now. Please feel free to open if closed in 
error.
   @sandeep-krishnamurthy Can you please close this issue?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209073735
 
 

 ##
 File path: scala-package/core/src/main/scala/org/apache/mxnet/IO.scala
 ##
 @@ -136,11 +142,29 @@ class DataBatch(val data: IndexedSeq[NDArray],
 val pad: Int,
 // the key for the bucket that should be used for this batch,
 // for bucketing io only
-val bucketKey: AnyRef = null,
+val bucketKey: AnyRef,
 // use ListMap to indicate the order of data/label loading
 // (must match the order of input data/label)
-private val providedData: ListMap[String, Shape] = null,
-private val providedLabel: ListMap[String, Shape] = null) {
+private val providedData: ListMap[String, Shape],
+private val providedLabel: ListMap[String, Shape],
+val dataDType: DType,
+val labelDType: DType,
+val dataLayout: String,
+val labelLayout: String) {
 
 Review comment:
   +1 
   as discussed offline, all of these constitute DataDesc
   `IndexedSeq[DataDesc]` for both data and label descriptors?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209074718
 
 

 ##
 File path: scala-package/core/src/main/scala/org/apache/mxnet/IO.scala
 ##
 @@ -216,6 +244,30 @@ object DataBatch {
   this
 }
 
+/**
+  * Set the dtype.
+  * @param dataDType The dtype of the data, default is Float32
+  * @param labelDType The dtype of the label, default is Int32
+  * @return this
+  */
+def setDType(dataDType: DType, labelDType: DType): Builder = {
+  this.dataDType = dataDType
+  this.labelDType = labelDType
+  this
+}
+
+/**
+  * Set the layout.
+  * @param dataLayout The layout of the data, default is NCHW
+  * @param labelLayout The layout of the label, default is N
+  * @return this
+  */
+def setLayout(dataLayout: String, labelLayout: String): Builder = {
 
 Review comment:
   same


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc type for the Scala Package

[GitBox]

nswamy commented on a change in pull request #11844: [MXNET-689] add DataDesc 
type for the Scala Package
URL: https://github.com/apache/incubator-mxnet/pull/11844#discussion_r209081258
 
 

 ##
 File path: 
scala-package/core/src/main/scala/org/apache/mxnet/io/NDArrayIter.scala
 ##
 @@ -42,27 +43,40 @@ import scala.collection.immutable.ListMap
 class NDArrayIter(data: IndexedSeq[(String, NDArray)],
   label: IndexedSeq[(String, NDArray)],
   private val dataBatchSize: Int, shuffle: Boolean,
-  lastBatchHandle: String) extends DataIter {
-
+  lastBatchHandle: String,
+  dataDType: DType, labelDType: DType,
+  dataLayout: String, labelLayout: String) extends DataIter {
+
+  // scalastyle:off
 
 Review comment:
   why?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] safrooze commented on issue #12001: SyncBatchNorm problems

[GitBox]

safrooze commented on issue #12001: SyncBatchNorm problems
URL: 
https://github.com/apache/incubator-mxnet/issues/12001#issuecomment-411905755
 
 
   @kaleidoscopical I haven't personally used SyncBatchNorm, but the operator 
is available in both `nd` and `sym` format. What are you using?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #12110: [MXNET-730][WIP] Scala test in nightly

[GitBox]

nswamy commented on a change in pull request #12110: [MXNET-730][WIP] Scala 
test in nightly
URL: https://github.com/apache/incubator-mxnet/pull/12110#discussion_r209083159
 
 

 ##
 File path: 
scala-package/examples/src/main/scala/org/apache/mxnetexamples/Util.scala
 ##
 @@ -42,4 +48,30 @@ object Util {
 }
if (!success) throw new Exception(s"$url Download failed!")
   }
+
+  /**
+* This Util is designed to manage the tests in CI
+* @param name the name of the test
+* @return runTest and number of epoch
+*/
+  def testManager(name: String) : (Boolean, Int) = {
 
 Review comment:
   when you look at what tests are running for unit tests/integration tests you 
look at configuration of Maven and immediately know that, with this I have to 
dig through some code to find what tests are running on 
CPU/GPU/Integration/Unit Tests. 
   Also we should not use this `SCALA_TEST_INTEGRATION`.  The environment 
variable we introduced for `GPU` tests(that was a temporary hack) should also 
not be done that way instead through a program argument. `--context`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] lanking520 commented on a change in pull request #12110: [MXNET-730][WIP] Scala test in nightly

[GitBox]

lanking520 commented on a change in pull request #12110: [MXNET-730][WIP] Scala 
test in nightly
URL: https://github.com/apache/incubator-mxnet/pull/12110#discussion_r209081704
 
 

 ##
 File path: 
scala-package/examples/src/main/scala/org/apache/mxnetexamples/Util.scala
 ##
 @@ -42,4 +48,30 @@ object Util {
 }
if (!success) throw new Exception(s"$url Download failed!")
   }
+
+  /**
+* This Util is designed to manage the tests in CI
+* @param name the name of the test
+* @return runTest and number of epoch
+*/
+  def testManager(name: String) : (Boolean, Int) = {
 
 Review comment:
   Let's say this way. If someday you want to add a new flag to split the test 
further for some other use cases, you need to open each test and add that flag 
there. Assume you have 10 tests in the plate, then 10 duplicated code change 
will be applied in the tests. This solution, one change would be necessary. 
Apart from that, this holds the same scenario of downloading files, we create a 
standard input and output into a single place will definite helps us for future 
maintenance.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services