[incubator-mxnet] branch master updated: Adjust MKLDNN compile flags order to resolve some corner case (#9936)

[zhasheng]

This is an automated email from the ASF dual-hosted git repository.

zhasheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new b87c63d  Adjust MKLDNN compile flags order to resolve some corner case 
(#9936)
b87c63d is described below

commit b87c63de391a9e837e1ff2ace0b0ed808882a966
Author: Jin Huang <34262351+jinhuang...@users.noreply.github.com>
AuthorDate: Fri Mar 16 13:55:27 2018 +0800

Adjust MKLDNN compile flags order to resolve some corner case (#9936)
---
 Makefile | 24 
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/Makefile b/Makefile
index 862a40c..2bef8e8 100644
--- a/Makefile
+++ b/Makefile
@@ -118,6 +118,18 @@ ifndef LINT_LANG
LINT_LANG="all"
 endif
 
+ifeq ($(USE_MKLDNN), 1)
+   CFLAGS += -DMXNET_USE_MKLDNN=1
+   CFLAGS += -DUSE_MKL=1
+   CFLAGS += -I$(ROOTDIR)/src/operator/nn/mkldnn/
+   ifneq ($(MKLDNNROOT), $(MKLROOT))
+   CFLAGS += -I$(MKLROOT)/include
+   LDFLAGS += -L$(MKLROOT)/lib
+   endif
+   CFLAGS += -I$(MKLDNNROOT)/include
+   LDFLAGS += -L$(MKLDNNROOT)/lib -lmkldnn -Wl,-rpath,'$${ORIGIN}'
+endif
+
 # setup opencv
 ifeq ($(USE_OPENCV), 1)
CFLAGS += -DMXNET_USE_OPENCV=1 $(shell pkg-config --cflags opencv)
@@ -138,18 +150,6 @@ ifeq ($(USE_NNPACK), 1)
LDFLAGS += -lnnpack
 endif
 
-ifeq ($(USE_MKLDNN), 1)
-   CFLAGS += -DMXNET_USE_MKLDNN=1
-   CFLAGS += -DUSE_MKL=1
-   CFLAGS += -I$(ROOTDIR)/src/operator/nn/mkldnn/
-   ifneq ($(MKLDNNROOT), $(MKLROOT))
-   CFLAGS += -I$(MKLROOT)/include
-   LDFLAGS += -L$(MKLROOT)/lib
-   endif
-   CFLAGS += -I$(MKLDNNROOT)/include
-   LDFLAGS += -L$(MKLDNNROOT)/lib -lmkldnn -Wl,-rpath,'$${ORIGIN}'
-endif
-
 ifeq ($(USE_OPERATOR_TUNING), 1)
CFLAGS += -DMXNET_USE_OPERATOR_TUNING=1
 endif

-- 
To stop receiving notification emails like this one, please contact
zhash...@apache.org.

[GitHub] szha closed pull request #9936: Adjust MKLDNN compile flags order to resolve some corner case

[GitBox]

szha closed pull request #9936: Adjust MKLDNN compile flags order to resolve 
some corner case
URL: https://github.com/apache/incubator-mxnet/pull/9936
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/Makefile b/Makefile
index 5d81c7fbb16..55c1fab9d15 100644
--- a/Makefile
+++ b/Makefile
@@ -118,6 +118,18 @@ ifndef LINT_LANG
LINT_LANG="all"
 endif
 
+ifeq ($(USE_MKLDNN), 1)
+   CFLAGS += -DMXNET_USE_MKLDNN=1
+   CFLAGS += -DUSE_MKL=1
+   CFLAGS += -I$(ROOTDIR)/src/operator/nn/mkldnn/
+   ifneq ($(MKLDNNROOT), $(MKLROOT))
+   CFLAGS += -I$(MKLROOT)/include
+   LDFLAGS += -L$(MKLROOT)/lib
+   endif
+   CFLAGS += -I$(MKLDNNROOT)/include
+   LDFLAGS += -L$(MKLDNNROOT)/lib -lmkldnn -Wl,-rpath,'$${ORIGIN}'
+endif
+
 # setup opencv
 ifeq ($(USE_OPENCV), 1)
CFLAGS += -DMXNET_USE_OPENCV=1 $(shell pkg-config --cflags opencv)
@@ -138,18 +150,6 @@ ifeq ($(USE_NNPACK), 1)
LDFLAGS += -lnnpack
 endif
 
-ifeq ($(USE_MKLDNN), 1)
-   CFLAGS += -DMXNET_USE_MKLDNN=1
-   CFLAGS += -DUSE_MKL=1
-   CFLAGS += -I$(ROOTDIR)/src/operator/nn/mkldnn/
-   ifneq ($(MKLDNNROOT), $(MKLROOT))
-   CFLAGS += -I$(MKLROOT)/include
-   LDFLAGS += -L$(MKLROOT)/lib
-   endif
-   CFLAGS += -I$(MKLDNNROOT)/include
-   LDFLAGS += -L$(MKLDNNROOT)/lib -lmkldnn -Wl,-rpath,'$${ORIGIN}'
-endif
-
 ifeq ($(USE_OPERATOR_TUNING), 1)
CFLAGS += -DMXNET_USE_OPERATOR_TUNING=1
 endif


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference APIs

[GitBox]

yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference 
APIs
URL: https://github.com/apache/incubator-mxnet/pull/9678#discussion_r174998313
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/Predictor.scala
 ##
 @@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet.io.NDArrayIter
+import ml.dmlc.mxnet.{DataDesc, NDArray, Shape}
+import ml.dmlc.mxnet.module.Module
+
+import scala.collection.mutable.ListBuffer
+import org.slf4j.LoggerFactory
+
+/**
+ * Base Trait for MXNet Predictor classes.
+ */
+private[mxnet] trait PredictBase {
+
+  /**
+   * This method will take input as IndexedSeq one dimensional arrays and 
creates
+   * NDArray needed for inference. The array will be reshaped based on the 
input descriptors.
+   * @param input: A IndexedSequence of Scala one-dimensional array, An 
IndexedSequence is
+   * is needed when the model has more than one input/output
+   * @return IndexedSequence array of outputs.
+   */
+  def predict(input: IndexedSeq[Array[Float]]): IndexedSeq[Array[Float]]
+
+  /**
+   * Predict using NDArray as input. This method is useful when the input is a 
batch of data
+   * or when multiple operations on the input/output have to performed.
+   * Note: User is responsible for managing allocation/deallocation of 
NDArrays.
+   * @param input: IndexedSequence NDArrays.
+   * @return output of Predictions as NDArrays.
+   */
+  def predictWithNDArray(input: IndexedSeq[NDArray]): IndexedSeq[NDArray]
+
+}
+
+/**
+ * Implementation of predict routines.
+ *
+ * @param modelPathPrefix PathPrefix from where to load the model.
+ *Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json,
+ * @param inputDescriptors Descriptors defining the input node names, shape,
+ * layout and Type parameters.
+ * Note: If the input Descriptors is missing batchSize('N' in layout),
+ * a batchSize of 1 is assumed for the model.
+ * 
+ */
+class Predictor(modelPathPrefix: String, protected val inputDescriptors: 
IndexedSeq[DataDesc])
+  extends PredictBase {
+
+  private val logger = LoggerFactory.getLogger(classOf[Predictor])
+
+  protected var batchIndex = inputDescriptors(0).layout.indexOf('N')
+  protected var batchSize = if (batchIndex != -1) 
inputDescriptors(0).shape(batchIndex) else 1
+
+  protected var iDescriptors = inputDescriptors
+
+  inputDescriptors.foreach((f: DataDesc) => require(f.layout.indexOf('N') == 
batchIndex,
+"batch size should be in the same index for all inputs"))
+
+  if (batchIndex != -1) {
+inputDescriptors.foreach((f: DataDesc) => require(f.shape(batchIndex) == 
batchSize,
+  "batch size should be same for all inputs"))
+  } else {
+// Note: this is assuming that the input needs a batch
+logger.warn("InputDescriptor does not have batchSize, using 1 as the 
default batchSize")
+iDescriptors = inputDescriptors.map((f: DataDesc) => new DataDesc(f.name,
+  Shape(1 +: f.shape.toVector), f.dtype, 'N' +: f.layout))
+batchIndex = 1
+  }
+
+  protected val mxNetHandler = MXNetHandler()
+
+  protected val mod = loadModule()
+
+  /**
+   * This method will take input as IndexedSeq one dimensional arrays and 
creates
+   * NDArray needed for inference. The array will be reshaped based on the 
input descriptors.
+   *
+   * @param input : A IndexedSequence of Scala one-dimensional array, An 
IndexedSequence is
+   *  is needed when the model has more than one input/output
+   * @return IndexedSequence array of outputs.
+   */
+  override def predict(input: IndexedSeq[Array[Float]])
+  : IndexedSeq[Array[Float]] = {
+
+require(input.length == inputDescriptors.length, "number of inputs 
provided: %d" +
+  " does not match number of inputs in inputDescriptors: 
%d".format(input.length,
+inputDescriptors.length))
+
+for((i, d) <- input.zip(inputDescriptors)) {
+  require (i.length == d.shape.product/batchSize, "number of elements:" +
+" %d in the input does not match the shape:%s".format( i.length, 
d.shape.toString()))
+}
+var inputND: 

[GitHub] yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference APIs

[GitBox]

yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference 
APIs
URL: https://github.com/apache/incubator-mxnet/pull/9678#discussion_r174996978
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/Predictor.scala
 ##
 @@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet.io.NDArrayIter
+import ml.dmlc.mxnet.{Context, DataDesc, NDArray, Shape}
+import ml.dmlc.mxnet.module.Module
+
+import scala.collection.mutable.ListBuffer
+import org.slf4j.LoggerFactory
+
+/**
+ * Base Trait for MXNet Predictor classes.
+ */
+private[mxnet] trait PredictBase {
+
+  /**
+   * This method will take input as IndexedSeq one dimensional arrays and 
creates
+   * NDArray needed for inference. The array will be reshaped based on the 
input descriptors.
+   * @param input: A IndexedSequence of Scala one-dimensional array, An 
IndexedSequence is
+   * is needed when the model has more than one input/output
 
 Review comment:
   Feel confused with 'An IndexedSequence is is needed when the model has more 
than one input/output'. The `IndexedSeq` is not about model, it's just an array 
of input data, one entry for one input sample, right?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference APIs

[GitBox]

yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference 
APIs
URL: https://github.com/apache/incubator-mxnet/pull/9678#discussion_r174996566
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/Predictor.scala
 ##
 @@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet.io.NDArrayIter
+import ml.dmlc.mxnet.{Context, DataDesc, NDArray, Shape}
+import ml.dmlc.mxnet.module.Module
+
+import scala.collection.mutable.ListBuffer
+import org.slf4j.LoggerFactory
+
+/**
+ * Base Trait for MXNet Predictor classes.
+ */
+private[mxnet] trait PredictBase {
 
 Review comment:
   `private[infer]` ? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference APIs

[GitBox]

yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference 
APIs
URL: https://github.com/apache/incubator-mxnet/pull/9678#discussion_r174995928
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/Classifier.scala
 ##
 @@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet.{Context, DataDesc, NDArray}
+import java.io.File
+
+import org.slf4j.LoggerFactory
+
+import scala.io
+import scala.collection.mutable.ListBuffer
+
+trait ClassifierBase {
+
+  /**
+* Takes an Array of Floats and returns corresponding labels, score tuples.
+* @param input: IndexedSequence one-dimensional array of Floats.
+* @param topK: (Optional) How many top_k(sorting will be based on the last 
axis)
+* elements to return, if not passed returns unsorted output.
+* @return IndexedSequence of (Label, Score) tuples.
+*/
+  def classify(input: IndexedSeq[Array[Float]],
+   topK: Option[Int] = None): IndexedSeq[(String, Float)]
+
+  /**
+* Takes a Sequence of NDArrays and returns Label, Score tuples.
+* @param input: Indexed Sequence of NDArrays
+* @param topK: (Optional) How many top_k(sorting will be based on the last 
axis)
+* elements to return, if not passed returns unsorted output.
+* @return Traversable Sequence of (Label, Score) tuple, Score will be in 
the form of NDArray
+*/
+  def classifyWithNDArray(input: IndexedSeq[NDArray],
+  topK: Option[Int] = None): 
IndexedSeq[IndexedSeq[(String, Float)]]
+}
+
+/**
+  * A class for classifier tasks
+  * @param modelPathPrefix PathPrefix from where to load the symbol, 
parameters and synset.txt
+  *Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json
+  *file://model-dir/synset.txt
+  * @param inputDescriptors Descriptors defining the input node names, shape,
+  * layout and Type parameters
+  * @param contexts Device Contexts on which you want to run Inference, 
defaults to CPU.
+  * @param epoch Model epoch to load, defaults to 0.
+  */
+class Classifier(modelPathPrefix: String,
+ protected val inputDescriptors: IndexedSeq[DataDesc],
+ protected val contexts: Array[Context] = Context.cpu(),
+ protected val epoch: Option[Int] = Some(0))
+  extends ClassifierBase {
+
+  private val logger = LoggerFactory.getLogger(classOf[Classifier])
+
+  protected[mxnet] val predictor: PredictBase = getPredictor()
+
+  protected[mxnet] val synsetFilePath = getSynsetFilePath(modelPathPrefix)
+
+  protected[mxnet] val synset = readSynsetFile(synsetFilePath)
+
+  protected[mxnet] val handler = MXNetHandler()
+
+  /**
+* Takes a flat arrays as input and returns a List of (Label, tuple)
+* @param input: IndexedSequence one-dimensional array of Floats.
+* @param topK: (Optional) How many top_k(sorting will be based on the last 
axis)
+* elements to return, if not passed returns unsorted output.
+* @return IndexedSequence of (Label, Score) tuples.
+*/
+  override def classify(input: IndexedSeq[Array[Float]],
+topK: Option[Int] = None): IndexedSeq[(String, Float)] 
= {
+
+// considering only the first output
+val predictResult = predictor.predict(input)(0)
+var result: IndexedSeq[(String, Float)] = IndexedSeq.empty
+
+if (topK.isDefined) {
+  val sortedIndex = 
predictResult.zipWithIndex.sortBy(-_._1).map(_._2).take(topK.get)
+  result = sortedIndex.map(i => (synset(i), predictResult(i))).toIndexedSeq
+} else {
+  result = synset.zip(predictResult).toIndexedSeq
+}
+result
+  }
+
+  /**
+* Takes input as NDArrays, useful when you want to perform multiple 
operations on
+* the input Array or when you want to pass a batch of input.
+* @param input: Indexed Sequence of NDArrays
+* @param topK: (Optional) How many top_k(sorting will be based on the last 
axis)
+* elements to return, if not passed returns unsorted output.
+* @return Traversable Sequence of 

[GitHub] yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference APIs

[GitBox]

yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference 
APIs
URL: https://github.com/apache/incubator-mxnet/pull/9678#discussion_r174998313
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/Predictor.scala
 ##
 @@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet.io.NDArrayIter
+import ml.dmlc.mxnet.{DataDesc, NDArray, Shape}
+import ml.dmlc.mxnet.module.Module
+
+import scala.collection.mutable.ListBuffer
+import org.slf4j.LoggerFactory
+
+/**
+ * Base Trait for MXNet Predictor classes.
+ */
+private[mxnet] trait PredictBase {
+
+  /**
+   * This method will take input as IndexedSeq one dimensional arrays and 
creates
+   * NDArray needed for inference. The array will be reshaped based on the 
input descriptors.
+   * @param input: A IndexedSequence of Scala one-dimensional array, An 
IndexedSequence is
+   * is needed when the model has more than one input/output
+   * @return IndexedSequence array of outputs.
+   */
+  def predict(input: IndexedSeq[Array[Float]]): IndexedSeq[Array[Float]]
+
+  /**
+   * Predict using NDArray as input. This method is useful when the input is a 
batch of data
+   * or when multiple operations on the input/output have to performed.
+   * Note: User is responsible for managing allocation/deallocation of 
NDArrays.
+   * @param input: IndexedSequence NDArrays.
+   * @return output of Predictions as NDArrays.
+   */
+  def predictWithNDArray(input: IndexedSeq[NDArray]): IndexedSeq[NDArray]
+
+}
+
+/**
+ * Implementation of predict routines.
+ *
+ * @param modelPathPrefix PathPrefix from where to load the model.
+ *Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json,
+ * @param inputDescriptors Descriptors defining the input node names, shape,
+ * layout and Type parameters.
+ * Note: If the input Descriptors is missing batchSize('N' in layout),
+ * a batchSize of 1 is assumed for the model.
+ * 
+ */
+class Predictor(modelPathPrefix: String, protected val inputDescriptors: 
IndexedSeq[DataDesc])
+  extends PredictBase {
+
+  private val logger = LoggerFactory.getLogger(classOf[Predictor])
+
+  protected var batchIndex = inputDescriptors(0).layout.indexOf('N')
+  protected var batchSize = if (batchIndex != -1) 
inputDescriptors(0).shape(batchIndex) else 1
+
+  protected var iDescriptors = inputDescriptors
+
+  inputDescriptors.foreach((f: DataDesc) => require(f.layout.indexOf('N') == 
batchIndex,
+"batch size should be in the same index for all inputs"))
+
+  if (batchIndex != -1) {
+inputDescriptors.foreach((f: DataDesc) => require(f.shape(batchIndex) == 
batchSize,
+  "batch size should be same for all inputs"))
+  } else {
+// Note: this is assuming that the input needs a batch
+logger.warn("InputDescriptor does not have batchSize, using 1 as the 
default batchSize")
+iDescriptors = inputDescriptors.map((f: DataDesc) => new DataDesc(f.name,
+  Shape(1 +: f.shape.toVector), f.dtype, 'N' +: f.layout))
+batchIndex = 1
+  }
+
+  protected val mxNetHandler = MXNetHandler()
+
+  protected val mod = loadModule()
+
+  /**
+   * This method will take input as IndexedSeq one dimensional arrays and 
creates
+   * NDArray needed for inference. The array will be reshaped based on the 
input descriptors.
+   *
+   * @param input : A IndexedSequence of Scala one-dimensional array, An 
IndexedSequence is
+   *  is needed when the model has more than one input/output
+   * @return IndexedSequence array of outputs.
+   */
+  override def predict(input: IndexedSeq[Array[Float]])
+  : IndexedSeq[Array[Float]] = {
+
+require(input.length == inputDescriptors.length, "number of inputs 
provided: %d" +
+  " does not match number of inputs in inputDescriptors: 
%d".format(input.length,
+inputDescriptors.length))
+
+for((i, d) <- input.zip(inputDescriptors)) {
+  require (i.length == d.shape.product/batchSize, "number of elements:" +
+" %d in the input does not match the shape:%s".format( i.length, 
d.shape.toString()))
+}
+var inputND: 

[GitHub] yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference APIs

[GitBox]

yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference 
APIs
URL: https://github.com/apache/incubator-mxnet/pull/9678#discussion_r174996385
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/MXNetHandler.scala
 ##
 @@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import java.util.concurrent._
+
+import org.slf4j.LoggerFactory
+
+private[infer] trait MXNetHandler {
+
+  def execute[T](f: => T): T
+
+  val executor: ExecutorService
+
+}
+
+private[infer] object MXNetHandlerType extends Enumeration {
+
+  type MXNetHandlerType = Value
+  val SingleThreadHandler = Value("MXNetSingleThreadHandler")
+  val OneThreadPerModelHandler = Value("MXNetOneThreadPerModelHandler")
+}
+
+private[infer] class MXNetThreadPoolHandler(numThreads: Option[Int] = Some(1))
+  extends MXNetHandler {
+  private val logger = LoggerFactory.getLogger(classOf[MXNetThreadPoolHandler])
+
+  private val threadFactory = new ThreadFactory {
+
+override def newThread(r: Runnable): Thread = new Thread(r) {
+  setName(classOf[MXNetThreadPoolHandler].getCanonicalName
++ "-numThreads: %d".format(numThreads.get))
+}
+  }
+
+  override val executor: ExecutorService =
+Executors.newFixedThreadPool(numThreads.get, threadFactory)
+
+  private val creatorThread = executor.submit(new Callable[Thread] {
+override def call(): Thread = Thread.currentThread()
+  }).get()
+
+  override def execute[T](f: => T): T = {
+
+if (Thread.currentThread() eq creatorThread) {
+  f
 
 Review comment:
   better to write `f()` to distinct from a return of a function variable.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference APIs

[GitBox]

yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference 
APIs
URL: https://github.com/apache/incubator-mxnet/pull/9678#discussion_r174995230
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/Classifier.scala
 ##
 @@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet.{Context, DataDesc, NDArray}
+import java.io.File
+
+import org.slf4j.LoggerFactory
+
+import scala.io
+import scala.collection.mutable.ListBuffer
+
+trait ClassifierBase {
+
+  /**
+* Takes an Array of Floats and returns corresponding labels, score tuples.
+* @param input: IndexedSequence one-dimensional array of Floats.
+* @param topK: (Optional) How many top_k(sorting will be based on the last 
axis)
+* elements to return, if not passed returns unsorted output.
+* @return IndexedSequence of (Label, Score) tuples.
+*/
+  def classify(input: IndexedSeq[Array[Float]],
+   topK: Option[Int] = None): IndexedSeq[(String, Float)]
+
+  /**
+* Takes a Sequence of NDArrays and returns Label, Score tuples.
+* @param input: Indexed Sequence of NDArrays
+* @param topK: (Optional) How many top_k(sorting will be based on the last 
axis)
+* elements to return, if not passed returns unsorted output.
+* @return Traversable Sequence of (Label, Score) tuple, Score will be in 
the form of NDArray
+*/
+  def classifyWithNDArray(input: IndexedSeq[NDArray],
+  topK: Option[Int] = None): 
IndexedSeq[IndexedSeq[(String, Float)]]
+}
+
+/**
+  * A class for classifier tasks
+  * @param modelPathPrefix PathPrefix from where to load the symbol, 
parameters and synset.txt
+  *Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json
+  *file://model-dir/synset.txt
+  * @param inputDescriptors Descriptors defining the input node names, shape,
+  * layout and Type parameters
+  * @param contexts Device Contexts on which you want to run Inference, 
defaults to CPU.
+  * @param epoch Model epoch to load, defaults to 0.
+  */
+class Classifier(modelPathPrefix: String,
+ protected val inputDescriptors: IndexedSeq[DataDesc],
+ protected val contexts: Array[Context] = Context.cpu(),
+ protected val epoch: Option[Int] = Some(0))
+  extends ClassifierBase {
+
+  private val logger = LoggerFactory.getLogger(classOf[Classifier])
+
+  protected[mxnet] val predictor: PredictBase = getPredictor()
+
+  protected[mxnet] val synsetFilePath = getSynsetFilePath(modelPathPrefix)
+
+  protected[mxnet] val synset = readSynsetFile(synsetFilePath)
+
+  protected[mxnet] val handler = MXNetHandler()
+
+  /**
+* Takes a flat arrays as input and returns a List of (Label, tuple)
+* @param input: IndexedSequence one-dimensional array of Floats.
+* @param topK: (Optional) How many top_k(sorting will be based on the last 
axis)
+* elements to return, if not passed returns unsorted output.
+* @return IndexedSequence of (Label, Score) tuples.
+*/
+  override def classify(input: IndexedSeq[Array[Float]],
+topK: Option[Int] = None): IndexedSeq[(String, Float)] 
= {
+
+// considering only the first output
+val predictResult = predictor.predict(input)(0)
+var result: IndexedSeq[(String, Float)] = IndexedSeq.empty
+
+if (topK.isDefined) {
+  val sortedIndex = 
predictResult.zipWithIndex.sortBy(-_._1).map(_._2).take(topK.get)
+  result = sortedIndex.map(i => (synset(i), predictResult(i))).toIndexedSeq
+} else {
+  result = synset.zip(predictResult).toIndexedSeq
+}
+result
+  }
+
+  /**
+* Takes input as NDArrays, useful when you want to perform multiple 
operations on
+* the input Array or when you want to pass a batch of input.
+* @param input: Indexed Sequence of NDArrays
+* @param topK: (Optional) How many top_k(sorting will be based on the last 
axis)
+* elements to return, if not passed returns unsorted output.
+* @return Traversable Sequence of 

[GitHub] yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference APIs

[GitBox]

yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference 
APIs
URL: https://github.com/apache/incubator-mxnet/pull/9678#discussion_r174996407
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/MXNetHandler.scala
 ##
 @@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import java.util.concurrent._
+
+import org.slf4j.LoggerFactory
+
+private[infer] trait MXNetHandler {
+
+  def execute[T](f: => T): T
+
+  val executor: ExecutorService
+
+}
+
+private[infer] object MXNetHandlerType extends Enumeration {
+
+  type MXNetHandlerType = Value
+  val SingleThreadHandler = Value("MXNetSingleThreadHandler")
+  val OneThreadPerModelHandler = Value("MXNetOneThreadPerModelHandler")
+}
+
+private[infer] class MXNetThreadPoolHandler(numThreads: Option[Int] = Some(1))
+  extends MXNetHandler {
+  private val logger = LoggerFactory.getLogger(classOf[MXNetThreadPoolHandler])
+
+  private val threadFactory = new ThreadFactory {
+
+override def newThread(r: Runnable): Thread = new Thread(r) {
+  setName(classOf[MXNetThreadPoolHandler].getCanonicalName
++ "-numThreads: %d".format(numThreads.get))
+}
+  }
+
+  override val executor: ExecutorService =
+Executors.newFixedThreadPool(numThreads.get, threadFactory)
+
+  private val creatorThread = executor.submit(new Callable[Thread] {
+override def call(): Thread = Thread.currentThread()
+  }).get()
+
+  override def execute[T](f: => T): T = {
+
+if (Thread.currentThread() eq creatorThread) {
+  f
+}
+
+val task = new Callable[T] {
+  override def call(): T = {
+logger.debug("threadId: %s".format(Thread.currentThread().getId()))
+f
 
 Review comment:
   better to write f() to distinct from a return of a function variable.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference APIs

[GitBox]

yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference 
APIs
URL: https://github.com/apache/incubator-mxnet/pull/9678#discussion_r174994914
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/Classifier.scala
 ##
 @@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet.{Context, DataDesc, NDArray}
+import java.io.File
+
+import org.slf4j.LoggerFactory
+
+import scala.io
+import scala.collection.mutable.ListBuffer
+
+trait ClassifierBase {
+
+  /**
+* Takes an Array of Floats and returns corresponding labels, score tuples.
+* @param input: IndexedSequence one-dimensional array of Floats.
+* @param topK: (Optional) How many top_k(sorting will be based on the last 
axis)
+* elements to return, if not passed returns unsorted output.
+* @return IndexedSequence of (Label, Score) tuples.
+*/
+  def classify(input: IndexedSeq[Array[Float]],
+   topK: Option[Int] = None): IndexedSeq[(String, Float)]
+
+  /**
+* Takes a Sequence of NDArrays and returns Label, Score tuples.
+* @param input: Indexed Sequence of NDArrays
+* @param topK: (Optional) How many top_k(sorting will be based on the last 
axis)
+* elements to return, if not passed returns unsorted output.
+* @return Traversable Sequence of (Label, Score) tuple, Score will be in 
the form of NDArray
+*/
+  def classifyWithNDArray(input: IndexedSeq[NDArray],
+  topK: Option[Int] = None): 
IndexedSeq[IndexedSeq[(String, Float)]]
+}
+
+/**
+  * A class for classifier tasks
+  * @param modelPathPrefix PathPrefix from where to load the symbol, 
parameters and synset.txt
+  *Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json
+  *file://model-dir/synset.txt
+  * @param inputDescriptors Descriptors defining the input node names, shape,
+  * layout and Type parameters
+  * @param contexts Device Contexts on which you want to run Inference, 
defaults to CPU.
+  * @param epoch Model epoch to load, defaults to 0.
+  */
+class Classifier(modelPathPrefix: String,
+ protected val inputDescriptors: IndexedSeq[DataDesc],
+ protected val contexts: Array[Context] = Context.cpu(),
+ protected val epoch: Option[Int] = Some(0))
+  extends ClassifierBase {
+
+  private val logger = LoggerFactory.getLogger(classOf[Classifier])
+
+  protected[mxnet] val predictor: PredictBase = getPredictor()
+
+  protected[mxnet] val synsetFilePath = getSynsetFilePath(modelPathPrefix)
+
+  protected[mxnet] val synset = readSynsetFile(synsetFilePath)
+
+  protected[mxnet] val handler = MXNetHandler()
 
 Review comment:
   can we make them `protected[infer]`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference APIs

[GitBox]

yzhliu commented on a change in pull request #9678: [MXNET-50] Scala Inference 
APIs
URL: https://github.com/apache/incubator-mxnet/pull/9678#discussion_r174997037
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/Predictor.scala
 ##
 @@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet.io.NDArrayIter
+import ml.dmlc.mxnet.{Context, DataDesc, NDArray, Shape}
+import ml.dmlc.mxnet.module.Module
+
+import scala.collection.mutable.ListBuffer
+import org.slf4j.LoggerFactory
+
+/**
+ * Base Trait for MXNet Predictor classes.
+ */
+private[mxnet] trait PredictBase {
+
+  /**
+   * This method will take input as IndexedSeq one dimensional arrays and 
creates
+   * NDArray needed for inference. The array will be reshaped based on the 
input descriptors.
+   * @param input: A IndexedSequence of Scala one-dimensional array, An 
IndexedSequence is
+   * is needed when the model has more than one input/output
+   * @return IndexedSequence array of outputs.
+   */
+  def predict(input: IndexedSeq[Array[Float]]): IndexedSeq[Array[Float]]
+
+  /**
+   * Predict using NDArray as input. This method is useful when the input is a 
batch of data
+   * or when multiple operations on the input/output have to performed.
+   * Note: User is responsible for managing allocation/deallocation of 
NDArrays.
+   * @param input: IndexedSequence NDArrays.
+   * @return output of Predictions as NDArrays.
+   */
+  def predictWithNDArray(input: IndexedSeq[NDArray]): IndexedSeq[NDArray]
+
+}
+
+/**
+ * Implementation of predict routines.
+ *
+ * @param modelPathPrefix PathPrefix from where to load the model.
+ *Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json,
+ * @param inputDescriptors Descriptors defining the input node names, shape,
+ * layout and Type parameters.
+ * Note: If the input Descriptors is missing batchSize('N' in layout),
+ * a batchSize of 1 is assumed for the model.
+ * 
+ * @param contexts Device Contexts on which you want to run Inference, 
defaults to CPU.
+ * @param epoch Model epoch to load, defaults to 0.
+ */
+class Predictor(modelPathPrefix: String,
+protected val inputDescriptors: IndexedSeq[DataDesc],
+protected val contexts: Array[Context] = Context.cpu(),
+protected val epoch: Option[Int] = Some(0))
+extends PredictBase {
+
+  private val logger = LoggerFactory.getLogger(classOf[Predictor])
+
+  require(inputDescriptors.head.layout.size != 0, "layout size should not be 
zero")
+
+  protected[mxnet] var batchIndex = inputDescriptors(0).layout.indexOf('N')
+  protected[mxnet] var batchSize = if (batchIndex != -1) 
inputDescriptors(0).shape(batchIndex)
+else 1
+
+  protected[mxnet] var iDescriptors = inputDescriptors
 
 Review comment:
   `protected[infer]` ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] CoinCheung commented on a change in pull request #10000: [MXNET-80] Fix average pooling kernel size assignment error

[GitBox]

CoinCheung commented on a change in pull request #1: [MXNET-80] Fix average 
pooling kernel size assignment error
URL: https://github.com/apache/incubator-mxnet/pull/1#discussion_r174997176
 
 

 ##
 File path: src/operator/nn/pooling-inl.h
 ##
 @@ -132,19 +132,21 @@ class PoolingOp {
 using namespace mshadow;
 Stream *s = ctx.get_stream();
 const TShape& ishape = in_data.shape_;
+TShape kernel = param_.kernel;
 TShape padding = param_.pad;
+TShape stride = param_.stride;
 if (param_.global_pool) {
-  for (index_t i = 0; i < padding.ndim(); i++) {
+  kernel = TShape(ishape.data()+ishape.ndim()-param_.kernel.ndim(), 
ishape.data()+ishape.ndim());
 
 Review comment:
   @sxjscience 
   It still does not work even though kernel is set accordingly. By the way, I 
do not think the original implementation does wrong here, as it also considers 
the global_pool and resets the kernel.  
   Also I am not familiar with the source codes and the codes are in such a 
mass that I failed to find the definition of ```TShape```. Would you please 
show me where ```TShape``` is defined?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha closed pull request #9705: [MXNET-91] Added unittest for benchmarking metric performance

[GitBox]

szha closed pull request #9705: [MXNET-91] Added unittest for benchmarking 
metric performance
URL: https://github.com/apache/incubator-mxnet/pull/9705
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/tests/python/unittest/test_metric_perf.py 
b/tests/python/unittest/test_metric_perf.py
new file mode 100644
index 000..36cbc685797
--- /dev/null
+++ b/tests/python/unittest/test_metric_perf.py
@@ -0,0 +1,124 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import print_function
+
+import itertools
+import mxnet as mx
+import sys
+import time
+
+
+class MetricDataGen(object):
+""" Base class for generating random data for metric benchmarking """
+def __init__(self, n, c, pred_ctx, label_ctx):
+self.n = n
+self.c = c
+self.pred_ctx = pred_ctx
+self.label_ctx = label_ctx
+
+def data(self):
+mx.random.seed(0)
+pred = mx.nd.random_uniform(0.0, 1.0, (self.n, self.c), 
ctx=self.pred_ctx)
+label = mx.nd.random_uniform(0.0, self.c - 1, (self.n,), 
ctx=self.label_ctx).round()
+return label, pred
+
+@property
+def batch_size(self):
+return self.n
+
+@property
+def output_dim(self):
+return self.c
+
+
+class F1MetricDataGen(MetricDataGen):
+""" Class for generating random data for F1 metric benchmarking """
+def __init__(self, n, c, pred_ctx, label_ctx):
+super(F1MetricDataGen, self).__init__(n, 2, pred_ctx, label_ctx)
+
+
+class PearsonMetricDataGen(MetricDataGen):
+""" Class for generating random data for Pearson Correlation metric 
benchmarking """
+def __init__(self, n, c, pred_ctx, label_ctx):
+super(PearsonMetricDataGen, self).__init__(n, c, pred_ctx, label_ctx)
+
+def data(self):
+mx.random.seed(0)
+pred = mx.nd.random_uniform(0.0, 1.0, (self.n, self.c), 
ctx=self.pred_ctx)
+label = mx.nd.random_uniform(0.0, 1.0, (self.n, self.c), 
ctx=self.label_ctx)
+return label, pred
+
+
+def run_metric(name, data_gen_cls, i, n, c, pred_ctx, label_ctx, **kwargs):
+""" Helper function for running one metric benchmark """
+metric = mx.metric.create(name, **kwargs)
+data_gen = data_gen_cls(n, c, pred_ctx, label_ctx)
+try:
+label, pred = data_gen.data()
+mx.nd.waitall()
+before = time.time()
+metric.update([label] * i, [pred] * i)
+mx.nd.waitall()
+elapsed = time.time() - before
+elapsed_str = "{:<.5}".format(elapsed)
+except mx.MXNetError:
+elapsed_str = "FAILED"
+
print("{metric:<15}{pctx:<10}{lctx:<12}{niter:<12}{bs:<15}{out_dim:<15}{elapsed:<}".format(
+metric=name, pctx=str(pred_ctx), lctx=str(label_ctx), niter=i * n, 
bs=data_gen.batch_size,
+out_dim=data_gen.output_dim, elapsed=elapsed_str), file=sys.stderr)
+
+
+def test_metric_performance():
+""" unittest entry for metric performance benchmarking """
+# Each dictionary entry is (metric_name:(kwargs, DataGenClass))
+metrics = [
+('acc', ({}, MetricDataGen)),
+('top_k_acc', ({'top_k': 5}, MetricDataGen)),
+('F1', ({}, F1MetricDataGen)),
+('Perplexity', ({'ignore_label': -1}, MetricDataGen)),
+('MAE', ({}, MetricDataGen)),
+('MSE', ({}, MetricDataGen)),
+('RMSE', ({}, MetricDataGen)),
+('ce', ({}, MetricDataGen)),
+('nll_loss', ({}, MetricDataGen)),
+('pearsonr', ({}, PearsonMetricDataGen)),
+]
+
+data_size = 1024 * 128
+
+batch_sizes = [16, 64, 256, 1024]
+output_dims = [128, 1024, 8192]
+ctxs = [mx.cpu(), mx.gpu()]
+
+print("\nmx.metric benchmarks", file=sys.stderr)
+print(
+"{:15}{:10}{:12}{:12}{:15}{:15}{}".format(
+'Metric', 'Data-Ctx', 'Label-Ctx', 'Data Size', 'Batch Size', 
'Output Dim', 'Elapsed Time'),
+file=sys.stderr)
+print("{:-^90}".format(''), file=sys.stderr)
+for k, v in metrics:
+for c in output_dims:
+for 

[incubator-mxnet] branch master updated: Added test_metric_perf.py as a separate file (#9705)

[zhasheng]

This is an automated email from the ASF dual-hosted git repository.

zhasheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 4ad37d8  Added test_metric_perf.py as a separate file (#9705)
4ad37d8 is described below

commit 4ad37d891372ad06c83deb3c2c5cf413b6c9cde5
Author: Sina Afrooze 
AuthorDate: Thu Mar 15 20:57:43 2018 -0700

Added test_metric_perf.py as a separate file (#9705)

- Output of the benchmark is sent to stderr
- random is seeded
- nd.wait_all() used before starting timing and before ending timing
- Added batch-size values of 16, 64, 256, and 1024
- Datasize varies by number of output channels to keep total runtime down 
to a few minutes
---
 tests/python/unittest/test_metric_perf.py | 124 ++
 1 file changed, 124 insertions(+)

diff --git a/tests/python/unittest/test_metric_perf.py 
b/tests/python/unittest/test_metric_perf.py
new file mode 100644
index 000..36cbc68
--- /dev/null
+++ b/tests/python/unittest/test_metric_perf.py
@@ -0,0 +1,124 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import print_function
+
+import itertools
+import mxnet as mx
+import sys
+import time
+
+
+class MetricDataGen(object):
+""" Base class for generating random data for metric benchmarking """
+def __init__(self, n, c, pred_ctx, label_ctx):
+self.n = n
+self.c = c
+self.pred_ctx = pred_ctx
+self.label_ctx = label_ctx
+
+def data(self):
+mx.random.seed(0)
+pred = mx.nd.random_uniform(0.0, 1.0, (self.n, self.c), 
ctx=self.pred_ctx)
+label = mx.nd.random_uniform(0.0, self.c - 1, (self.n,), 
ctx=self.label_ctx).round()
+return label, pred
+
+@property
+def batch_size(self):
+return self.n
+
+@property
+def output_dim(self):
+return self.c
+
+
+class F1MetricDataGen(MetricDataGen):
+""" Class for generating random data for F1 metric benchmarking """
+def __init__(self, n, c, pred_ctx, label_ctx):
+super(F1MetricDataGen, self).__init__(n, 2, pred_ctx, label_ctx)
+
+
+class PearsonMetricDataGen(MetricDataGen):
+""" Class for generating random data for Pearson Correlation metric 
benchmarking """
+def __init__(self, n, c, pred_ctx, label_ctx):
+super(PearsonMetricDataGen, self).__init__(n, c, pred_ctx, label_ctx)
+
+def data(self):
+mx.random.seed(0)
+pred = mx.nd.random_uniform(0.0, 1.0, (self.n, self.c), 
ctx=self.pred_ctx)
+label = mx.nd.random_uniform(0.0, 1.0, (self.n, self.c), 
ctx=self.label_ctx)
+return label, pred
+
+
+def run_metric(name, data_gen_cls, i, n, c, pred_ctx, label_ctx, **kwargs):
+""" Helper function for running one metric benchmark """
+metric = mx.metric.create(name, **kwargs)
+data_gen = data_gen_cls(n, c, pred_ctx, label_ctx)
+try:
+label, pred = data_gen.data()
+mx.nd.waitall()
+before = time.time()
+metric.update([label] * i, [pred] * i)
+mx.nd.waitall()
+elapsed = time.time() - before
+elapsed_str = "{:<.5}".format(elapsed)
+except mx.MXNetError:
+elapsed_str = "FAILED"
+
print("{metric:<15}{pctx:<10}{lctx:<12}{niter:<12}{bs:<15}{out_dim:<15}{elapsed:<}".format(
+metric=name, pctx=str(pred_ctx), lctx=str(label_ctx), niter=i * n, 
bs=data_gen.batch_size,
+out_dim=data_gen.output_dim, elapsed=elapsed_str), file=sys.stderr)
+
+
+def test_metric_performance():
+""" unittest entry for metric performance benchmarking """
+# Each dictionary entry is (metric_name:(kwargs, DataGenClass))
+metrics = [
+('acc', ({}, MetricDataGen)),
+('top_k_acc', ({'top_k': 5}, MetricDataGen)),
+('F1', ({}, F1MetricDataGen)),
+('Perplexity', ({'ignore_label': -1}, MetricDataGen)),
+('MAE', ({}, MetricDataGen)),
+('MSE', ({}, MetricDataGen)),
+('RMSE', ({}, MetricDataGen)),
+('ce', ({}, MetricDataGen)),
+('nll_loss', ({}, MetricDataGen)),
+('pearsonr', ({}, 

[GitHub] haojin2 commented on a change in pull request #10125: [MXNET-100] Support float16 in Correlation operator

[GitBox]

haojin2 commented on a change in pull request #10125: [MXNET-100] Support 
float16 in Correlation operator 
URL: https://github.com/apache/incubator-mxnet/pull/10125#discussion_r174990270
 
 

 ##
 File path: src/operator/correlation-inl.h
 ##
 @@ -79,14 +79,14 @@ class CorrelationOp : public Operator {
 CHECK_EQ(in_data.size(), 2U);
 CHECK_EQ(out_data.size(), 3U);
 Stream *s = ctx.get_stream();
-Tensor data1 = in_data[Correlation::kData1].get(s);
-Tensor data2 = in_data[Correlation::kData2].get(s);
-Tensor out   = out_data[Correlation::kOut].get(s);
-Tensor tmp1  = out_data[Correlation::kTemp1].get(s);
-Tensor tmp2  = out_data[Correlation::kTemp2].get(s);
-tmp1 = 0.0f;
-tmp2 = 0.0f;
-out = 0.0f;
+Tensor data1 = in_data[Correlation::kData1].get(s);
+Tensor data2 = in_data[Correlation::kData2].get(s);
+Tensor out   = out_data[Correlation::kOut].get(s);
+Tensor tmp1  = out_data[Correlation::kTemp1].get(s);
+Tensor tmp2  = out_data[Correlation::kTemp2].get(s);
+tmp1 = DType(0.0f);
+tmp2 = DType(0.0f);
 
 Review comment:
   Seems like static_cast is causing some compilation errors, taking a look at 
it now.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 commented on a change in pull request #10125: [MXNET-100] Support float16 in Correlation operator

[GitBox]

haojin2 commented on a change in pull request #10125: [MXNET-100] Support 
float16 in Correlation operator 
URL: https://github.com/apache/incubator-mxnet/pull/10125#discussion_r174989030
 
 

 ##
 File path: src/operator/correlation-inl.h
 ##
 @@ -228,6 +228,22 @@ void Init(const std::vector >& kwargs) overr
 out_shape->push_back(Shape4(dshape1[0], paddedbottomheight, 
paddedbottomwidth, dshape1[1]));
 return true;
   }
+  bool InferType(std::vector *in_type,
+ std::vector *out_type,
+ std::vector *aux_type) const override {
+int dtype = (*in_type)[0];
+type_assign(&(*in_type)[1], dtype);
+type_assign(&(*out_type)[0], dtype);
+type_assign(&(*out_type)[1], dtype);
+type_assign(&(*out_type)[2], dtype);
+
+TYPE_ASSIGN_CHECK(*in_type, 0, dtype);
 
 Review comment:
   If the reduced datatype is not -1 then all inputs and outputs will then have 
the same datatype at the end of this function.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 commented on a change in pull request #10125: [MXNET-100] Support float16 in Correlation operator

[GitBox]

haojin2 commented on a change in pull request #10125: [MXNET-100] Support 
float16 in Correlation operator 
URL: https://github.com/apache/incubator-mxnet/pull/10125#discussion_r174988891
 
 

 ##
 File path: src/operator/correlation-inl.h
 ##
 @@ -79,14 +79,14 @@ class CorrelationOp : public Operator {
 CHECK_EQ(in_data.size(), 2U);
 CHECK_EQ(out_data.size(), 3U);
 Stream *s = ctx.get_stream();
-Tensor data1 = in_data[Correlation::kData1].get(s);
-Tensor data2 = in_data[Correlation::kData2].get(s);
-Tensor out   = out_data[Correlation::kOut].get(s);
-Tensor tmp1  = out_data[Correlation::kTemp1].get(s);
-Tensor tmp2  = out_data[Correlation::kTemp2].get(s);
-tmp1 = 0.0f;
-tmp2 = 0.0f;
-out = 0.0f;
+Tensor data1 = in_data[Correlation::kData1].get(s);
+Tensor data2 = in_data[Correlation::kData2].get(s);
+Tensor out   = out_data[Correlation::kOut].get(s);
+Tensor tmp1  = out_data[Correlation::kTemp1].get(s);
+Tensor tmp2  = out_data[Correlation::kTemp2].get(s);
+tmp1 = DType(0.0f);
+tmp2 = DType(0.0f);
 
 Review comment:
   Sure, will address this issue shortly


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 commented on a change in pull request #10125: [MXNET-100] Support float16 in Correlation operator

[GitBox]

haojin2 commented on a change in pull request #10125: [MXNET-100] Support 
float16 in Correlation operator 
URL: https://github.com/apache/incubator-mxnet/pull/10125#discussion_r174988873
 
 

 ##
 File path: src/operator/correlation-inl.h
 ##
 @@ -228,6 +228,22 @@ void Init(const std::vector >& kwargs) overr
 out_shape->push_back(Shape4(dshape1[0], paddedbottomheight, 
paddedbottomwidth, dshape1[1]));
 return true;
   }
+  bool InferType(std::vector *in_type,
+ std::vector *out_type,
+ std::vector *aux_type) const override {
+int dtype = (*in_type)[0];
+type_assign(&(*in_type)[1], dtype);
+type_assign(&(*out_type)[0], dtype);
+type_assign(&(*out_type)[1], dtype);
+type_assign(&(*out_type)[2], dtype);
+
+TYPE_ASSIGN_CHECK(*in_type, 0, dtype);
 
 Review comment:
   It was advice from Jun, he suggested that we can do a mutual inference by 
reducing all datatypes to one and then assign the reduced type back to 
everything.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Jerryzcn commented on issue #10104: [WIP][MXNET-107] Fused RNN implementation for CPU

[GitBox]

Jerryzcn commented on issue #10104: [WIP][MXNET-107] Fused RNN implementation 
for CPU
URL: https://github.com/apache/incubator-mxnet/pull/10104#issuecomment-373587916
 
 
   should we create a separate branch for cpu rnn. Once all the changes are 
checked in, we merge the rnn branch with the master. This way the master won't 
break people's code.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudh2290 commented on a change in pull request #10125: [MXNET-100] Support float16 in Correlation operator

[GitBox]

anirudh2290 commented on a change in pull request #10125: [MXNET-100] Support 
float16 in Correlation operator 
URL: https://github.com/apache/incubator-mxnet/pull/10125#discussion_r174977144
 
 

 ##
 File path: src/operator/correlation-inl.h
 ##
 @@ -79,14 +79,14 @@ class CorrelationOp : public Operator {
 CHECK_EQ(in_data.size(), 2U);
 CHECK_EQ(out_data.size(), 3U);
 Stream *s = ctx.get_stream();
-Tensor data1 = in_data[Correlation::kData1].get(s);
-Tensor data2 = in_data[Correlation::kData2].get(s);
-Tensor out   = out_data[Correlation::kOut].get(s);
-Tensor tmp1  = out_data[Correlation::kTemp1].get(s);
-Tensor tmp2  = out_data[Correlation::kTemp2].get(s);
-tmp1 = 0.0f;
-tmp2 = 0.0f;
-out = 0.0f;
+Tensor data1 = in_data[Correlation::kData1].get(s);
+Tensor data2 = in_data[Correlation::kData2].get(s);
+Tensor out   = out_data[Correlation::kOut].get(s);
+Tensor tmp1  = out_data[Correlation::kTemp1].get(s);
+Tensor tmp2  = out_data[Correlation::kTemp2].get(s);
+tmp1 = DType(0.0f);
+tmp2 = DType(0.0f);
 
 Review comment:
   Preferably, Use static_cast.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudh2290 commented on a change in pull request #10125: [MXNET-100] Support float16 in Correlation operator

[GitBox]

anirudh2290 commented on a change in pull request #10125: [MXNET-100] Support 
float16 in Correlation operator 
URL: https://github.com/apache/incubator-mxnet/pull/10125#discussion_r174978587
 
 

 ##
 File path: src/operator/correlation-inl.h
 ##
 @@ -228,6 +228,22 @@ void Init(const std::vector >& kwargs) overr
 out_shape->push_back(Shape4(dshape1[0], paddedbottomheight, 
paddedbottomwidth, dshape1[1]));
 return true;
   }
+  bool InferType(std::vector *in_type,
+ std::vector *out_type,
+ std::vector *aux_type) const override {
+int dtype = (*in_type)[0];
+type_assign(&(*in_type)[1], dtype);
+type_assign(&(*out_type)[0], dtype);
+type_assign(&(*out_type)[1], dtype);
+type_assign(&(*out_type)[2], dtype);
+
+TYPE_ASSIGN_CHECK(*in_type, 0, dtype);
 
 Review comment:
   Why do you need both type_assign and TYPE_ASSIGN_CHECK ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-mxnet-site] branch asf-site updated (0024b23 -> c921d4f)

[liuyizhi]

This is an automated email from the ASF dual-hosted git repository.

liuyizhi pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-site.git.


from 0024b23  Merge pull request #58 from aaronmarkham/redirect
 add 50e6250  fixed why mxnet redirection issue
 add 08d3713  fixed why mxnet redirection issue
 add c4b8a4c  updated index.html
 add 5ac8462  updated get_started redirection rule and removed get_started 
directory
 new c921d4f  Merge pull request #59 from 
thinksanky/fix_why_mxnet_redirection

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .htaccess   |   2 +
 {get_started => faq}/why_mxnet.html |   0
 get_started/index.html  | 253 
 index.html  |   2 +-
 4 files changed, 3 insertions(+), 254 deletions(-)
 rename {get_started => faq}/why_mxnet.html (100%)
 delete mode 100644 get_started/index.html

-- 
To stop receiving notification emails like this one, please contact
liuyi...@apache.org.

[GitHub] eric-haibin-lin commented on issue #8658: mxnet random seed does not work for mx.init.Xavier on both CPU and GPU

[GitBox]

eric-haibin-lin commented on issue #8658: mxnet random seed does not work for 
mx.init.Xavier on both CPU and GPU
URL: 
https://github.com/apache/incubator-mxnet/issues/8658#issuecomment-373569323
 
 
   Fixed with https://github.com/apache/incubator-mxnet/pull/9658


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin closed issue #8658: mxnet random seed does not work for mx.init.Xavier on both CPU and GPU

[GitBox]

eric-haibin-lin closed issue #8658: mxnet random seed does not work for 
mx.init.Xavier on both CPU and GPU
URL: https://github.com/apache/incubator-mxnet/issues/8658
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kazizzad opened a new issue #10126: Change a block

[GitBox]

kazizzad opened a new issue #10126: Change a block
URL: https://github.com/apache/incubator-mxnet/issues/10126
 
 
   Hi, 
   I am trying to implement the following work
   https://openreview.net/pdf?id=B1QRgziT-
   It has the tensor flow, chainer, and pytorch implementation, but I could not 
find gluon version of it.
   It requires changing the structure of the layers. (sample code for pytorch  
https://github.com/godisboy/SN-GAN/blob/master/src/snlayers/snconv2d.py)
   
   Can you help me to design such block in gluon?
   This page 
http://gluon.mxnet.io/chapter03_deep-neural-networks/custom-layer.html
   shows how to build a new layer, but not modifying the existing one.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on issue #8651: How to combine softmax and regression

[GitBox]

szha commented on issue #8651: How to combine softmax  and regression
URL: 
https://github.com/apache/incubator-mxnet/issues/8651#issuecomment-373564865
 
 
   @apache/mxnet-committers: This issue has been inactive for the past 90 days. 
It has no label and needs triage.
   
   For general "how-to" questions, our [user forum](https://discuss.mxnet.io/) 
(and [Chinese version](https://discuss.gluon.ai/)) is a good place to get help.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhanghang1989 commented on issue #9688: [MXNET-108] Adding BilinearResize2D and AdaptiveAvgPool2d operators

[GitBox]

zhanghang1989 commented on issue #9688: [MXNET-108] Adding BilinearResize2D and 
AdaptiveAvgPool2d operators
URL: https://github.com/apache/incubator-mxnet/pull/9688#issuecomment-373563075
 
 
   Hi guys@piiswrong @szha, could you give some comments about this PR? Thx!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin commented on issue #9396: inference speed drop after updating mxnet from 0.10.0 to 1.0.0

[GitBox]

eric-haibin-lin commented on issue #9396: inference speed drop after updating 
mxnet from 0.10.0 to 1.0.0
URL: 
https://github.com/apache/incubator-mxnet/issues/9396#issuecomment-373557291
 
 
   Let me try v100


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhanghang1989 commented on issue #9688: Adding BilinearResize2D and AdaptiveAvgPool2d operators

[GitBox]

zhanghang1989 commented on issue #9688: Adding BilinearResize2D and 
AdaptiveAvgPool2d operators
URL: https://github.com/apache/incubator-mxnet/pull/9688#issuecomment-371360712
 
 
   @cjolivier01 I guess opencv won't support batch dimension. We have to 
support 4D input (Batch x Channel x Height x Width). We also need backward :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #7928: Cannot reproduce the results of faster-rcnn

[GitBox]

rahul003 commented on issue #7928: Cannot reproduce the results of faster-rcnn
URL: 
https://github.com/apache/incubator-mxnet/issues/7928#issuecomment-373526022
 
 
   @cjolivier01  Could you please label this with question, performance


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #8257: How can I view the network in rnn example?

[GitBox]

rahul003 commented on issue #8257: How can I view the network in rnn example?
URL: 
https://github.com/apache/incubator-mxnet/issues/8257#issuecomment-373518062
 
 
   @eric-haibin-lin Please label this with question, visualization, rnn


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #8060: Evaluation measure remains after several iterations, epochs...What could be the reason?

[GitBox]

rahul003 commented on issue #8060: Evaluation measure remains after several 
iterations, epochs...What could be the reason?
URL: 
https://github.com/apache/incubator-mxnet/issues/8060#issuecomment-373525221
 
 
   @cjolivier01 Could you help label this with Question and close it.
   
   @chowkamlee81 If you are still facing this, please ask on our discussion 
forum https://discuss.mxnet.io/


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #8257: How can I view the network in rnn example?

[GitBox]

rahul003 commented on issue #8257: How can I view the network in rnn example?
URL: 
https://github.com/apache/incubator-mxnet/issues/8257#issuecomment-373518062
 
 
   @eric-haibin-lin Please label this with question, visualization


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #8257: How can I view the network in rnn example?

[GitBox]

rahul003 commented on issue #8257: How can I view the network in rnn example?
URL: 
https://github.com/apache/incubator-mxnet/issues/8257#issuecomment-373518062
 
 
   @eric-haibin-lin Please label this with doc, question, visualization


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #8257: How can I view the network in rnn example?

[GitBox]

rahul003 commented on issue #8257: How can I view the network in rnn example?
URL: 
https://github.com/apache/incubator-mxnet/issues/8257#issuecomment-373518062
 
 
   @eric-haibin-lin Please label this withquestion, visualization


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rahul003 commented on issue #8257: How can I view the network in rnn example?

[GitBox]

rahul003 commented on issue #8257: How can I view the network in rnn example?
URL: 
https://github.com/apache/incubator-mxnet/issues/8257#issuecomment-373516414
 
 
   Hi, did eric-haibin-lin's suggestion solve your problem?
   If not, could you use https://discuss.mxnet.io/ to ask this again. We 
request you to use that forum for questions.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] hdjsjyl commented on issue #8910: Possible Bug in Mxnet when training a Network with no labels

[GitBox]

hdjsjyl commented on issue #8910: Possible Bug in Mxnet when training a Network 
with no labels
URL: 
https://github.com/apache/incubator-mxnet/issues/8910#issuecomment-373513845
 
 
   @adrianloy Hi, I meet the same problem as you. And I have made my own 
DataIterator like you. But I do not know how to change the update_metric part. 
Any advice will be appreciated. Thank you very much.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham commented on issue #59: Fixed Learn More to the correct landing page

[GitBox]

aaronmarkham commented on issue #59: Fixed Learn More to the correct landing 
page
URL: 
https://github.com/apache/incubator-mxnet-site/pull/59#issuecomment-373508165
 
 
   Gtg! Ship it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] pakmarkthub commented on issue #4858: Race Conditions on Pascal GPUs?

[GitBox]

pakmarkthub commented on issue #4858: Race Conditions on Pascal GPUs?
URL: 
https://github.com/apache/incubator-mxnet/issues/4858#issuecomment-373508012
 
 
   Have you already found a workaround or a solution? 
   
   I encountered similar problem when running train_cifar10.py on four P100 
GPUs. I found that compiled with DEBUG=1 also created similar problem, not only 
limit to using NaiveEngine. I have created my ticket here #10123. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] DickJC123 commented on issue #9396: inference speed drop after updating mxnet from 0.10.0 to 1.0.0

[GitBox]

DickJC123 commented on issue #9396: inference speed drop after updating mxnet 
from 0.10.0 to 1.0.0
URL: 
https://github.com/apache/incubator-mxnet/issues/9396#issuecomment-373499028
 
 
   I was not.  Is it easy for you to run your perf test on newer architectures 
than Kepler?
   
   > On March 15, 2018 at 11:38 AM Haibin Lin  wrote:
   > 
   > 
   > @nicklhy https://github.com/nicklhy I bisected the changes between 
0.10.0 and 1.0 and found the following on a p2.xlarge(K80) instance. The 
commits are patched with the fix in PR 9055
   > 
   > git checkout xxx; 
   > git submodule update --recursive; 
   > git cherry-pick 9cc8ea3be23fb7adf4630e4cf065a2473094fbc8 -X theirs
   > make
   > 
   > and below is the result
   > 
   > ff21e1f Changed FullyConnected to use new linalg gemm, plus TensorCore 
if fp16 I/O. (#7505)
   > 
   > speed test for batch size: 1
   > avg forward speed: 24.484983 samples/s
   > avg forward time: mean = 0.040839 s, std = 0.95 s
   > 
   > 56eae58 Fixed Makefile so a null CUDA_ARCH is treated like an unset 
one. (#7515) - Fast
   > 
   > 
   > speed test for batch size: 1
   > avg forward speed: 25.461191 samples/s
   > avg forward time: mean = 0.039270 s, std = 0.95 s
   > 
   > 
   > Looks like the commit ff21e1f 
https://github.com/apache/incubator-mxnet/commit/ff21e1fd41f118dbbaf55d8f02a9669842ef565f
 caused the 4% slowdown during inference.
   > 
   > @DickJC123 https://github.com/dickjc123 were you aware of this?
   > 
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub 
https://github.com/apache/incubator-mxnet/issues/9396#issuecomment-373480684 , 
or mute the thread 
https://github.com/notifications/unsubscribe-auth/ABn95cDNeZmmZxQZ_Jskw0yivlr9C3eNks5terUOgaJpZM4RcJ9U
 .
   > 
   >  
   > 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhanghang1989 commented on issue #9688: Adding BilinearResize2D and AdaptiveAvgPool2d operators

[GitBox]

zhanghang1989 commented on issue #9688: Adding BilinearResize2D and 
AdaptiveAvgPool2d operators
URL: https://github.com/apache/incubator-mxnet/pull/9688#issuecomment-373497114
 
 
   Hi @marcoabreu , I have added unit tests using python. Let me know if you 
have other comments.
   Thx


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] thinksanky commented on issue #10081: [MXNET-82] [WIP] Sparse op tutorial for developers

[GitBox]

thinksanky commented on issue #10081: [MXNET-82] [WIP] Sparse op tutorial for 
developers
URL: https://github.com/apache/incubator-mxnet/pull/10081#issuecomment-373496815
 
 
   All how_to are moved to faq and we should not use how_to anymore. Is there a 
reason for updating how_to ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham commented on issue #10081: [MXNET-82] [WIP] Sparse op tutorial for developers

[GitBox]

aaronmarkham commented on issue #10081: [MXNET-82] [WIP] Sparse op tutorial for 
developers
URL: https://github.com/apache/incubator-mxnet/pull/10081#issuecomment-373494205
 
 
   Prerequisite add op in backend doc updates:
   
   * References a move that seems to have happened, but now the files are in 
both places... 
   
https://github.com/apache/incubator-mxnet/blame/master/docs/faq/add_op_in_backend.md#L48-L51
   
   * Link to specific instructions on adding descriptions
   
https://github.com/apache/incubator-mxnet/blame/master/docs/faq/add_op_in_backend.md#L95
   
   * Reference to FCompute is pretty light, and I'm unclear on the comparison 
with FComputeX in this sparse tutorial.
   
   For my review on the tutorial for sparse ops, it looks like it was outdated 
before I finished it, so please take a look at all of my "outdated" comments!
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] Sparse op tutorial for developers

[GitBox]

aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] 
Sparse op tutorial for developers
URL: https://github.com/apache/incubator-mxnet/pull/10081#discussion_r174884953
 
 

 ##
 File path: docs/how_to/add_sparse_op_in_backend.md
 ##
 @@ -0,0 +1,429 @@
+# A Guide to Implementing Sparse Operators in MXNet Backend
+
+## Prerequisites
+- Basic knowledge of [how to implement a dense operator in MXNet 
backend](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html)
+- Basic knowledge of 
[CSRNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/csr.html) and 
[RowSparseNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/row_sparse.html)
 in MXNet
+
+## Introduction
+In the [previous 
tutorial](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html),
+we went through the steps to implementing an operator using C++ in the MXNet 
backend.
+In this tutorial, we will cover how sparse operators are implemented
+in the backend. Specifically, we will practice adding CSRNDArray support to 
the forward function of the `quadratic` operator.
+
+## Implementation
+### A Sparse Operator Example
+
+Let's consider the quadratic function `f(x) = ax^2+bx+c` when x is a 
CSRNDArray. 
+Notice that if the input x is sparse and c is 0.0, the output is also sparse.
+If c is non-zero, the output is dense. In MXNet frontend, the operator works 
like this:
+
+```python
+>>> x = mx.nd.array([[0,1],[2,0]).tostype('csr')
+>>> x
+
+>>> y = mx.nd.sparse.quadratic(x, a=1, b=2, c=0)
+>>> y
+
+>>> z = mx.nd.quadratic(x, a=1, b=2, c=3)
+>>> z
+[[  3.   6.]
+ [ 11.   3.]]
+
+```
+
+The statement `z = mx.nd.quadratic(x, a=1, b=2, c=3)` generates a warning 
message which says
+the sparse input is converted to dense storage, and the dense operator is used 
to compute the dense output.
+This is the "storage fallback" mechanism in MXNet, where a dense operator is 
automatically used for
+inputs that a sparse operator doesn't have special kernels for.
+
+In this tutorial, we will implement the forward function of the sparse 
quadratic operator.
+The storage type of the output depends on the inputs:
+- quadratic('csr', a, b, 0.0) outputs 'csr'
+- otherwise, outputs 'default'
+
+To implement this, we first register the storage type inference property of 
the operator, from which the operator
+infers the output storage type based on operator arguments and inputs types. 
Then we implement the forward
+function for the case where c is 0.0 and x is a CSRNDArray.
+
+Next, we are going to
 
 Review comment:
   Describe what we're generally going to do with the next bullets. Should you 
summarize the rest of the doc and include the unit test section?
   "The next steps will go into detail on how to create a sparse operator in 
C++:"


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] Sparse op tutorial for developers

[GitBox]

aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] 
Sparse op tutorial for developers
URL: https://github.com/apache/incubator-mxnet/pull/10081#discussion_r174888482
 
 

 ##
 File path: docs/how_to/add_sparse_op_in_backend.md
 ##
 @@ -0,0 +1,429 @@
+# A Guide to Implementing Sparse Operators in MXNet Backend
+
+## Prerequisites
+- Basic knowledge of [how to implement a dense operator in MXNet 
backend](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html)
+- Basic knowledge of 
[CSRNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/csr.html) and 
[RowSparseNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/row_sparse.html)
 in MXNet
+
+## Introduction
+In the [previous 
tutorial](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html),
+we went through the steps to implementing an operator using C++ in the MXNet 
backend.
+In this tutorial, we will cover how sparse operators are implemented
+in the backend. Specifically, we will practice adding CSRNDArray support to 
the forward function of the `quadratic` operator.
+
+## Implementation
+### A Sparse Operator Example
+
+Let's consider the quadratic function `f(x) = ax^2+bx+c` when x is a 
CSRNDArray. 
+Notice that if the input x is sparse and c is 0.0, the output is also sparse.
+If c is non-zero, the output is dense. In MXNet frontend, the operator works 
like this:
+
+```python
+>>> x = mx.nd.array([[0,1],[2,0]).tostype('csr')
+>>> x
+
+>>> y = mx.nd.sparse.quadratic(x, a=1, b=2, c=0)
+>>> y
+
+>>> z = mx.nd.quadratic(x, a=1, b=2, c=3)
+>>> z
+[[  3.   6.]
+ [ 11.   3.]]
+
+```
+
+The statement `z = mx.nd.quadratic(x, a=1, b=2, c=3)` generates a warning 
message which says
+the sparse input is converted to dense storage, and the dense operator is used 
to compute the dense output.
+This is the "storage fallback" mechanism in MXNet, where a dense operator is 
automatically used for
+inputs that a sparse operator doesn't have special kernels for.
+
+In this tutorial, we will implement the forward function of the sparse 
quadratic operator.
+The storage type of the output depends on the inputs:
+- quadratic('csr', a, b, 0.0) outputs 'csr'
+- otherwise, outputs 'default'
+
+To implement this, we first register the storage type inference property of 
the operator, from which the operator
+infers the output storage type based on operator arguments and inputs types. 
Then we implement the forward
+function for the case where c is 0.0 and x is a CSRNDArray.
+
+Next, we are going to
+
+- Understand the FComputeEx and relevant NDArray interfaces in backend.
+- Define storage type inference functions in quadratic_op-inl.h.
+- Define the forward function in quadratic_op-inl.h.
+- Register the sparse operator using nnvm in quadratic_op.cc and 
quadratic_op.cu for CPU and GPU computing, respectively.
+
+Now let's walk through the process step by step.
+
+### The FComputeEx and Relevant NDArray Interfaces in Backend
+
+Before we dive into the details of relevant interfaces, here are two 
differences between
+dense and sparse operators:
+- Dense operators only handle dense inputs and outputs. Sparse operators 
support various combinations of
+storage types.
+- Memories of inputs and outputs are pre-allocated based their shapes for 
dense operators. However, with sparse representations, memories for sparse 
inputs and outputs depend on the number of non-zero elements they have,
+which is only known at runtime.
+
+With these differences in mind, let's review the `FCompute` interface 
introduced in the previous operator tutorial:
 
 Review comment:
   add link to previous tutorial and maybe reference it as "dense op"


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] Sparse op tutorial for developers

[GitBox]

aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] 
Sparse op tutorial for developers
URL: https://github.com/apache/incubator-mxnet/pull/10081#discussion_r174886511
 
 

 ##
 File path: docs/how_to/add_sparse_op_in_backend.md
 ##
 @@ -0,0 +1,429 @@
+# A Guide to Implementing Sparse Operators in MXNet Backend
+
+## Prerequisites
+- Basic knowledge of [how to implement a dense operator in MXNet 
backend](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html)
+- Basic knowledge of 
[CSRNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/csr.html) and 
[RowSparseNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/row_sparse.html)
 in MXNet
+
+## Introduction
+In the [previous 
tutorial](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html),
+we went through the steps to implementing an operator using C++ in the MXNet 
backend.
+In this tutorial, we will cover how sparse operators are implemented
+in the backend. Specifically, we will practice adding CSRNDArray support to 
the forward function of the `quadratic` operator.
+
+## Implementation
+### A Sparse Operator Example
+
+Let's consider the quadratic function `f(x) = ax^2+bx+c` when x is a 
CSRNDArray. 
+Notice that if the input x is sparse and c is 0.0, the output is also sparse.
+If c is non-zero, the output is dense. In MXNet frontend, the operator works 
like this:
+
+```python
+>>> x = mx.nd.array([[0,1],[2,0]).tostype('csr')
+>>> x
+
+>>> y = mx.nd.sparse.quadratic(x, a=1, b=2, c=0)
+>>> y
+
+>>> z = mx.nd.quadratic(x, a=1, b=2, c=3)
+>>> z
+[[  3.   6.]
+ [ 11.   3.]]
+
+```
+
+The statement `z = mx.nd.quadratic(x, a=1, b=2, c=3)` generates a warning 
message which says
+the sparse input is converted to dense storage, and the dense operator is used 
to compute the dense output.
+This is the "storage fallback" mechanism in MXNet, where a dense operator is 
automatically used for
 
 Review comment:
   Reword? De-complicate.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] Sparse op tutorial for developers

[GitBox]

aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] 
Sparse op tutorial for developers
URL: https://github.com/apache/incubator-mxnet/pull/10081#discussion_r174885696
 
 

 ##
 File path: docs/how_to/add_sparse_op_in_backend.md
 ##
 @@ -0,0 +1,429 @@
+# A Guide to Implementing Sparse Operators in MXNet Backend
 
 Review comment:
   in C++ ?
   Shouldn't you also add this to the tutorials page under the C++ section?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] Sparse op tutorial for developers

[GitBox]

aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] 
Sparse op tutorial for developers
URL: https://github.com/apache/incubator-mxnet/pull/10081#discussion_r174887547
 
 

 ##
 File path: docs/how_to/add_sparse_op_in_backend.md
 ##
 @@ -0,0 +1,429 @@
+# A Guide to Implementing Sparse Operators in MXNet Backend
+
+## Prerequisites
+- Basic knowledge of [how to implement a dense operator in MXNet 
backend](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html)
 
 Review comment:
   This moved:
   https://mxnet.incubator.apache.org/faq/add_op_in_backend.html
   And isn't really titled with "dense", but maybe it should be. Maybe rephrase 
to say that previously, backend ops were dense.
   Also, I think you might want to update that page cross-referencing this one 
and introduce that you now have this dense v sparse option for ops. And also 
retitle it?
   
   As a prerequisite I'm reading that as well and noted that it needs some 
updates too. I'll circle back on those as a comment to this PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] Sparse op tutorial for developers

[GitBox]

aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] 
Sparse op tutorial for developers
URL: https://github.com/apache/incubator-mxnet/pull/10081#discussion_r174902335
 
 

 ##
 File path: docs/how_to/add_sparse_op_in_backend.md
 ##
 @@ -0,0 +1,429 @@
+# A Guide to Implementing Sparse Operators in MXNet Backend
+
+## Prerequisites
+- Basic knowledge of [how to implement a dense operator in MXNet 
backend](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html)
+- Basic knowledge of 
[CSRNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/csr.html) and 
[RowSparseNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/row_sparse.html)
 in MXNet
+
+## Introduction
+In the [previous 
tutorial](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html),
+we went through the steps to implementing an operator using C++ in the MXNet 
backend.
+In this tutorial, we will cover how sparse operators are implemented
+in the backend. Specifically, we will practice adding CSRNDArray support to 
the forward function of the `quadratic` operator.
+
+## Implementation
+### A Sparse Operator Example
+
+Let's consider the quadratic function `f(x) = ax^2+bx+c` when x is a 
CSRNDArray. 
+Notice that if the input x is sparse and c is 0.0, the output is also sparse.
+If c is non-zero, the output is dense. In MXNet frontend, the operator works 
like this:
+
+```python
+>>> x = mx.nd.array([[0,1],[2,0]).tostype('csr')
+>>> x
+
+>>> y = mx.nd.sparse.quadratic(x, a=1, b=2, c=0)
+>>> y
+
+>>> z = mx.nd.quadratic(x, a=1, b=2, c=3)
+>>> z
+[[  3.   6.]
+ [ 11.   3.]]
+
+```
+
+The statement `z = mx.nd.quadratic(x, a=1, b=2, c=3)` generates a warning 
message which says
+the sparse input is converted to dense storage, and the dense operator is used 
to compute the dense output.
+This is the "storage fallback" mechanism in MXNet, where a dense operator is 
automatically used for
+inputs that a sparse operator doesn't have special kernels for.
+
+In this tutorial, we will implement the forward function of the sparse 
quadratic operator.
+The storage type of the output depends on the inputs:
+- quadratic('csr', a, b, 0.0) outputs 'csr'
+- otherwise, outputs 'default'
+
+To implement this, we first register the storage type inference property of 
the operator, from which the operator
+infers the output storage type based on operator arguments and inputs types. 
Then we implement the forward
+function for the case where c is 0.0 and x is a CSRNDArray.
+
+Next, we are going to
+
+- Understand the FComputeEx and relevant NDArray interfaces in backend.
+- Define storage type inference functions in quadratic_op-inl.h.
+- Define the forward function in quadratic_op-inl.h.
+- Register the sparse operator using nnvm in quadratic_op.cc and 
quadratic_op.cu for CPU and GPU computing, respectively.
+
+Now let's walk through the process step by step.
+
+### The FComputeEx and Relevant NDArray Interfaces in Backend
+
+Before we dive into the details of relevant interfaces, here are two 
differences between
+dense and sparse operators:
+- Dense operators only handle dense inputs and outputs. Sparse operators 
support various combinations of
+storage types.
+- Memories of inputs and outputs are pre-allocated based their shapes for 
dense operators. However, with sparse representations, memories for sparse 
inputs and outputs depend on the number of non-zero elements they have,
+which is only known at runtime.
+
+With these differences in mind, let's review the `FCompute` interface 
introduced in the previous operator tutorial:
+```cpp
+void (const nnvm::NodeAttrs& attrs,
+  const OpContext& ctx,
+  const std::vector& inputs,
+  const std::vector& req,
+  const std::vector& outputs);
+```
+Notice the `FCompute` interface doesn't include data structures that could be 
used to query storage
+types of inputs, nor manipulate auxiliary arrays like `indices` and `indptr`. 
+Therefore, instead of the `FCompute` interface, sparse operators are 
registered with the following `FComputeEx` interface:
+```cpp
+void (const nnvm::NodeAttrs& attrs,
+  const OpContext& ctx,
+  const std::vector& inputs,
+  const std::vector& req,
+  const std::vector& outputs);
+```
+where the vectors of TBlobs are replaced with vectors of NDArrays. Now, let's 
go through a few important methods in the NDArray class.
+
+In the python frontend, there are three types of NDArrays, namely 
`mx.nd.NDArray`, `mx.nd.sparse.RowSparseNDArray` and `mx.nd.sparse.CSRNDArray`. 
In the C++ backend, however, all of them are represented by the 
`mxnet::NDArray` class.
+The `storage_type()` method indicates the storage type of the NDArray:
+```cpp
+enum NDArrayStorageType {
+  kUndefinedStorage = -1,  // undefined storage
+  kDefaultStorage, // dense
+  kRowSparseStorage,   // row sparse
+  kCSRStorage, // csr
+};
+
+// return the type of storage 

[GitHub] aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] Sparse op tutorial for developers

[GitBox]

aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] 
Sparse op tutorial for developers
URL: https://github.com/apache/incubator-mxnet/pull/10081#discussion_r174902292
 
 

 ##
 File path: docs/how_to/add_sparse_op_in_backend.md
 ##
 @@ -0,0 +1,429 @@
+# A Guide to Implementing Sparse Operators in MXNet Backend
+
+## Prerequisites
+- Basic knowledge of [how to implement a dense operator in MXNet 
backend](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html)
+- Basic knowledge of 
[CSRNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/csr.html) and 
[RowSparseNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/row_sparse.html)
 in MXNet
+
+## Introduction
+In the [previous 
tutorial](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html),
+we went through the steps to implementing an operator using C++ in the MXNet 
backend.
+In this tutorial, we will cover how sparse operators are implemented
+in the backend. Specifically, we will practice adding CSRNDArray support to 
the forward function of the `quadratic` operator.
+
+## Implementation
+### A Sparse Operator Example
+
+Let's consider the quadratic function `f(x) = ax^2+bx+c` when x is a 
CSRNDArray. 
+Notice that if the input x is sparse and c is 0.0, the output is also sparse.
+If c is non-zero, the output is dense. In MXNet frontend, the operator works 
like this:
+
+```python
+>>> x = mx.nd.array([[0,1],[2,0]).tostype('csr')
+>>> x
+
+>>> y = mx.nd.sparse.quadratic(x, a=1, b=2, c=0)
+>>> y
+
+>>> z = mx.nd.quadratic(x, a=1, b=2, c=3)
+>>> z
+[[  3.   6.]
+ [ 11.   3.]]
+
+```
+
+The statement `z = mx.nd.quadratic(x, a=1, b=2, c=3)` generates a warning 
message which says
+the sparse input is converted to dense storage, and the dense operator is used 
to compute the dense output.
+This is the "storage fallback" mechanism in MXNet, where a dense operator is 
automatically used for
+inputs that a sparse operator doesn't have special kernels for.
+
+In this tutorial, we will implement the forward function of the sparse 
quadratic operator.
+The storage type of the output depends on the inputs:
+- quadratic('csr', a, b, 0.0) outputs 'csr'
+- otherwise, outputs 'default'
+
+To implement this, we first register the storage type inference property of 
the operator, from which the operator
+infers the output storage type based on operator arguments and inputs types. 
Then we implement the forward
+function for the case where c is 0.0 and x is a CSRNDArray.
+
+Next, we are going to
+
+- Understand the FComputeEx and relevant NDArray interfaces in backend.
+- Define storage type inference functions in quadratic_op-inl.h.
+- Define the forward function in quadratic_op-inl.h.
+- Register the sparse operator using nnvm in quadratic_op.cc and 
quadratic_op.cu for CPU and GPU computing, respectively.
+
+Now let's walk through the process step by step.
+
+### The FComputeEx and Relevant NDArray Interfaces in Backend
+
+Before we dive into the details of relevant interfaces, here are two 
differences between
+dense and sparse operators:
+- Dense operators only handle dense inputs and outputs. Sparse operators 
support various combinations of
+storage types.
+- Memories of inputs and outputs are pre-allocated based their shapes for 
dense operators. However, with sparse representations, memories for sparse 
inputs and outputs depend on the number of non-zero elements they have,
+which is only known at runtime.
+
+With these differences in mind, let's review the `FCompute` interface 
introduced in the previous operator tutorial:
+```cpp
+void (const nnvm::NodeAttrs& attrs,
+  const OpContext& ctx,
+  const std::vector& inputs,
+  const std::vector& req,
+  const std::vector& outputs);
+```
+Notice the `FCompute` interface doesn't include data structures that could be 
used to query storage
+types of inputs, nor manipulate auxiliary arrays like `indices` and `indptr`. 
+Therefore, instead of the `FCompute` interface, sparse operators are 
registered with the following `FComputeEx` interface:
+```cpp
+void (const nnvm::NodeAttrs& attrs,
+  const OpContext& ctx,
+  const std::vector& inputs,
+  const std::vector& req,
+  const std::vector& outputs);
+```
+where the vectors of TBlobs are replaced with vectors of NDArrays. Now, let's 
go through a few important methods in the NDArray class.
+
+In the python frontend, there are three types of NDArrays, namely 
`mx.nd.NDArray`, `mx.nd.sparse.RowSparseNDArray` and `mx.nd.sparse.CSRNDArray`. 
In the C++ backend, however, all of them are represented by the 
`mxnet::NDArray` class.
+The `storage_type()` method indicates the storage type of the NDArray:
+```cpp
+enum NDArrayStorageType {
+  kUndefinedStorage = -1,  // undefined storage
+  kDefaultStorage, // dense
+  kRowSparseStorage,   // row sparse
+  kCSRStorage, // csr
+};
+
+// return the type of storage 

[GitHub] aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] Sparse op tutorial for developers

[GitBox]

aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] 
Sparse op tutorial for developers
URL: https://github.com/apache/incubator-mxnet/pull/10081#discussion_r174901434
 
 

 ##
 File path: docs/how_to/add_sparse_op_in_backend.md
 ##
 @@ -0,0 +1,429 @@
+# A Guide to Implementing Sparse Operators in MXNet Backend
+
+## Prerequisites
+- Basic knowledge of [how to implement a dense operator in MXNet 
backend](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html)
+- Basic knowledge of 
[CSRNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/csr.html) and 
[RowSparseNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/row_sparse.html)
 in MXNet
+
+## Introduction
+In the [previous 
tutorial](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html),
+we went through the steps to implementing an operator using C++ in the MXNet 
backend.
+In this tutorial, we will cover how sparse operators are implemented
+in the backend. Specifically, we will practice adding CSRNDArray support to 
the forward function of the `quadratic` operator.
+
+## Implementation
+### A Sparse Operator Example
+
+Let's consider the quadratic function `f(x) = ax^2+bx+c` when x is a 
CSRNDArray. 
+Notice that if the input x is sparse and c is 0.0, the output is also sparse.
+If c is non-zero, the output is dense. In MXNet frontend, the operator works 
like this:
+
+```python
+>>> x = mx.nd.array([[0,1],[2,0]).tostype('csr')
+>>> x
+
+>>> y = mx.nd.sparse.quadratic(x, a=1, b=2, c=0)
+>>> y
+
+>>> z = mx.nd.quadratic(x, a=1, b=2, c=3)
+>>> z
+[[  3.   6.]
+ [ 11.   3.]]
+
+```
+
+The statement `z = mx.nd.quadratic(x, a=1, b=2, c=3)` generates a warning 
message which says
+the sparse input is converted to dense storage, and the dense operator is used 
to compute the dense output.
+This is the "storage fallback" mechanism in MXNet, where a dense operator is 
automatically used for
+inputs that a sparse operator doesn't have special kernels for.
+
+In this tutorial, we will implement the forward function of the sparse 
quadratic operator.
+The storage type of the output depends on the inputs:
+- quadratic('csr', a, b, 0.0) outputs 'csr'
+- otherwise, outputs 'default'
+
+To implement this, we first register the storage type inference property of 
the operator, from which the operator
+infers the output storage type based on operator arguments and inputs types. 
Then we implement the forward
+function for the case where c is 0.0 and x is a CSRNDArray.
+
+Next, we are going to
+
+- Understand the FComputeEx and relevant NDArray interfaces in backend.
+- Define storage type inference functions in quadratic_op-inl.h.
+- Define the forward function in quadratic_op-inl.h.
+- Register the sparse operator using nnvm in quadratic_op.cc and 
quadratic_op.cu for CPU and GPU computing, respectively.
+
+Now let's walk through the process step by step.
+
+### The FComputeEx and Relevant NDArray Interfaces in Backend
+
+Before we dive into the details of relevant interfaces, here are two 
differences between
+dense and sparse operators:
+- Dense operators only handle dense inputs and outputs. Sparse operators 
support various combinations of
+storage types.
+- Memories of inputs and outputs are pre-allocated based their shapes for 
dense operators. However, with sparse representations, memories for sparse 
inputs and outputs depend on the number of non-zero elements they have,
+which is only known at runtime.
+
+With these differences in mind, let's review the `FCompute` interface 
introduced in the previous operator tutorial:
+```cpp
+void (const nnvm::NodeAttrs& attrs,
+  const OpContext& ctx,
+  const std::vector& inputs,
+  const std::vector& req,
+  const std::vector& outputs);
+```
+Notice the `FCompute` interface doesn't include data structures that could be 
used to query storage
+types of inputs, nor manipulate auxiliary arrays like `indices` and `indptr`. 
+Therefore, instead of the `FCompute` interface, sparse operators are 
registered with the following `FComputeEx` interface:
+```cpp
+void (const nnvm::NodeAttrs& attrs,
+  const OpContext& ctx,
+  const std::vector& inputs,
+  const std::vector& req,
+  const std::vector& outputs);
+```
+where the vectors of TBlobs are replaced with vectors of NDArrays. Now, let's 
go through a few important methods in the NDArray class.
+
+In the python frontend, there are three types of NDArrays, namely 
`mx.nd.NDArray`, `mx.nd.sparse.RowSparseNDArray` and `mx.nd.sparse.CSRNDArray`. 
In the C++ backend, however, all of them are represented by the 
`mxnet::NDArray` class.
+The `storage_type()` method indicates the storage type of the NDArray:
+```cpp
+enum NDArrayStorageType {
+  kUndefinedStorage = -1,  // undefined storage
+  kDefaultStorage, // dense
+  kRowSparseStorage,   // row sparse
+  kCSRStorage, // csr
+};
+
+// return the type of storage 

[GitHub] aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] Sparse op tutorial for developers

[GitBox]

aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] 
Sparse op tutorial for developers
URL: https://github.com/apache/incubator-mxnet/pull/10081#discussion_r174903015
 
 

 ##
 File path: docs/how_to/add_sparse_op_in_backend.md
 ##
 @@ -0,0 +1,429 @@
+# A Guide to Implementing Sparse Operators in MXNet Backend
+
+## Prerequisites
+- Basic knowledge of [how to implement a dense operator in MXNet 
backend](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html)
+- Basic knowledge of 
[CSRNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/csr.html) and 
[RowSparseNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/row_sparse.html)
 in MXNet
+
+## Introduction
+In the [previous 
tutorial](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html),
+we went through the steps to implementing an operator using C++ in the MXNet 
backend.
+In this tutorial, we will cover how sparse operators are implemented
+in the backend. Specifically, we will practice adding CSRNDArray support to 
the forward function of the `quadratic` operator.
+
+## Implementation
+### A Sparse Operator Example
+
+Let's consider the quadratic function `f(x) = ax^2+bx+c` when x is a 
CSRNDArray. 
+Notice that if the input x is sparse and c is 0.0, the output is also sparse.
+If c is non-zero, the output is dense. In MXNet frontend, the operator works 
like this:
+
+```python
+>>> x = mx.nd.array([[0,1],[2,0]).tostype('csr')
+>>> x
+
+>>> y = mx.nd.sparse.quadratic(x, a=1, b=2, c=0)
+>>> y
+
+>>> z = mx.nd.quadratic(x, a=1, b=2, c=3)
+>>> z
+[[  3.   6.]
+ [ 11.   3.]]
+
+```
+
+The statement `z = mx.nd.quadratic(x, a=1, b=2, c=3)` generates a warning 
message which says
+the sparse input is converted to dense storage, and the dense operator is used 
to compute the dense output.
+This is the "storage fallback" mechanism in MXNet, where a dense operator is 
automatically used for
+inputs that a sparse operator doesn't have special kernels for.
+
+In this tutorial, we will implement the forward function of the sparse 
quadratic operator.
+The storage type of the output depends on the inputs:
+- quadratic('csr', a, b, 0.0) outputs 'csr'
+- otherwise, outputs 'default'
+
+To implement this, we first register the storage type inference property of 
the operator, from which the operator
+infers the output storage type based on operator arguments and inputs types. 
Then we implement the forward
+function for the case where c is 0.0 and x is a CSRNDArray.
+
+Next, we are going to
+
+- Understand the FComputeEx and relevant NDArray interfaces in backend.
+- Define storage type inference functions in quadratic_op-inl.h.
+- Define the forward function in quadratic_op-inl.h.
+- Register the sparse operator using nnvm in quadratic_op.cc and 
quadratic_op.cu for CPU and GPU computing, respectively.
+
+Now let's walk through the process step by step.
+
+### The FComputeEx and Relevant NDArray Interfaces in Backend
+
+Before we dive into the details of relevant interfaces, here are two 
differences between
+dense and sparse operators:
+- Dense operators only handle dense inputs and outputs. Sparse operators 
support various combinations of
+storage types.
+- Memories of inputs and outputs are pre-allocated based their shapes for 
dense operators. However, with sparse representations, memories for sparse 
inputs and outputs depend on the number of non-zero elements they have,
+which is only known at runtime.
+
+With these differences in mind, let's review the `FCompute` interface 
introduced in the previous operator tutorial:
+```cpp
+void (const nnvm::NodeAttrs& attrs,
+  const OpContext& ctx,
+  const std::vector& inputs,
+  const std::vector& req,
+  const std::vector& outputs);
+```
+Notice the `FCompute` interface doesn't include data structures that could be 
used to query storage
+types of inputs, nor manipulate auxiliary arrays like `indices` and `indptr`. 
+Therefore, instead of the `FCompute` interface, sparse operators are 
registered with the following `FComputeEx` interface:
+```cpp
+void (const nnvm::NodeAttrs& attrs,
+  const OpContext& ctx,
+  const std::vector& inputs,
+  const std::vector& req,
+  const std::vector& outputs);
+```
+where the vectors of TBlobs are replaced with vectors of NDArrays. Now, let's 
go through a few important methods in the NDArray class.
+
+In the python frontend, there are three types of NDArrays, namely 
`mx.nd.NDArray`, `mx.nd.sparse.RowSparseNDArray` and `mx.nd.sparse.CSRNDArray`. 
In the C++ backend, however, all of them are represented by the 
`mxnet::NDArray` class.
+The `storage_type()` method indicates the storage type of the NDArray:
+```cpp
+enum NDArrayStorageType {
+  kUndefinedStorage = -1,  // undefined storage
+  kDefaultStorage, // dense
+  kRowSparseStorage,   // row sparse
+  kCSRStorage, // csr
+};
+
+// return the type of storage 

[GitHub] aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] Sparse op tutorial for developers

[GitBox]

aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] 
Sparse op tutorial for developers
URL: https://github.com/apache/incubator-mxnet/pull/10081#discussion_r174901142
 
 

 ##
 File path: docs/how_to/add_sparse_op_in_backend.md
 ##
 @@ -0,0 +1,429 @@
+# A Guide to Implementing Sparse Operators in MXNet Backend
+
+## Prerequisites
+- Basic knowledge of [how to implement a dense operator in MXNet 
backend](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html)
+- Basic knowledge of 
[CSRNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/csr.html) and 
[RowSparseNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/row_sparse.html)
 in MXNet
+
+## Introduction
+In the [previous 
tutorial](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html),
+we went through the steps to implementing an operator using C++ in the MXNet 
backend.
+In this tutorial, we will cover how sparse operators are implemented
+in the backend. Specifically, we will practice adding CSRNDArray support to 
the forward function of the `quadratic` operator.
+
+## Implementation
+### A Sparse Operator Example
+
+Let's consider the quadratic function `f(x) = ax^2+bx+c` when x is a 
CSRNDArray. 
+Notice that if the input x is sparse and c is 0.0, the output is also sparse.
+If c is non-zero, the output is dense. In MXNet frontend, the operator works 
like this:
+
+```python
+>>> x = mx.nd.array([[0,1],[2,0]).tostype('csr')
+>>> x
+
+>>> y = mx.nd.sparse.quadratic(x, a=1, b=2, c=0)
+>>> y
+
+>>> z = mx.nd.quadratic(x, a=1, b=2, c=3)
+>>> z
+[[  3.   6.]
+ [ 11.   3.]]
+
+```
+
+The statement `z = mx.nd.quadratic(x, a=1, b=2, c=3)` generates a warning 
message which says
+the sparse input is converted to dense storage, and the dense operator is used 
to compute the dense output.
+This is the "storage fallback" mechanism in MXNet, where a dense operator is 
automatically used for
+inputs that a sparse operator doesn't have special kernels for.
+
+In this tutorial, we will implement the forward function of the sparse 
quadratic operator.
+The storage type of the output depends on the inputs:
+- quadratic('csr', a, b, 0.0) outputs 'csr'
+- otherwise, outputs 'default'
+
+To implement this, we first register the storage type inference property of 
the operator, from which the operator
+infers the output storage type based on operator arguments and inputs types. 
Then we implement the forward
+function for the case where c is 0.0 and x is a CSRNDArray.
+
+Next, we are going to
+
+- Understand the FComputeEx and relevant NDArray interfaces in backend.
+- Define storage type inference functions in quadratic_op-inl.h.
+- Define the forward function in quadratic_op-inl.h.
+- Register the sparse operator using nnvm in quadratic_op.cc and 
quadratic_op.cu for CPU and GPU computing, respectively.
+
+Now let's walk through the process step by step.
+
+### The FComputeEx and Relevant NDArray Interfaces in Backend
+
+Before we dive into the details of relevant interfaces, here are two 
differences between
+dense and sparse operators:
+- Dense operators only handle dense inputs and outputs. Sparse operators 
support various combinations of
+storage types.
+- Memories of inputs and outputs are pre-allocated based their shapes for 
dense operators. However, with sparse representations, memories for sparse 
inputs and outputs depend on the number of non-zero elements they have,
+which is only known at runtime.
+
+With these differences in mind, let's review the `FCompute` interface 
introduced in the previous operator tutorial:
+```cpp
+void (const nnvm::NodeAttrs& attrs,
+  const OpContext& ctx,
+  const std::vector& inputs,
+  const std::vector& req,
+  const std::vector& outputs);
+```
+Notice the `FCompute` interface doesn't include data structures that could be 
used to query storage
+types of inputs, nor manipulate auxiliary arrays like `indices` and `indptr`. 
+Therefore, instead of the `FCompute` interface, sparse operators are 
registered with the following `FComputeEx` interface:
+```cpp
+void (const nnvm::NodeAttrs& attrs,
+  const OpContext& ctx,
+  const std::vector& inputs,
+  const std::vector& req,
+  const std::vector& outputs);
+```
+where the vectors of TBlobs are replaced with vectors of NDArrays. Now, let's 
go through a few important methods in the NDArray class.
 
 Review comment:
   Start a new sentence.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] Sparse op tutorial for developers

[GitBox]

aaronmarkham commented on a change in pull request #10081: [MXNET-82] [WIP] 
Sparse op tutorial for developers
URL: https://github.com/apache/incubator-mxnet/pull/10081#discussion_r174885951
 
 

 ##
 File path: docs/how_to/add_sparse_op_in_backend.md
 ##
 @@ -0,0 +1,429 @@
+# A Guide to Implementing Sparse Operators in MXNet Backend
+
+## Prerequisites
+- Basic knowledge of [how to implement a dense operator in MXNet 
backend](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html)
+- Basic knowledge of 
[CSRNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/csr.html) and 
[RowSparseNDArray](http://mxnet.incubator.apache.org/tutorials/sparse/row_sparse.html)
 in MXNet
+
+## Introduction
+In the [previous 
tutorial](https://mxnet.incubator.apache.org/versions/master/how_to/add_op_in_backend.html),
+we went through the steps to implementing an operator using C++ in the MXNet 
backend.
+In this tutorial, we will cover how sparse operators are implemented
+in the backend. Specifically, we will practice adding CSRNDArray support to 
the forward function of the `quadratic` operator.
+
+## Implementation
+### A Sparse Operator Example
+
+Let's consider the quadratic function `f(x) = ax^2+bx+c` when x is a 
CSRNDArray. 
+Notice that if the input x is sparse and c is 0.0, the output is also sparse.
+If c is non-zero, the output is dense. In MXNet frontend, the operator works 
like this:
+
+```python
+>>> x = mx.nd.array([[0,1],[2,0]).tostype('csr')
+>>> x
+
+>>> y = mx.nd.sparse.quadratic(x, a=1, b=2, c=0)
+>>> y
+
+>>> z = mx.nd.quadratic(x, a=1, b=2, c=3)
+>>> z
+[[  3.   6.]
+ [ 11.   3.]]
+
+```
+
+The statement `z = mx.nd.quadratic(x, a=1, b=2, c=3)` generates a warning 
message which says
 
 Review comment:
   Your output above didn't have a warning message...


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhreshold commented on issue #10096: Fix multi worker

[GitBox]

zhreshold commented on issue #10096: Fix multi worker
URL: https://github.com/apache/incubator-mxnet/pull/10096#issuecomment-373492995
 
 
   @Jerryzcn The latest commit should fix #10042 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] thinksanky commented on issue #59: Fixed Learn More to the correct landing page

[GitBox]

thinksanky commented on issue #59: Fixed Learn More to the correct landing page
URL: 
https://github.com/apache/incubator-mxnet-site/pull/59#issuecomment-373216639
 
 
   Fixed the issue.
   
   Test website - http://54.175.164.236/
   - Check "Learn More" page, should point to faq/why_mxnet.html
   - To test redirection  from stale cached index . Test - 
http://54.175.164.236/get_started/why_mxnet.html
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] thinksanky commented on issue #59: Fixed Learn More to the correct landing page

[GitBox]

thinksanky commented on issue #59: Fixed Learn More to the correct landing page
URL: 
https://github.com/apache/incubator-mxnet-site/pull/59#issuecomment-373216639
 
 
   Fixed the issue


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image classifier for scala-infer package

[GitBox]

nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image 
classifier for scala-infer package
URL: https://github.com/apache/incubator-mxnet/pull/10054#discussion_r174892529
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/ImageClassifier.scala
 ##
 @@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet._
+
+import scala.collection.mutable.ListBuffer
+
+// scalastyle:off
+import java.awt.image.BufferedImage
+// scalastyle:on
+import java.io.File
+import javax.imageio.ImageIO
+
+
+/**
+  * A class for classifier tasks
+  *
+  * @param modelPathPrefix  PathPrefix from where to load the symbol, 
parameters and synset.txt
+  * Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json
+  * file://model-dir/synset.txt
+  * @param inputDescriptors Descriptors defining the input node names, shape,
+  * layout and Type parameters
+  */
+class ImageClassifier(modelPathPrefix: String,
+  inputDescriptors: IndexedSeq[DataDesc])
+  extends Classifier(modelPathPrefix,
+  inputDescriptors) {
+
+  val classifier: Classifier = getClassifier(modelPathPrefix, inputDescriptors)
+
+  // Loading image from file
+  def loadImageFromFile(inputImagePath: String): BufferedImage = {
+  val img = ImageIO.read(new File(inputImagePath))
+  img
+  }
+
+  /**
+* Loading input batch of images
+* @param inputImageDirPath
+* @return List of buffered images
+*/
+  def loadInputBatch(inputImageDirPath: String): List[BufferedImage] = {
+val dir = new File(inputImageDirPath)
+val inputBatch = ListBuffer[BufferedImage]()
+for (imgFile: File <- dir.listFiles()){
+  val img = ImageIO.read(imgFile)
+  inputBatch += img
+}
+inputBatch.toList
+  }
+
+  /**
+* Reshape the input image to new shape
+*
+* @param img   image
+* @param newWidth  rescale to new width
+* @param newHeight rescale to new height
+* @return Rescaled BufferedImage
+*/
+  def getScaledImage(img: BufferedImage, newWidth: Int, newHeight: Int): 
BufferedImage = {
+val resizedImage = new BufferedImage(newWidth, newHeight, 
BufferedImage.TYPE_INT_RGB)
+val g = resizedImage.createGraphics()
+g.drawImage(img, 0, 0, newWidth, newHeight, null)
+g.dispose()
+
+resizedImage
+  }
+
+  /**
+* Read image file from provided path
+*
+* @param resizedImage BufferedImage to get pixels from
+* @return NDArray pixels array
+*/
+  def getPixelsFromImage(resizedImage: BufferedImage): NDArray = {
+val w = resizedImage.getWidth
+val h = resizedImage.getHeight
+
+val pixels = new ListBuffer[Float]()
+
+for (x <- 0 until h) {
+  for (y <- 0 until w) {
+val color = resizedImage.getRGB(y, x)
+val red = (color & 0xff) >> 16
+val green = (color & 0xff00) >> 8
+val blue = color & 0xff
+pixels += red
+pixels += green
+pixels += blue
+  }
+}
+
+val reshaped_pixels = NDArray.array(pixels.toArray, shape = Shape(224, 
224, 3))
+
+val swapped_axis = NDArray.swapaxes(reshaped_pixels, 0, 2)
+val pixelsNdarray = NDArray.swapaxes(swapped_axis, 1, 2)
+
+pixelsNdarray
+  }
+
+  /**
+* To classify the image according to the provided model
+*
+* @param inputImage PathPrefix of the input image
+* @param topK Get top k elements with maximum probability
+* @return List of list of tuples of (class, probability)
+*/
+  def classifyImage(inputImage: BufferedImage,
+topK: Option[Int] = None): IndexedSeq[List[(String, 
Float)]] = {
+
+val width = inputDescriptors(0).shape(2)
 
 Review comment:
   This should be based on the layout parameter of the input descriptor


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact 

[GitHub] haojin2 commented on issue #2302: General support for Float16 and other DTypes

[GitBox]

haojin2 commented on issue #2302: General support for Float16 and other DTypes
URL: 
https://github.com/apache/incubator-mxnet/issues/2302#issuecomment-373488161
 
 
   Seems like crop, slice_channel, softmax_activation are all deprecated 
operators, I think maybe we can skip the support for FP16 for those operators?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on issue #10054: [MXNET-53][WIP] Image classifier for scala-infer package

[GitBox]

nswamy commented on issue #10054: [MXNET-53][WIP] Image classifier for 
scala-infer package
URL: https://github.com/apache/incubator-mxnet/pull/10054#issuecomment-373488118
 
 
   @marcoabreu currently we don't have any hooks to run Scala integration 
tests. We will take up adding examples to CI later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image classifier for scala-infer package

[GitBox]

nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image 
classifier for scala-infer package
URL: https://github.com/apache/incubator-mxnet/pull/10054#discussion_r174895078
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/ImageClassifier.scala
 ##
 @@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet._
+
+import scala.collection.mutable.ListBuffer
+
+// scalastyle:off
+import java.awt.image.BufferedImage
+// scalastyle:on
+import java.io.File
+import javax.imageio.ImageIO
+
+
+/**
+  * A class for classifier tasks
+  *
+  * @param modelPathPrefix  PathPrefix from where to load the symbol, 
parameters and synset.txt
+  * Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json
+  * file://model-dir/synset.txt
+  * @param inputDescriptors Descriptors defining the input node names, shape,
+  * layout and Type parameters
+  */
+class ImageClassifier(modelPathPrefix: String,
+  inputDescriptors: IndexedSeq[DataDesc])
+  extends Classifier(modelPathPrefix,
+  inputDescriptors) {
+
+  val classifier: Classifier = getClassifier(modelPathPrefix, inputDescriptors)
+
+  // Loading image from file
+  def loadImageFromFile(inputImagePath: String): BufferedImage = {
+  val img = ImageIO.read(new File(inputImagePath))
+  img
+  }
+
+  /**
+* Loading input batch of images
+* @param inputImageDirPath
+* @return List of buffered images
+*/
+  def loadInputBatch(inputImageDirPath: String): List[BufferedImage] = {
+val dir = new File(inputImageDirPath)
+val inputBatch = ListBuffer[BufferedImage]()
+for (imgFile: File <- dir.listFiles()){
+  val img = ImageIO.read(imgFile)
+  inputBatch += img
+}
+inputBatch.toList
+  }
+
+  /**
+* Reshape the input image to new shape
+*
+* @param img   image
+* @param newWidth  rescale to new width
+* @param newHeight rescale to new height
+* @return Rescaled BufferedImage
+*/
+  def getScaledImage(img: BufferedImage, newWidth: Int, newHeight: Int): 
BufferedImage = {
+val resizedImage = new BufferedImage(newWidth, newHeight, 
BufferedImage.TYPE_INT_RGB)
+val g = resizedImage.createGraphics()
+g.drawImage(img, 0, 0, newWidth, newHeight, null)
+g.dispose()
+
+resizedImage
+  }
+
+  /**
+* Read image file from provided path
+*
+* @param resizedImage BufferedImage to get pixels from
+* @return NDArray pixels array
+*/
+  def getPixelsFromImage(resizedImage: BufferedImage): NDArray = {
+val w = resizedImage.getWidth
+val h = resizedImage.getHeight
+
+val pixels = new ListBuffer[Float]()
+
+for (x <- 0 until h) {
+  for (y <- 0 until w) {
+val color = resizedImage.getRGB(y, x)
+val red = (color & 0xff) >> 16
+val green = (color & 0xff00) >> 8
+val blue = color & 0xff
+pixels += red
+pixels += green
+pixels += blue
+  }
+}
+
+val reshaped_pixels = NDArray.array(pixels.toArray, shape = Shape(224, 
224, 3))
+
+val swapped_axis = NDArray.swapaxes(reshaped_pixels, 0, 2)
+val pixelsNdarray = NDArray.swapaxes(swapped_axis, 1, 2)
+
+pixelsNdarray
+  }
+
+  /**
+* To classify the image according to the provided model
+*
+* @param inputImage PathPrefix of the input image
+* @param topK Get top k elements with maximum probability
+* @return List of list of tuples of (class, probability)
+*/
+  def classifyImage(inputImage: BufferedImage,
+topK: Option[Int] = None): IndexedSeq[List[(String, 
Float)]] = {
+
+val width = inputDescriptors(0).shape(2)
+val height = inputDescriptors(0).shape(3)
 
 Review comment:
   same as above.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact 

[GitHub] nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image classifier for scala-infer package

[GitBox]

nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image 
classifier for scala-infer package
URL: https://github.com/apache/incubator-mxnet/pull/10054#discussion_r174895695
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/ImageClassifier.scala
 ##
 @@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet._
+
+import scala.collection.mutable.ListBuffer
+
+// scalastyle:off
+import java.awt.image.BufferedImage
+// scalastyle:on
+import java.io.File
+import javax.imageio.ImageIO
+
+
+/**
+  * A class for classifier tasks
+  *
+  * @param modelPathPrefix  PathPrefix from where to load the symbol, 
parameters and synset.txt
+  * Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json
+  * file://model-dir/synset.txt
+  * @param inputDescriptors Descriptors defining the input node names, shape,
+  * layout and Type parameters
+  */
+class ImageClassifier(modelPathPrefix: String,
+  inputDescriptors: IndexedSeq[DataDesc])
+  extends Classifier(modelPathPrefix,
+  inputDescriptors) {
+
+  val classifier: Classifier = getClassifier(modelPathPrefix, inputDescriptors)
+
+  // Loading image from file
+  def loadImageFromFile(inputImagePath: String): BufferedImage = {
+  val img = ImageIO.read(new File(inputImagePath))
+  img
+  }
+
+  /**
+* Loading input batch of images
+* @param inputImageDirPath
+* @return List of buffered images
+*/
+  def loadInputBatch(inputImageDirPath: String): List[BufferedImage] = {
+val dir = new File(inputImageDirPath)
+val inputBatch = ListBuffer[BufferedImage]()
+for (imgFile: File <- dir.listFiles()){
+  val img = ImageIO.read(imgFile)
+  inputBatch += img
+}
+inputBatch.toList
+  }
+
+  /**
+* Reshape the input image to new shape
+*
+* @param img   image
+* @param newWidth  rescale to new width
+* @param newHeight rescale to new height
+* @return Rescaled BufferedImage
+*/
+  def getScaledImage(img: BufferedImage, newWidth: Int, newHeight: Int): 
BufferedImage = {
+val resizedImage = new BufferedImage(newWidth, newHeight, 
BufferedImage.TYPE_INT_RGB)
+val g = resizedImage.createGraphics()
+g.drawImage(img, 0, 0, newWidth, newHeight, null)
+g.dispose()
+
+resizedImage
+  }
+
+  /**
+* Read image file from provided path
+*
+* @param resizedImage BufferedImage to get pixels from
+* @return NDArray pixels array
+*/
+  def getPixelsFromImage(resizedImage: BufferedImage): NDArray = {
+val w = resizedImage.getWidth
+val h = resizedImage.getHeight
+
+val pixels = new ListBuffer[Float]()
+
+for (x <- 0 until h) {
+  for (y <- 0 until w) {
+val color = resizedImage.getRGB(y, x)
+val red = (color & 0xff) >> 16
+val green = (color & 0xff00) >> 8
+val blue = color & 0xff
+pixels += red
+pixels += green
+pixels += blue
+  }
+}
+
+val reshaped_pixels = NDArray.array(pixels.toArray, shape = Shape(224, 
224, 3))
+
+val swapped_axis = NDArray.swapaxes(reshaped_pixels, 0, 2)
+val pixelsNdarray = NDArray.swapaxes(swapped_axis, 1, 2)
+
+pixelsNdarray
+  }
+
+  /**
+* To classify the image according to the provided model
+*
+* @param inputImage PathPrefix of the input image
+* @param topK Get top k elements with maximum probability
+* @return List of list of tuples of (class, probability)
+*/
+  def classifyImage(inputImage: BufferedImage,
+topK: Option[Int] = None): IndexedSeq[List[(String, 
Float)]] = {
+
+val width = inputDescriptors(0).shape(2)
+val height = inputDescriptors(0).shape(3)
+
+val scaledImage = this.getScaledImage(inputImage, width, height)
+val pixelsNdarray = this.getPixelsFromImage(scaledImage)
+
+val input = IndexedSeq(pixelsNdarray.reshape(inputDescriptors(0).shape))
+
+val output = super.classifyWithNDArray(input, topK)
+
+IndexedSeq(output(0))
 
 Review comment:

[GitHub] nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image classifier for scala-infer package

[GitBox]

nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image 
classifier for scala-infer package
URL: https://github.com/apache/incubator-mxnet/pull/10054#discussion_r174895483
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/ImageClassifier.scala
 ##
 @@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet._
+
+import scala.collection.mutable.ListBuffer
+
+// scalastyle:off
+import java.awt.image.BufferedImage
+// scalastyle:on
+import java.io.File
+import javax.imageio.ImageIO
+
+
+/**
+  * A class for classifier tasks
+  *
+  * @param modelPathPrefix  PathPrefix from where to load the symbol, 
parameters and synset.txt
+  * Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json
+  * file://model-dir/synset.txt
+  * @param inputDescriptors Descriptors defining the input node names, shape,
+  * layout and Type parameters
+  */
+class ImageClassifier(modelPathPrefix: String,
+  inputDescriptors: IndexedSeq[DataDesc])
+  extends Classifier(modelPathPrefix,
+  inputDescriptors) {
+
+  val classifier: Classifier = getClassifier(modelPathPrefix, inputDescriptors)
+
+  // Loading image from file
+  def loadImageFromFile(inputImagePath: String): BufferedImage = {
+  val img = ImageIO.read(new File(inputImagePath))
+  img
+  }
+
+  /**
+* Loading input batch of images
+* @param inputImageDirPath
+* @return List of buffered images
+*/
+  def loadInputBatch(inputImageDirPath: String): List[BufferedImage] = {
+val dir = new File(inputImageDirPath)
+val inputBatch = ListBuffer[BufferedImage]()
+for (imgFile: File <- dir.listFiles()){
+  val img = ImageIO.read(imgFile)
+  inputBatch += img
+}
+inputBatch.toList
+  }
+
+  /**
+* Reshape the input image to new shape
+*
+* @param img   image
+* @param newWidth  rescale to new width
+* @param newHeight rescale to new height
+* @return Rescaled BufferedImage
+*/
+  def getScaledImage(img: BufferedImage, newWidth: Int, newHeight: Int): 
BufferedImage = {
+val resizedImage = new BufferedImage(newWidth, newHeight, 
BufferedImage.TYPE_INT_RGB)
+val g = resizedImage.createGraphics()
+g.drawImage(img, 0, 0, newWidth, newHeight, null)
+g.dispose()
+
+resizedImage
+  }
+
+  /**
+* Read image file from provided path
+*
+* @param resizedImage BufferedImage to get pixels from
+* @return NDArray pixels array
+*/
+  def getPixelsFromImage(resizedImage: BufferedImage): NDArray = {
 
 Review comment:
   take layout parameter as input and make it static


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image classifier for scala-infer package

[GitBox]

nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image 
classifier for scala-infer package
URL: https://github.com/apache/incubator-mxnet/pull/10054#discussion_r174892088
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/ImageClassifier.scala
 ##
 @@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet._
 
 Review comment:
   Please import only the required modules


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image classifier for scala-infer package

[GitBox]

nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image 
classifier for scala-infer package
URL: https://github.com/apache/incubator-mxnet/pull/10054#discussion_r174894313
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/ImageClassifier.scala
 ##
 @@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet._
+
+import scala.collection.mutable.ListBuffer
+
+// scalastyle:off
+import java.awt.image.BufferedImage
+// scalastyle:on
+import java.io.File
+import javax.imageio.ImageIO
+
+
+/**
+  * A class for classifier tasks
+  *
+  * @param modelPathPrefix  PathPrefix from where to load the symbol, 
parameters and synset.txt
+  * Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json
+  * file://model-dir/synset.txt
+  * @param inputDescriptors Descriptors defining the input node names, shape,
+  * layout and Type parameters
+  */
+class ImageClassifier(modelPathPrefix: String,
+  inputDescriptors: IndexedSeq[DataDesc])
+  extends Classifier(modelPathPrefix,
+  inputDescriptors) {
+
+  val classifier: Classifier = getClassifier(modelPathPrefix, inputDescriptors)
+
+  // Loading image from file
+  def loadImageFromFile(inputImagePath: String): BufferedImage = {
+  val img = ImageIO.read(new File(inputImagePath))
+  img
+  }
+
+  /**
+* Loading input batch of images
+* @param inputImageDirPath
+* @return List of buffered images
+*/
+  def loadInputBatch(inputImageDirPath: String): List[BufferedImage] = {
+val dir = new File(inputImageDirPath)
+val inputBatch = ListBuffer[BufferedImage]()
+for (imgFile: File <- dir.listFiles()){
+  val img = ImageIO.read(imgFile)
+  inputBatch += img
+}
+inputBatch.toList
+  }
+
+  /**
+* Reshape the input image to new shape
+*
+* @param img   image
+* @param newWidth  rescale to new width
+* @param newHeight rescale to new height
+* @return Rescaled BufferedImage
+*/
+  def getScaledImage(img: BufferedImage, newWidth: Int, newHeight: Int): 
BufferedImage = {
+val resizedImage = new BufferedImage(newWidth, newHeight, 
BufferedImage.TYPE_INT_RGB)
+val g = resizedImage.createGraphics()
+g.drawImage(img, 0, 0, newWidth, newHeight, null)
+g.dispose()
+
+resizedImage
+  }
+
+  /**
+* Read image file from provided path
+*
+* @param resizedImage BufferedImage to get pixels from
+* @return NDArray pixels array
+*/
+  def getPixelsFromImage(resizedImage: BufferedImage): NDArray = {
+val w = resizedImage.getWidth
+val h = resizedImage.getHeight
+
+val pixels = new ListBuffer[Float]()
+
+for (x <- 0 until h) {
 
 Review comment:
   loops should be based on layout of the input descriptor. find out if `H` is 
before `W`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image classifier for scala-infer package

[GitBox]

nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image 
classifier for scala-infer package
URL: https://github.com/apache/incubator-mxnet/pull/10054#discussion_r174892964
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/ImageClassifier.scala
 ##
 @@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet._
+
+import scala.collection.mutable.ListBuffer
+
+// scalastyle:off
+import java.awt.image.BufferedImage
+// scalastyle:on
+import java.io.File
+import javax.imageio.ImageIO
+
+
+/**
+  * A class for classifier tasks
+  *
+  * @param modelPathPrefix  PathPrefix from where to load the symbol, 
parameters and synset.txt
+  * Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json
+  * file://model-dir/synset.txt
+  * @param inputDescriptors Descriptors defining the input node names, shape,
+  * layout and Type parameters
+  */
+class ImageClassifier(modelPathPrefix: String,
+  inputDescriptors: IndexedSeq[DataDesc])
+  extends Classifier(modelPathPrefix,
+  inputDescriptors) {
+
+  val classifier: Classifier = getClassifier(modelPathPrefix, inputDescriptors)
+
+  // Loading image from file
+  def loadImageFromFile(inputImagePath: String): BufferedImage = {
+  val img = ImageIO.read(new File(inputImagePath))
+  img
+  }
+
+  /**
+* Loading input batch of images
+* @param inputImageDirPath
+* @return List of buffered images
+*/
+  def loadInputBatch(inputImageDirPath: String): List[BufferedImage] = {
+val dir = new File(inputImageDirPath)
+val inputBatch = ListBuffer[BufferedImage]()
+for (imgFile: File <- dir.listFiles()){
+  val img = ImageIO.read(imgFile)
+  inputBatch += img
+}
+inputBatch.toList
+  }
+
+  /**
+* Reshape the input image to new shape
+*
+* @param img   image
+* @param newWidth  rescale to new width
+* @param newHeight rescale to new height
+* @return Rescaled BufferedImage
+*/
+  def getScaledImage(img: BufferedImage, newWidth: Int, newHeight: Int): 
BufferedImage = {
 
 Review comment:
   why not call this reshape Image? instead of getScaledImage. Also this can be 
static method, so users can just call this method without having to create an 
object. ie.., a method on the companion object


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image classifier for scala-infer package

[GitBox]

nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image 
classifier for scala-infer package
URL: https://github.com/apache/incubator-mxnet/pull/10054#discussion_r174894433
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/ImageClassifier.scala
 ##
 @@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet._
+
+import scala.collection.mutable.ListBuffer
+
+// scalastyle:off
+import java.awt.image.BufferedImage
+// scalastyle:on
+import java.io.File
+import javax.imageio.ImageIO
+
+
+/**
+  * A class for classifier tasks
+  *
+  * @param modelPathPrefix  PathPrefix from where to load the symbol, 
parameters and synset.txt
+  * Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json
+  * file://model-dir/synset.txt
+  * @param inputDescriptors Descriptors defining the input node names, shape,
+  * layout and Type parameters
+  */
+class ImageClassifier(modelPathPrefix: String,
+  inputDescriptors: IndexedSeq[DataDesc])
+  extends Classifier(modelPathPrefix,
+  inputDescriptors) {
+
+  val classifier: Classifier = getClassifier(modelPathPrefix, inputDescriptors)
+
+  // Loading image from file
+  def loadImageFromFile(inputImagePath: String): BufferedImage = {
+  val img = ImageIO.read(new File(inputImagePath))
+  img
+  }
+
+  /**
+* Loading input batch of images
+* @param inputImageDirPath
+* @return List of buffered images
+*/
+  def loadInputBatch(inputImageDirPath: String): List[BufferedImage] = {
+val dir = new File(inputImageDirPath)
+val inputBatch = ListBuffer[BufferedImage]()
+for (imgFile: File <- dir.listFiles()){
+  val img = ImageIO.read(imgFile)
+  inputBatch += img
+}
+inputBatch.toList
+  }
+
+  /**
+* Reshape the input image to new shape
+*
+* @param img   image
+* @param newWidth  rescale to new width
+* @param newHeight rescale to new height
+* @return Rescaled BufferedImage
+*/
+  def getScaledImage(img: BufferedImage, newWidth: Int, newHeight: Int): 
BufferedImage = {
+val resizedImage = new BufferedImage(newWidth, newHeight, 
BufferedImage.TYPE_INT_RGB)
+val g = resizedImage.createGraphics()
+g.drawImage(img, 0, 0, newWidth, newHeight, null)
+g.dispose()
+
+resizedImage
+  }
+
+  /**
+* Read image file from provided path
+*
+* @param resizedImage BufferedImage to get pixels from
+* @return NDArray pixels array
+*/
+  def getPixelsFromImage(resizedImage: BufferedImage): NDArray = {
+val w = resizedImage.getWidth
+val h = resizedImage.getHeight
+
+val pixels = new ListBuffer[Float]()
+
+for (x <- 0 until h) {
+  for (y <- 0 until w) {
+val color = resizedImage.getRGB(y, x)
+val red = (color & 0xff) >> 16
+val green = (color & 0xff00) >> 8
+val blue = color & 0xff
+pixels += red
+pixels += green
+pixels += blue
+  }
+}
+
+val reshaped_pixels = NDArray.array(pixels.toArray, shape = Shape(224, 
224, 3))
 
 Review comment:
   Shape based on input descriptor


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image classifier for scala-infer package

[GitBox]

nswamy commented on a change in pull request #10054: [MXNET-53][WIP] Image 
classifier for scala-infer package
URL: https://github.com/apache/incubator-mxnet/pull/10054#discussion_r174894688
 
 

 ##
 File path: 
scala-package/infer/src/main/scala/ml/dmlc/mxnet/infer/ImageClassifier.scala
 ##
 @@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ml.dmlc.mxnet.infer
+
+import ml.dmlc.mxnet._
+
+import scala.collection.mutable.ListBuffer
+
+// scalastyle:off
+import java.awt.image.BufferedImage
+// scalastyle:on
+import java.io.File
+import javax.imageio.ImageIO
+
+
+/**
+  * A class for classifier tasks
+  *
+  * @param modelPathPrefix  PathPrefix from where to load the symbol, 
parameters and synset.txt
+  * Example: file://model-dir/resnet-152(containing 
resnet-152-symbol.json
+  * file://model-dir/synset.txt
+  * @param inputDescriptors Descriptors defining the input node names, shape,
+  * layout and Type parameters
+  */
+class ImageClassifier(modelPathPrefix: String,
+  inputDescriptors: IndexedSeq[DataDesc])
+  extends Classifier(modelPathPrefix,
+  inputDescriptors) {
+
+  val classifier: Classifier = getClassifier(modelPathPrefix, inputDescriptors)
+
+  // Loading image from file
+  def loadImageFromFile(inputImagePath: String): BufferedImage = {
+  val img = ImageIO.read(new File(inputImagePath))
+  img
+  }
+
+  /**
+* Loading input batch of images
+* @param inputImageDirPath
+* @return List of buffered images
+*/
+  def loadInputBatch(inputImageDirPath: String): List[BufferedImage] = {
+val dir = new File(inputImageDirPath)
+val inputBatch = ListBuffer[BufferedImage]()
+for (imgFile: File <- dir.listFiles()){
+  val img = ImageIO.read(imgFile)
+  inputBatch += img
+}
+inputBatch.toList
+  }
+
+  /**
+* Reshape the input image to new shape
+*
+* @param img   image
+* @param newWidth  rescale to new width
+* @param newHeight rescale to new height
+* @return Rescaled BufferedImage
+*/
+  def getScaledImage(img: BufferedImage, newWidth: Int, newHeight: Int): 
BufferedImage = {
+val resizedImage = new BufferedImage(newWidth, newHeight, 
BufferedImage.TYPE_INT_RGB)
+val g = resizedImage.createGraphics()
+g.drawImage(img, 0, 0, newWidth, newHeight, null)
+g.dispose()
+
+resizedImage
+  }
+
+  /**
+* Read image file from provided path
+*
+* @param resizedImage BufferedImage to get pixels from
+* @return NDArray pixels array
+*/
+  def getPixelsFromImage(resizedImage: BufferedImage): NDArray = {
+val w = resizedImage.getWidth
+val h = resizedImage.getHeight
+
+val pixels = new ListBuffer[Float]()
+
+for (x <- 0 until h) {
+  for (y <- 0 until w) {
+val color = resizedImage.getRGB(y, x)
+val red = (color & 0xff) >> 16
+val green = (color & 0xff00) >> 8
+val blue = color & 0xff
+pixels += red
+pixels += green
+pixels += blue
+  }
+}
+
+val reshaped_pixels = NDArray.array(pixels.toArray, shape = Shape(224, 
224, 3))
+
+val swapped_axis = NDArray.swapaxes(reshaped_pixels, 0, 2)
 
 Review comment:
   would you need to swap if you build the Input according to the input 
descriptor?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] haojin2 opened a new pull request #10125: [MXNET-100] Support float16 in Correlation operator

[GitBox]

haojin2 opened a new pull request #10125: [MXNET-100] Support float16 in 
Correlation operator 
URL: https://github.com/apache/incubator-mxnet/pull/10125
 
 
   ## Description ##
   Add support for any datatype for Correlation operator, which is not 
mentioned in issue #2302.
   
   ## Checklist ##
   ### Essentials ###
   - [x] Passed code style checking (`make lint`)
   - [x] Changes are complete (i.e. I finished coding on this PR)
   - [x] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - [x] Code is well-documented: 
   - [x] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [x] Change Correlation operator from only supporting real_t to supporting 
any datatype
   - [x] Add additional test cases for float16
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin opened a new pull request #10124: [WIP][DO NOT MERGE] Bug fix for libsvm iter

[GitBox]

eric-haibin-lin opened a new pull request #10124: [WIP][DO NOT MERGE] Bug fix 
for libsvm iter
URL: https://github.com/apache/incubator-mxnet/pull/10124
 
 
   ## Description ##
   Will add a JIRA task soon
   
   ## Checklist ##
   ### Essentials ###
   - [ ] Passed code style checking (`make lint`)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing 
distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a 
new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments 
are documented. 
   - For new examples, README.md is added to explain the what the example does, 
the source of the dataset, expected performance on test set and reference to 
the original paper if applicable
   - [ ] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [ ] Feature1, tests, (and when applicable, API doc)
   - [ ] Feature2, tests, (and when applicable, API doc)
   
   ## Comments ##
   - If this change is a backward incompatible change, why must this change be 
made.
   - Interesting edge cases to note here
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin commented on issue #9396: inference speed drop after updating mxnet from 0.10.0 to 1.0.0

[GitBox]

eric-haibin-lin commented on issue #9396: inference speed drop after updating 
mxnet from 0.10.0 to 1.0.0
URL: 
https://github.com/apache/incubator-mxnet/issues/9396#issuecomment-373480684
 
 
   @nicklhy I bisected the changes between 0.10.0 and 1.0 and found the 
following on a p2.xlarge(K80) instance. The commits are patched with the fix in 
PR 9055
   ```bash
   git checkout xxx; 
   git submodule update --recursive; 
   git cherry-pick 9cc8ea3be23fb7adf4630e4cf065a2473094fbc8 -X theirs
   make
   ```
   and below is the result
   ```
   ff21e1f Changed FullyConnected to use new linalg gemm, plus TensorCore if 
fp16 I/O. (#7505)
   
   speed test for batch size: 1
   avg forward speed: 24.484983 samples/s
   avg forward time: mean = 0.040839 s, std = 0.95 s
   ```
   ```
   56eae58 Fixed Makefile so a null CUDA_ARCH is treated like an unset one. 
(#7515) - Fast
   
   
   speed test for batch size: 1
   avg forward speed: 25.461191 samples/s
   avg forward time: mean = 0.039270 s, std = 0.95 s
   
   ```
   Looks like the commit ff21e1f caused the 4% slowdown during inference. 
   
   @DickJC123 were you aware of this? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudh2290 commented on issue #10094: [MXNET-89] Bug fix for bucketing module

[GitBox]

anirudh2290 commented on issue #10094: [MXNET-89] Bug fix for bucketing module
URL: https://github.com/apache/incubator-mxnet/pull/10094#issuecomment-373472404
 
 
   Failed once for : #9972 and #10114 . Retriggered


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudh2290 commented on issue #10094: [MXNET-89] Bug fix for bucketing module

[GitBox]

anirudh2290 commented on issue #10094: [MXNET-89] Bug fix for bucketing module
URL: https://github.com/apache/incubator-mxnet/pull/10094#issuecomment-373472404
 
 
   Failed once for : #9972 and #10114


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-mxnet] 04/04: Add vocabulary and embedding (#10074)

[zhasheng]

This is an automated email from the ASF dual-hosted git repository.

zhasheng pushed a commit to branch nlp_toolkit
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 9e4f9d18ec6827799f5b787f4c0697df7c017ef2
Author: Aston Zhang <22279212+astonzh...@users.noreply.github.com>
AuthorDate: Thu Mar 15 09:39:05 2018 -0700

Add vocabulary and embedding (#10074)

* [MXNET-67] Sync master with v1.1.0 branch (#10031)

* [REVIEW REQUIRED] Revert PR #9484 & add additional dependency licenses to 
LICENSE file (#9701)

* Revert "[Review Required] Fixing Licenses: Cleaning up the Top Level 
LICENSE file (#9484)"

This reverts commit 8930d96b265560a797c5554a9617f607cea7740f.

* Some more LICENSE fixes

* Adding some more packages to the LICENSE file

* Adding dependencies of dependencies

* update v1.1.0 change log to NEWS.md

* sync README.md from v1.1.0 branch

* revert to correct jenkins url in README

* Parallelization for ROIpooling OP (#9958)

* parallelization for roipooling

* remove some useless computation

* remove useless muls

* add author and retriggering

* retrigger again

* comments to copy and copyto are corrected (#10040)

* Bug Fix and performance optimized for rtc (#10018)

* Bug Fix and performance optimized for rtc

1. "super().__init__()" bug is fixed in python 2.
2. Kernel is initialized in the stage of operator init.

* Update custom_softmax_rtc.py

fix unnessesary format

* set embedding

* Code and test revised

* api implementation done

* license and news

* readme and cpp

* pylint disable

* Add API doc

* less pylint disable

* remove contrib

* move to gluon, revise api doc

* fix import order

* re-test

* relative imports

* re-run test

* revise implementation, test case, and api doc

* re-test
---
 docs/api/python/gluon/text.md| 347 +++
 python/mxnet/gluon/text/__init__.py  |  26 ++
 python/mxnet/gluon/text/_constants.py| 704 +++
 python/mxnet/gluon/text/embedding.py | 581 +
 python/mxnet/gluon/text/utils.py |  77 
 python/mxnet/gluon/text/vocab.py | 323 ++
 tests/python/unittest/test_gluon_text.py | 681 ++
 7 files changed, 2739 insertions(+)

diff --git a/docs/api/python/gluon/text.md b/docs/api/python/gluon/text.md
new file mode 100644
index 000..f9fcf33
--- /dev/null
+++ b/docs/api/python/gluon/text.md
@@ -0,0 +1,347 @@
+# Gluon Text API
+
+## Overview
+
+The `mxnet.gluon.text` APIs refer to classes and functions related to text 
data processing, such
+as bulding indices and loading pre-trained embedding vectors for text tokens 
and storing them in the
+`mxnet.ndarray.NDArray` format.
+
+This document lists the text APIs in `mxnet.gluon`:
+
+```eval_rst
+.. autosummary::
+:nosignatures:
+
+mxnet.gluon.text.embedding
+mxnet.gluon.text.vocab
+mxnet.gluon.text.utils
+```
+
+All the code demonstrated in this document assumes that the following modules 
or packages are
+imported.
+
+```python
+>>> from mxnet import gluon
+>>> from mxnet import nd
+>>> from mxnet.gluon import text
+>>> import collections
+
+```
+
+### Indexing words and using pre-trained word embeddings in `gluon`
+
+As a common use case, let us index words, attach pre-trained word embeddings 
for them, and use
+such embeddings in `gluon` in just a few lines of code. 
+
+To begin with, suppose that we have a simple text data set in the string 
format. We can count word
+frequency in the data set.
+
+```python
+>>> text_data = " hello world \n hello nice world \n hi world \n"
+>>> counter = text.count_tokens_from_str(text_data)
+
+```
+
+The obtained `counter` has key-value pairs whose keys are words and values are 
word frequencies.
+This allows us to filter out infrequent words (See details at
+[Vocabulary API specifications](#mxnet.gluon.text.vocab.Vocabulary)).
+Suppose that we want to build indices for all the keys in `counter`. We need a 
Vocabulary instance
+with `counter` as its argument.
+
+```python
+>>> my_vocab = text.Vocabulary(counter)
+
+```
+
+To attach word embedding to indexed words in `my_vocab`, let us go on to 
create a fastText word
+embedding instance by specifying the embedding name `fasttext` and the 
pre-trained file name
+`wiki.simple.vec`.
+
+```python
+>>> fasttext = text.embedding.create('fasttext', file_name='wiki.simple.vec')
+
+```
+
+So we can attach word embedding `fasttext` to indexed words `my_vocab`.
+
+```python
+>>> my_vocab.set_embedding(fasttext)
+
+```
+
+Now we are ready to access the fastText word embedding vectors for indexed 
words, such as 'hello'
+and 'world'.
+
+```python

[incubator-mxnet] branch nlp_toolkit updated (092df8f -> 9e4f9d1)

[zhasheng]

This is an automated email from the ASF dual-hosted git repository.

zhasheng pushed a change to branch nlp_toolkit
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


 discard 092df8f  Add vocabulary and embedding (#10074)
 discard 7e59d74  fix CorpusReader (#10079)
omit 453fc23  CorpusReader (#10061)
omit a61fd99  gluon language modeling dataset and text token reader (#9986)
 add ccf0469  [MXNET-67] Sync master with v1.1.0 branch (#10031)
 add 884408a  Parallelization for ROIpooling OP (#9958)
 add c153a6c  comments to copy and copyto are corrected (#10040)
 add a7ba921  Bug Fix and performance optimized for rtc (#10018)
 add 85215b6  CI docker revamp; Add Jetson, Raspberry and CentOS 7 build 
[MXNET-42][MXNET-43] (#9995)
 add 279ccb1  [MXNET-58]Layer Normalization in C++ (#10029)
 add 17643e7  Update test_sparse_operator.py (#10050)
 add bf8658b  fixed calling error while preparing data in ssd (#10024)
 add 13ae4d1  Cpu lstm inference (#9977)
 add 1794a2a  [MXNET-38]add reshape predicator function to c_predict_api 
(#9984)
 add 94f68fc  Fixes for profiler (#9932)
 add 0bcf334  Adding back comments to index.md that cause nightly test to 
fail (#10058)
 add a9c1717  [TENSOR] Fix DLTensor conversion for int64 (#10083)
 add dd1f21b  Support single array input for metric (#9930)
 add c9ec311  [MXNET-86] Revert to pre-profile-changes copy code (#10090)
 add 1dc01ef   [MXNET-47] Documentation for build_version_doc scripts 
folder (#10010)
 add faa4f9f  [MXNET-83] Fix CMake build issue with MKL. (#10075)
 add f202f10  [MXNET-74]Update mkldnn to the newest & Add clang build test 
with mkldnn. (#9918)
 add 93e0ceb  [MXNET-34] Onnx Module to import onnx models into mxnet 
(#9963)
 add f7fa49e  Compile MKLDNN in Mac. (#10115)
 new 929f5ef  gluon language modeling dataset and text token reader (#9986)
 new 85f564e  CorpusReader (#10061)
 new 911ff3b  fix CorpusReader (#10079)
 new 9e4f9d1  Add vocabulary and embedding (#10074)

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (092df8f)
\
 N -- N -- N   refs/heads/nlp_toolkit (9e4f9d1)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 example/utils/__init__.py => .mxnet_root   |   0
 3rdparty/mkldnn|   2 +-
 CMakeLists.txt |  15 +-
 Jenkinsfile| 351 ++
 LICENSE| 312 +++--
 Makefile   |  10 +-
 NEWS.md|  41 ++
 README.md  |   1 +
 ci/README.md   |  50 ++
 ci/build.py| 179 
 ci/docker/Dockerfile.build.amzn_linux_cpu  |  44 ++
 ci/docker/Dockerfile.build.android_arm64   |  66 +++
 .../docker/Dockerfile.build.android_armv7  |  24 +-
 .../Makefile => ci/docker/Dockerfile.build.arm64   |  27 +-
 .../docker/Dockerfile.build.armv7  |  22 +-
 .../docker/Dockerfile.build.centos7_cpu|  25 +-
 .../docker/Dockerfile.build.centos7_gpu|  23 +-
 .../docker}/Dockerfile.build.jetson|  74 ++-
 ci/docker/Dockerfile.build.ubuntu_build_cuda   |  57 +++
 ci/docker/Dockerfile.build.ubuntu_cpu  |  55 +++
 ci/docker/Dockerfile.build.ubuntu_gpu  |  55 +++
 .../docker/install/amzn_linux_core.sh  |  39 +-
 .../docker/install/amzn_linux_julia.sh |  13 +-
 .../docker/install/amzn_linux_library.sh   |  11 +-
 .../docker/install/amzn_linux_maven.sh |  13 +-
 .../docker/install/amzn_linux_openblas.sh  |  10 +-
 .../docker/install/amzn_linux_opencv.sh|  19 +-
 .../docker/install/amzn_linux_python2.sh   |  15 +-
 .../docker/install/amzn_linux_python3.sh   |  18 +-
 .../docker/install/amzn_linux_testdeps.sh  |  11 +-
 .../docker/install/android_arm64_ndk.sh|  31 +-
 .../docker/install/android_arm64_openblas.sh  

[incubator-mxnet] 02/04: CorpusReader (#10061)

[zhasheng]

This is an automated email from the ASF dual-hosted git repository.

zhasheng pushed a commit to branch nlp_toolkit
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 85f564e59ab26e4896225842d803d2213aa84234
Author: Sheng Zha 
AuthorDate: Sat Mar 10 14:40:50 2018 -0800

CorpusReader (#10061)

* corpus reader

* update

* delete pair
---
 python/mxnet/gluon/data/datareader.py |  2 +-
 python/mxnet/gluon/data/text/base.py  | 99 +--
 python/mxnet/gluon/data/text/lm.py| 10 +--
 python/mxnet/gluon/data/text/utils.py | 27 ++--
 tests/python/unittest/test_gluon_data_text.py | 22 +++---
 5 files changed, 87 insertions(+), 73 deletions(-)

diff --git a/python/mxnet/gluon/data/datareader.py 
b/python/mxnet/gluon/data/datareader.py
index 9b94ed4..b96575b 100644
--- a/python/mxnet/gluon/data/datareader.py
+++ b/python/mxnet/gluon/data/datareader.py
@@ -31,4 +31,4 @@ class DataReader(object):
 raise NotImplementedError
 
 def read_iter(self):
-return self.read()
+return iter(self.read())
diff --git a/python/mxnet/gluon/data/text/base.py 
b/python/mxnet/gluon/data/text/base.py
index a9fa25b..6c4cf87 100644
--- a/python/mxnet/gluon/data/text/base.py
+++ b/python/mxnet/gluon/data/text/base.py
@@ -27,12 +27,56 @@ import os
 
 from ..dataset import SimpleDataset
 from ..datareader import DataReader
-from .utils import flatten_samples, collate, pair
+from .utils import flatten_samples, collate
 
-class WordLanguageReader(DataReader):
+class CorpusReader(DataReader):
 """Text reader that reads a whole corpus and produces a dataset based on 
provided
 sample splitter and word tokenizer.
 
+The returned dataset includes samples, each of which can either be a list 
of tokens if tokenizer
+is specified, or a single string segment from the result of 
sample_splitter.
+
+Parameters
+--
+filename : str
+Path to the input text file.
+encoding : str, default 'utf8'
+File encoding format.
+flatten : bool, default False
+Whether to return all samples as flattened tokens. If True, each 
sample is a token.
+sample_splitter : function, default str.splitlines
+A function that splits the dataset string into samples.
+tokenizer : function or None, default str.split
+A function that splits each sample string into list of tokens. If 
None, raw samples are
+returned according to `sample_splitter`.
+"""
+def __init__(self, filename, encoding='utf8', flatten=False,
+ sample_splitter=lambda s: s.splitlines(),
+ tokenizer=lambda s: s.split()):
+assert sample_splitter, 'sample_splitter must be specified.'
+self._filename = os.path.expanduser(filename)
+self._encoding = encoding
+self._flatten = flatten
+self._sample_splitter = sample_splitter
+self._tokenizer = tokenizer
+
+def read(self):
+with io.open(self._filename, 'r', encoding=self._encoding) as fin:
+content = fin.read()
+samples = (s.strip() for s in self._sample_splitter(content))
+if self._tokenizer:
+samples = [self._tokenizer(s) for s in samples if s]
+if self._flatten:
+samples = flatten(samples)
+else:
+samples = [s for s in samples if s]
+return SimpleDataset(samples)
+
+
+class WordLanguageReader(CorpusReader):
+"""Text reader that reads a whole corpus and produces a language modeling 
dataset given
+the provided sample splitter and word tokenizer.
+
 The returned dataset includes data (current word) and label (next word).
 
 Parameters
@@ -46,8 +90,9 @@ class WordLanguageReader(DataReader):
 tokenizer : function, default str.split
 A function that splits each sample string into list of tokens.
 seq_len : int or None
-The length of each of the samples. If None, samples are divided 
according to
-`sample_splitter` only, and may have variable lengths.
+The length of each of the samples regardless of sample boundary.
+If None, samples are divided according to `sample_splitter` only,
+and may have variable lengths.
 bos : str or None, default None
 The token to add at the begining of each sentence. If None, nothing is 
added.
 eos : str or None, default None
@@ -61,43 +106,27 @@ class WordLanguageReader(DataReader):
 """
 def __init__(self, filename, encoding='utf8', sample_splitter=lambda s: 
s.splitlines(),
  tokenizer=lambda s: s.split(), seq_len=None, bos=None, 
eos=None, pad=None):
-self._filename = os.path.expanduser(filename)
-self._encoding = encoding
-self._sample_splitter = sample_splitter
-self._tokenizer = tokenizer
-
-if bos and eos:
-def 

[incubator-mxnet] 03/04: fix CorpusReader (#10079)

[zhasheng]

This is an automated email from the ASF dual-hosted git repository.

zhasheng pushed a commit to branch nlp_toolkit
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 911ff3bf0cbca4dc626edfcbd63c87388d23b153
Author: Xingjian Shi 
AuthorDate: Mon Mar 12 15:02:28 2018 -0700

fix CorpusReader (#10079)

try to accelerate

update

Revert "update"

This reverts commit 7d409b54e0fdb006e6d8be50749a90e24b12715d.

Revert "try to accelerate"

This reverts commit b7d93f9184323909ae59d77445d45588ea76aaa5.
---
 python/mxnet/gluon/data/text/base.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/python/mxnet/gluon/data/text/base.py 
b/python/mxnet/gluon/data/text/base.py
index 6c4cf87..3c02424 100644
--- a/python/mxnet/gluon/data/text/base.py
+++ b/python/mxnet/gluon/data/text/base.py
@@ -20,7 +20,7 @@
 
 """Base classes for text datasets and readers."""
 
-__all__ = ['WordLanguageReader']
+__all__ = ['CorpusReader', 'WordLanguageReader']
 
 import io
 import os
@@ -67,7 +67,7 @@ class CorpusReader(DataReader):
 if self._tokenizer:
 samples = [self._tokenizer(s) for s in samples if s]
 if self._flatten:
-samples = flatten(samples)
+samples = flatten_samples(samples)
 else:
 samples = [s for s in samples if s]
 return SimpleDataset(samples)

-- 
To stop receiving notification emails like this one, please contact
zhash...@apache.org.

[incubator-mxnet] 01/04: gluon language modeling dataset and text token reader (#9986)

[zhasheng]

This is an automated email from the ASF dual-hosted git repository.

zhasheng pushed a commit to branch nlp_toolkit
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 929f5ef630f8f9e5cf6bef6b419282932f531fd0
Author: Sheng Zha 
AuthorDate: Thu Mar 8 16:49:51 2018 -0500

gluon language modeling dataset and text token reader (#9986)

* language modeling dataset and text token reader.

* update

* add padding

* update bos insert

* update doc
---
 example/gluon/word_language_model/train.py |  44 ---
 python/mxnet/gluon/data/__init__.py|   2 +
 .../gluon/data/{__init__.py => datareader.py}  |  18 ++-
 python/mxnet/gluon/data/{ => text}/__init__.py |  10 +-
 .../gluon/data/{__init__.py => text/_constants.py} |  12 +-
 python/mxnet/gluon/data/text/base.py   | 103 +++
 python/mxnet/gluon/data/text/lm.py | 145 +
 python/mxnet/gluon/data/text/utils.py  |  73 +++
 tests/python/unittest/test_gluon_data_text.py  |  50 +++
 9 files changed, 420 insertions(+), 37 deletions(-)

diff --git a/example/gluon/word_language_model/train.py 
b/example/gluon/word_language_model/train.py
index b69fd17..c732393 100644
--- a/example/gluon/word_language_model/train.py
+++ b/example/gluon/word_language_model/train.py
@@ -16,13 +16,13 @@
 # under the License.
 
 import argparse
+import collections
 import time
 import math
 import mxnet as mx
-from mxnet import gluon, autograd
-from mxnet.gluon import contrib
+from mxnet import gluon, autograd, contrib
+from mxnet.gluon import data
 import model
-import data
 
 parser = argparse.ArgumentParser(description='MXNet Autograd RNN/LSTM Language 
Model on Wikitext-2.')
 parser.add_argument('--model', type=str, default='lstm',
@@ -71,32 +71,40 @@ if args.cuda:
 else:
 context = mx.cpu(0)
 
-train_dataset = contrib.data.text.WikiText2('./data', 'train', 
seq_len=args.bptt)
-vocab = train_dataset.vocabulary
-val_dataset, test_dataset = [contrib.data.text.WikiText2('./data', segment,
- vocab=vocab,
- seq_len=args.bptt)
- for segment in ['validation', 'test']]
+train_dataset = data.text.lm.WikiText2('./data', 'train', seq_len=args.bptt,
+   eos='')
+
+def get_frequencies(dataset):
+return collections.Counter(x for tup in dataset for x in tup[0] if x)
+
+vocab = contrib.text.vocab.Vocabulary(get_frequencies(train_dataset))
+def index_tokens(data, label):
+return vocab.to_indices(data), vocab.to_indices(label)
+
+val_dataset, test_dataset = [data.text.lm.WikiText2('./data', segment,
+seq_len=args.bptt,
+eos='')
+ for segment in ['val', 'test']]
 
 nbatch_train = len(train_dataset) // args.batch_size
-train_data = gluon.data.DataLoader(train_dataset,
+train_data = gluon.data.DataLoader(train_dataset.transform(index_tokens),
batch_size=args.batch_size,
-   
sampler=contrib.data.IntervalSampler(len(train_dataset),
-
nbatch_train),
+   
sampler=gluon.contrib.data.IntervalSampler(len(train_dataset),
+  
nbatch_train),
last_batch='discard')
 
 nbatch_val = len(val_dataset) // args.batch_size
-val_data = gluon.data.DataLoader(val_dataset,
+val_data = gluon.data.DataLoader(val_dataset.transform(index_tokens),
  batch_size=args.batch_size,
- 
sampler=contrib.data.IntervalSampler(len(val_dataset),
-  
nbatch_val),
+ 
sampler=gluon.contrib.data.IntervalSampler(len(val_dataset),
+
nbatch_val),
  last_batch='discard')
 
 nbatch_test = len(test_dataset) // args.batch_size
-test_data = gluon.data.DataLoader(test_dataset,
+test_data = gluon.data.DataLoader(test_dataset.transform(index_tokens),
   batch_size=args.batch_size,
-  
sampler=contrib.data.IntervalSampler(len(test_dataset),
-   
nbatch_test),
+  
sampler=gluon.contrib.data.IntervalSampler(len(test_dataset),
+ 
nbatch_test),

[GitHub] pakmarkthub opened a new issue #10123: train_cifar10.py hangs on first epoch in debug mode (4 P100 GPUs)

[GitBox]

pakmarkthub opened a new issue #10123: train_cifar10.py hangs on first epoch in 
debug mode (4 P100 GPUs)
URL: https://github.com/apache/incubator-mxnet/issues/10123
 
 
   ## Description
   train_cifar10.py hangs before outputting validation-accuracy on the first 
epoch when using four P100 GPUs, compiled with debug=1 or 
MXNET_ENGINE_TYPE=NaiveEngine (recompile with debug=0 in the later case).
   
   ## Environment info (Required)
   --Python Info--
   ('Version  :', '2.7.5')
   ('Compiler :', 'GCC 4.8.5 20150623 (Red Hat 4.8.5-16)')
   ('Build:', ('default', 'Aug  4 2017 00:39:18'))
   ('Arch :', ('64bit', 'ELF'))
   Pip Info---
   ('Version  :', '9.0.1')
   ('Directory:', '/usr/lib/python2.7/site-packages/pip')
   --MXNet Info---
   ('Version  :', '1.1.0')
   ('Directory:', 
'/home/cc/Projects/uvm-reduce/experiments/mxnet/original/python/mxnet')
   Hashtag not found. Not installed from pre-built package.
   --System Info--
   ('Platform :', 
'Linux-3.10.0-514.6.1.el7.x86_64-x86_64-with-centos-7.4.1708-Core')
   ('system   :', 'Linux')
   ('node :', 'ftg-p100.novalocal')
   ('release  :', '3.10.0-514.6.1.el7.x86_64')
   ('version  :', '#1 SMP Wed Jan 18 13:06:36 UTC 2017')
   --Hardware Info--
   ('machine  :', 'x86_64')
   ('processor:', 'x86_64')
   Architecture:  x86_64
   CPU op-mode(s):32-bit, 64-bit
   Byte Order:Little Endian
   CPU(s):56
   On-line CPU(s) list:   0-55
   Thread(s) per core:2
   Core(s) per socket:14
   Socket(s): 2
   NUMA node(s):  2
   Vendor ID: GenuineIntel
   CPU family:6
   Model: 79
   Model name:Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz
   Stepping:  1
   CPU MHz:   2499.921
   CPU max MHz:   3200.
   CPU min MHz:   1200.
   BogoMIPS:  3999.88
   Virtualization:VT-x
   Hypervisor vendor: vertical
   Virtualization type:   full
   L1d cache: 32K
   L1i cache: 32K
   L2 cache:  256K
   L3 cache:  35840K
   NUMA node0 CPU(s): 
0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54
   NUMA node1 CPU(s): 
1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55
   Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx 
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology 
nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est 
tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt 
tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch ida arat 
epb pln pts dtherm intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase 
tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap 
xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local
   --Network Test--
   Setting timeout: 10
   Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0129 
sec, LOAD: 0.7237 sec.
   Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0127 sec, LOAD: 
0.2568 sec.
   Timing for FashionMNIST: 
https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz,
 DNS: 0.0640 sec, LOAD: 0.2466 sec.
   Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0161 sec, 
LOAD: 0.0744 sec.
   Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0646 sec, LOAD: 
0.2254 sec.
   Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0478 sec, LOAD: 
0.4038 sec.
   ---CUDA
   CUDA Driver Version / Runtime Version: 9.1 / 9.1
   Driver version: 390.30
   GPUs: four Tesla P100-SXM2-16GB GPUs
   Peer access: Yes (all GPU pairs)
   ---OpenCv--
   Version: 3.4.1
   
   
   Package used (Python/R/Scala/Julia):
   I'm using python2.7.5
   
   ## Build info (Required if built from source)
   
   Compiler (gcc/clang/mingw/visual studio): gcc
   
   MXNet commit hash:
   07a83a0325a3d782513a04f47d711710972cb144 (tag 1.1.0)
   f7fa49e3068ab30a7379bbce1b288bf285fe5cde (master)
   
   Build config:
   #-
   # choice of compiler
   #
   
   export CC = gcc
   export CXX = g++
   export NVCC = nvcc
   
   # whether compile with options for MXNet developer
   DEV = 0
   
   # whether compile with debug
   DEBUG = 1
   
   # whether compile with profiler
   USE_PROFILER =
   
   # whether to turn on segfault signal handler to log the stack trace
   USE_SIGNAL_HANDLER =
   
   # the additional link flags you want to add
   ADD_LDFLAGS = -lopencv_core -lopencv_imgcodecs -lopencv_imgproc
   
   # the 

[GitHub] eric-haibin-lin opened a new pull request #10122: [MXNET-16] [DO NOT MERGE] Move submodules to 3rdparty

[GitBox]

eric-haibin-lin opened a new pull request #10122: [MXNET-16] [DO NOT MERGE] 
Move submodules to 3rdparty
URL: https://github.com/apache/incubator-mxnet/pull/10122
 
 
   ## Description ##
   (Brief description on what this PR is about)
   
   ## Checklist ##
   ### Essentials ###
   - [ ] Passed code style checking (`make lint`)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing 
distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a 
new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments 
are documented. 
   - For new examples, README.md is added to explain the what the example does, 
the source of the dataset, expected performance on test set and reference to 
the original paper if applicable
   - [ ] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [ ] Feature1, tests, (and when applicable, API doc)
   - [ ] Feature2, tests, (and when applicable, API doc)
   
   ## Comments ##
   - If this change is a backward incompatible change, why must this change be 
made.
   - Interesting edge cases to note here
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha closed pull request #10074: Add vocabulary and embedding

[GitBox]

szha closed pull request #10074: Add vocabulary and embedding
URL: https://github.com/apache/incubator-mxnet/pull/10074
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/docs/api/python/gluon/text.md b/docs/api/python/gluon/text.md
new file mode 100644
index 000..f9fcf335356
--- /dev/null
+++ b/docs/api/python/gluon/text.md
@@ -0,0 +1,347 @@
+# Gluon Text API
+
+## Overview
+
+The `mxnet.gluon.text` APIs refer to classes and functions related to text 
data processing, such
+as bulding indices and loading pre-trained embedding vectors for text tokens 
and storing them in the
+`mxnet.ndarray.NDArray` format.
+
+This document lists the text APIs in `mxnet.gluon`:
+
+```eval_rst
+.. autosummary::
+:nosignatures:
+
+mxnet.gluon.text.embedding
+mxnet.gluon.text.vocab
+mxnet.gluon.text.utils
+```
+
+All the code demonstrated in this document assumes that the following modules 
or packages are
+imported.
+
+```python
+>>> from mxnet import gluon
+>>> from mxnet import nd
+>>> from mxnet.gluon import text
+>>> import collections
+
+```
+
+### Indexing words and using pre-trained word embeddings in `gluon`
+
+As a common use case, let us index words, attach pre-trained word embeddings 
for them, and use
+such embeddings in `gluon` in just a few lines of code. 
+
+To begin with, suppose that we have a simple text data set in the string 
format. We can count word
+frequency in the data set.
+
+```python
+>>> text_data = " hello world \n hello nice world \n hi world \n"
+>>> counter = text.count_tokens_from_str(text_data)
+
+```
+
+The obtained `counter` has key-value pairs whose keys are words and values are 
word frequencies.
+This allows us to filter out infrequent words (See details at
+[Vocabulary API specifications](#mxnet.gluon.text.vocab.Vocabulary)).
+Suppose that we want to build indices for all the keys in `counter`. We need a 
Vocabulary instance
+with `counter` as its argument.
+
+```python
+>>> my_vocab = text.Vocabulary(counter)
+
+```
+
+To attach word embedding to indexed words in `my_vocab`, let us go on to 
create a fastText word
+embedding instance by specifying the embedding name `fasttext` and the 
pre-trained file name
+`wiki.simple.vec`.
+
+```python
+>>> fasttext = text.embedding.create('fasttext', file_name='wiki.simple.vec')
+
+```
+
+So we can attach word embedding `fasttext` to indexed words `my_vocab`.
+
+```python
+>>> my_vocab.set_embedding(fasttext)
+
+```
+
+Now we are ready to access the fastText word embedding vectors for indexed 
words, such as 'hello'
+and 'world'.
+
+```python
+>>> my_vocab.embedding[['hello', 'world']]
+
+[[  3.95669997e-01   2.14540005e-01  -3.53889987e-02  -2.42990002e-01
+...
+   -7.54180014e-01  -3.14429998e-01   2.40180008e-02  -7.61009976e-02]
+ [  1.04440004e-01  -1.08580001e-01   2.7211e-01   1.32990003e-01
+...
+   -3.7340e-01   5.67310005e-02   5.60180008e-01   2.9019e-02]]
+
+
+```
+
+To demonstrate how to use pre-trained word embeddings in the `gluon` package, 
let us first obtain
+indices of the words 'hello' and 'world'.
+
+```python
+>>> my_vocab[['hello', 'world']]
+[2, 1]
+
+```
+
+We can obtain the vector representation for the words 'hello' and 'world' by 
specifying their
+indices (2 and 1) and the weight matrix `my_vocab.embedding.idx_to_vec` in
+`mxnet.gluon.nn.Embedding`.
+ 
+```python
+>>> input_dim, output_dim = my_vocab.embedding.idx_to_vec.shape
+>>> layer = gluon.nn.Embedding(input_dim, output_dim)
+>>> layer.initialize()
+>>> layer.weight.set_data(my_vocab.embedding.idx_to_vec)
+>>> layer(nd.array([2, 1]))
+
+[[  3.95669997e-01   2.14540005e-01  -3.53889987e-02  -2.42990002e-01
+...
+   -7.54180014e-01  -3.14429998e-01   2.40180008e-02  -7.61009976e-02]
+ [  1.04440004e-01  -1.08580001e-01   2.7211e-01   1.32990003e-01
+...
+   -3.7340e-01   5.67310005e-02   5.60180008e-01   2.9019e-02]]
+
+
+```
+
+## Vocabulary
+
+The vocabulary builds indices for text tokens and can be attached with token 
embeddings. The input
+counter whose keys are candidate indices may be obtained via
+[`count_tokens_from_str`](#mxnet.gluon.text.utils.count_tokens_from_str).
+
+
+```eval_rst
+.. currentmodule:: mxnet.gluon.text.vocab
+.. autosummary::
+:nosignatures:
+
+Vocabulary
+```
+
+Suppose that we have a simple text data set in the string format. We can count 
word frequency in the
+data set.
+
+```python
+>>> text_data = " hello world \n hello nice world \n hi world \n"
+>>> counter = text.utils.count_tokens_from_str(text_data)
+
+```
+
+The obtained `counter` has key-value pairs whose keys are words and values are 
word frequencies.
+This allows us to filter out infrequent words. Suppose that we want to build 
indices for the 2 most
+frequent 

[GitHub] marcoabreu opened a new pull request #10121: test ci

[GitBox]

marcoabreu opened a new pull request #10121: test ci
URL: https://github.com/apache/incubator-mxnet/pull/10121
 
 
   Testing CI


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-mxnet] branch nlp_toolkit updated: Add vocabulary and embedding (#10074)

[zhasheng]

This is an automated email from the ASF dual-hosted git repository.

zhasheng pushed a commit to branch nlp_toolkit
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/nlp_toolkit by this push:
 new 092df8f  Add vocabulary and embedding (#10074)
092df8f is described below

commit 092df8f8c6b4b3a29020340ec6b6b3de41ca022e
Author: Aston Zhang <22279212+astonzh...@users.noreply.github.com>
AuthorDate: Thu Mar 15 09:39:05 2018 -0700

Add vocabulary and embedding (#10074)

* [MXNET-67] Sync master with v1.1.0 branch (#10031)

* [REVIEW REQUIRED] Revert PR #9484 & add additional dependency licenses to 
LICENSE file (#9701)

* Revert "[Review Required] Fixing Licenses: Cleaning up the Top Level 
LICENSE file (#9484)"

This reverts commit 8930d96b265560a797c5554a9617f607cea7740f.

* Some more LICENSE fixes

* Adding some more packages to the LICENSE file

* Adding dependencies of dependencies

* update v1.1.0 change log to NEWS.md

* sync README.md from v1.1.0 branch

* revert to correct jenkins url in README

* Parallelization for ROIpooling OP (#9958)

* parallelization for roipooling

* remove some useless computation

* remove useless muls

* add author and retriggering

* retrigger again

* comments to copy and copyto are corrected (#10040)

* Bug Fix and performance optimized for rtc (#10018)

* Bug Fix and performance optimized for rtc

1. "super().__init__()" bug is fixed in python 2.
2. Kernel is initialized in the stage of operator init.

* Update custom_softmax_rtc.py

fix unnessesary format

* set embedding

* Code and test revised

* api implementation done

* license and news

* readme and cpp

* pylint disable

* Add API doc

* less pylint disable

* remove contrib

* move to gluon, revise api doc

* fix import order

* re-test

* relative imports

* re-run test

* revise implementation, test case, and api doc

* re-test
---
 docs/api/python/gluon/text.md| 347 +++
 python/mxnet/gluon/text/__init__.py  |  26 ++
 python/mxnet/gluon/text/_constants.py| 704 +++
 python/mxnet/gluon/text/embedding.py | 581 +
 python/mxnet/gluon/text/utils.py |  77 
 python/mxnet/gluon/text/vocab.py | 323 ++
 tests/python/unittest/test_gluon_text.py | 681 ++
 7 files changed, 2739 insertions(+)

diff --git a/docs/api/python/gluon/text.md b/docs/api/python/gluon/text.md
new file mode 100644
index 000..f9fcf33
--- /dev/null
+++ b/docs/api/python/gluon/text.md
@@ -0,0 +1,347 @@
+# Gluon Text API
+
+## Overview
+
+The `mxnet.gluon.text` APIs refer to classes and functions related to text 
data processing, such
+as bulding indices and loading pre-trained embedding vectors for text tokens 
and storing them in the
+`mxnet.ndarray.NDArray` format.
+
+This document lists the text APIs in `mxnet.gluon`:
+
+```eval_rst
+.. autosummary::
+:nosignatures:
+
+mxnet.gluon.text.embedding
+mxnet.gluon.text.vocab
+mxnet.gluon.text.utils
+```
+
+All the code demonstrated in this document assumes that the following modules 
or packages are
+imported.
+
+```python
+>>> from mxnet import gluon
+>>> from mxnet import nd
+>>> from mxnet.gluon import text
+>>> import collections
+
+```
+
+### Indexing words and using pre-trained word embeddings in `gluon`
+
+As a common use case, let us index words, attach pre-trained word embeddings 
for them, and use
+such embeddings in `gluon` in just a few lines of code. 
+
+To begin with, suppose that we have a simple text data set in the string 
format. We can count word
+frequency in the data set.
+
+```python
+>>> text_data = " hello world \n hello nice world \n hi world \n"
+>>> counter = text.count_tokens_from_str(text_data)
+
+```
+
+The obtained `counter` has key-value pairs whose keys are words and values are 
word frequencies.
+This allows us to filter out infrequent words (See details at
+[Vocabulary API specifications](#mxnet.gluon.text.vocab.Vocabulary)).
+Suppose that we want to build indices for all the keys in `counter`. We need a 
Vocabulary instance
+with `counter` as its argument.
+
+```python
+>>> my_vocab = text.Vocabulary(counter)
+
+```
+
+To attach word embedding to indexed words in `my_vocab`, let us go on to 
create a fastText word
+embedding instance by specifying the embedding name `fasttext` and the 
pre-trained file name
+`wiki.simple.vec`.
+
+```python
+>>> fasttext = text.embedding.create('fasttext', file_name='wiki.simple.vec')
+
+```
+
+So we can attach word embedding `fasttext` to indexed words `my_vocab`.
+
+```python
+>>> 

[GitHub] eric-haibin-lin closed pull request #10112: [MXNET-16] [DO NOT MERGE] Move submodules to 3rdparty

[GitBox]

eric-haibin-lin closed pull request #10112: [MXNET-16] [DO NOT MERGE] Move 
submodules to 3rdparty
URL: https://github.com/apache/incubator-mxnet/pull/10112
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.gitmodules b/.gitmodules
index cdb8a553679..51d7d009197 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -11,7 +11,7 @@
path = nnvm
url = https://github.com/dmlc/nnvm
 [submodule "dlpack"]
-   path = dlpack
+   path = 3rdparty/dlpack
url = https://github.com/dmlc/dlpack
 [submodule "3rdparty/openmp"]
path = 3rdparty/openmp
diff --git a/dlpack b/3rdparty/dlpack
similarity index 100%
rename from dlpack
rename to 3rdparty/dlpack
diff --git a/CMakeLists.txt b/CMakeLists.txt
index b3a895583b4..d5df5a3c837 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -248,7 +248,7 @@ include_directories("3rdparty/cub")
 include_directories("nnvm/include")
 include_directories("nnvm/tvm/include")
 include_directories("dmlc-core/include")
-include_directories("dlpack/include")
+include_directories("3rdparty/dlpack/include")
 
 # commented out until PR goes through
 #if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/dlpack)
diff --git a/Makefile b/Makefile
index 862a40c3406..b404c86af47 100644
--- a/Makefile
+++ b/Makefile
@@ -45,7 +45,7 @@ ifndef NNVM_PATH
 endif
 
 ifndef DLPACK_PATH
-   DLPACK_PATH = $(ROOTDIR)/dlpack
+   DLPACK_PATH = $(ROOTDIR)/3rdparty/dlpack
 endif
 
 ifndef AMALGAMATION_PATH
diff --git a/amalgamation/Makefile b/amalgamation/Makefile
index bd9403e620c..effb3db751d 100644
--- a/amalgamation/Makefile
+++ b/amalgamation/Makefile
@@ -80,7 +80,7 @@ mxnet_predict0.d: mxnet_predict0.cc nnvm.d dmlc.d
${CXX} ${CFLAGS} -M -MT mxnet_predict0.o \
-I ${MXNET_ROOT}/ -I ${MXNET_ROOT}/mshadow/ -I 
${MXNET_ROOT}/dmlc-core/include -I ${MXNET_ROOT}/dmlc-core/src \
-I ${MXNET_ROOT}/nnvm/include \
-   -I ${MXNET_ROOT}/dlpack/include \
+   -I ${MXNET_ROOT}/3rdparty/dlpack/include \
-I ${MXNET_ROOT}/include \
-D__MIN__=$(MIN) mxnet_predict0.cc > mxnet_predict0.d
cat dmlc.d >> mxnet_predict0.d
diff --git a/tools/license_header.py b/tools/license_header.py
index 14d3f587d93..039872dd44f 100755
--- a/tools/license_header.py
+++ b/tools/license_header.py
@@ -63,7 +63,7 @@
 _WHITE_LIST = ['R-package/',
'cub/',
'docker/Dockerfiles',
-   'dlpack/',
+   #'dlpack/',
'dmlc-core/',
'mshadow/',
'nnvm',


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] marcoabreu commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move submodules to 3rdparty

[GitBox]

marcoabreu commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move submodules 
to 3rdparty
URL: https://github.com/apache/incubator-mxnet/pull/10112#issuecomment-373435280
 
 
   Hmm, doesn't work either.. Can you try opening a new PR?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move submodules to 3rdparty

[GitBox]

eric-haibin-lin commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move 
submodules to 3rdparty
URL: https://github.com/apache/incubator-mxnet/pull/10112#issuecomment-373427664
 
 
   Oh right, I didn't notice my branch is already behind master. Will do rebase


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] marcoabreu commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move submodules to 3rdparty

[GitBox]

marcoabreu commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move submodules 
to 3rdparty
URL: https://github.com/apache/incubator-mxnet/pull/10112#issuecomment-373407753
 
 
   No, I don't know if that's happening. Please try rebasing within the same PR 
and see if it succeedes. If it doesn't, but works in a new PR, then I will have 
to take action


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] marcoabreu commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move submodules to 3rdparty

[GitBox]

marcoabreu commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move submodules 
to 3rdparty
URL: https://github.com/apache/incubator-mxnet/pull/10112#issuecomment-373407385
 
 
   CI merges automatically if your branch diverged from the master. Since your 
branch is behind it, it will try to merge your commit. Please try to rebase


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move submodules to 3rdparty

[GitBox]

eric-haibin-lin commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move 
submodules to 3rdparty
URL: https://github.com/apache/incubator-mxnet/pull/10112#issuecomment-373406907
 
 
   You mean CI may not clean up the workspace if it is corrupted in some case? 
If so maybe we want to prevent it from happening next time 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eric-haibin-lin commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move submodules to 3rdparty

[GitBox]

eric-haibin-lin commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move 
submodules to 3rdparty
URL: https://github.com/apache/incubator-mxnet/pull/10112#issuecomment-373406256
 
 
   No, I didn’t merge anything. I just added two commits on top of master


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] BTaskaya closed pull request #9603: Btaskaya/stringformatting

[GitBox]

BTaskaya closed pull request #9603: Btaskaya/stringformatting
URL: https://github.com/apache/incubator-mxnet/pull/9603
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] meanmee commented on issue #9763: Too many header files need to be included when using C++ api

[GitBox]

meanmee commented on issue #9763: Too many header files need to be included 
when using C++ api
URL: 
https://github.com/apache/incubator-mxnet/issues/9763#issuecomment-373382367
 
 
   error:
   ../../include/mxnet-cpp/optimizer.hpp:37:26: fatal error: mxnet-cpp/op.h: No 
such file or directory
   
   But I can't find any files named op.h in mxnet-cpp, is this a bug?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nihui commented on issue #10098: [MXNET-97] implement DepthwiseConv2dBackwardFilterKernel from tensorflow codebase

[GitBox]

nihui commented on issue #10098: [MXNET-97] implement 
DepthwiseConv2dBackwardFilterKernel from tensorflow codebase
URL: https://github.com/apache/incubator-mxnet/pull/10098#issuecomment-373380857
 
 
   @chinakook hi, the special path for small maps has already been implemented 
in mxnet.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nihui commented on issue #10098: [MXNET-97] implement DepthwiseConv2dBackwardFilterKernel from tensorflow codebase

[GitBox]

nihui commented on issue #10098: [MXNET-97] implement 
DepthwiseConv2dBackwardFilterKernel from tensorflow codebase
URL: https://github.com/apache/incubator-mxnet/pull/10098#issuecomment-373380857
 
 
   @chinakook hi, the special path for small maps has already implemented in 
mxnet.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] wkcn commented on issue #9939: [MXNET-40]add multi proposal operator (cpu version) and fix the bug in proposal op (gpu version)

[GitBox]

wkcn commented on issue #9939: [MXNET-40]add multi proposal operator (cpu 
version) and fix the bug in proposal op (gpu version)
URL: https://github.com/apache/incubator-mxnet/pull/9939#issuecomment-373370119
 
 
   @marcoabreu @szha @cjolivier01 
   
   Hello!
   
   For Changes Requested, I have removed the USE_STABLE_SORT_FOR_PROPOSAL flag 
and recovered the building setting.
   
   Could you please review and merge the PR?
   
   Thank you!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] marcoabreu commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move submodules to 3rdparty

[GitBox]

marcoabreu commented on issue #10112: [MXNET-16] [DO NOT MERGE] Move submodules 
to 3rdparty
URL: https://github.com/apache/incubator-mxnet/pull/10112#issuecomment-373366285
 
 
   Git seems to have a problem if you try changing a 3rd party module and merge 
at the same time. Could you try to rebase to make sure no merge happens? 
Additionally, it could be possible that this PR left the workspace (these 
actions happen on the master) in a corrupt state. In that case, please open a 
new PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] chinakook commented on issue #10098: [MXNET-97] implement DepthwiseConv2dBackwardFilterKernel from tensorflow codebase

[GitBox]

chinakook commented on issue #10098: [MXNET-97] implement 
DepthwiseConv2dBackwardFilterKernel from tensorflow codebase
URL: https://github.com/apache/incubator-mxnet/pull/10098#issuecomment-373362959
 
 
   Tensorflow has also a 32x32 DepthwiseConv when feature maps are small. Could 
you add that too?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] chinakook commented on a change in pull request #10120: Update profiler example code

[GitBox]

chinakook commented on a change in pull request #10120: Update profiler example 
code
URL: https://github.com/apache/incubator-mxnet/pull/10120#discussion_r174761552
 
 

 ##
 File path: docs/faq/perf.md
 ##
 @@ -228,12 +228,12 @@ See 
[example/profiler](https://github.com/dmlc/mxnet/tree/master/example/profile
 for complete examples of how to use the profiler in code, but briefly, the 
Python code looks like:
 
 ```
-mx.profiler.profiler_set_config(mode='all', filename='profile_output.json')
-mx.profiler.profiler_set_state('run')
+mx.profiler.set_config(profile_all=True, filename='profile_output.json')
+mx.profiler.set_state('run')
 
 Review comment:
   ```https://github.com/apache/incubator-mxnet/tree/master/example/profiler``` 
also need to change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] gydlcc commented on issue #10102: how to install version of 0.9.3

[GitBox]

gydlcc commented on issue #10102: how to install version of 0.9.3
URL: 
https://github.com/apache/incubator-mxnet/issues/10102#issuecomment-373301437
 
 
   Thanks pretty much, i get it


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services