[GitHub] lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing maintenance on NDArray
lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing
maintenance on NDArray
URL: https://github.com/apache/incubator-mxnet/pull/11126#discussion_r194818676
##
File path:
scala-package/core/src/main/scala/org/apache/mxnet/annotation/Experimental.scala
##
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mxnet.annotation
+
+import java.lang.annotation.{ElementType, Retention, Target, _}
+
+@Retention(RetentionPolicy.RUNTIME)
+@Target(Array(ElementType.TYPE, ElementType.FIELD, ElementType.METHOD,
ElementType.PARAMETER,
+ ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE))
+class Experimental {}
Review comment:
Pushed into backlog now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing maintenance on NDArray
lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing
maintenance on NDArray
URL: https://github.com/apache/incubator-mxnet/pull/11126#discussion_r194804382
##
File path:
scala-package/core/src/main/scala/org/apache/mxnet/annotation/Experimental.scala
##
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mxnet.annotation
+
+import java.lang.annotation.{ElementType, Retention, Target, _}
+
+@Retention(RetentionPolicy.RUNTIME)
+@Target(Array(ElementType.TYPE, ElementType.FIELD, ElementType.METHOD,
ElementType.PARAMETER,
+ ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE))
+class Experimental {}
Review comment:
@nswamy , Scala does not support interface, we need to use an Class to
replace that. Shall we follow the "Scala way" to do the annotation rather than
java?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing maintenance on NDArray
lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing
maintenance on NDArray
URL: https://github.com/apache/incubator-mxnet/pull/11126#discussion_r193255902
##
File path:
scala-package/macros/src/main/scala/org/apache/mxnet/APIDocGenerator.scala
##
@@ -97,9 +97,11 @@ private[mxnet] object APIDocGenerator{
argDef += "name : String = null"
argDef += "attr : Map[String, String] = null"
} else {
+ argDef += "out : Option[NDArray] = None"
returnType = "org.apache.mxnet.NDArrayFuncReturn"
}
-s"def ${func.name} (${argDef.mkString(", ")}) : ${returnType}"
+val experimentalTag = "@Experimental"
Review comment:
How about print("Caution! This is an Experimental function") in every
generated methods?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing maintenance on NDArray
lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing
maintenance on NDArray
URL: https://github.com/apache/incubator-mxnet/pull/11126#discussion_r192968711
##
File path:
scala-package/macros/src/main/scala/org/apache/mxnet/NDArrayMacro.scala
##
@@ -129,6 +119,8 @@ private[mxnet] object NDArrayMacro {
}
impl += base
})
+ // add default out parameter
+ argDef += "out : Option[NDArray] = None"
Review comment:
The usage is to save memory space, you pass in a pointer and C will store
the value in there...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing maintenance on NDArray
lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing
maintenance on NDArray
URL: https://github.com/apache/incubator-mxnet/pull/11126#discussion_r192968485
##
File path:
scala-package/core/src/main/scala/org/apache/mxnet/ExperimentalTag.scala
##
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mxnet
+
+import java.lang.annotation.{ElementType, Retention, Target, _}
+
+@Retention(RetentionPolicy.RUNTIME)
Review comment:
No implementation, I follow the Apache Spark notations to add Experimental
tag here
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing maintenance on NDArray
lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing
maintenance on NDArray
URL: https://github.com/apache/incubator-mxnet/pull/11126#discussion_r192968334
##
File path: scala-package/core/src/main/scala/org/apache/mxnet/NDArray.scala
##
@@ -64,22 +64,36 @@ object NDArray {
val function = functions(funcName)
val ndArgs = ArrayBuffer.empty[NDArray]
val posArgs = ArrayBuffer.empty[String]
-args.foreach {
- case arr: NDArray =>
-ndArgs.append(arr)
- case arrFunRet: NDArrayFuncReturn =>
-arrFunRet.arr.foreach(ndArgs.append(_))
- case arg =>
-posArgs.append(arg.toString)
+if (args != null) {
+ args.foreach {
+case arr: NDArray =>
+ ndArgs.append(arr)
+case arrFunRet: NDArrayFuncReturn =>
+ arrFunRet.arr.foreach(ndArgs.append(_))
+case arg =>
+ posArgs.append(arg.toString)
+ }
}
-
require(posArgs.length <= function.arguments.length,
s"len(posArgs) = ${posArgs.length}, should be less or equal to
len(arguments) " +
s"= ${function.arguments.length}")
val updatedKwargs: Map[String, String] =
(Option(kwargs).getOrElse(Map.empty[String, String])
++ function.arguments.slice(0, posArgs.length).zip(posArgs) - "out"
- ).map { case (k, v) => k -> v.toString }
+ ).filter{ case (key, value) =>
+!value.isInstanceOf[NDArray] && !value.isInstanceOf[NDArrayFuncReturn]
Review comment:
For new API, NDarray are mixed in the kwargs, we don’t want these to become
string... instead to store into a ndArgs list
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing maintenance on NDArray
lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing
maintenance on NDArray
URL: https://github.com/apache/incubator-mxnet/pull/11126#discussion_r192968127
##
File path: scala-package/core/src/main/scala/org/apache/mxnet/NDArray.scala
##
@@ -537,6 +551,10 @@ object NDArray {
new NDArray(handleRef.value)
}
+ private def _crop_assign(kwargs: Map[String, Any] = null)(args: Any*) :
NDArrayFuncReturn = {
Review comment:
Since this function is the only underscore function used for old api, I
choose to block the rest underscore function and keep it there.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing maintenance on NDArray
lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing
maintenance on NDArray
URL: https://github.com/apache/incubator-mxnet/pull/11126#discussion_r192967931
##
File path:
scala-package/macros/src/main/scala/org/apache/mxnet/APIDocGenerator.scala
##
@@ -97,9 +97,11 @@ private[mxnet] object APIDocGenerator{
argDef += "name : String = null"
argDef += "attr : Map[String, String] = null"
} else {
+ argDef += "out : Option[NDArray] = None"
returnType = "org.apache.mxnet.NDArrayFuncReturn"
}
-s"def ${func.name} (${argDef.mkString(", ")}) : ${returnType}"
+val experimentalTag = "@Experimental"
Review comment:
Add experimental to label API is experimental...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing maintenance on NDArray
lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing
maintenance on NDArray
URL: https://github.com/apache/incubator-mxnet/pull/11126#discussion_r192967726
##
File path: scala-package/core/src/main/scala/org/apache/mxnet/NDArray.scala
##
@@ -105,6 +105,57 @@ object NDArray {
})
}
+ /**
+* Used by NDArrayMacro for New Scala NDArray API.
+* Invoke this function by passing in parameters.
+* Parameters
+* --
+* @param kwargs Key-value arguments of input scalars
+* @return The result NDArrays of result of computation.
+*/
+ private[mxnet] def genericNewAPINDArrayFunctionInvoke(
+funcName: String, kwargs: Map[String, Any] = null) : NDArrayFuncReturn = {
+val function = functions(funcName)
+val ndArgs = ArrayBuffer.empty[NDArray]
+val updatedKwargs: Map[String, String] =
+ Option(kwargs).getOrElse(Map.empty[String, String]).filter{ case (key,
value) =>
+!value.isInstanceOf[NDArray] && !value.isInstanceOf[NDArrayFuncReturn]
+ } .map { case (k, v) => k -> v.toString }
+
+Option(kwargs).getOrElse(Map.empty[String, String]).filter{ case (key,
value) =>
+ value.isInstanceOf[NDArray] || value.isInstanceOf[NDArrayFuncReturn]
+} .filter{ case (key, value) => key != "out"}.foreach{
+ case (key, value) => value match {
+case nd : NDArray =>
+ ndArgs.append(nd.asInstanceOf[NDArray])
+case arrFunRet: NDArrayFuncReturn =>
+
arrFunRet.asInstanceOf[NDArrayFuncReturn].arr.foreach(ndArgs.append(_))
+ }
+}
Review comment:
I see your idea, two ways come in my mind. Change Macros design using Ndargs
to store all NDArray element to match the order and store others into kv store.
Second way I think we may need to find another C function to handle this method
to ensure the correct order
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing maintenance on NDArray
lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing
maintenance on NDArray
URL: https://github.com/apache/incubator-mxnet/pull/11126#discussion_r192605853
##
File path: scala-package/core/src/main/scala/org/apache/mxnet/NDArray.scala
##
@@ -105,6 +105,57 @@ object NDArray {
})
}
+ /**
+* Used by NDArrayMacro for New Scala NDArray API.
+* Invoke this function by passing in parameters.
+* Parameters
+* --
+* @param kwargs Key-value arguments of input scalars
+* @return The result NDArrays of result of computation.
+*/
+ private[mxnet] def genericNewAPINDArrayFunctionInvoke(
+funcName: String, kwargs: Map[String, Any] = null) : NDArrayFuncReturn = {
+val function = functions(funcName)
+val ndArgs = ArrayBuffer.empty[NDArray]
+val updatedKwargs: Map[String, String] =
+ Option(kwargs).getOrElse(Map.empty[String, String]).filter{ case (key,
value) =>
+!value.isInstanceOf[NDArray] && !value.isInstanceOf[NDArrayFuncReturn]
+ } .map { case (k, v) => k -> v.toString }
+
+Option(kwargs).getOrElse(Map.empty[String, String]).filter{ case (key,
value) =>
+ value.isInstanceOf[NDArray] || value.isInstanceOf[NDArrayFuncReturn]
+} .filter{ case (key, value) => key != "out"}.foreach{
+ case (key, value) => value match {
+case nd : NDArray =>
+ ndArgs.append(nd.asInstanceOf[NDArray])
+case arrFunRet: NDArrayFuncReturn =>
+
arrFunRet.asInstanceOf[NDArrayFuncReturn].arr.foreach(ndArgs.append(_))
+ }
+}
Review comment:
It's working now, send the PR...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing maintenance on NDArray
lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing
maintenance on NDArray
URL: https://github.com/apache/incubator-mxnet/pull/11126#discussion_r192571898
##
File path: scala-package/core/src/main/scala/org/apache/mxnet/NDArray.scala
##
@@ -105,6 +105,57 @@ object NDArray {
})
}
+ /**
+* Used by NDArrayMacro for New Scala NDArray API.
+* Invoke this function by passing in parameters.
+* Parameters
+* --
+* @param kwargs Key-value arguments of input scalars
+* @return The result NDArrays of result of computation.
+*/
+ private[mxnet] def genericNewAPINDArrayFunctionInvoke(
+funcName: String, kwargs: Map[String, Any] = null) : NDArrayFuncReturn = {
+val function = functions(funcName)
+val ndArgs = ArrayBuffer.empty[NDArray]
+val updatedKwargs: Map[String, String] =
+ Option(kwargs).getOrElse(Map.empty[String, String]).filter{ case (key,
value) =>
+!value.isInstanceOf[NDArray] && !value.isInstanceOf[NDArrayFuncReturn]
+ } .map { case (k, v) => k -> v.toString }
+
+Option(kwargs).getOrElse(Map.empty[String, String]).filter{ case (key,
value) =>
+ value.isInstanceOf[NDArray] || value.isInstanceOf[NDArrayFuncReturn]
+} .filter{ case (key, value) => key != "out"}.foreach{
+ case (key, value) => value match {
+case nd : NDArray =>
+ ndArgs.append(nd.asInstanceOf[NDArray])
+case arrFunRet: NDArrayFuncReturn =>
+
arrFunRet.asInstanceOf[NDArrayFuncReturn].arr.foreach(ndArgs.append(_))
+ }
+}
Review comment:
Last time when I try to combine them, some strange issue appeared. It breaks
the original api. I will try a second time to combine them and send error log
if I failed again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing maintenance on NDArray
lanking520 commented on a change in pull request #11126: [MXNET-386] ongoing
maintenance on NDArray
URL: https://github.com/apache/incubator-mxnet/pull/11126#discussion_r192571877
##
File path:
scala-package/macros/src/main/scala/org/apache/mxnet/NDArrayMacro.scala
##
@@ -57,14 +57,13 @@ private[mxnet] object NDArrayMacro {
val newNDArrayFunctions = {
if (isContrib) ndarrayFunctions.filter(_.name.startsWith("_contrib_"))
- else ndarrayFunctions.filter(!_.name.startsWith("_contrib_"))
+ else ndarrayFunctions.filterNot(_.name.startsWith("_"))
Review comment:
Yes, if you enable the contrib operators.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
