[incubator-mxnet] branch master updated (7fde59a -> be92cc6)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


from 7fde59a  Fix formatting and typos in CD README.md (#16703)
 add be92cc6  change the wording and log level to be more in line with the 
general use (#16626)

No new revisions were added by this update.

Summary of changes:
 .../core/src/main/scala/org/apache/mxnet/Base.scala | 13 ++---
 1 file changed, 6 insertions(+), 7 deletions(-)

[incubator-mxnet] branch master updated (1210c7d -> 904ba00)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


from 1210c7d  Fallback to official NumPy operators (#16698)
 add 904ba00  improve example docs with installation instructions (#16625)

No new revisions were added by this update.

Summary of changes:
 contrib/clojure-package/examples/bert/README.md   |  7 +++
 .../examples/cnn-text-classification/README.md|  7 +++
 contrib/clojure-package/examples/gan/README.md|  7 +++
 contrib/clojure-package/examples/imclassification/README.md   | 11 +++
 contrib/clojure-package/examples/multi-label/README.md|  9 +
 contrib/clojure-package/examples/neural-style/README.md   |  7 +++
 contrib/clojure-package/examples/pre-trained-models/README.md |  7 +++
 contrib/clojure-package/examples/profiler/README.md   | 11 ++-
 contrib/clojure-package/examples/rnn/README.md|  7 +++
 contrib/clojure-package/examples/tutorial/README.md   |  9 +
 contrib/clojure-package/examples/visualization/README.md  | 10 ++
 11 files changed, 91 insertions(+), 1 deletion(-)

[incubator-mxnet] branch master updated (1210c7d -> 904ba00)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


from 1210c7d  Fallback to official NumPy operators (#16698)
 add 904ba00  improve example docs with installation instructions (#16625)

No new revisions were added by this update.

Summary of changes:
 contrib/clojure-package/examples/bert/README.md   |  7 +++
 .../examples/cnn-text-classification/README.md|  7 +++
 contrib/clojure-package/examples/gan/README.md|  7 +++
 contrib/clojure-package/examples/imclassification/README.md   | 11 +++
 contrib/clojure-package/examples/multi-label/README.md|  9 +
 contrib/clojure-package/examples/neural-style/README.md   |  7 +++
 contrib/clojure-package/examples/pre-trained-models/README.md |  7 +++
 contrib/clojure-package/examples/profiler/README.md   | 11 ++-
 contrib/clojure-package/examples/rnn/README.md|  7 +++
 contrib/clojure-package/examples/tutorial/README.md   |  9 +
 contrib/clojure-package/examples/visualization/README.md  | 10 ++
 11 files changed, 91 insertions(+), 1 deletion(-)

[incubator-mxnet] 01/01: change the wording and log level to be more in line with the general use case of loading from the jar

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch change-warning-on-base-loading-of-scala-native
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 2533bd6e70b824b8158195d0b54daf82af85d7ec
Author: gigasquid 
AuthorDate: Fri Oct 25 11:44:09 2019 -0400

change the wording and log level to be more in line with the general use
case of loading from the jar
---
 .../core/src/main/scala/org/apache/mxnet/Base.scala | 13 ++---
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/Base.scala 
b/scala-package/core/src/main/scala/org/apache/mxnet/Base.scala
index 001bd04..c3378ec 100644
--- a/scala-package/core/src/main/scala/org/apache/mxnet/Base.scala
+++ b/scala-package/core/src/main/scala/org/apache/mxnet/Base.scala
@@ -62,18 +62,17 @@ private[mxnet] object Base {
   // The primitives currently supported for NDArray operations
   val MX_PRIMITIVES = new Group ((Double, Float))
 
+
+  /* Find the native libray either on the path or copy it from
+   * the jar in the dependency
+   * jar into a temp directory and load it
+   */
   try {
 try {
   tryLoadLibraryOS("mxnet-scala")
 } catch {
   case e: UnsatisfiedLinkError =>
-logger.warn("MXNet Scala native library not found in path. " +
-  "Copying native library from the archive. " +
-  "Consider installing the library somewhere in the path " +
-  "(for Windows: PATH, for Linux: LD_LIBRARY_PATH), " +
-  "or specifying by Java cmd option -Djava.library.path=[lib path].")
-logger.warn("LD_LIBRARY_PATH=" + System.getenv("LD_LIBRARY_PATH"))
-logger.warn("java.library.path=" + 
System.getProperty("java.library.path"))
+logger.info("Copying and loading native library from the jar archive")
 NativeLibraryLoader.loadLibrary("mxnet-scala")
 }
   } catch {

[incubator-mxnet] branch change-warning-on-base-loading-of-scala-native created (now 2533bd6)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch change-warning-on-base-loading-of-scala-native
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at 2533bd6  change the wording and log level to be more in line with the 
general use case of loading from the jar

This branch includes the following new commits:

 new 2533bd6  change the wording and log level to be more in line with the 
general use case of loading from the jar

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-mxnet] branch clojure-release-script created (now 1a8f72d)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-release-script
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at 1a8f72d  validation

This branch includes the following new commits:

 new ff5759b  add repostiories for deployment
 new 92b7351  wip
 new edcf660  Get the deploy side working - next will come the validate of 
the deployed jar
 new 32c06fe  make next steps more clear
 new 1a8f72d  validation

The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-mxnet] 04/05: make next steps more clear

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-release-script
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 32c06feb26d69576a7a4dc923ad67c710aa7db2c
Author: gigasquid 
AuthorDate: Sat Oct 26 08:06:57 2019 -0400

make next steps more clear
---
 contrib/clojure-package/src/dev/package_release.clj | 1 +
 1 file changed, 1 insertion(+)

diff --git a/contrib/clojure-package/src/dev/package_release.clj 
b/contrib/clojure-package/src/dev/package_release.clj
index 541fe6d..433f0f0 100644
--- a/contrib/clojure-package/src/dev/package_release.clj
+++ b/contrib/clojure-package/src/dev/package_release.clj
@@ -145,6 +145,7 @@
 (println "Please validate that logs are all correct and the jar looks 
good.")
 (print "Then run the following command to push the jar to apache staging:")
 (println "   lein deploy :staging")
+(println "   \n\n\n\n")
 (flush)))
 
 (defn run-build [args]

[incubator-mxnet] 02/05: wip

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-release-script
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 92b73514ad522517790b74cd5d444a5ea1eb6a8c
Author: gigasquid 
AuthorDate: Fri Oct 25 16:08:55 2019 -0400

wip
---
 .../examples/imclassification/project.clj  |   4 +-
 .../clojure-package/src/dev/package_release.clj| 224 +
 2 files changed, 227 insertions(+), 1 deletion(-)

diff --git a/contrib/clojure-package/examples/imclassification/project.clj 
b/contrib/clojure-package/examples/imclassification/project.clj
index 702a33d..08e7ae3 100644
--- a/contrib/clojure-package/examples/imclassification/project.clj
+++ b/contrib/clojure-package/examples/imclassification/project.clj
@@ -21,4 +21,6 @@
   :dependencies [[org.clojure/clojure "1.9.0"]
  [org.apache.mxnet.contrib.clojure/clojure-mxnet 
"1.6.0-SNAPSHOT"]]
   :pedantic? :skip
-  :main imclassification.train-mnist)
+  :main imclassification.train-mnist
+  :repositories [["download-staging" {:url 
"https://repository.apache.org/content/repositories/staging;  
:snapshots true
+:update :always}]])
diff --git a/contrib/clojure-package/src/dev/package_release.clj 
b/contrib/clojure-package/src/dev/package_release.clj
new file mode 100644
index 000..cc3c946
--- /dev/null
+++ b/contrib/clojure-package/src/dev/package_release.clj
@@ -0,0 +1,224 @@
+(ns dev.package-release
+  (:require [clojure.java.io :as io]
+[clojure.edn :as edn]
+[clojure.java.shell :refer [sh]]
+[clojure.spec.alpha :as s]
+[clojure.string :as string]
+[dev.generator]))
+
+(s/def ::build-type #{:osx-cpu :linux-gpu :linux-cpu})
+(s/def ::release-number string?)
+(s/def ::apache-gpg-key string?)
+(s/def ::release-args (s/keys :req-un [::build-type ::release-number
+   ::apache-gpg-key]))
+
+(def scala-jar-name
+  {:linux-cpu "org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu"
+   :linux-gpu "org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu"
+   :osx-cpu "org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu"})
+
+(defn backup-project-file
+  "Backs up a copy of the original project file"
+  []
+  (do
+(sh "cp" "project.clj" "project.clj.bak")
+(sh "cp" "examples/imclassification/project.clj" 
"examples/imclassification/project.clj.bak")))
+
+(defn make-new-project-file
+  "Creates a new project file for the relase by
+  changing the build type and the version along
+  with the release number and the signer's public key
+  which can be an email"
+  [{:keys [build-type release-number apache-gpg-key]}]
+
+  (let [project-data (read-string (slurp "project.clj"))
+project-header (take 3 project-data)
+project-rest-map (->> project-data
+  (drop 3)
+  (apply hash-map))
+new-header `[~'defproject
+ ~(symbol (str 
"org.apache.mxnet.contrib.clojure/clojure-mxnet"
+   "-" (name build-type)))
+ ~release-number]
+new-dependencies (->> (:dependencies project-rest-map)
+  (remove #(string/includes? % "org.apache.mxnet"))
+  (into [[(symbol (scala-jar-name build-type)) 
release-number]]))
+new-project-rest-map (-> project-rest-map
+ (assoc :dependencies new-dependencies)
+ (assoc-in [:deploy-repositories 0 1 
:signing :gpg-key]
+   apache-gpg-key))]
+
+(as-> (into [] new-project-rest-map) p
+  (into new-header (mapcat identity p))
+  (apply list p)
+  (with-out-str (clojure.pprint/pprint p))
+  (str dev.generator/license p)
+  (spit "project.clj" p
+
+(defn run-commands
+  "Run shell commands and report exit and out"
+  ([commands text]
+   (commands text nil))
+  ([commands text dir]
+   (do
+ (println "=  " text  "  =")
+ (println "Please wait ")
+ (flush))
+   (let [{:keys [out exit err]} (apply sh (if dir
+(into commands [:dir dir])
+commands))]
+ (do
+   (println out)
+   (flush)
+   (when-not (zero? exit)
+ (println "Errors:")
+ (println err)
+ (flush))
+   (zero? exit)
+
+(defn reset-project-files
+  "Resets the projects files back to the original saved version"
+  []
+  (d

[incubator-mxnet] 05/05: validation

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-release-script
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 1a8f72dd76b5dd76315270b221ca496d978dd872
Author: gigasquid 
AuthorDate: Sat Oct 26 08:56:24 2019 -0400

validation
---
 .../clojure-package/src/dev/package_release.clj| 70 +-
 1 file changed, 56 insertions(+), 14 deletions(-)

diff --git a/contrib/clojure-package/src/dev/package_release.clj 
b/contrib/clojure-package/src/dev/package_release.clj
index 433f0f0..51a5c6c 100644
--- a/contrib/clojure-package/src/dev/package_release.clj
+++ b/contrib/clojure-package/src/dev/package_release.clj
@@ -12,6 +12,8 @@
 (s/def ::release-args (s/keys :req-un [::build-type ::release-number
::apache-gpg-key]))
 
+(s/def ::validate-args (s/keys :req-un [::build-type ::release-number]))
+
 (def scala-jar-name
   {:linux-cpu "org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu"
:linux-gpu "org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu"
@@ -99,23 +101,27 @@
   "This will move maven downloads of the release to test the
   download of the deployed jar from staging"
   [build-type release-number]
-  (let [m2-path (str (System/getProperty "user.home")
+  (let [mv-id (str release-number "-" (java.util.UUID/randomUUID) "-bak")
+m2-path (str (System/getProperty "user.home")
  
"/.m2/repository/org/apache/mxnet/contrib/clojure/clojure-mxnet"
- "-" build-type "/"
- release-number)
+ "-" build-type)
 files-to-be-cleared (:out (sh "ls" m2-path))]
-(when files-to-be-cleared
+(when-not (string/blank? files-to-be-cleared)
   (println "Installed m2 jars found")
   (print files-to-be-cleared)
-  (println "Do you want to mv the the m2 paths of " m2-path "to" (str 
m2-path "-bak") "?")
+  (println "Do you want to mv the dir " release-number " in " m2-path "to" 
mv-id)
   (println "Type \"yes\" to confirm: ")
   (flush)
   (let [answer (read-line)]
 (if (= "yes" answer)
-  (println (sh "mv" m2-path (str m2-path "-bak")))
+  (println (sh "mv" release-number mv-id :dir m2-path))
   (println "Not moving"))
 
-(defn test-installed-jar [build-type release-number]
+
+(defn test-installed-jar
+  "Tests the example with a jar locally if in maven
+  or downloads from the apache repo if needed"
+  [build-type release-number]
   (let [project-data (read-string (slurp 
"./examples/imclassification/project.clj"))
 project-header (into [] (take 3 project-data))
 project-rest-map (->> project-data
@@ -133,9 +139,13 @@
 (with-out-str (clojure.pprint/pprint p))
 (str dev.generator/license p)
 (spit "./examples/imclassification/project.clj" p))
-  (run-commands ["lein" "run"]
+  (run-commands ["lein" "run" ":cpu"]
 "Running Image Classification Example"
-"examples/imclassification"
+"examples/imclassification")
+  (when (= :linux-gpu build-type)
+(run-commands ["lein" "run" ":gpu"]
+  "Running Image Classification Example"
+  "examples/imclassification")
 
 (defn print-deploy-instructions []
   (do
@@ -148,8 +158,15 @@
 (println "   \n\n\n\n")
 (flush)))
 
+(defn print-test-deploy-conf []
+  (do
+(println "**")
+(println "*SUCCESS!!!*")
+(println "**")
+(flush)))
+
 (defn run-build [args]
-  (let [[build-type release-number apache-gpg-key] args
+  (let [[_ build-type release-number apache-gpg-key] args
 release-args {:build-type (keyword build-type)
   :release-number release-number
   :apache-gpg-key apache-gpg-key}]
@@ -166,13 +183,38 @@
 (System/exit 0))
   (do
 (println "Error with Args" release-args)
-(s/explain ::release-args release-args))
-  )))
+(s/explain ::release-args release-args)
+
+(defn validate-deploy [args]
+  (let [[_ build-type release-number] args
+validate-args {:build-type (keyword build-type)
+  :release-number release-number}]
+
+(if (s/valid? ::validate-args validate-args)
+  (do
+(clean-m2-release build-type releas

[incubator-mxnet] 01/05: add repostiories for deployment

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-release-script
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit ff5759b01920e3378ca76140aa57ae7e35e37698
Author: gigasquid 
AuthorDate: Fri Oct 25 14:37:43 2019 -0400

add repostiories for deployment
---
 contrib/clojure-package/project.clj | 14 +++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/contrib/clojure-package/project.clj 
b/contrib/clojure-package/project.clj
index 718fbf6..3bbb6d4 100644
--- a/contrib/clojure-package/project.clj
+++ b/contrib/clojure-package/project.clj
@@ -41,7 +41,15 @@
 [lein-cljfmt "0.5.7"]]
   :codox {:namespaces [#"^org\.apache\.clojure-mxnet\.(?!gen).*"]}
   :aot [dev.generator]
-  :repositories [["staging" {:url 
"https://repository.apache.org/content/repositories/staging;  
:snapshots true
- :update :always}]
+  :deploy-repositories [["staging" {:sign-releases true
+:url 
"https://repository.apache.org/service/local/staging/deploy/maven2;
+:creds :gpg
+:signing {:gpg-key "your key"} ;; this can 
be email of key-id
+:snapshots true
+:checksum :fail
+:update :always
+:releases {:checksum :fail :update 
:always}}]]
+  :repositories [["download-staging" {:url 
"https://repository.apache.org/content/repositories/staging;  
:snapshots true
+:update :always}]
  ["snapshots" {:url 
"https://repository.apache.org/content/repositories/snapshots;   
:snapshots true
-   :update :always}]])
+  :update :always}]])

[incubator-mxnet] 03/05: Get the deploy side working - next will come the validate of the deployed jar

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-release-script
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit edcf660e647c527100ca4e66ac79c5f2819eee25
Author: gigasquid 
AuthorDate: Sat Oct 26 07:56:36 2019 -0400

Get the deploy side working
- next will come the validate of the deployed jar
---
 .../clojure-package/scripts/setup_deploy_env.sh| 28 +++
 .../clojure-package/src/dev/package_release.clj| 91 ++
 2 files changed, 50 insertions(+), 69 deletions(-)

diff --git a/contrib/clojure-package/scripts/setup_deploy_env.sh 
b/contrib/clojure-package/scripts/setup_deploy_env.sh
new file mode 100755
index 000..ab3ae89
--- /dev/null
+++ b/contrib/clojure-package/scripts/setup_deploy_env.sh
@@ -0,0 +1,28 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Run this from the main Clojure project directory with 2 arguments
+# old-version and new-version
+# Ex: scripts/update_version 1.6.0-SNAPSHOT 1.6.0-SNAPSHOT
+
+set -evx
+echo "Setting up env for deploy"
+export GPG_TTY=$(tty)
+echo "Done!"
+
diff --git a/contrib/clojure-package/src/dev/package_release.clj 
b/contrib/clojure-package/src/dev/package_release.clj
index cc3c946..541fe6d 100644
--- a/contrib/clojure-package/src/dev/package_release.clj
+++ b/contrib/clojure-package/src/dev/package_release.clj
@@ -58,7 +58,7 @@
 (defn run-commands
   "Run shell commands and report exit and out"
   ([commands text]
-   (commands text nil))
+   (run-commands commands text nil))
   ([commands text dir]
(do
  (println "=  " text  "  =")
@@ -76,14 +76,6 @@
  (flush))
(zero? exit)
 
-(defn reset-project-files
-  "Resets the projects files back to the original saved version"
-  []
-  (do
-(sh "cp" "project.clj" "new-project-clj")
-(sh "cp" "project.clj.bak" "project.clj")
-(sh "cp" "examples/imclassification/project.clj.bak" 
"examples/imclassification/project.clj")))
-
 (defn run-tests-and-install
   "With the project file in place, use lein to
   dowload deps, run tests and then install the generated
@@ -145,80 +137,41 @@
 "Running Image Classification Example"
 "examples/imclassification"
 
+(defn print-deploy-instructions []
+  (do
+(println "**")
+(println "*READY FOR DEPLOY*")
+(println "**")
+(println "Please validate that logs are all correct and the jar looks 
good.")
+(print "Then run the following command to push the jar to apache staging:")
+(println "   lein deploy :staging")
+(flush)))
 
 (defn run-build [args]
   (let [[build-type release-number apache-gpg-key] args
 release-args {:build-type (keyword build-type)
   :release-number release-number
   :apache-gpg-key apache-gpg-key}]
-
+
 (if (s/valid? ::release-args release-args)
-  (try
-(do
-  (test-installed-jar build-type release-number)
-  (System/exit 0)
-  )
-#_(do
-(make-new-project-file {:build-type :osx-cpu
-:release-number "1.5.1"
-:apache-gpg-key "cme...@apache.org"} )
-(run-tests-and-install)
-(System/exit 0)
-)
-(finally (reset-project-files)))
+  (do
+(make-new-project-file {:build-type :osx-cpu
+:release-number "1.5.1"
+:apache-gpg-key "cme...@apache.org"} )
+(run-tests-and-install)
+(test-installed-jar build-type release-number)
+(run-commands ["scripts/setup_deploy_env.sh"] "Setting up display for 
deploy")
+(print-deploy-instructions)
+(System/ex

[incubator-mxnet] branch clojure-release-script created (now 1a8f72d)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-release-script
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at 1a8f72d  validation

This branch includes the following new commits:

 new ff5759b  add repostiories for deployment
 new 92b7351  wip
 new edcf660  Get the deploy side working - next will come the validate of 
the deployed jar
 new 32c06fe  make next steps more clear
 new 1a8f72d  validation

The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-mxnet] 03/05: Get the deploy side working - next will come the validate of the deployed jar

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-release-script
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit edcf660e647c527100ca4e66ac79c5f2819eee25
Author: gigasquid 
AuthorDate: Sat Oct 26 07:56:36 2019 -0400

Get the deploy side working
- next will come the validate of the deployed jar
---
 .../clojure-package/scripts/setup_deploy_env.sh| 28 +++
 .../clojure-package/src/dev/package_release.clj| 91 ++
 2 files changed, 50 insertions(+), 69 deletions(-)

diff --git a/contrib/clojure-package/scripts/setup_deploy_env.sh 
b/contrib/clojure-package/scripts/setup_deploy_env.sh
new file mode 100755
index 000..ab3ae89
--- /dev/null
+++ b/contrib/clojure-package/scripts/setup_deploy_env.sh
@@ -0,0 +1,28 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Run this from the main Clojure project directory with 2 arguments
+# old-version and new-version
+# Ex: scripts/update_version 1.6.0-SNAPSHOT 1.6.0-SNAPSHOT
+
+set -evx
+echo "Setting up env for deploy"
+export GPG_TTY=$(tty)
+echo "Done!"
+
diff --git a/contrib/clojure-package/src/dev/package_release.clj 
b/contrib/clojure-package/src/dev/package_release.clj
index cc3c946..541fe6d 100644
--- a/contrib/clojure-package/src/dev/package_release.clj
+++ b/contrib/clojure-package/src/dev/package_release.clj
@@ -58,7 +58,7 @@
 (defn run-commands
   "Run shell commands and report exit and out"
   ([commands text]
-   (commands text nil))
+   (run-commands commands text nil))
   ([commands text dir]
(do
  (println "=  " text  "  =")
@@ -76,14 +76,6 @@
  (flush))
(zero? exit)
 
-(defn reset-project-files
-  "Resets the projects files back to the original saved version"
-  []
-  (do
-(sh "cp" "project.clj" "new-project-clj")
-(sh "cp" "project.clj.bak" "project.clj")
-(sh "cp" "examples/imclassification/project.clj.bak" 
"examples/imclassification/project.clj")))
-
 (defn run-tests-and-install
   "With the project file in place, use lein to
   dowload deps, run tests and then install the generated
@@ -145,80 +137,41 @@
 "Running Image Classification Example"
 "examples/imclassification"
 
+(defn print-deploy-instructions []
+  (do
+(println "**")
+(println "*READY FOR DEPLOY*")
+(println "**")
+(println "Please validate that logs are all correct and the jar looks 
good.")
+(print "Then run the following command to push the jar to apache staging:")
+(println "   lein deploy :staging")
+(flush)))
 
 (defn run-build [args]
   (let [[build-type release-number apache-gpg-key] args
 release-args {:build-type (keyword build-type)
   :release-number release-number
   :apache-gpg-key apache-gpg-key}]
-
+
 (if (s/valid? ::release-args release-args)
-  (try
-(do
-  (test-installed-jar build-type release-number)
-  (System/exit 0)
-  )
-#_(do
-(make-new-project-file {:build-type :osx-cpu
-:release-number "1.5.1"
-:apache-gpg-key "cme...@apache.org"} )
-(run-tests-and-install)
-(System/exit 0)
-)
-(finally (reset-project-files)))
+  (do
+(make-new-project-file {:build-type :osx-cpu
+:release-number "1.5.1"
+:apache-gpg-key "cme...@apache.org"} )
+(run-tests-and-install)
+(test-installed-jar build-type release-number)
+(run-commands ["scripts/setup_deploy_env.sh"] "Setting up display for 
deploy")
+(print-deploy-instructions)
+(System/ex

[incubator-mxnet] 05/05: validation

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-release-script
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 1a8f72dd76b5dd76315270b221ca496d978dd872
Author: gigasquid 
AuthorDate: Sat Oct 26 08:56:24 2019 -0400

validation
---
 .../clojure-package/src/dev/package_release.clj| 70 +-
 1 file changed, 56 insertions(+), 14 deletions(-)

diff --git a/contrib/clojure-package/src/dev/package_release.clj 
b/contrib/clojure-package/src/dev/package_release.clj
index 433f0f0..51a5c6c 100644
--- a/contrib/clojure-package/src/dev/package_release.clj
+++ b/contrib/clojure-package/src/dev/package_release.clj
@@ -12,6 +12,8 @@
 (s/def ::release-args (s/keys :req-un [::build-type ::release-number
::apache-gpg-key]))
 
+(s/def ::validate-args (s/keys :req-un [::build-type ::release-number]))
+
 (def scala-jar-name
   {:linux-cpu "org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu"
:linux-gpu "org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu"
@@ -99,23 +101,27 @@
   "This will move maven downloads of the release to test the
   download of the deployed jar from staging"
   [build-type release-number]
-  (let [m2-path (str (System/getProperty "user.home")
+  (let [mv-id (str release-number "-" (java.util.UUID/randomUUID) "-bak")
+m2-path (str (System/getProperty "user.home")
  
"/.m2/repository/org/apache/mxnet/contrib/clojure/clojure-mxnet"
- "-" build-type "/"
- release-number)
+ "-" build-type)
 files-to-be-cleared (:out (sh "ls" m2-path))]
-(when files-to-be-cleared
+(when-not (string/blank? files-to-be-cleared)
   (println "Installed m2 jars found")
   (print files-to-be-cleared)
-  (println "Do you want to mv the the m2 paths of " m2-path "to" (str 
m2-path "-bak") "?")
+  (println "Do you want to mv the dir " release-number " in " m2-path "to" 
mv-id)
   (println "Type \"yes\" to confirm: ")
   (flush)
   (let [answer (read-line)]
 (if (= "yes" answer)
-  (println (sh "mv" m2-path (str m2-path "-bak")))
+  (println (sh "mv" release-number mv-id :dir m2-path))
   (println "Not moving"))
 
-(defn test-installed-jar [build-type release-number]
+
+(defn test-installed-jar
+  "Tests the example with a jar locally if in maven
+  or downloads from the apache repo if needed"
+  [build-type release-number]
   (let [project-data (read-string (slurp 
"./examples/imclassification/project.clj"))
 project-header (into [] (take 3 project-data))
 project-rest-map (->> project-data
@@ -133,9 +139,13 @@
 (with-out-str (clojure.pprint/pprint p))
 (str dev.generator/license p)
 (spit "./examples/imclassification/project.clj" p))
-  (run-commands ["lein" "run"]
+  (run-commands ["lein" "run" ":cpu"]
 "Running Image Classification Example"
-"examples/imclassification"
+"examples/imclassification")
+  (when (= :linux-gpu build-type)
+(run-commands ["lein" "run" ":gpu"]
+  "Running Image Classification Example"
+  "examples/imclassification")
 
 (defn print-deploy-instructions []
   (do
@@ -148,8 +158,15 @@
 (println "   \n\n\n\n")
 (flush)))
 
+(defn print-test-deploy-conf []
+  (do
+(println "**")
+(println "*SUCCESS!!!*")
+(println "**")
+(flush)))
+
 (defn run-build [args]
-  (let [[build-type release-number apache-gpg-key] args
+  (let [[_ build-type release-number apache-gpg-key] args
 release-args {:build-type (keyword build-type)
   :release-number release-number
   :apache-gpg-key apache-gpg-key}]
@@ -166,13 +183,38 @@
 (System/exit 0))
   (do
 (println "Error with Args" release-args)
-(s/explain ::release-args release-args))
-  )))
+(s/explain ::release-args release-args)
+
+(defn validate-deploy [args]
+  (let [[_ build-type release-number] args
+validate-args {:build-type (keyword build-type)
+  :release-number release-number}]
+
+(if (s/valid? ::validate-args validate-args)
+  (do
+(clean-m2-release build-type releas

[incubator-mxnet] 04/05: make next steps more clear

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-release-script
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 32c06feb26d69576a7a4dc923ad67c710aa7db2c
Author: gigasquid 
AuthorDate: Sat Oct 26 08:06:57 2019 -0400

make next steps more clear
---
 contrib/clojure-package/src/dev/package_release.clj | 1 +
 1 file changed, 1 insertion(+)

diff --git a/contrib/clojure-package/src/dev/package_release.clj 
b/contrib/clojure-package/src/dev/package_release.clj
index 541fe6d..433f0f0 100644
--- a/contrib/clojure-package/src/dev/package_release.clj
+++ b/contrib/clojure-package/src/dev/package_release.clj
@@ -145,6 +145,7 @@
 (println "Please validate that logs are all correct and the jar looks 
good.")
 (print "Then run the following command to push the jar to apache staging:")
 (println "   lein deploy :staging")
+(println "   \n\n\n\n")
 (flush)))
 
 (defn run-build [args]

[incubator-mxnet] 01/05: add repostiories for deployment

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-release-script
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit ff5759b01920e3378ca76140aa57ae7e35e37698
Author: gigasquid 
AuthorDate: Fri Oct 25 14:37:43 2019 -0400

add repostiories for deployment
---
 contrib/clojure-package/project.clj | 14 +++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/contrib/clojure-package/project.clj 
b/contrib/clojure-package/project.clj
index 718fbf6..3bbb6d4 100644
--- a/contrib/clojure-package/project.clj
+++ b/contrib/clojure-package/project.clj
@@ -41,7 +41,15 @@
 [lein-cljfmt "0.5.7"]]
   :codox {:namespaces [#"^org\.apache\.clojure-mxnet\.(?!gen).*"]}
   :aot [dev.generator]
-  :repositories [["staging" {:url 
"https://repository.apache.org/content/repositories/staging;  
:snapshots true
- :update :always}]
+  :deploy-repositories [["staging" {:sign-releases true
+:url 
"https://repository.apache.org/service/local/staging/deploy/maven2;
+:creds :gpg
+:signing {:gpg-key "your key"} ;; this can 
be email of key-id
+:snapshots true
+:checksum :fail
+:update :always
+:releases {:checksum :fail :update 
:always}}]]
+  :repositories [["download-staging" {:url 
"https://repository.apache.org/content/repositories/staging;  
:snapshots true
+:update :always}]
  ["snapshots" {:url 
"https://repository.apache.org/content/repositories/snapshots;   
:snapshots true
-   :update :always}]])
+  :update :always}]])

[incubator-mxnet] 02/05: wip

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-release-script
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 92b73514ad522517790b74cd5d444a5ea1eb6a8c
Author: gigasquid 
AuthorDate: Fri Oct 25 16:08:55 2019 -0400

wip
---
 .../examples/imclassification/project.clj  |   4 +-
 .../clojure-package/src/dev/package_release.clj| 224 +
 2 files changed, 227 insertions(+), 1 deletion(-)

diff --git a/contrib/clojure-package/examples/imclassification/project.clj 
b/contrib/clojure-package/examples/imclassification/project.clj
index 702a33d..08e7ae3 100644
--- a/contrib/clojure-package/examples/imclassification/project.clj
+++ b/contrib/clojure-package/examples/imclassification/project.clj
@@ -21,4 +21,6 @@
   :dependencies [[org.clojure/clojure "1.9.0"]
  [org.apache.mxnet.contrib.clojure/clojure-mxnet 
"1.6.0-SNAPSHOT"]]
   :pedantic? :skip
-  :main imclassification.train-mnist)
+  :main imclassification.train-mnist
+  :repositories [["download-staging" {:url 
"https://repository.apache.org/content/repositories/staging;  
:snapshots true
+:update :always}]])
diff --git a/contrib/clojure-package/src/dev/package_release.clj 
b/contrib/clojure-package/src/dev/package_release.clj
new file mode 100644
index 000..cc3c946
--- /dev/null
+++ b/contrib/clojure-package/src/dev/package_release.clj
@@ -0,0 +1,224 @@
+(ns dev.package-release
+  (:require [clojure.java.io :as io]
+[clojure.edn :as edn]
+[clojure.java.shell :refer [sh]]
+[clojure.spec.alpha :as s]
+[clojure.string :as string]
+[dev.generator]))
+
+(s/def ::build-type #{:osx-cpu :linux-gpu :linux-cpu})
+(s/def ::release-number string?)
+(s/def ::apache-gpg-key string?)
+(s/def ::release-args (s/keys :req-un [::build-type ::release-number
+   ::apache-gpg-key]))
+
+(def scala-jar-name
+  {:linux-cpu "org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu"
+   :linux-gpu "org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu"
+   :osx-cpu "org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu"})
+
+(defn backup-project-file
+  "Backs up a copy of the original project file"
+  []
+  (do
+(sh "cp" "project.clj" "project.clj.bak")
+(sh "cp" "examples/imclassification/project.clj" 
"examples/imclassification/project.clj.bak")))
+
+(defn make-new-project-file
+  "Creates a new project file for the relase by
+  changing the build type and the version along
+  with the release number and the signer's public key
+  which can be an email"
+  [{:keys [build-type release-number apache-gpg-key]}]
+
+  (let [project-data (read-string (slurp "project.clj"))
+project-header (take 3 project-data)
+project-rest-map (->> project-data
+  (drop 3)
+  (apply hash-map))
+new-header `[~'defproject
+ ~(symbol (str 
"org.apache.mxnet.contrib.clojure/clojure-mxnet"
+   "-" (name build-type)))
+ ~release-number]
+new-dependencies (->> (:dependencies project-rest-map)
+  (remove #(string/includes? % "org.apache.mxnet"))
+  (into [[(symbol (scala-jar-name build-type)) 
release-number]]))
+new-project-rest-map (-> project-rest-map
+ (assoc :dependencies new-dependencies)
+ (assoc-in [:deploy-repositories 0 1 
:signing :gpg-key]
+   apache-gpg-key))]
+
+(as-> (into [] new-project-rest-map) p
+  (into new-header (mapcat identity p))
+  (apply list p)
+  (with-out-str (clojure.pprint/pprint p))
+  (str dev.generator/license p)
+  (spit "project.clj" p
+
+(defn run-commands
+  "Run shell commands and report exit and out"
+  ([commands text]
+   (commands text nil))
+  ([commands text dir]
+   (do
+ (println "=  " text  "  =")
+ (println "Please wait ")
+ (flush))
+   (let [{:keys [out exit err]} (apply sh (if dir
+(into commands [:dir dir])
+commands))]
+ (do
+   (println out)
+   (flush)
+   (when-not (zero? exit)
+ (println "Errors:")
+ (println err)
+ (flush))
+   (zero? exit)
+
+(defn reset-project-files
+  "Resets the projects files back to the original saved version"
+  []
+  (d

[incubator-mxnet] branch master updated (1c67928 -> 9a9c5f8)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


from 1c67928  [MXNET-1294] Add KVSTORE PushPull API (#15559)
 add 9a9c5f8  Fix failing tests (#16117)

No new revisions were added by this update.

Summary of changes:
 .../test/org/apache/clojure_mxnet/operator_test.clj   | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

[incubator-mxnet] branch master updated (c742ef1 -> 4333a7b)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


from c742ef1  add numpy operator remainder (#16080)
 add 4333a7b  Update readme and project.clj comment (#16084)

No new revisions were added by this update.

Summary of changes:
 contrib/clojure-package/README.md   | 8 
 contrib/clojure-package/project.clj | 4 ++--
 2 files changed, 6 insertions(+), 6 deletions(-)

[incubator-mxnet] branch fix-flaky-clojure-profiler-test updated (fa5e35a -> 6ce48b2)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch fix-flaky-clojure-profiler-test
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


from fa5e35a  Fix flaky clojure profile test
 add 6ce48b2  disable profiler test

No new revisions were added by this update.

Summary of changes:
 contrib/clojure-package/integration-tests.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[incubator-mxnet] branch fix-flaky-clojure-profiler-test updated: Fix flaky clojure profile test

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch fix-flaky-clojure-profiler-test
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to 
refs/heads/fix-flaky-clojure-profiler-test by this push:
 new fa5e35a  Fix flaky clojure profile test
fa5e35a is described below

commit fa5e35a23d5680009b8d504caa0b2379ff2d40cd
Author: gigasquid 
AuthorDate: Sat Aug 31 11:27:31 2019 -0400

Fix flaky clojure profile test
---
 .../examples/profiler/test/core_test.clj   |   3 +-
 .../profiler/test/profile-matmul-20iter.json.ref   | 271 -
 2 files changed, 1 insertion(+), 273 deletions(-)

diff --git a/contrib/clojure-package/examples/profiler/test/core_test.clj 
b/contrib/clojure-package/examples/profiler/test/core_test.clj
index 1173f07..9f03c94 100644
--- a/contrib/clojure-package/examples/profiler/test/core_test.clj
+++ b/contrib/clojure-package/examples/profiler/test/core_test.clj
@@ -27,5 +27,4 @@
 (deftest run-profiler
(profiler/run)
(let [new-file (clojure.java.io/as-file profiler/profiler-name)]
-   (is (.exists new-file))
-   (is (> 1 (- (System/currentTimeMillis) (.lastModified 
new-file))
\ No newline at end of file
+   (is (.exists new-file
diff --git 
a/contrib/clojure-package/examples/profiler/test/profile-matmul-20iter.json.ref 
b/contrib/clojure-package/examples/profiler/test/profile-matmul-20iter.json.ref
deleted file mode 100644
index d6baa42..000
--- 
a/contrib/clojure-package/examples/profiler/test/profile-matmul-20iter.json.ref
+++ /dev/null
@@ -1,271 +0,0 @@
-{
-"traceEvents": [
-{
-"ph": "M",
-"args": {
-"name": "cpu/0"
-},
-"pid": 0,
-"name": "process_name"
-},
-{
-"ph": "M",
-"args": {
-"name": "cpu/1"
-},
-"pid": 1,
-"name": "process_name"
-},
-{
-"ph": "M",
-"args": {
-"name": "cpu/2"
-},
-"pid": 2,
-"name": "process_name"
-},
-{
-"ph": "M",
-"args": {
-"name": "cpu/3"
-},
-"pid": 3,
-"name": "process_name"
-},
-{
-"ph": "M",
-"args": {
-"name": "cpu pinned/"
-},
-"pid": 4,
-"name": "process_name"
-},
-{
-"ph": "M",
-"args": {
-"name": "cpu shared/"
-},
-"pid": 5,
-"name": "process_name"
-},{
-"ph": "M",
-"args": {
-"name": "MXNET_C_API"
-},
-"pid": 13841910479334118176,
-"name": "process_name"
-},
-
-{
-"name": "MXNet C API Calls",
-"cat": "MXNET_C_API",
-"ph": "C",
-"ts": 51195258331,
-"args": { "MXNet C API Calls": 1 },
-"pid": 13841910479334118176,
-"tid": 6902988396839073221
-}
-,
-{
-"name": "MXNet C API Concurrency",
-"cat": "MXNET_C_API",
-"ph": "C",
-"ts": 51195258338,
-"args": { "MXNet C API Concurrency": 1 },
-"pid": 13841910479334118176,
-"tid": 6902988396839073221
-}
-,
-{
-"name": "MXExecutorForward",
-"cat": "MXNET_C_API",
-"ph": "b",
-"ts": 51195258348,
-"id": 6902988396839073221,
-"pid": 13841910479334118176,
-"tid": 6902988396839073221
-}
-,
-{
-"name": "MXExecutorForward",
-"cat": "MXNET_C_API",
-"ph": "e",
-"ts": 51195258357,
-"id": 6902988396839073221,
-"pid": 13841910479334118176,
-"tid"

[incubator-mxnet] branch fix-flaky-clojure-profiler-test created (now aab4ded)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch fix-flaky-clojure-profiler-test
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at aab4ded  Numpy compatible max min (#16046)

No new revisions were added by this update.

[incubator-mxnet] branch v1.5.x updated: Clojure package remove source images (#15828)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch v1.5.x
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/v1.5.x by this push:
 new 52ce718  Clojure package remove source images (#15828)
52ce718 is described below

commit 52ce718a076ca56385360a9b45d6afb44b5f1e27
Author: Carin Meier 
AuthorDate: Sun Aug 11 00:47:47 2019 -0400

Clojure package remove source images (#15828)

* remove test images

* add script and .gitignore

* add test helper to download images

* remove unlicensed pic

* add license header
---
 contrib/clojure-package/.gitignore |   1 +
 .../infer/objectdetector/images/marcel.jpg | Bin 116161 -> 0 bytes
 contrib/clojure-package/scripts/get_test_images.sh |  33 +
 .../test/org/apache/clojure_mxnet/image_test.clj   |   6 +++-
 .../clojure_mxnet/infer/imageclassifier_test.clj   |   5 +++-
 .../test/test-images/Pug-Cookie.jpg| Bin 104323 -> 0 bytes
 .../clojure-package/test/test-images/kitten.jpg| Bin 110969 -> 0 bytes
 contrib/clojure-package/test/test_helper.clj   |  26 
 8 files changed, 69 insertions(+), 2 deletions(-)

diff --git a/contrib/clojure-package/.gitignore 
b/contrib/clojure-package/.gitignore
index 884834e..c304e79 100644
--- a/contrib/clojure-package/.gitignore
+++ b/contrib/clojure-package/.gitignore
@@ -46,5 +46,6 @@ test/test-symbol.clj
 test/test-symbol-random.clj
 test/test-symbol-random-api.clj
 test/test-symbol-api.clj
+test/test-images/*
 src/org/apache/clojure_mxnet/gen/*
 
diff --git 
a/contrib/clojure-package/examples/infer/objectdetector/images/marcel.jpg 
b/contrib/clojure-package/examples/infer/objectdetector/images/marcel.jpg
deleted file mode 100644
index 1bf7387..000
Binary files 
a/contrib/clojure-package/examples/infer/objectdetector/images/marcel.jpg and 
/dev/null differ
diff --git a/contrib/clojure-package/scripts/get_test_images.sh 
b/contrib/clojure-package/scripts/get_test_images.sh
new file mode 100755
index 000..e850948
--- /dev/null
+++ b/contrib/clojure-package/scripts/get_test_images.sh
@@ -0,0 +1,33 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -evx
+
+MXNET_ROOT=$(cd "$(dirname $0)/.."; pwd)
+
+image_path=$MXNET_ROOT/test/test-images
+
+if [ ! -d "$image_path" ]; then
+mkdir -p "$image_path"
+fi
+
+if [ ! -f "$image_path/kitten.jpg" ]; then
+wget 
https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/kitten.jpg
 -P $image_path
+wget https://s3.amazonaws.com/model-server/inputs/Pug-Cookie.jpg -P 
$image_path
+fi
diff --git 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/image_test.clj 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/image_test.clj
index fd200f1..a5b6803 100644
--- a/contrib/clojure-package/test/org/apache/clojure_mxnet/image_test.clj
+++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/image_test.clj
@@ -19,10 +19,14 @@
   (:require [org.apache.clojure-mxnet.image :as image]
 [org.apache.clojure-mxnet.ndarray :as ndarray]
 [clojure.java.io :as io]
-[clojure.test :refer [deftest is use-fixtures]])
+[clojure.test :refer [deftest is use-fixtures run-tests]]
+[test-helper])
   (:import (javax.imageio ImageIO)
(java.io File)))
 
+
+(test-helper/load-test-images)
+
 (def tmp-dir (System/getProperty "java.io.tmpdir"))
 (def image-path (.getAbsolutePath (io/file tmp-dir "Pug-Cookie.jpg")))
 (def image-src-path "test/test-images/Pug-Cookie.jpg")
diff --git 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/imageclassifier_test.clj
 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/imageclassifier_test.clj
index b7f468f..5890f75 100644
--- 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/imageclassifier_test.clj
+++ 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/imageclassifier_test.clj
@@ -22,7 +22,10 @@
 [org.apache.clojur

[incubator-mxnet] branch clojure-autoencoder deleted (was 119f135)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-autoencoder
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


 was 119f135  tweaks

This change permanently discards the following revisions:

 discard 119f135  tweaks
 discard efb14d4  cleaning up
 discard ef3451a  basic autoencoder
 discard f64dd20  wip working
 discard d0f97a7  wip

[incubator-mxnet] branch clojure-autoencoder-2 deleted (was 35ba0ab)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-autoencoder-2
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


 was 35ba0ab  use one hot encoding

This change permanently discards the following revisions:

 discard 35ba0ab  use one hot encoding
 discard a6119ae  input with label
 discard 119f135  tweaks
 discard efb14d4  cleaning up
 discard ef3451a  basic autoencoder
 discard f64dd20  wip working
 discard d0f97a7  wip

[incubator-mxnet] branch clojure-package-remove-source-images deleted (was f2bba64)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-package-remove-source-images
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


 was f2bba64  add license header

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

[incubator-mxnet] branch clojure-package-remove-source-images updated (cdbe7bd -> f2bba64)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-package-remove-source-images
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


from cdbe7bd  remove unlicensed pic
 add f2bba64  add license header

No new revisions were added by this update.

Summary of changes:
 contrib/clojure-package/test/test_helper.clj | 17 +
 1 file changed, 17 insertions(+)

[incubator-mxnet] 01/04: remove test images

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-package-remove-source-images
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 46dbfe92074d65d41f7e7ade938ad9e287993822
Author: gigasquid 
AuthorDate: Fri Aug 9 14:48:47 2019 -0400

remove test images
---
 .../clojure-package/test/test-images/Pug-Cookie.jpg| Bin 104323 -> 0 bytes
 contrib/clojure-package/test/test-images/kitten.jpg| Bin 110969 -> 0 bytes
 2 files changed, 0 insertions(+), 0 deletions(-)

diff --git a/contrib/clojure-package/test/test-images/Pug-Cookie.jpg 
b/contrib/clojure-package/test/test-images/Pug-Cookie.jpg
deleted file mode 100644
index 56f5dc1..000
Binary files a/contrib/clojure-package/test/test-images/Pug-Cookie.jpg and 
/dev/null differ
diff --git a/contrib/clojure-package/test/test-images/kitten.jpg 
b/contrib/clojure-package/test/test-images/kitten.jpg
deleted file mode 100644
index ffcd2be..000
Binary files a/contrib/clojure-package/test/test-images/kitten.jpg and 
/dev/null differ

[incubator-mxnet] branch clojure-package-remove-source-images created (now cdbe7bd)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-package-remove-source-images
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at cdbe7bd  remove unlicensed pic

This branch includes the following new commits:

 new 46dbfe9  remove test images
 new ab8fbb1  add script and .gitignore
 new 56abeb8  add test helper to download images
 new cdbe7bd  remove unlicensed pic

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-mxnet] 03/04: add test helper to download images

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-package-remove-source-images
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 56abeb81262e9661194ae617efbd3b52f8dc70d7
Author: gigasquid 
AuthorDate: Fri Aug 9 15:04:51 2019 -0400

add test helper to download images
---
 .../clojure-package/test/org/apache/clojure_mxnet/image_test.clj | 6 +-
 .../test/org/apache/clojure_mxnet/infer/imageclassifier_test.clj | 5 -
 contrib/clojure-package/test/test_helper.clj | 9 +
 3 files changed, 18 insertions(+), 2 deletions(-)

diff --git 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/image_test.clj 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/image_test.clj
index fd200f1..a5b6803 100644
--- a/contrib/clojure-package/test/org/apache/clojure_mxnet/image_test.clj
+++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/image_test.clj
@@ -19,10 +19,14 @@
   (:require [org.apache.clojure-mxnet.image :as image]
 [org.apache.clojure-mxnet.ndarray :as ndarray]
 [clojure.java.io :as io]
-[clojure.test :refer [deftest is use-fixtures]])
+[clojure.test :refer [deftest is use-fixtures run-tests]]
+[test-helper])
   (:import (javax.imageio ImageIO)
(java.io File)))
 
+
+(test-helper/load-test-images)
+
 (def tmp-dir (System/getProperty "java.io.tmpdir"))
 (def image-path (.getAbsolutePath (io/file tmp-dir "Pug-Cookie.jpg")))
 (def image-src-path "test/test-images/Pug-Cookie.jpg")
diff --git 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/imageclassifier_test.clj
 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/imageclassifier_test.clj
index b7f468f..5890f75 100644
--- 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/imageclassifier_test.clj
+++ 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/imageclassifier_test.clj
@@ -22,7 +22,10 @@
 [org.apache.clojure-mxnet.ndarray :as ndarray]
 [clojure.java.io :as io]
 [clojure.java.shell :refer [sh]]
-[clojure.test :refer :all]))
+[clojure.test :refer :all]
+[test-helper]))
+
+(test-helper/load-test-images)
 
 (def model-dir "data/")
 (def model-path-prefix (str model-dir "resnet-18/resnet-18"))
diff --git a/contrib/clojure-package/test/test_helper.clj 
b/contrib/clojure-package/test/test_helper.clj
new file mode 100644
index 000..5a0bcb9
--- /dev/null
+++ b/contrib/clojure-package/test/test_helper.clj
@@ -0,0 +1,9 @@
+(ns test-helper
+  (:require [clojure.java.io :as io]
+[clojure.java.shell :refer [sh]]))
+
+(def data-dir "test/test-images/")
+
+(defn load-test-images []
+  (when-not (.exists (io/file (str data-dir "Pug-Cookie.jpg")))
+(sh "./scripts/get_test_images.sh")))

[incubator-mxnet] 04/04: remove unlicensed pic

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-package-remove-source-images
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit cdbe7bde8f2b7ea83bfcd06faf2af488cc704a1d
Author: gigasquid 
AuthorDate: Fri Aug 9 15:11:34 2019 -0400

remove unlicensed pic
---
 .../examples/infer/objectdetector/images/marcel.jpg| Bin 116161 -> 0 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)

diff --git 
a/contrib/clojure-package/examples/infer/objectdetector/images/marcel.jpg 
b/contrib/clojure-package/examples/infer/objectdetector/images/marcel.jpg
deleted file mode 100644
index 1bf7387..000
Binary files 
a/contrib/clojure-package/examples/infer/objectdetector/images/marcel.jpg and 
/dev/null differ

[incubator-mxnet] 02/04: add script and .gitignore

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-package-remove-source-images
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit ab8fbb107cf00fe77b5f0bda231675b231d45d58
Author: gigasquid 
AuthorDate: Fri Aug 9 14:49:51 2019 -0400

add script and .gitignore
---
 contrib/clojure-package/.gitignore |  1 +
 contrib/clojure-package/scripts/get_test_images.sh | 33 ++
 2 files changed, 34 insertions(+)

diff --git a/contrib/clojure-package/.gitignore 
b/contrib/clojure-package/.gitignore
index 884834e..c304e79 100644
--- a/contrib/clojure-package/.gitignore
+++ b/contrib/clojure-package/.gitignore
@@ -46,5 +46,6 @@ test/test-symbol.clj
 test/test-symbol-random.clj
 test/test-symbol-random-api.clj
 test/test-symbol-api.clj
+test/test-images/*
 src/org/apache/clojure_mxnet/gen/*
 
diff --git a/contrib/clojure-package/scripts/get_test_images.sh 
b/contrib/clojure-package/scripts/get_test_images.sh
new file mode 100755
index 000..e850948
--- /dev/null
+++ b/contrib/clojure-package/scripts/get_test_images.sh
@@ -0,0 +1,33 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -evx
+
+MXNET_ROOT=$(cd "$(dirname $0)/.."; pwd)
+
+image_path=$MXNET_ROOT/test/test-images
+
+if [ ! -d "$image_path" ]; then
+mkdir -p "$image_path"
+fi
+
+if [ ! -f "$image_path/kitten.jpg" ]; then
+wget 
https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/kitten.jpg
 -P $image_path
+wget https://s3.amazonaws.com/model-server/inputs/Pug-Cookie.jpg -P 
$image_path
+fi

[incubator-mxnet] branch clojure-autoencoder-2 created (now 35ba0ab)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-autoencoder-2
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at 35ba0ab  use one hot encoding

This branch includes the following new commits:

 new a6119ae  input with label
 new 35ba0ab  use one hot encoding

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-mxnet] 01/02: input with label

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-autoencoder-2
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit a6119aed919c1984df942377633f3a213c945e47
Author: gigasquid 
AuthorDate: Fri Jul 19 19:26:51 2019 -0400

input with label
---
 .../examples/gan/src/gan/auto_encoder.clj  | 75 ++
 1 file changed, 47 insertions(+), 28 deletions(-)

diff --git a/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj 
b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
index e039a99..591c387 100644
--- a/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
+++ b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
@@ -25,34 +25,35 @@
 
 ;;; Load the MNIST datasets
 ;;; note that the label is the same as the image
-(def train-data (mx-io/mnist-iter {:image (str data-dir 
"train-images-idx3-ubyte")
-   :label (str data-dir 
"train-images-idx3-ubyte")
-;:input-shape [1 28 28]
-:input-shape [784]
-:label-shape [10]
-:flat true
-:batch-size batch-size
-:shuffle true}))
-
-(def test-data (mx-io/mnist-iter {:image (str data-dir 
"train-images-idx3-ubyte")
-   :label (str data-dir 
"train-images-idx3-ubyte")
-   ;;:input-shape [1 28 28]
-  :input-shape [784]
+(defonce train-data (mx-io/mnist-iter {:image (str data-dir 
"train-images-idx3-ubyte")
+   :label (str data-dir 
"train-labels-idx1-ubyte")
+   :label-name "softmax_label"
+   :input-shape [784]
:batch-size batch-size
+   :shuffle true
:flat true
-   :shuffle true}))
+   :silent false
+   :seed 10}))
+
+(defonce test-data (mx-io/mnist-iter {:image (str data-dir 
"t10k-images-idx3-ubyte")
+  :label (str data-dir 
"t10k-labels-idx1-ubyte")
+  :input-shape [784]
+  :batch-size batch-size
+  :flat true
+  :silent false}))
+
 (def output (sym/variable "input_"))
 
-(defn get-symbol []
-  (as-> (sym/variable "input") data
-;; encode
-(sym/fully-connected "encode1" {:data data :num-hidden 100})
-(sym/activation "sigmoid1" {:data data :act-type "sigmoid"})
+(comment
+  (def x (mx-io/next test-data))
 
-;; encode
-(sym/fully-connected "encode2" {:data data :num-hidden 50})
-(sym/activation "sigmoid2" {:data data :act-type "sigmoid"})
+  (mx-io/batch-label x)
 
+
+  )
+
+(defn get-symbol []
+  (as-> (sym/variable "input") data
 ;; decode
 (sym/fully-connected "decode1" {:data data :num-hidden 50})
 (sym/activation "sigmoid3" {:data data :act-type "sigmoid"})
@@ -70,9 +71,10 @@
 ))
 
 (def data-desc (first (mx-io/provide-data-desc train-data)))
+(def label-desc (first (mx-io/provide-label-desc train-data)))
 
 (def model (-> (m/module (get-symbol) {:data-names ["input"] :label-names 
["input_"]})
-   (m/bind {:data-shapes [(assoc data-desc :name "input")]
+   (m/bind {:data-shapes [(assoc label-desc :name "input")]
 :label-shapes [(assoc data-desc :name "input_")]})
(m/init-params {:initializer  (initializer/uniform 1)})
(m/init-optimizer {:optimizer (optimizer/adam {:learning-rage 
0.001})})))
@@ -86,7 +88,7 @@
  train-data
  (fn [batch]
(-> model
-   (m/forward {:data (mx-io/batch-data batch) :label (mx-io/batch-data 
batch)})
+   (m/forward {:data (mx-io/batch-label batch) :label 
(mx-io/batch-data batch)})
(m/update-metric my-metric (mx-io/batch-data batch))
(m/backward)
(m/update
@@ -98,26 +100,43 @@
   (mx-io/provide-label train-data)
   (mx-io/reset train-data)
   (def my-batch (mx-io/next train-data))
+  (def train-labels (mx-io/batch-label my-batch))
   (def images (mx-io/batch-data my-batch))
   (ndarray/shape (ndarray/reshape (first images) [100 1 28 28]))
   (viz/im-sav {:title &q

[incubator-mxnet] 02/02: use one hot encoding

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-autoencoder-2
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 35ba0ab690eedd4e5c2e89101a53fb77e0cbb7d4
Author: gigasquid 
AuthorDate: Fri Aug 9 10:46:13 2019 -0400

use one hot encoding
---
 contrib/clojure-package/examples/gan/project.clj |  4 ++--
 .../examples/gan/src/gan/auto_encoder.clj| 20 
 2 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/contrib/clojure-package/examples/gan/project.clj 
b/contrib/clojure-package/examples/gan/project.clj
index 4393987..2748ac0 100644
--- a/contrib/clojure-package/examples/gan/project.clj
+++ b/contrib/clojure-package/examples/gan/project.clj
@@ -20,7 +20,7 @@
   :plugins [[lein-cljfmt "0.5.7"]]
   :repositories [["vendredi" {:url 
"https://repository.hellonico.info/repository/hellonico/"}]]
   :dependencies [[org.clojure/clojure "1.9.0"]
- [org.apache.mxnet.contrib.clojure/clojure-mxnet 
"1.5.0-SNAPSHOT"]
+ [org.apache.mxnet.contrib.clojure/clojure-mxnet-osx-cpu 
"1.5.0"]
  [origami "4.0.0-3"]
  ]
-  :main gan.gan-mnist)
\ No newline at end of file
+  :main gan.gan-mnist)
diff --git a/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj 
b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
index 591c387..90affa1 100644
--- a/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
+++ b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
@@ -71,7 +71,10 @@
 ))
 
 (def data-desc (first (mx-io/provide-data-desc train-data)))
-(def label-desc (first (mx-io/provide-label-desc train-data)))
+;; one hot encode the label
+(def label-desc (assoc
+ (first (mx-io/provide-label-desc train-data))
+ :shape [100 10]))
 
 (def model (-> (m/module (get-symbol) {:data-names ["input"] :label-names 
["input_"]})
(m/bind {:data-shapes [(assoc label-desc :name "input")]
@@ -81,6 +84,10 @@
 
 (def my-metric (eval-metric/mse))
 
+(defn one-hot-encode [labels]
+  (ndarray-api/one-hot labels 10))
+
+
 (defn train [num-epochs]
   (doseq [epoch-num (range 0 num-epochs)]
 (println "starting epoch " epoch-num)
@@ -88,7 +95,10 @@
  train-data
  (fn [batch]
(-> model
-   (m/forward {:data (mx-io/batch-label batch) :label 
(mx-io/batch-data batch)})
+   (m/forward {:data [(-> (mx-io/batch-label batch)
+  first
+  (one-hot-encode))]
+   :label (mx-io/batch-data batch)})
(m/update-metric my-metric (mx-io/batch-data batch))
(m/backward)
(m/update
@@ -107,16 +117,18 @@
 
 
   (ndarray/shape (first train-labels))
+  (ndarray/->vec (first train-labels))
   (train 3)
 
 
   (def my-test-batch (mx-io/next test-data))
   (def test-images (mx-io/batch-data my-test-batch))
   (def test-labels (mx-io/batch-label my-test-batch))
-  (def preds (m/predict-batch model {:data test-labels} ))
+  (def one-hot-labels [(one-hot-encode (first test-labels))])
+  (def preds (m/predict-batch model {:data one-hot-labels} ))
   (viz/im-sav {:title "preds" :output-path "results/" :x (ndarray/reshape 
(first preds) [100 1 28 28])})
 
-  (def x (ndarray/array (into [] (repeat 100 2)) [100]))
+  (def x (ndarray/array (into [] (repeatedly 100 #(rand-int 9))) [100]))
   (def new-preds (m/predict-batch model {:data [x]} ))
   (viz/im-sav {:title "preds" :output-path "results/" :x (ndarray/reshape 
(first new-preds) [100 1 28 28])})
 

[incubator-mxnet] branch clojure-autoencoder created (now 119f135)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-autoencoder
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at 119f135  tweaks

This branch includes the following new commits:

 new d0f97a7  wip
 new f64dd20  wip working
 new ef3451a  basic autoencoder
 new efb14d4  cleaning up
 new 119f135  tweaks

The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-mxnet] 02/05: wip working

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-autoencoder
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit f64dd20a19131a1e85493603e7ec3e6f8ec930d7
Author: gigasquid 
AuthorDate: Sun Jul 7 17:02:00 2019 -0400

wip working
---
 contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj | 10 +-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj 
b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
index 93b..ee1a9a6 100644
--- a/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
+++ b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
@@ -75,11 +75,18 @@
 
   (mx-io/provide-data train-data)
   (mx-io/provide-label train-data)
+  (mx-io/reset train-data)
   (def my-batch (mx-io/next train-data))
   (def images (mx-io/batch-data my-batch))
-  (ndarray/shape (first images))
+  (ndarray/shape (ndarray/reshape (first images) [100 1 28 28]))
   (viz/im-sav {:title "first" :output-path "results/" :x (first images)})
+  (viz/im-sav {:title "cm-first" :output-path "results/" :x (ndarray/reshape 
(first images) [100 1 28 28])})
 
+
+  (def preds (m/predict-batch my-mod {:data images} ))
+  (ndarray/shape (ndarray/reshape (first preds) [100 1 28 28]))
+(viz/im-sav {:title "cm-preds" :output-path "results/" :x (ndarray/reshape 
(first preds) [100 1 28 28])})
+  
   (def my-metric (eval-metric/mse))
 
 
@@ -119,6 +126,7 @@
 ;;;high level score (returs the eval values)
 (let [score (m/score mod {:eval-data test-data :eval-metric 
(eval-metric/accuracy)})]
   (println "High level predict score is " score)))
+
   
 
 

[incubator-mxnet] 01/05: wip

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-autoencoder
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit d0f97a76ff24158754a37d4e669ca24652e81937
Author: gigasquid 
AuthorDate: Sun Jul 7 16:42:56 2019 -0400

wip
---
 .../examples/gan/src/gan/auto_encoder.clj  | 129 +
 1 file changed, 129 insertions(+)

diff --git a/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj 
b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
new file mode 100644
index 000..93b
--- /dev/null
+++ b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
@@ -0,0 +1,129 @@
+(ns gan.auto-encoder
+  (:require [clojure.java.io :as io]
+[clojure.java.shell :refer [sh]]
+[org.apache.clojure-mxnet.io :as mx-io]
+[org.apache.clojure-mxnet.ndarray :as ndarray]
+[org.apache.clojure-mxnet.ndarray-api :as ndarray-api]
+[org.apache.clojure-mxnet.image :as image]
+[org.apache.clojure-mxnet.dtype :as dtype]
+[gan.viz :as viz]
+[org.apache.clojure-mxnet.symbol :as sym]
+[org.apache.clojure-mxnet.module :as m]
+[org.apache.clojure-mxnet.eval-metric :as eval-metric]
+[org.apache.clojure-mxnet.initializer :as initializer]
+[org.apache.clojure-mxnet.optimizer :as optimizer])
+  (:import (javax.imageio ImageIO)
+   (org.apache.mxnet Image NDArray)
+   (java.io File)))
+
+(def data-dir "data/")
+(def batch-size 100)
+
+(when-not (.exists (io/file (str data-dir "train-images-idx3-ubyte")))
+  (sh "../../scripts/get_mnist_data.sh"))
+
+
+;;; Load the MNIST datasets
+;;; note that the label is the same as the image
+(def train-data (mx-io/mnist-iter {:image (str data-dir 
"train-images-idx3-ubyte")
+   :label (str data-dir 
"train-images-idx3-ubyte")
+;:input-shape [1 28 28]
+:input-shape [784]
+:label-shape [784]
+:flat true
+:batch-size batch-size
+:shuffle true}))
+
+(def test-data (mx-io/mnist-iter {:image (str data-dir 
"train-images-idx3-ubyte")
+   :label (str data-dir 
"train-images-idx3-ubyte")
+   ;;:input-shape [1 28 28]
+  :input-shape [784]
+   :batch-size batch-size
+   :flat true
+   :shuffle true}))
+(def output (sym/variable "input_"))
+
+(defn get-symbol []
+  (as-> (sym/variable "input") data
+;; encode
+(sym/fully-connected "encode1" {:data data :num-hidden 100})
+(sym/activation "sigmoid1" {:data data :act-type "sigmoid"})
+
+;; encode
+(sym/fully-connected "encode2" {:data data :num-hidden 50})
+(sym/activation "sigmoid2" {:data data :act-type "sigmoid"})
+
+;; decode
+(sym/fully-connected "decode1" {:data data :num-hidden 50})
+(sym/activation "sigmoid3" {:data data :act-type "sigmoid"})
+
+;; decode
+(sym/fully-connected "decode2" {:data data :num-hidden 100})
+(sym/activation "sigmoid4" {:data data :act-type "sigmoid"})
+
+;;output
+(sym/fully-connected "result" {:data data :num-hidden 784})
+(sym/activation "sigmoid5" {:data data :act-type "sigmoid"})
+
+(sym/linear-regression-output {:data data :label output})
+
+#_(sym/make-loss "loss" {:data (sym/- data label)})
+
+))
+
+(comment
+
+  (mx-io/provide-data train-data)
+  (mx-io/provide-label train-data)
+  (def my-batch (mx-io/next train-data))
+  (def images (mx-io/batch-data my-batch))
+  (ndarray/shape (first images))
+  (viz/im-sav {:title "first" :output-path "results/" :x (first images)})
+
+  (def my-metric (eval-metric/mse))
+
+
+  (sym/list-arguments (m/symbol my-mod))
+  (def data-desc (first (mx-io/provide-data-desc train-data)))
+
+  (def my-mod (-> (m/module (get-symbol) {:data-names ["input"] :label-names 
["input_"]})
+  (m/bind {:data-shapes [(assoc data-desc :name "input")]
+   :label-shapes [(assoc data-desc :name "input_")]})
+  (m/init-params {:initializer  (initializer/uniform 1)})
+  (m/init-optimizer {:optimizer (optimizer/adam 
{:learning-rage 0.001})})))
+
+
+  (doseq [epoch-num (range 0 1)]
+  (println 

[incubator-mxnet] 04/05: cleaning up

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-autoencoder
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit efb14d42c75385589bd36ca5c77f8dcc4864f850
Author: gigasquid 
AuthorDate: Fri Jul 19 10:11:40 2019 -0400

cleaning up
---
 contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj 
b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
index db44b6a..ca3befc 100644
--- a/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
+++ b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
@@ -67,8 +67,6 @@
 
 (sym/linear-regression-output {:data data :label output})
 
-#_(sym/make-loss "loss" {:data (sym/- data label)})
-
 ))
 
 (def data-desc (first (mx-io/provide-data-desc train-data)))
@@ -105,7 +103,7 @@
   (viz/im-sav {:title "originals" :output-path "results/" :x (ndarray/reshape 
(first images) [100 1 28 28])})
 
 
-  (train 1)
+  (train 3)
 
   
   (def preds (m/predict-batch model {:data images} ))

[incubator-mxnet] 05/05: tweaks

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-autoencoder
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 119f135624df3a99f6aa0dda6391832b8958b83f
Author: gigasquid 
AuthorDate: Fri Jul 19 13:23:59 2019 -0400

tweaks
---
 contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj 
b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
index ca3befc..e039a99 100644
--- a/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
+++ b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
@@ -29,7 +29,7 @@
:label (str data-dir 
"train-images-idx3-ubyte")
 ;:input-shape [1 28 28]
 :input-shape [784]
-:label-shape [784]
+:label-shape [10]
 :flat true
 :batch-size batch-size
 :shuffle true}))
@@ -105,8 +105,10 @@
 
   (train 3)
 
-  
-  (def preds (m/predict-batch model {:data images} ))
+
+  (def my-test-batch (mx-io/next test-data))
+  (def test-images (mx-io/batch-data my-test-batch))
+  (def preds (m/predict-batch model {:data test-images} ))
   (viz/im-sav {:title "preds" :output-path "results/" :x (ndarray/reshape 
(first preds) [100 1 28 28])})
   
 

[incubator-mxnet] 03/05: basic autoencoder

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-autoencoder
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit ef3451a38c99b116fddead55eff8145b0f73bafb
Author: gigasquid 
AuthorDate: Fri Jul 19 10:08:08 2019 -0400

basic autoencoder
---
 .../examples/gan/src/gan/auto_encoder.clj  | 56 --
 1 file changed, 32 insertions(+), 24 deletions(-)

diff --git a/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj 
b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
index ee1a9a6..db44b6a 100644
--- a/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
+++ b/contrib/clojure-package/examples/gan/src/gan/auto_encoder.clj
@@ -71,6 +71,29 @@
 
 ))
 
+(def data-desc (first (mx-io/provide-data-desc train-data)))
+
+(def model (-> (m/module (get-symbol) {:data-names ["input"] :label-names 
["input_"]})
+   (m/bind {:data-shapes [(assoc data-desc :name "input")]
+:label-shapes [(assoc data-desc :name "input_")]})
+   (m/init-params {:initializer  (initializer/uniform 1)})
+   (m/init-optimizer {:optimizer (optimizer/adam {:learning-rage 
0.001})})))
+
+(def my-metric (eval-metric/mse))
+
+(defn train [num-epochs]
+  (doseq [epoch-num (range 0 num-epochs)]
+(println "starting epoch " epoch-num)
+(mx-io/do-batches
+ train-data
+ (fn [batch]
+   (-> model
+   (m/forward {:data (mx-io/batch-data batch) :label (mx-io/batch-data 
batch)})
+   (m/update-metric my-metric (mx-io/batch-data batch))
+   (m/backward)
+   (m/update
+(println "result for epoch " epoch-num " is " (eval-metric/get-and-reset 
my-metric
+
 (comment
 
   (mx-io/provide-data train-data)
@@ -79,38 +102,23 @@
   (def my-batch (mx-io/next train-data))
   (def images (mx-io/batch-data my-batch))
   (ndarray/shape (ndarray/reshape (first images) [100 1 28 28]))
-  (viz/im-sav {:title "first" :output-path "results/" :x (first images)})
-  (viz/im-sav {:title "cm-first" :output-path "results/" :x (ndarray/reshape 
(first images) [100 1 28 28])})
+  (viz/im-sav {:title "originals" :output-path "results/" :x (ndarray/reshape 
(first images) [100 1 28 28])})
+
 
+  (train 1)
 
-  (def preds (m/predict-batch my-mod {:data images} ))
-  (ndarray/shape (ndarray/reshape (first preds) [100 1 28 28]))
-(viz/im-sav {:title "cm-preds" :output-path "results/" :x (ndarray/reshape 
(first preds) [100 1 28 28])})
   
-  (def my-metric (eval-metric/mse))
+  (def preds (m/predict-batch model {:data images} ))
+  (viz/im-sav {:title "preds" :output-path "results/" :x (ndarray/reshape 
(first preds) [100 1 28 28])})
+  
 
 
   (sym/list-arguments (m/symbol my-mod))
   (def data-desc (first (mx-io/provide-data-desc train-data)))
 
-  (def my-mod (-> (m/module (get-symbol) {:data-names ["input"] :label-names 
["input_"]})
-  (m/bind {:data-shapes [(assoc data-desc :name "input")]
-   :label-shapes [(assoc data-desc :name "input_")]})
-  (m/init-params {:initializer  (initializer/uniform 1)})
-  (m/init-optimizer {:optimizer (optimizer/adam 
{:learning-rage 0.001})})))
-
-
-  (doseq [epoch-num (range 0 1)]
-  (println "starting epoch " epoch-num)
-  (mx-io/do-batches
-   train-data
-   (fn [batch]
- (-> my-mod
- (m/forward {:data (mx-io/batch-data batch) :label 
(mx-io/batch-data batch)})
- (m/update-metric my-metric (mx-io/batch-data batch))
- (m/backward)
- (m/update
-  (println "result for epoch " epoch-num " is " (eval-metric/get-and-reset 
my-metric)))
+
+
+
 
 
 

[incubator-mxnet] branch master updated (06df38c -> b869ecd)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


from 06df38c  point fix the vector declaration in MultiBoxDetection (#15300)
 add b869ecd  [Clojure] Add fastText example (#15340)

No new revisions were added by this update.

Summary of changes:
 .../examples/cnn-text-classification/README.md | 22 ++---
 .../get_fasttext_data.sh}  |  8 ++---
 .../src/cnn_text_classification/data_helper.clj| 37 --
 3 files changed, 48 insertions(+), 19 deletions(-)
 copy contrib/clojure-package/examples/{rnn/get_data.sh => 
cnn-text-classification/get_fasttext_data.sh} (87%)

[incubator-mxnet] branch master updated (b30949f -> 2113cb7)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


from b30949f  [MXNet-1211] Factor and "Like" modes in BilinearResize2D 
operator (#13226)
 add 2113cb7  [Clojure] Add Fine Tuning Sentence Pair Classification BERT 
Example (#14769)

No new revisions were added by this update.

Summary of changes:
 .gitignore |   2 +-
 contrib/clojure-package/.gitignore |   2 +
 .../examples/{bert-qa => bert}/.gitignore  |   8 +-
 .../examples/{bert-qa => bert}/README.md   |  80 +++-
 .../examples/bert/fine-tune-bert.ipynb | 510 +
 .../examples/bert/fine-tune-bert.md| 371 +++
 .../examples/{bert-qa => bert}/get_bert_data.sh|   5 +-
 .../examples/{bert-qa => bert}/project.clj |  16 +-
 .../examples/{bert-qa => bert}/squad-samples.edn   |   0
 .../bert/src/bert/bert_sentence_classification.clj | 160 +++
 .../src/bert_qa => bert/src/bert}/infer.clj|  67 +--
 .../examples/bert/src/bert/util.clj|  52 +++
 .../bert/bert_sentence_classification_test.clj |  86 
 .../test/bert_qa => bert/test/bert}/infer_test.clj |   9 +-
 .../src/org/apache/clojure_mxnet/callback.clj  |   9 +-
 15 files changed, 1310 insertions(+), 67 deletions(-)
 rename contrib/clojure-package/examples/{bert-qa => bert}/.gitignore (69%)
 rename contrib/clojure-package/examples/{bert-qa => bert}/README.md (50%)
 create mode 100644 contrib/clojure-package/examples/bert/fine-tune-bert.ipynb
 create mode 100644 contrib/clojure-package/examples/bert/fine-tune-bert.md
 rename contrib/clojure-package/examples/{bert-qa => bert}/get_bert_data.sh 
(73%)
 rename contrib/clojure-package/examples/{bert-qa => bert}/project.clj (63%)
 rename contrib/clojure-package/examples/{bert-qa => bert}/squad-samples.edn 
(100%)
 create mode 100644 
contrib/clojure-package/examples/bert/src/bert/bert_sentence_classification.clj
 rename contrib/clojure-package/examples/{bert-qa/src/bert_qa => 
bert/src/bert}/infer.clj (73%)
 create mode 100644 contrib/clojure-package/examples/bert/src/bert/util.clj
 create mode 100644 
contrib/clojure-package/examples/bert/test/bert/bert_sentence_classification_test.clj
 rename contrib/clojure-package/examples/{bert-qa/test/bert_qa => 
bert/test/bert}/infer_test.clj (91%)

[incubator-mxnet] branch master updated (36c3306 -> 1540a84)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


from 36c3306  Update base CUDA image for CI to v10.0 cuDNN 7.3.1 (#14513)
 add 1540a84  [Clojure] Remove unneeded test files (#14813)

No new revisions were added by this update.

Summary of changes:
 contrib/clojure-package/.gitignore |   2 +
 .../test/test-ndarray-random-api.clj   |  95 -
 .../test/test-symbol-random-api.clj| 118 -
 3 files changed, 2 insertions(+), 213 deletions(-)
 delete mode 100644 contrib/clojure-package/test/test-ndarray-random-api.clj
 delete mode 100644 contrib/clojure-package/test/test-symbol-random-api.clj

[incubator-mxnet] branch revert-14800-arthur/clojure-ndarray-api-random-merged deleted (was 95d624d)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch 
revert-14800-arthur/clojure-ndarray-api-random-merged
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


 was 95d624d  Revert "[clojure][generator] ndarray/symbol api random merged 
(#14800)"

This change permanently discards the following revisions:

 discard 95d624d  Revert "[clojure][generator] ndarray/symbol api random merged 
(#14800)"

[incubator-mxnet] branch remove-unneeded-clojure-test-files deleted (was b7f6cfb)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch remove-unneeded-clojure-test-files
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


 was b7f6cfb  add test files to gitignore

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

[incubator-mxnet] 01/01: add test files to gitignore

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch remove-unneeded-clojure-test-files
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit b7f6cfb085c1e0d06ac8b4cd925b29f7fc8bf11f
Author: gigasquid 
AuthorDate: Tue Apr 30 16:38:01 2019 -0400

add test files to gitignore
---
 contrib/clojure-package/.gitignore | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/contrib/clojure-package/.gitignore 
b/contrib/clojure-package/.gitignore
index 71d812e..8efd090 100644
--- a/contrib/clojure-package/.gitignore
+++ b/contrib/clojure-package/.gitignore
@@ -39,8 +39,10 @@ examples/visualization/test-vis.pdf
 src/.DS_Store
 src/org/.DS_Store
 test/test-ndarray.clj
+test/test-ndarray-random.clj
 test/test-ndarray-api.clj
 test/test-symbol.clj
+test/test-symbol-random.clj
 test/test-symbol-api.clj
 src/org/apache/clojure_mxnet/gen/*
 

[incubator-mxnet] branch remove-unneeded-clojure-test-files created (now b7f6cfb)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch remove-unneeded-clojure-test-files
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at b7f6cfb  add test files to gitignore

This branch includes the following new commits:

 new b7f6cfb  add test files to gitignore

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-mxnet] branch master updated: Fix Clojure BERT example's context argument (#14843)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new cdd7087  Fix Clojure BERT example's context argument (#14843)
cdd7087 is described below

commit cdd708745e33aa85d30188b8440a23235add3a82
Author: Dave Liepmann 
AuthorDate: Tue Apr 30 22:37:15 2019 +0200

Fix Clojure BERT example's context argument (#14843)

* Clojure BERT example: minor code cleanup

* Remove unused requires
* Remove unused vars & function
* Use `io` alias

* Clojure BERT example: whitespace fix

* Clojure BERT example: allow running with GPU

The `infer` function accepts a CPU/GPU context, which the command line
version of this example exposes as a `:cpu`/`:gpu`
keyword. Previously, these options were ignored and the context was
overridden to the default context (CPU). This commit allows
users (both REPL and shell) to pass in a GPU context.
---
 contrib/clojure-package/examples/bert-qa/README.md |  5 +-
 .../examples/bert-qa/src/bert_qa/infer.clj | 53 ++
 2 files changed, 26 insertions(+), 32 deletions(-)

diff --git a/contrib/clojure-package/examples/bert-qa/README.md 
b/contrib/clojure-package/examples/bert-qa/README.md
index 9a21bcd..55f13e6 100644
--- a/contrib/clojure-package/examples/bert-qa/README.md
+++ b/contrib/clojure-package/examples/bert-qa/README.md
@@ -57,9 +57,8 @@ Some sample questions and answers are provide in the 
`squad-sample.edn` file. So
 
 * `lein install` in the root of the main project directory
 * cd into this project directory and do `lein run`. This will execute the cpu 
version.
-
-`lein run :cpu` - to run with cpu
-`lein run :gpu` - to run with gpu
+  * `lein run` or `lein run :cpu` to run with cpu
+  * `lein run :gpu` to run with gpu
 
 ## Background
 
diff --git a/contrib/clojure-package/examples/bert-qa/src/bert_qa/infer.clj 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/infer.clj
index 836684e..9dcc783 100644
--- a/contrib/clojure-package/examples/bert-qa/src/bert_qa/infer.clj
+++ b/contrib/clojure-package/examples/bert-qa/src/bert_qa/infer.clj
@@ -15,13 +15,10 @@
 ;; limitations under the License.
 ;;
 
-
 (ns bert-qa.infer
   (:require [clojure.string :as string]
-[clojure.reflect :as r]
 [cheshire.core :as json]
 [clojure.java.io :as io]
-[clojure.set :as set]
 [org.apache.clojure-mxnet.dtype :as dtype]
 [org.apache.clojure-mxnet.context :as context]
 [org.apache.clojure-mxnet.layout :as layout]
@@ -30,11 +27,7 @@
 [clojure.pprint :as pprint]))
 
 (def model-path-prefix "model/static_bert_qa")
-;; epoch number of the model
-(def epoch 2)
-;; the vocabulary used in the model
-(def model-vocab "model/vocab.json")
-;; the input question
+
 ;; the maximum length of the sequence
 (def seq-length 384)
 
@@ -60,16 +53,13 @@
 (into tokens (repeat (- num (count tokens)) pad-item
 
 (defn get-vocab []
-  (let [vocab (json/parse-stream (clojure.java.io/reader "model/vocab.json"))]
+  (let [vocab (json/parse-stream (io/reader "model/vocab.json"))]
 {:idx->token (get vocab "idx_to_token")
  :token->idx (get vocab "token_to_idx")}))
 
 (defn tokens->idxs [token->idx tokens]
   (let [unk-idx (get token->idx "[UNK]")]
-   (mapv #(get token->idx % unk-idx) tokens)))
-
-(defn idxs->tokens [idx->token idxs]
-  (mapv #(get idx->token %) idxs))
+(mapv #(get token->idx % unk-idx) tokens)))
 
 (defn post-processing [result tokens]
   (let [output1 (ndarray/slice-axis result 2 0 1)
@@ -131,22 +121,23 @@
  :tokens tokens
  :qa-map qa-map}))
 
-(defn infer [ctx]
-  (let [ctx (context/default-context)
-predictor (make-predictor ctx)
-{:keys [idx->token token->idx]} (get-vocab)
+(defn infer
+  ([] (infer (context/default-context)))
+  ([ctx]
+   (let [predictor (make-predictor ctx)
+ {:keys [idx->token token->idx]} (get-vocab)
 ;;; samples taken from 
https://rajpurkar.github.io/SQuAD-explorer/explore/v2.0/dev/
-question-answers (clojure.edn/read-string (slurp "squad-samples.edn"))]
-(doseq [qa-map question-answers]
-  (let [{:keys [input-batch tokens qa-map]} (pre-processing ctx idx->token 
token->idx qa-map)
-result (first (infer/predict-with-ndarray predictor input-batch))
-answer (post-processing result tokens)]
-(println "===")
-(println "  Question Answer Data")
-(pprint/pprint qa-map)
-(println)
-(println "  Predicted Answer: " answer)
-(println "==

[incubator-mxnet] branch revert-14800-arthur/clojure-ndarray-api-random-merged created (now 95d624d)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch 
revert-14800-arthur/clojure-ndarray-api-random-merged
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at 95d624d  Revert "[clojure][generator] ndarray/symbol api random merged 
(#14800)"

This branch includes the following new commits:

 new 95d624d  Revert "[clojure][generator] ndarray/symbol api random merged 
(#14800)"

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-mxnet] 01/01: Revert "[clojure][generator] ndarray/symbol api random merged (#14800)"

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch 
revert-14800-arthur/clojure-ndarray-api-random-merged
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 95d624d5d7badd7d3450c6cfc1f8398eb83c723f
Author: Carin Meier 
AuthorDate: Fri Apr 26 13:32:09 2019 -0400

Revert "[clojure][generator] ndarray/symbol api random merged (#14800)"

This reverts commit 5dd9fa27d8bdd2a8677b7c275a494d17082c0e1c.
---
 contrib/clojure-package/src/dev/generator.clj  | 298 +++--
 .../src/org/apache/clojure_mxnet/ndarray_api.clj   |   8 +-
 .../apache/clojure_mxnet/ndarray_random_api.clj|  28 --
 .../org/apache/clojure_mxnet/symbol_random_api.clj |  32 ---
 .../clojure-package/test/dev/generator_test.clj|  47 +---
 .../test/good-test-ndarray-random-api.clj  |  95 ---
 .../test/good-test-symbol-random-api.clj   | 118 
 .../test/test-ndarray-random-api.clj   |  95 ---
 .../test/test-symbol-random-api.clj| 118 
 9 files changed, 108 insertions(+), 731 deletions(-)

diff --git a/contrib/clojure-package/src/dev/generator.clj 
b/contrib/clojure-package/src/dev/generator.clj
index d1f59dc..864c67f 100644
--- a/contrib/clojure-package/src/dev/generator.clj
+++ b/contrib/clojure-package/src/dev/generator.clj
@@ -29,7 +29,6 @@
 
 
 (defn clojure-case
-  "Transforms a scala string (function name) to clojure case"
   [string]
   (-> string
   (clojure.string/replace #"(\s+)([A-Z][a-z]+)" "$1-$2")
@@ -58,9 +57,10 @@
count
pos?))
 
+
 (defn increment-param-name [pname]
   (if-let [num-str (re-find #"-\d" pname)]
-(str
+(str 
  (first (clojure.string/split pname #"-"))
  "-"
  (inc (Integer/parseInt (last (clojure.string/split num-str #"-")
@@ -130,33 +130,15 @@
   (.write w fstr))
 (.write w "\n"
 
-(defn remove-prefix
-  [prefix s]
-  (let [regex (re-pattern (str prefix "(.*)"))
-replacement "$1"]
-  (clojure.string/replace s regex replacement)))
-
-(defn in-namespace-random? [op-name]
-  (or (clojure.string/includes? op-name "random_")
-  (clojure.string/includes? op-name "sample_")))
-
-(defn op-name->namespace-type [op-name]
-  (cond
-(#{"uniform" "normal"} op-name)  :deprecated
-(clojure.string/includes? op-name "random_") :random
-(clojure.string/includes? op-name "sample_") :random
-:else:core))
-
 ;;; Common operations
 
 (def libinfo (Base/_LIB))
-
 (def op-names
   (let [l ($ ListBuffer/empty)]
-(.mxListAllOpNames libinfo l)
-(->> l
- (util/buffer->vec)
- (remove #(or (= "Custom" %) (re-matches #"^_.*" %))
+(do (.mxListAllOpNames libinfo l)
+(remove #(or (= "Custom" %)
+ (re-matches #"^_.*" %))
+(util/buffer->vec l)
 
 (defn- parse-arg-type [s]
   (let [[_ var-arg-type _ set-arg-type arg-spec _ type-req _ default-val] 
(re-find 
#"(([\w-\[\]\s]+)|\{([^}]+)\})\s*(\([^)]+\))?(,\s*(optional|required)(,\s*default=(.*))?)?"
 s)]
@@ -306,6 +288,8 @@
  `(~'defn ~function-name
~@(remove nil? (gen-symbol-function-arity op-name op-values 
function-name))
 
+
+
 (def symbol-gen-ns "(ns org.apache.clojure-mxnet.symbol
   (:refer-clojure :exclude [* - + > >= < <= / cast concat identity flatten 
load max
 min repeat reverse set sort take to-array empty sin
@@ -316,9 +300,7 @@
 
 (defn generate-symbol-file []
   (println "Generating symbol file")
-  (write-to-file all-symbol-functions
- symbol-gen-ns
- "src/org/apache/clojure_mxnet/gen/symbol.clj"))
+  (write-to-file all-symbol-functions symbol-gen-ns 
"src/org/apache/clojure_mxnet/gen/symbol.clj"))
 
 ;;; NDArray
 
@@ -340,17 +322,21 @@
count
pos?))
 
+
 (def ndarray-public-to-hand-gen
   (filter is-ndarray-hand-gen? ndarray-public-no-default))
 (def ndarray-public-to-gen
   (get-public-to-gen-methods ndarray-public-to-hand-gen
  ndarray-public-no-default))
 
+
 (count ndarray-public-to-hand-gen) ;=> 15
 (count ndarray-public-to-gen) ;=> 486
 
 (->> ndarray-public-to-hand-gen (map :name) (into #{}))
 
+
+
 (defn gen-ndarray-function-arity [op-name op-values]
   (for [[param-count info] op-values]
 (let [targets (->> (mapv :parameter-types info)
@@ -394,8 +380,7 @@
 (def all-ndarray-functions
   (gen-ndarray-functions ndarray-public-to-gen))
 
-(def ndarray-gen-ns
-  "(ns org.apache.clojure-mxnet.ndarray
+(def ndarray-gen-ns "(ns org.apache.clojure-mxne

[incubator-mxnet] branch master updated: [clojure][generator] ndarray/symbol api random merged (#14800)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 5dd9fa2  [clojure][generator] ndarray/symbol api random merged (#14800)
5dd9fa2 is described below

commit 5dd9fa27d8bdd2a8677b7c275a494d17082c0e1c
Author: Arthur Caillau 
AuthorDate: Fri Apr 26 19:30:41 2019 +0200

[clojure][generator] ndarray/symbol api random merged (#14800)

* [clojure][generator] add random namespace generation

* `ndarray_random_api`
* `symbol_random_api`

* fix tests
---
 contrib/clojure-package/src/dev/generator.clj  | 298 ++---
 .../src/org/apache/clojure_mxnet/ndarray_api.clj   |   8 +-
 .../{ndarray_api.clj => ndarray_random_api.clj}|  10 +-
 .../{ndarray_api.clj => symbol_random_api.clj} |  20 +-
 .../clojure-package/test/dev/generator_test.clj|  47 +++-
 .../test/good-test-ndarray-random-api.clj  |  95 +++
 .../test/good-test-symbol-random-api.clj   | 118 
 .../test/test-ndarray-random-api.clj   |  95 +++
 .../test/test-symbol-random-api.clj| 118 
 9 files changed, 684 insertions(+), 125 deletions(-)

diff --git a/contrib/clojure-package/src/dev/generator.clj 
b/contrib/clojure-package/src/dev/generator.clj
index 864c67f..d1f59dc 100644
--- a/contrib/clojure-package/src/dev/generator.clj
+++ b/contrib/clojure-package/src/dev/generator.clj
@@ -29,6 +29,7 @@
 
 
 (defn clojure-case
+  "Transforms a scala string (function name) to clojure case"
   [string]
   (-> string
   (clojure.string/replace #"(\s+)([A-Z][a-z]+)" "$1-$2")
@@ -57,10 +58,9 @@
count
pos?))
 
-
 (defn increment-param-name [pname]
   (if-let [num-str (re-find #"-\d" pname)]
-(str 
+(str
  (first (clojure.string/split pname #"-"))
  "-"
  (inc (Integer/parseInt (last (clojure.string/split num-str #"-")
@@ -130,15 +130,33 @@
   (.write w fstr))
 (.write w "\n"
 
+(defn remove-prefix
+  [prefix s]
+  (let [regex (re-pattern (str prefix "(.*)"))
+replacement "$1"]
+  (clojure.string/replace s regex replacement)))
+
+(defn in-namespace-random? [op-name]
+  (or (clojure.string/includes? op-name "random_")
+  (clojure.string/includes? op-name "sample_")))
+
+(defn op-name->namespace-type [op-name]
+  (cond
+(#{"uniform" "normal"} op-name)  :deprecated
+(clojure.string/includes? op-name "random_") :random
+(clojure.string/includes? op-name "sample_") :random
+:else:core))
+
 ;;; Common operations
 
 (def libinfo (Base/_LIB))
+
 (def op-names
   (let [l ($ ListBuffer/empty)]
-(do (.mxListAllOpNames libinfo l)
-(remove #(or (= "Custom" %)
- (re-matches #"^_.*" %))
-(util/buffer->vec l)
+(.mxListAllOpNames libinfo l)
+(->> l
+ (util/buffer->vec)
+ (remove #(or (= "Custom" %) (re-matches #"^_.*" %))
 
 (defn- parse-arg-type [s]
   (let [[_ var-arg-type _ set-arg-type arg-spec _ type-req _ default-val] 
(re-find 
#"(([\w-\[\]\s]+)|\{([^}]+)\})\s*(\([^)]+\))?(,\s*(optional|required)(,\s*default=(.*))?)?"
 s)]
@@ -288,8 +306,6 @@
  `(~'defn ~function-name
~@(remove nil? (gen-symbol-function-arity op-name op-values 
function-name))
 
-
-
 (def symbol-gen-ns "(ns org.apache.clojure-mxnet.symbol
   (:refer-clojure :exclude [* - + > >= < <= / cast concat identity flatten 
load max
 min repeat reverse set sort take to-array empty sin
@@ -300,7 +316,9 @@
 
 (defn generate-symbol-file []
   (println "Generating symbol file")
-  (write-to-file all-symbol-functions symbol-gen-ns 
"src/org/apache/clojure_mxnet/gen/symbol.clj"))
+  (write-to-file all-symbol-functions
+ symbol-gen-ns
+ "src/org/apache/clojure_mxnet/gen/symbol.clj"))
 
 ;;; NDArray
 
@@ -322,21 +340,17 @@
count
pos?))
 
-
 (def ndarray-public-to-hand-gen
   (filter is-ndarray-hand-gen? ndarray-public-no-default))
 (def ndarray-public-to-gen
   (get-public-to-gen-methods ndarray-public-to-hand-gen
  ndarray-public-no-default))
 
-
 (count ndarray-public-to-hand-gen) ;=> 15
 (count ndarray-public-to-gen) ;=> 486
 
 (->> ndarray-public-to-hand-gen (map :name) (into #{}))
 
-
-
 (defn gen-ndarray-function-arity [op-name op-values]
   (for [[param-count info] op-values]
 (let [targets (->> (mapv :parameter-types info)
@@ -380,7 +394,8 @@
 (def all-ndarray-fu

[incubator-mxnet] branch new-bert-example-with-finetuning deleted (was 7de5a26)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch new-bert-example-with-finetuning
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


 was 7de5a26  add license

This change permanently discards the following revisions:

 discard 7de5a26  add license

[incubator-mxnet] branch clojure-bert-sentence-pair-classification deleted (was 1c6e734)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-bert-sentence-pair-classification
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


 was 1c6e734  gradients not exploding

This change permanently discards the following revisions:

 discard 1c6e734  gradients not exploding
 discard cb8ff3b  clean up extra stuff
 discard 7295d8c  base working (although slow)
 discard 5270b89  more wip
 discard 3f1058e  wip
 discard 8e89a9c  util to update roster

[incubator-mxnet] 01/01: add license

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch new-bert-example-with-finetuning
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 7de5a2669527dbd4ade0c4c939af41943c499c2d
Author: gigasquid 
AuthorDate: Fri Apr 26 10:08:32 2019 -0400

add license
---
 .gitignore  |  6 ++
 contrib/clojure-package/examples/bert/fine-tune-bert.md | 16 
 contrib/clojure-package/examples/bert/src/bert/util.clj | 17 +
 3 files changed, 39 insertions(+)

diff --git a/.gitignore b/.gitignore
index 705ef92..f3b3305 100644
--- a/.gitignore
+++ b/.gitignore
@@ -162,3 +162,9 @@ coverage.xml
 
 # Local CMake build config
 cmake_options.yml
+contrib/clojure-package/examples/bert/.lein-failures
+contrib/clojure-package/examples/bert/.lein-repl-history
+contrib/clojure-package/examples/bert/.nrepl-port
+contrib/clojure-package/examples/bert/target/classes/META-INF/maven/bert-qa/bert-qa/pom.properties
+contrib/clojure-package/examples/bert/target/repl-port
+contrib/clojure-package/examples/bert/target/stale/leiningen.core.classpath.extract-native-dependencies
\ No newline at end of file
diff --git a/contrib/clojure-package/examples/bert/fine-tune-bert.md 
b/contrib/clojure-package/examples/bert/fine-tune-bert.md
index 1c71998..08cf290 100644
--- a/contrib/clojure-package/examples/bert/fine-tune-bert.md
+++ b/contrib/clojure-package/examples/bert/fine-tune-bert.md
@@ -1,3 +1,19 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 # Fine-tuning Sentence Pair Classification with BERT
 
diff --git a/contrib/clojure-package/examples/bert/src/bert/util.clj 
b/contrib/clojure-package/examples/bert/src/bert/util.clj
index 295b5a0..7c8e36f 100644
--- a/contrib/clojure-package/examples/bert/src/bert/util.clj
+++ b/contrib/clojure-package/examples/bert/src/bert/util.clj
@@ -1,3 +1,20 @@
+;;
+;; Licensed to the Apache Software Foundation (ASF) under one or more
+;; contributor license agreements.  See the NOTICE file distributed with
+;; this work for additional information regarding copyright ownership.
+;; The ASF licenses this file to You under the Apache License, Version 2.0
+;; (the "License"); you may not use this file except in compliance with
+;; the License.  You may obtain a copy of the License at
+;;
+;;http://www.apache.org/licenses/LICENSE-2.0
+;;
+;; Unless required by applicable law or agreed to in writing, software
+;; distributed under the License is distributed on an "AS IS" BASIS,
+;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+;; See the License for the specific language governing permissions and
+;; limitations under the License.
+;;
+
 (ns bert.util
   (:require [clojure.java.io :as io]
 [clojure.string :as string]

[incubator-mxnet] branch new-bert-example-with-finetuning created (now 7de5a26)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch new-bert-example-with-finetuning
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at 7de5a26  add license

This branch includes the following new commits:

 new 7de5a26  add license

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-mxnet] branch master updated: [Clojure] Better api docstrings by replacing newlines (#14752)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new a1b0a3a  [Clojure] Better api docstrings by replacing newlines (#14752)
a1b0a3a is described below

commit a1b0a3a9d7f44a917ec25bb59e8500d92f0e69f1
Author: Kedar Bellare 
AuthorDate: Tue Apr 23 09:07:50 2019 -0700

[Clojure] Better api docstrings by replacing newlines (#14752)
---
 contrib/clojure-package/src/dev/generator.clj  | 16 +++-
 .../clojure-package/test/good-test-ndarray-api.clj | 85 -
 .../clojure-package/test/good-test-symbol-api.clj  | 87 +-
 3 files changed, 181 insertions(+), 7 deletions(-)

diff --git a/contrib/clojure-package/src/dev/generator.clj 
b/contrib/clojure-package/src/dev/generator.clj
index 34210be..864c67f 100644
--- a/contrib/clojure-package/src/dev/generator.clj
+++ b/contrib/clojure-package/src/dev/generator.clj
@@ -123,7 +123,11 @@
 (.write w "\n\n")
 (.write w "\n\n")
   (doseq [f functions]
-(clojure.pprint/pprint f w)
+(let [fstr (-> f
+   clojure.pprint/pprint
+   with-out-str
+   (clojure.string/replace #"\\n\\n" "\n"))]
+  (.write w fstr))
 (.write w "\n"
 
 ;;; Common operations
@@ -447,7 +451,10 @@
 :type "Map[String, String]"
 :optional? true
 :description "Attributes of the symbol"}))
-doc (gen-symbol-api-doc fn-description params)
+doc (clojure.string/join
+ "\n\n  "
+ (-> (gen-symbol-api-doc fn-description params)
+ (clojure.string/split #"\n")))
 default-call (gen-symbol-api-default-arity op-name params)]
 `(~'defn ~(symbol fn-name)
   ~doc
@@ -520,7 +527,10 @@
  :type "NDArray-or-Symbol"
  :optional? true
  :description "Output array."}))
-doc (gen-ndarray-api-doc fn-description params)
+doc (clojure.string/join
+ "\n\n  "
+ (-> (gen-ndarray-api-doc fn-description params)
+ (clojure.string/split #"\n")))
 opt-params (filter :optional? params)
 req-params (remove :optional? params)
 req-call (gen-ndarray-api-required-arity fn-name req-params)
diff --git a/contrib/clojure-package/test/good-test-ndarray-api.clj 
b/contrib/clojure-package/test/good-test-ndarray-api.clj
index 1b83a7b..7554089 100644
--- a/contrib/clojure-package/test/good-test-ndarray-api.clj
+++ b/contrib/clojure-package/test/good-test-ndarray-api.clj
@@ -31,7 +31,23 @@
 
 (defn
  activation
- "Applies an activation function element-wise to the input.\n\nThe following 
activation functions are supported:\n\n- `relu`: Rectified Linear Unit, 
:math:`y = max(x, 0)`\n- `sigmoid`: :math:`y = \\frac{1}{1 + exp(-x)}`\n- 
`tanh`: Hyperbolic tangent, :math:`y = \\frac{exp(x) - exp(-x)}{exp(x) + 
exp(-x)}`\n- `softrelu`: Soft ReLU, or SoftPlus, :math:`y = log(1 + exp(x))`\n- 
`softsign`: :math:`y = \\frac{x}{1 + abs(x)}`\n\n\n\nDefined in 
src/operator/nn/activation.cc:L167\n\n`data`: The in [...]
+ "Applies an activation function element-wise to the input.
+  
+  The following activation functions are supported:
+  
+  - `relu`: Rectified Linear Unit, :math:`y = max(x, 0)`
+  - `sigmoid`: :math:`y = \\frac{1}{1 + exp(-x)}`
+  - `tanh`: Hyperbolic tangent, :math:`y = \\frac{exp(x) - exp(-x)}{exp(x) + 
exp(-x)}`
+  - `softrelu`: Soft ReLU, or SoftPlus, :math:`y = log(1 + exp(x))`
+  - `softsign`: :math:`y = \\frac{x}{1 + abs(x)}`
+  
+  
+  
+  Defined in src/operator/nn/activation.cc:L167
+  
+  `data`: The input array.
+  `act-type`: Activation function to be applied.
+  `out`: Output array. (optional)"
  ([data act-type] (activation {:data data, :act-type act-type}))
  ([{:keys [data act-type out], :or {out nil}, :as opts}]
   (util/coerce-return
@@ -39,7 +55,72 @@
 
 (defn
  batch-norm
- "Batch normalization.\n\nNormalizes a data batch by mean and variance, and 
applies a scale ``gamma`` as\nwell as offset ``beta``.\n\nAssume the input has 
more than one dimension and we normalize along axis 1.\nWe first compute the 
mean and variance along this axis:\n\n.. math::\n\n  data\\_mean[i] = 
mean(data[:,i,:,...]) \n  data\\_var[i] = var(data[:,i,:,...])\n\nThen 
compute the normalized output, which has the same shape as input, as 
following:\n\n.. math::\n\n  out[:,i,:,...] =  [...]
+ "Batch normalization.
+  
+  Normalizes a data batch by mean and variance, and applies a scale ``gamma`` 
as
+  well as offset ``beta``.
+  
+  Assume the

[incubator-mxnet] 05/06: clean up extra stuff

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-bert-sentence-pair-classification
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit cb8ff3b0cd0e90aa0e5e900e3a6bf324b111bdfb
Author: gigasquid 
AuthorDate: Fri Apr 19 17:49:38 2019 -0400

clean up extra stuff
---
 .../src/bert_qa/bert_sentence_classification.clj   | 30 --
 1 file changed, 30 deletions(-)

diff --git 
a/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
index 7f6723e..b7d4425 100644
--- 
a/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
+++ 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
@@ -28,31 +28,6 @@
 (def seq-length 128)
 
 
-
-
-;;; Data preprocessing
-
-#_(defn fit [devs msymbol arg-params aux-params]
-  (let [mod (-> (m/module msymbol {:contexts devs})
-(m/bind {:data-shapes (mx-io/provide-data-desc train-iter) 
:label-shapes (mx-io/provide-label-desc val-iter)})
-(m/init-params {:arg-params arg-params :aux-params aux-params
-:allow-missing true}))]
-(m/fit mod
-   {:train-data train-iter
-:eval-data val-iter
-:num-epoch 1
-:fit-params (m/fit-params {:intializer (init/xavier {:rand-type 
"gaussian"
- :factor-type 
"in"
- :magnitude 2})
-   :batch-end-callback 
(callback/speedometer batch-size 10)})})))
-
-#_(defn fine-tune! [devs]
-  (let [{:keys [msymbol arg-params aux-params] :as model} (get-model)
-new-model (fine-tune-model (merge model {:num-classes 2}))]
-new-model
-#_(fit devs net new-args arg-params)))
-
-
 (defn pre-processing [ctx idx->token token->idx train-item]
 (let [[sentence-a sentence-b label] train-item
;;; pre-processing tokenize sentence
@@ -116,11 +91,6 @@
  :dtype dtype/FLOAT32
  :layout layout/NT}])
 
-  #_(def base-mod (-> bert-base
-(m/bind {:data-shapes input-descs})
-(m/init-params {:arg-params arg-params :aux-params 
aux-params
-:allow-missing true})))
-
   ;;; Data Preprocessing for BERT
 
   ;; For demonstration purpose, we use the dev set of the Microsoft Research 
Paraphrase Corpus dataset. The file is named ‘dev.tsv’. Let’s take a look at 
the raw dataset.

[incubator-mxnet] 04/06: base working (although slow)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-bert-sentence-pair-classification
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 7295d8c3cc3ed72ff23afc4372719382fb16c0dc
Author: gigasquid 
AuthorDate: Fri Apr 19 17:43:41 2019 -0400

base working (although slow)
---
 .../src/bert_qa/bert_sentence_classification.clj   | 69 ++
 1 file changed, 30 insertions(+), 39 deletions(-)

diff --git 
a/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
index 053dade..7f6723e 100644
--- 
a/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
+++ 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
@@ -11,6 +11,7 @@
 [org.apache.clojure-mxnet.symbol :as sym]
 [org.apache.clojure-mxnet.module :as m]
 [org.apache.clojure-mxnet.infer :as infer]
+[org.apache.clojure-mxnet.eval-metric :as eval-metric]
 [org.apache.clojure-mxnet.optimizer :as optimizer]
 [clojure.pprint :as pprint]
 [clojure-csv.core :as csv]
@@ -115,15 +116,7 @@
  :dtype dtype/FLOAT32
  :layout layout/NT}])
 
-  ;; now create the module
-  (def mod (-> (m/module model-sym {:contexts devs
-:data-names ["data0" "data1" "data2"]})
-   (m/bind {:data-shapes input-descs :label-shapes label-descs})
-   (m/init-params {:arg-params arg-params :aux-params aux-params
-   :allow-missing true})
-   (m/init-optimizer {:optimizer (optimizer/sgd {:learning-rate 
0.01 :momentum 0.9})})))
-
-  (def base-mod (-> bert-base
+  #_(def base-mod (-> bert-base
 (m/bind {:data-shapes input-descs})
 (m/init-params {:arg-params arg-params :aux-params 
aux-params
 :allow-missing true})))
@@ -152,8 +145,9 @@
   (vals (select-keys (first raw-file) [3 4 0]))
   ;=> ("#1 String" "#2 String" "Quality")
   (def data-train-raw (->> raw-file
-   (map #(vals (select-keys % [3 4 0])))
+   (mapv #(vals (select-keys % [3 4 0])))
(rest) ;;drop header
+   (into [])
))
   (def sample (first data-train-raw))
   (nth sample 0) ;;;sentence a
@@ -174,34 +168,31 @@
   ;;; our sample item
   (def sample-data (pre-processing (context/default-context) idx->token 
token->idx sample))
 
-  
-
-  ;; with a predictor
-  (defn make-predictor [ctx]
-  (let [input-descs [{:name "data0"
-  :shape [1 seq-length]
-  :dtype dtype/FLOAT32
-  :layout layout/NT}
- {:name "data1"
-  :shape [1 seq-length]
-  :dtype dtype/FLOAT32
-  :layout layout/NT}
- {:name "data2"
-  :shape [1]
-  :dtype dtype/FLOAT32
-  :layout layout/N}]
-factory (infer/model-factory model-path-prefix input-descs)]
-(infer/create-predictor
- factory
- {:contexts [ctx]
-  :epoch 0})))
-
-  (def predictor (make-predictor (context/default-context)))
-  (def sample-result (first (infer/predict-with-ndarray predictor 
(:input-batch sample-data
-  
-
-
-  
- 
+  (def train-count (count data-train-raw));=> 389
+
+;; now create the module
+  (def model (-> (m/module model-sym {:contexts devs
+  :data-names ["data0" "data1" "data2"]})
+ (m/bind {:data-shapes input-descs :label-shapes label-descs})
+ (m/init-params {:arg-params arg-params :aux-params aux-params
+ :allow-missing true})
+ (m/init-optimizer {:optimizer (optimizer/sgd {:learning-rate 
0.01 :momentum 0.9})})))
+
+  (def metric (eval-metric/accuracy))
+  (def num-epoch 3)
+  (def processed-datas (mapv #(pre-processing (context/default-context) 
idx->token token->idx %)
+ data-train-raw))
+
+  (doseq [epoch-num (range num-epoch)]
+(doall (map-indexed (fn [i batch-data]
+  (-> model
+  (m/forward {:data (:input-batch batch-data)})
+  (m/update-metric metric [(:label batch-data)])
+  (m/backward)
+  (m/update))
+  (when (mod i 10)
+(println 

[incubator-mxnet] 02/06: wip

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-bert-sentence-pair-classification
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 3f1058edba47ee114c13284d566ab5a3389c77f1
Author: gigasquid 
AuthorDate: Fri Apr 19 15:13:40 2019 -0400

wip
---
 .../clojure-package/examples/bert-qa/project.clj   |   3 +-
 .../src/bert_qa/bert_sentence_classification.clj   | 138 +
 2 files changed, 140 insertions(+), 1 deletion(-)

diff --git a/contrib/clojure-package/examples/bert-qa/project.clj 
b/contrib/clojure-package/examples/bert-qa/project.clj
index d256d44..7d79dc6 100644
--- a/contrib/clojure-package/examples/bert-qa/project.clj
+++ b/contrib/clojure-package/examples/bert-qa/project.clj
@@ -21,7 +21,8 @@
   :plugins [[lein-cljfmt "0.5.7"]]
   :dependencies [[org.clojure/clojure "1.9.0"]
  [org.apache.mxnet.contrib.clojure/clojure-mxnet 
"1.5.0-SNAPSHOT"]
- [cheshire "5.8.1"]]
+ [cheshire "5.8.1"]
+ [clojure-csv/clojure-csv "2.0.1"]]
   :pedantic? :skip
   :java-source-paths ["src/java"]
   :main bert-qa.infer
diff --git 
a/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
new file mode 100644
index 000..e53d5da
--- /dev/null
+++ 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
@@ -0,0 +1,138 @@
+(ns bert-qa.bert-sentence-classification
+  (:require [clojure.string :as string]
+[clojure.reflect :as r]
+[cheshire.core :as json]
+[clojure.java.io :as io]
+[clojure.set :as set]
+[org.apache.clojure-mxnet.dtype :as dtype]
+[org.apache.clojure-mxnet.context :as context]
+[org.apache.clojure-mxnet.layout :as layout]
+[org.apache.clojure-mxnet.ndarray :as ndarray]
+[org.apache.clojure-mxnet.symbol :as sym]
+[org.apache.clojure-mxnet.module :as m]
+[clojure.pprint :as pprint]
+[clojure-csv.core :as csv]
+[bert-qa.infer :as bert-infer]))
+
+
+(def model-path-prefix "model/bert-base")
+;; epoch number of the model
+(def epoch 0)
+;; the vocabulary used in the model
+(def model-vocab "model/vocab.json")
+;; the input question
+;; the maximum length of the sequence
+(def seq-length 384)
+
+
+(defn fine-tune-model
+  "msymbol: the pretrained network symbol
+arg-params: the argument parameters of the pretrained model
+num-classes: the number of classes for the fine-tune datasets"
+  [msymbol num-classes]
+  (as-> msymbol data
+(sym/flatten "flatten-finetune" {:data data})
+(sym/fully-connected "fc-finetune" {:data data :num-hidden num-classes})
+(sym/softmax-output "softmax" {:data data})))
+
+;;; Data preprocessing
+
+#_(defn fit [devs msymbol arg-params aux-params]
+  (let [mod (-> (m/module msymbol {:contexts devs})
+(m/bind {:data-shapes (mx-io/provide-data-desc train-iter) 
:label-shapes (mx-io/provide-label-desc val-iter)})
+(m/init-params {:arg-params arg-params :aux-params aux-params
+:allow-missing true}))]
+(m/fit mod
+   {:train-data train-iter
+:eval-data val-iter
+:num-epoch 1
+:fit-params (m/fit-params {:intializer (init/xavier {:rand-type 
"gaussian"
+ :factor-type 
"in"
+ :magnitude 2})
+   :batch-end-callback 
(callback/speedometer batch-size 10)})})))
+
+#_(defn fine-tune! [devs]
+  (let [{:keys [msymbol arg-params aux-params] :as model} (get-model)
+new-model (fine-tune-model (merge model {:num-classes 2}))]
+new-model
+#_(fit devs net new-args arg-params)))
+
+
+(comment
+
+  ;;; load the pre-trained BERT model using the module api
+  (def bert-base (m/load-checkpoint {:prefix model-path-prefix :epoch 0}))
+  ;;; now that we have loaded the BERT model we need to attach an additional 
layer for classification which is a dense layer with 2 classes
+  (def model (fine-tune-model (m/symbol bert-base) 2))
+
+  ;;; Data Preprocessing for BERT
+
+  ;; For demonstration purpose, we use the dev set of the Microsoft Research 
Paraphrase Corpus dataset. The file is named ‘dev.tsv’. Let’s take a look at 
the raw dataset.
+  ;; it contains 5 columns seperated by tabs
+  (def raw-file (->> (string/split (slurp "dev.tsv") #"\n")
+ (map #(string/split % #"\t") )))
+  (def raw-file (csv/parse-csv (slurp "dev.t

[incubator-mxnet] 03/06: more wip

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-bert-sentence-pair-classification
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 5270b89aaa218d84328a1c66ca251c66c45528e9
Author: gigasquid 
AuthorDate: Fri Apr 19 17:01:40 2019 -0400

more wip
---
 .../src/bert_qa/bert_sentence_classification.clj   | 141 +++--
 1 file changed, 105 insertions(+), 36 deletions(-)

diff --git 
a/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
index e53d5da..053dade 100644
--- 
a/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
+++ 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
@@ -10,6 +10,8 @@
 [org.apache.clojure-mxnet.ndarray :as ndarray]
 [org.apache.clojure-mxnet.symbol :as sym]
 [org.apache.clojure-mxnet.module :as m]
+[org.apache.clojure-mxnet.infer :as infer]
+[org.apache.clojure-mxnet.optimizer :as optimizer]
 [clojure.pprint :as pprint]
 [clojure-csv.core :as csv]
 [bert-qa.infer :as bert-infer]))
@@ -22,18 +24,10 @@
 (def model-vocab "model/vocab.json")
 ;; the input question
 ;; the maximum length of the sequence
-(def seq-length 384)
+(def seq-length 128)
+
 
 
-(defn fine-tune-model
-  "msymbol: the pretrained network symbol
-arg-params: the argument parameters of the pretrained model
-num-classes: the number of classes for the fine-tune datasets"
-  [msymbol num-classes]
-  (as-> msymbol data
-(sym/flatten "flatten-finetune" {:data data})
-(sym/fully-connected "fc-finetune" {:data data :num-hidden num-classes})
-(sym/softmax-output "softmax" {:data data})))
 
 ;;; Data preprocessing
 
@@ -58,12 +52,81 @@
 #_(fit devs net new-args arg-params)))
 
 
+(defn pre-processing [ctx idx->token token->idx train-item]
+(let [[sentence-a sentence-b label] train-item
+   ;;; pre-processing tokenize sentence
+  token-1 (bert-infer/tokenize (string/lower-case sentence-a))
+  token-2 (bert-infer/tokenize (string/lower-case sentence-b))
+  valid-length (+ (count token-1) (count token-2))
+;;; generate token types [......]
+  qa-embedded (into (bert-infer/pad [] 0 (count token-1))
+(bert-infer/pad [] 1 (count token-2)))
+  token-types (bert-infer/pad qa-embedded 0 seq-length)
+;;; make BERT pre-processing standard
+  token-2 (conj token-2 "[SEP]")
+  token-1 (into [] (concat ["[CLS]"] token-1 ["[SEP]"] token-2))
+  tokens (bert-infer/pad token-1 "[PAD]" seq-length)
+;;; pre-processing - token to index translation
+  indexes (bert-infer/tokens->idxs token->idx tokens)]
+{:input-batch [(ndarray/array indexes [1 seq-length] {:context ctx})
+   (ndarray/array token-types [1 seq-length] {:context ctx})
+   (ndarray/array [valid-length] [1] {:context ctx})]
+ :label (if (= "0" label)
+  (ndarray/array [1 0] [2] {:ctx ctx})
+  (ndarray/array [0 1] [2] {:ctx ctx}))
+ :tokens tokens
+ :train-item train-item}))
+
+(defn fine-tune-model
+  "msymbol: the pretrained network symbol
+arg-params: the argument parameters of the pretrained model
+num-classes: the number of classes for the fine-tune datasets"
+  [msymbol num-classes]
+  (as-> msymbol data
+(sym/flatten "flatten-finetune" {:data data})
+(sym/fully-connected "fc-finetune" {:data data :num-hidden num-classes})
+(sym/softmax-output "softmax" {:data data})))
+
+
 (comment
 
   ;;; load the pre-trained BERT model using the module api
   (def bert-base (m/load-checkpoint {:prefix model-path-prefix :epoch 0}))
   ;;; now that we have loaded the BERT model we need to attach an additional 
layer for classification which is a dense layer with 2 classes
-  (def model (fine-tune-model (m/symbol bert-base) 2))
+  (def model-sym (fine-tune-model (m/symbol bert-base) 2))
+  (def arg-params (m/arg-params bert-base))
+  (def aux-params (m/aux-params bert-base))
+
+  (def devs [(context/default-context)])
+  (def input-descs [{:name "data0"
+  :shape [1 seq-length]
+  :dtype dtype/FLOAT32
+  :layout layout/NT}
+ {:name "data1"
+  :shape [1 seq-length]
+  :dtype dtype/FLOAT32
+  :layout layout/NT}
+ {:name "data2"
+  :shape [1]
+  :dtype dtype/FLOAT32
+ 

[incubator-mxnet] 01/06: util to update roster

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-bert-sentence-pair-classification
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 8e89a9c406e88ffb7c1da433a98a94cdf9a0ca21
Author: gigasquid 
AuthorDate: Sun Apr 14 08:54:35 2019 -0400

util to update roster
---
 .../examples/bert-qa/test/bert_qa/roster.clj   | 35 ++
 1 file changed, 35 insertions(+)

diff --git a/contrib/clojure-package/examples/bert-qa/test/bert_qa/roster.clj 
b/contrib/clojure-package/examples/bert-qa/test/bert_qa/roster.clj
new file mode 100644
index 000..75843ce
--- /dev/null
+++ b/contrib/clojure-package/examples/bert-qa/test/bert_qa/roster.clj
@@ -0,0 +1,35 @@
+(ns bert-qa.roster
+  (:require  [clojure.test :as t]
+ [clojure.string :as string]))
+
+(require '[clojure.xml :as xml]
+ '[clojure.zip :as zip])
+
+;;convenience function, first seen at nakkaya.com later in clj.zip src
+(defn zip-str [s]
+  (zip/xml-zip 
+  (xml/parse (java.io.ByteArrayInputStream. (.getBytes s)
+
+(def x (zip-str (string/replace (slurp "ppmc-roster.txt") #"\"
+"")))
+
+
+;;; format
+(def people (->> (first x)
+ :content
+  (map :content)
+  (rest)
+  (first)
+  (map :content)
+  (map (fn [x] [(-> (first x) :content first :content first :attrs :href 
(string/split #"\/") last)
+(-> (nth x 2) :content first :content first)]))
+  
+  ))
+
+(defn emit-people [persons]
+  (doseq [[username fullname] persons]
+(xml/emit-element {:tag :tr :content [{:tag :td,:content ["."]}
+  {:tag :td,:content [username]}
+  {:tag :td,:content [fullname]}]})))
+
+(emit-people people)

[incubator-mxnet] branch clojure-bert-sentence-pair-classification created (now 1c6e734)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-bert-sentence-pair-classification
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at 1c6e734  gradients not exploding

This branch includes the following new commits:

 new 8e89a9c  util to update roster
 new 3f1058e  wip
 new 5270b89  more wip
 new 7295d8c  base working (although slow)
 new cb8ff3b  clean up extra stuff
 new 1c6e734  gradients not exploding

The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-mxnet] 06/06: gradients not exploding

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-bert-sentence-pair-classification
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 1c6e734dc53e16063f4c1dca18d8cb9d0fb5d2d2
Author: gigasquid 
AuthorDate: Fri Apr 19 20:13:38 2019 -0400

gradients not exploding
---
 .../src/bert_qa/bert_sentence_classification.clj   | 203 -
 1 file changed, 117 insertions(+), 86 deletions(-)

diff --git 
a/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
index b7d4425..20257d2 100644
--- 
a/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
+++ 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/bert_sentence_classification.clj
@@ -65,16 +65,19 @@
 
 
 (comment
+  
+ (do
 
-  ;;; load the pre-trained BERT model using the module api
-  (def bert-base (m/load-checkpoint {:prefix model-path-prefix :epoch 0}))
-  ;;; now that we have loaded the BERT model we need to attach an additional 
layer for classification which is a dense layer with 2 classes
-  (def model-sym (fine-tune-model (m/symbol bert-base) 2))
-  (def arg-params (m/arg-params bert-base))
-  (def aux-params (m/aux-params bert-base))
-
-  (def devs [(context/default-context)])
-  (def input-descs [{:name "data0"
+;;; load the pre-trained BERT model using the module api
+  
+   (def bert-base (m/load-checkpoint {:prefix model-path-prefix :epoch 0}))
+;;; now that we have loaded the BERT model we need to attach an additional 
layer for classification which is a dense layer with 2 classes
+   (def model-sym (fine-tune-model (m/symbol bert-base) 2))
+   (def arg-params (m/arg-params bert-base))
+   (def aux-params (m/aux-params bert-base))
+
+   (def devs [(context/default-context)])
+   (def input-descs [{:name "data0"
   :shape [1 seq-length]
   :dtype dtype/FLOAT32
   :layout layout/NT}
@@ -86,83 +89,111 @@
   :shape [1]
   :dtype dtype/FLOAT32
   :layout layout/N}])
-  (def label-descs [{:name "softmax_label"
- :shape [1 2]
- :dtype dtype/FLOAT32
- :layout layout/NT}])
-
-  ;;; Data Preprocessing for BERT
-
-  ;; For demonstration purpose, we use the dev set of the Microsoft Research 
Paraphrase Corpus dataset. The file is named ‘dev.tsv’. Let’s take a look at 
the raw dataset.
-  ;; it contains 5 columns seperated by tabs
-  (def raw-file (->> (string/split (slurp "dev.tsv") #"\n")
- (map #(string/split % #"\t") )))
-  (def raw-file (csv/parse-csv (slurp "dev.tsv") :delimiter \tab))
-  (take 3 raw-file)
- ;; (["Quality" "#1 ID" "#2 ID" "#1 String" "#2 String"]
- ;; ["1"
- ;;  "1355540"
- ;;  "1355592"
- ;;  "He said the foodservice pie business doesn 't fit the company 's 
long-term growth strategy ."
- ;;  "\" The foodservice pie business does not fit our long-term growth 
strategy ."]
- ;; ["0"
- ;;  "2029631"
- ;;  "2029565"
- ;;  "Magnarelli said Racicot hated the Iraqi regime and looked forward to 
using his long years of training in the war ."
- ;;  "His wife said he was \" 100 percent behind George Bush \" and looked 
forward to using his years of training in the war ."])
-
-  ;;; for our task we are only interested in the 0 3rd and 4th column
-  (vals (select-keys (first raw-file) [3 4 0]))
-  ;=> ("#1 String" "#2 String" "Quality")
-  (def data-train-raw (->> raw-file
-   (mapv #(vals (select-keys % [3 4 0])))
-   (rest) ;;drop header
-   (into [])
-   ))
-  (def sample (first data-train-raw))
-  (nth sample 0) ;;;sentence a
-  ;=> "He said the foodservice pie business doesn 't fit the company 's 
long-term growth strategy ."
-  (nth sample 1) ;; sentence b
-  "\" The foodservice pie business does not fit our long-term growth strategy 
."
-
-  (nth sample 2) ; 1 means equivalent, 0 means not equivalent
-   ;=> "1"
-
-  ;;; Now we need to turn these into ndarrays to make a Data Iterator
-  (def vocab (bert-infer/get-vocab))
-  (def idx->token (:idx->token vocab))
-  (def token->idx (:token->idx vocab))
+   (def label-descs [{:name "softmax_label"
+  :shape [1 2]
+  :dtype dtype/FLOAT32
+  :layout layout/NT}])
+
+;;; Data Preprocessing for BERT
+
+   ;; For demonstration purpose, we use the 

[incubator-mxnet] branch master updated: [docstring] improve docstring and indentation in `module.clj` (#14705)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 52a3553  [docstring] improve docstring and indentation in `module.clj` 
(#14705)
52a3553 is described below

commit 52a3553fe200214437c717e7b35e6ce39adb59d8
Author: Arthur Caillau 
AuthorDate: Tue Apr 16 15:31:01 2019 +0200

[docstring] improve docstring and indentation in `module.clj` (#14705)
---
 .../src/org/apache/clojure_mxnet/module.clj| 544 +
 .../src/org/apache/clojure_mxnet/util.clj  |   2 +-
 2 files changed, 345 insertions(+), 201 deletions(-)

diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/module.clj 
b/contrib/clojure-package/src/org/apache/clojure_mxnet/module.clj
index aa5ce39..09f17e5 100644
--- a/contrib/clojure-package/src/org/apache/clojure_mxnet/module.clj
+++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/module.clj
@@ -16,6 +16,7 @@
 ;;
 
 (ns org.apache.clojure-mxnet.module
+  "Module API for Clojure package."
   (:refer-clojure :exclude [update symbol])
   (:require [org.apache.clojure-mxnet.callback :as callback]
 [org.apache.clojure-mxnet.context :as context]
@@ -31,18 +32,29 @@
   (:import (org.apache.mxnet.module Module FitParams BaseModule)
(org.apache.mxnet.io MXDataIter NDArrayIter)
(org.apache.mxnet Initializer Optimizer NDArray DataBatch
- Context EvalMetric Monitor Callback$Speedometer 
DataDesc)))
+ Context EvalMetric Monitor Callback$Speedometer
+ DataDesc)))
 
 (defn module
-  "Module is a basic module that wrap a symbol.
-   sym : Symbol definition.
-   map of options
-   :data-names - Input data names.
-   :label-names - Input label names
-   :contexts - Default is cpu().
-   :workload-list - Default nil, indicating uniform workload.
-   :fixed-param-names Default nil, indicating no network parameters are 
fixed."
-  ([sym {:keys [data-names label-names contexts workload-list 
fixed-param-names] :as opts
+  "Module is a basic module that wrap a `symbol`.
+`sym`: Symbol definition.
+`opts-map` {
+  `data-names`: vector of strings - Default is [\"data\"]
+  Input data names
+  `label-names`: vector of strings - Default is [\"softmax_label\"]
+  Input label names
+  `contexts`: Context - Default is `context/cpu`.
+  `workload-list`: Default nil
+  Indicating uniform workload.
+  `fixed-param-names`: Default nil
+  Indicating no network parameters are fixed.
+}
+   Ex:
+ (module sym)
+ (module sym {:data-names [\"data\"]
+  :label-names [\"linear_regression_label\"]}"
+  ([sym {:keys [data-names label-names contexts
+workload-list fixed-param-names] :as opts
  :or {data-names ["data"]
   label-names ["softmax_label"]
   contexts [(context/default-context)]}}]
@@ -80,31 +92,41 @@
 (s/def ::force-rebind boolean?)
 (s/def ::shared-module #(instance? Module))
 (s/def ::grad-req string?)
-(s/def ::bind-opts (s/keys :req-un [::data-shapes] :opt-un [::label-shapes 
::for-training ::inputs-need-grad
-::force-rebind 
::shared-module ::grad-req]))
+(s/def ::bind-opts
+  (s/keys :req-un [::data-shapes]
+  :opt-un [::label-shapes ::for-training ::inputs-need-grad
+   ::force-rebind ::shared-module ::grad-req]))
 
 (defn bind
   "Bind the symbols to construct executors. This is necessary before one
can perform computation with the module.
-   mod : module
-   map of opts:
- :data-shapes Typically is  (provide-data-desc data-iter). Data shape must 
be in the form of io/data-desc with is a map of :name :shape :dtype and :layout
- :label-shapes Typically is  (provide-label-desc data-iter). map of :name 
:shape :dtype and :layout
- :for-training Default is `true`. Whether the executors should be bind for 
training.
- :inputs-need-grad Default is `false`.
-   Whether the gradients to the input data need to be 
computed.
-   Typically this is not needed.
-   But this might be needed when implementing composition 
of modules.
- :force-rebind Default is `false`.
-   This function does nothing if the executors are already 
binded.
-   But with this `true`, the executors will be forced to 
rebind.
- :shared-module Default is nil. This is used in bucketing.
-When not `None`, the shared module essentially corresponds 
to
-a different bucket -- a module with diffe

[incubator-mxnet] branch master updated: [Clojure] Clojure BERT QA example (#14691)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new c2ba51b  [Clojure] Clojure BERT QA example (#14691)
c2ba51b is described below

commit c2ba51b742229b245367a347f2d2cc0e9c8232a2
Author: Carin Meier 
AuthorDate: Sun Apr 14 15:37:03 2019 -0400

[Clojure] Clojure BERT QA example (#14691)

* Initial working example for bert qa

* add RAT
rename core to infer
add integration test

* add rat for project.clj

* Couldn’t resist adding a qa about Clojure

* rat for readme

* feedback from @kedarbellare
---
 .../clojure-package/examples/bert-qa/.gitignore|  12 ++
 contrib/clojure-package/examples/bert-qa/README.md |  91 
 .../examples/bert-qa/get_bert_data.sh  |  29 
 .../clojure-package/examples/bert-qa/project.clj   |  28 
 .../examples/bert-qa/squad-samples.edn |  39 +
 .../examples/bert-qa/src/bert_qa/infer.clj | 159 +
 .../examples/bert-qa/test/bert_qa/infer_test.clj   |  42 ++
 7 files changed, 400 insertions(+)

diff --git a/contrib/clojure-package/examples/bert-qa/.gitignore 
b/contrib/clojure-package/examples/bert-qa/.gitignore
new file mode 100644
index 000..d18f225
--- /dev/null
+++ b/contrib/clojure-package/examples/bert-qa/.gitignore
@@ -0,0 +1,12 @@
+/target
+/classes
+/checkouts
+profiles.clj
+pom.xml
+pom.xml.asc
+*.jar
+*.class
+/.lein-*
+/.nrepl-port
+.hgignore
+.hg/
diff --git a/contrib/clojure-package/examples/bert-qa/README.md 
b/contrib/clojure-package/examples/bert-qa/README.md
new file mode 100644
index 000..9a21bcd
--- /dev/null
+++ b/contrib/clojure-package/examples/bert-qa/README.md
@@ -0,0 +1,91 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# bert-qa
+
+**This example was based off of the Java API one. It shows how to do inference 
with a pre-trained BERT network that is trained on Questions and Answers using 
the [SQuAD Dataset](https://rajpurkar.github.io/SQuAD-explorer/)**
+
+The pretrained model was created using GluonNLP and then exported to the MXNet 
symbol format. You can find more information in the background section below.
+
+In this tutorial, we will walk through the BERT QA model trained by MXNet. 
+Users can provide a question with a paragraph contains answer to the model and
+the model will be able to find the best answer from the answer paragraph.
+
+Example:
+
+```
+{:input-answer "Steam engines are external combustion engines, where the 
working fluid is separate from the combustion products. Non-combustion heat 
sources such as solar power, nuclear power or geothermal energy may be used. 
The ideal thermodynamic cycle used to analyze this process is called the 
Rankine cycle. In the cycle, water is heated and transforms into steam within a 
boiler operating at a high pressure. When expanded through pistons or turbines, 
mechanical work is done. The redu [...]
+  :input-question "Along with geothermal and nuclear, what is a notable 
non-combustion heat source?"
+  :ground-truth-answers ["solar"
+ "solar power"
+ "solar power, nuclear power or geothermal 
energysolar"]}
+```
+
+The prediction in this case would be `solar power`
+
+## Setup Guide
+
+### Step 1: Download the model
+
+For this tutorial, you can get the model and vocabulary by running following 
bash file. This script will use `wget` to download these artifacts from AWS S3.
+
+From the example directory:
+
+```bash
+./get_bert_data.sh
+```
+
+Some sample questions and answers are provide in the `squad-sample.edn` file. 
Some are taken directly from the SQuAD dataset and one was just made up. Feel 
free to edit the file and add your own!
+
+
+## To run
+
+* `lein install` in the root of the main project directory
+* cd into this project directory and do `lein run`. This will execute the cpu 
version.
+
+`lein run :cpu` - to run with cpu
+`lein run :gpu` - to run with gpu
+
+## Background
+
+To learn more about how BERT works in MXNet, please follow this [MXNet Gluon 
tutorial on NLP using 
BERT](https://medium.com/apache-mxnet/gluon-nlp-bert-6a489bdd3340).
+
+The model was extracted from MXNet GluonNLP with static length settings.
+
+[Download link for the script](https://gluon-nlp.mxnet.io/_downloads/bert.zip)
+
+The original description can be found in the [MXNet GluonNLP model 
zoo](https://gluon-nlp.mxnet.io/model_zoo/bert/index.html#bert-base-on-squad-1-1).
+```bash
+python static_finetune_squad.py --optimizer adam --accumulate 2 --batch_size 6 
--lr 3e-5 --epochs 2 --gpu 0 --export
+
+```
+This script will generate `json` and `param` fles that are the standard MXNet 
model files.
+By default, this model are using `bert_12_768_12` model with extra layers f

[incubator-mxnet] branch master updated: [Clojure] Add methods based on NDArrayAPI/SymbolAPI (#14195)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new a5db391  [Clojure] Add methods based on NDArrayAPI/SymbolAPI (#14195)
a5db391 is described below

commit a5db391cb27342bf8e267cfbfb4db26b5ef66721
Author: Kedar Bellare 
AuthorDate: Sat Apr 13 12:19:06 2019 -0700

[Clojure] Add methods based on NDArrayAPI/SymbolAPI (#14195)

* [Clojure] Add methods based on NDArrayAPI/SymbolAPI

* Add symbol API methods and ndarray API unit tests

* Some more ndarray API unit tests

* Explore direct use of JNI

* Use library info directly instead of reflection

* Add tests for generation op info

* Fix ordering of keys using array-map

* Ignore generated test files

* Minor style changes

* Refactor code for better readability

* Address comments

* Small tweaks to symbol api coercion
---
 contrib/clojure-package/.gitignore |   2 +
 contrib/clojure-package/src/dev/generator.clj  | 460 -
 .../src/org/apache/clojure_mxnet/ndarray_api.clj   |  32 ++
 .../src/org/apache/clojure_mxnet/symbol_api.clj|  32 ++
 .../src/org/apache/clojure_mxnet/util.clj  |   6 +-
 .../clojure-package/test/dev/generator_test.clj| 148 ++-
 .../clojure-package/test/good-test-ndarray-api.clj |  89 
 .../clojure-package/test/good-test-symbol-api.clj  | 109 +
 .../test/org/apache/clojure_mxnet/conv_test.clj|  24 +-
 .../org/apache/clojure_mxnet/ndarray_api_test.clj  | 415 +++
 .../org/apache/clojure_mxnet/symbol_api_test.clj   |  61 +++
 11 files changed, 1257 insertions(+), 121 deletions(-)

diff --git a/contrib/clojure-package/.gitignore 
b/contrib/clojure-package/.gitignore
index f5d81dd..71d812e 100644
--- a/contrib/clojure-package/.gitignore
+++ b/contrib/clojure-package/.gitignore
@@ -39,6 +39,8 @@ examples/visualization/test-vis.pdf
 src/.DS_Store
 src/org/.DS_Store
 test/test-ndarray.clj
+test/test-ndarray-api.clj
 test/test-symbol.clj
+test/test-symbol-api.clj
 src/org/apache/clojure_mxnet/gen/*
 
diff --git a/contrib/clojure-package/src/dev/generator.clj 
b/contrib/clojure-package/src/dev/generator.clj
index ca93c34..34210be 100644
--- a/contrib/clojure-package/src/dev/generator.clj
+++ b/contrib/clojure-package/src/dev/generator.clj
@@ -17,10 +17,14 @@
 
 (ns dev.generator
   (:require [t6.from-scala.core :as scala]
+[t6.from-scala.core :refer [$ $$] :as $]
 [clojure.reflect :as r]
-[org.apache.clojure-mxnet.util :as util]
-[clojure.pprint])
-  (:import (org.apache.mxnet NDArray Symbol))
+[clojure.pprint]
+[org.apache.clojure-mxnet.util :as util])
+  (:import (org.apache.mxnet NDArray NDArrayAPI
+ Symbol SymbolAPI
+ Base Base$RefInt Base$RefLong Base$RefFloat 
Base$RefString)
+   (scala.collection.mutable ListBuffer ArrayBuffer))
   (:gen-class))
 
 
@@ -34,17 +38,17 @@
   (clojure.string/replace #"\_" "-")
   (clojure.string/replace #"\/" "div")))
 
-(defn symbol-transform-param-name [parameter-types]
+(defn transform-param-names [coerce-fn parameter-types]
   (->> parameter-types
(map str)
-   (map (fn [x] (or (util/symbol-param-coerce x) x)))
+   (map (fn [x] (or (coerce-fn x) x)))
(map (fn [x] (last (clojure.string/split x #"\."))
 
+(defn symbol-transform-param-name [parameter-types]
+  (transform-param-names util/symbol-param-coerce parameter-types))
+
 (defn ndarray-transform-param-name [parameter-types]
-  (->> parameter-types
-   (map str)
-   (map (fn [x] (or (util/ndarray-param-coerce x) x)))
-   (map (fn [x] (last (clojure.string/split x #"\."))
+  (transform-param-names util/ndarray-param-coerce parameter-types))
 
 (defn has-variadic? [params]
   (->> params
@@ -56,37 +60,136 @@
 
 (defn increment-param-name [pname]
   (if-let [num-str (re-find #"-\d" pname)]
-(str (first (clojure.string/split pname #"-")) "-" (inc (Integer/parseInt 
(last (clojure.string/split num-str #"-")
+(str 
+ (first (clojure.string/split pname #"-"))
+ "-"
+ (inc (Integer/parseInt (last (clojure.string/split num-str #"-")
 (str pname "-" 1)))
 
-(defn rename-duplicate-params [params]
-  (reduce (fn [known-names n] (conj known-names (if (contains? (set 
known-names) n)
-  (increment-param-name n)
-  n)))
-  []
-  params))
-
+(defn rename-duplicate-params [pnames]
+  (->> (reduce

[incubator-mxnet] branch clojure-bert-qa-example deleted (was 03d2f4f)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-bert-qa-example
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


 was 03d2f4f  move input to edn file and rearrange things

This change permanently discards the following revisions:

 discard 03d2f4f  move input to edn file and rearrange things
 discard 3cdf989  computational complexity
 discard 459c8d9  try another question
 discard 1106bac  cleaning up example
 discard 37e7b84  qa example working
 discard 9a15c06  prediction
 discard 665ed7d  wip

[incubator-mxnet] 05/07: try another question

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-bert-qa-example
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 459c8d92ef033a33ac44498445ef2262d9ef6fd4
Author: gigasquid 
AuthorDate: Fri Apr 12 19:52:16 2019 -0400

try another question
---
 .../examples/bert-qa/src/bert_qa/core.clj   | 21 +++--
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
index 02f2d34..e44ad40 100644
--- a/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
+++ b/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
@@ -16,9 +16,11 @@
 ;; the vocabulary used in the model
 (def model-vocab "model/vocab.json")
 ;; the input question
-(def input-q "When did BBC Japan start broadcasting?")
+#_(def input-q "When did BBC Japan start broadcasting?")
+(def input-q "When did Herve serve as a Byzantine general?")
 ;;; the input answer
-(def input-a (str  "BBC Japan was a general entertainment Channel.\n"
+(def input-a "One of the first Norman mercenaries to serve as a Byzantine 
general was Hervé in the 1050s. By then however, there were already Norman 
mercenaries serving as far away as Trebizond and Georgia. They were based at 
Malatya and Edessa, under the Byzantine duke of Antioch, Isaac Komnenos. In the 
1060s, Robert Crispin led the Normans of Edessa against the Turks. Roussel de 
Bailleul even tried to carve out an independent state in Asia Minor with 
support from the local population,  [...]
+#_(def input-a (str "BBC Japan was a general entertainment Channel.\n"
   " Which operated between December 2004 and April 2006.\n"
   "It ceased operations after its Japanese distributor 
folded."))
 ;; the maximum length of the sequence
@@ -51,7 +53,7 @@
  :token2idx (get vocab "token_to_idx")}))
 
 (defn tokens->idxs [token2idx tokens]
-  (mapv #(get token2idx % "[UNK]")  tokens))
+  (mapv #(get token2idx % (get token2idx "[UNK]"))  tokens))
 
 (defn idxs->tokens [idx2token idxs]
   (mapv #(get idx2token %) idxs))
@@ -70,10 +72,15 @@
 end-idx (-> (ndarray/argmax end-prob 1)
 (ndarray/->vec)
 (first))]
-(subvec tokens (dec start-idx) (inc end-idx
+(println "start-idx" start-idx "end-idx" end-idx)
+(if (> end-idx start-idx)
+  (subvec tokens (dec start-idx) (inc end-idx))
+  (subvec tokens (dec end-idx) (inc end-idx)) )
+))
 
 (defn infer [ctx]
-  (let [;;; pre-processing tokenize sentence
+  (let [ctx (context/default-context)
+;;; pre-processing tokenize sentence
 token-q (tokenizer (string/lower-case input-q))
 token-a (tokenizer (string/lower-case input-a))
 valid-length (+ (count token-q) (count token-a))
@@ -126,5 +133,7 @@
 
 (comment
 
-  (infer)
+  (infer :cpu)
+
+  (get (:token2idx (get-vocab)) "[UNK]")
 )

[incubator-mxnet] 03/07: qa example working

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-bert-qa-example
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 37e7b8483f07e0fb0f911cb5d149e6dadea007ea
Author: gigasquid 
AuthorDate: Fri Apr 12 19:18:20 2019 -0400

qa example working
---
 .../examples/bert-qa/src/bert_qa/core.clj  | 42 --
 .../examples/bert-qa/src/java/BertQA.java  |  3 ++
 2 files changed, 27 insertions(+), 18 deletions(-)

diff --git a/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
index 1876a83..1c7b14c 100644
--- a/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
+++ b/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
@@ -59,6 +59,22 @@
   (mapv #(get idx2token %) idxs))
 
 
+(defn post-processing [result tokens]
+  (let [output1 (ndarray/slice-axis result 2 0 1)
+output2 (ndarray/slice-axis result 2 1 2)
+;;; get the formatted logits result
+start-logits (ndarray/reshape output1 [0 -3])
+end-logits (ndarray/reshape output2 [0 -3])
+start-prob (ndarray/softmax start-logits)
+end-prob (ndarray/softmax end-logits)
+start-idx (-> (ndarray/argmax start-prob 1)
+  (ndarray/->vec)
+  (first))
+end-idx (-> (ndarray/argmax end-prob 1)
+(ndarray/->vec)
+(first))]
+(subvec tokens (dec start-idx) (inc end-idx
+
 (defn infer []
   (let [ctx (context/default-context)
 ;;; pre-processing tokenize sentence
@@ -99,25 +115,15 @@
factory
{:contexts [ctx]
 :epoch 2})
-result (first (infer/predict-with-ndarray predictor input-batch))]
-result)
-  
-  )
+;;; start predication
+result (first (infer/predict-with-ndarray predictor input-batch))
+answer (post-processing result tokens)]
+(println "Question: " input-q)
+(println "Answer paragraph: " input-a)
+(println "Answer: " answer)))
 
 
 (comment
 
-  (repeat 3 0)
-  (def x )
-  (keys x)
-  (get x "idx_to_token")
-  (def bert-parser (new BertDataParser))
-  (.parseJSON bert-parser model-vocab)
-  (.token2idx bert-parser (java.util.ArrayList. ["and" "where"]))
-  [1998 2073]
-  (.idx2token bert-parser (java.util.ArrayList. (map int [1998 2073])))
-  ["and" "where"]
-  (def bert-qa (new BertQA))
-
-  (BertQA/main (into-array ["--model-path-prefix" "model/static_bert_qa"]) )
-  (r/reflect bert-qa))
+  (infer)
+)
diff --git a/contrib/clojure-package/examples/bert-qa/src/java/BertQA.java 
b/contrib/clojure-package/examples/bert-qa/src/java/BertQA.java
index 7308bca..8521f0b 100644
--- a/contrib/clojure-package/examples/bert-qa/src/java/BertQA.java
+++ b/contrib/clojure-package/examples/bert-qa/src/java/BertQA.java
@@ -69,6 +69,7 @@ public class BertQA {
 static List postProcessing(NDArray result, List tokens) {
 NDArray[] output = NDArray.split(
 NDArray.new splitParam(result, 2).setAxis(2));
+   logger.info("Carin postprocessing output: " + Arrays.toString(output));
 // Get the formatted logits result
 NDArray startLogits = output[0].reshape(new int[]{0, -3});
 NDArray endLogits = output[1].reshape(new int[]{0, -3});
@@ -79,6 +80,8 @@ public class BertQA {
 NDArray.new softmaxParam(endLogits))[0].toArray();
 int startIdx = argmax(startProb);
 int endIdx = argmax(endProb);
+   logger.info("Carin startIdx "+ startIdx);
+   logger.info("Carin endIdx "+ startIdx);
 return tokens.subList(startIdx, endIdx + 1);
 }
 

[incubator-mxnet] 04/07: cleaning up example

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-bert-qa-example
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 1106bacaa96514af92abac57d171b4cb3934ddb8
Author: gigasquid 
AuthorDate: Fri Apr 12 19:27:58 2019 -0400

cleaning up example
---
 .../clojure-package/examples/bert-qa/project.clj   |   2 +-
 .../examples/bert-qa/src/bert_qa/core.clj  |  15 +-
 .../examples/bert-qa/src/java/BertDataParser.java  | 126 -
 .../examples/bert-qa/src/java/BertQA.java  | 152 -
 4 files changed, 9 insertions(+), 286 deletions(-)

diff --git a/contrib/clojure-package/examples/bert-qa/project.clj 
b/contrib/clojure-package/examples/bert-qa/project.clj
index 328d040..5bec165 100644
--- a/contrib/clojure-package/examples/bert-qa/project.clj
+++ b/contrib/clojure-package/examples/bert-qa/project.clj
@@ -3,8 +3,8 @@
   :plugins [[lein-cljfmt "0.5.7"]]
   :dependencies [[org.clojure/clojure "1.9.0"]
  [org.apache.mxnet.contrib.clojure/clojure-mxnet 
"1.5.0-SNAPSHOT"]
- [com.google.code.gson/gson "2.8.5"]
  [cheshire "5.8.1"]]
   :pedantic? :skip
   :java-source-paths ["src/java"]
+  :main bert-qa.core
   :repl-options {:init-ns bert-qa.core})
diff --git a/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
index 1c7b14c..02f2d34 100644
--- a/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
+++ b/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
@@ -8,9 +8,7 @@
 [org.apache.clojure-mxnet.context :as context]
 [org.apache.clojure-mxnet.layout :as layout]
 [org.apache.clojure-mxnet.ndarray :as ndarray]
-[org.apache.clojure-mxnet.infer :as infer])
-  (:import (bert BertDataParser)
-   (bert BertQA)))
+[org.apache.clojure-mxnet.infer :as infer]))
 
 (def model-path-prefix "model/static_bert_qa")
 ;; epoch number of the model
@@ -58,7 +56,6 @@
 (defn idxs->tokens [idx2token idxs]
   (mapv #(get idx2token %) idxs))
 
-
 (defn post-processing [result tokens]
   (let [output1 (ndarray/slice-axis result 2 0 1)
 output2 (ndarray/slice-axis result 2 1 2)
@@ -75,9 +72,8 @@
 (first))]
 (subvec tokens (dec start-idx) (inc end-idx
 
-(defn infer []
-  (let [ctx (context/default-context)
-;;; pre-processing tokenize sentence
+(defn infer [ctx]
+  (let [;;; pre-processing tokenize sentence
 token-q (tokenizer (string/lower-case input-q))
 token-a (tokenizer (string/lower-case input-a))
 valid-length (+ (count token-q) (count token-a))
@@ -122,6 +118,11 @@
 (println "Answer paragraph: " input-a)
 (println "Answer: " answer)))
 
+(defn -main [& args]
+  (let [[dev] args]
+(if (= dev ":gpu")
+  (infer (context/gpu))
+  (infer (context/cpu)
 
 (comment
 
diff --git 
a/contrib/clojure-package/examples/bert-qa/src/java/BertDataParser.java 
b/contrib/clojure-package/examples/bert-qa/src/java/BertDataParser.java
deleted file mode 100644
index a0a821a..000
--- a/contrib/clojure-package/examples/bert-qa/src/java/BertDataParser.java
+++ /dev/null
@@ -1,126 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.mxnetexamples.javaapi.infer.bert;
-
-import java.io.FileReader;
-import java.util.*;
-
-import com.google.gson.Gson;
-import com.google.gson.JsonArray;
-import com.google.gson.JsonElement;
-import com.google.gson.JsonObject;
-
-/**
- * This is the Utility for pre-processing the data for Bert Model
- * You can use this utility to parse Vocabulary JSON into Java Array and 
Dictionary,
- * clean and tokenize sentences and pad the text
- */
-public class BertDataParser {
-
-private Map token2idx;
-private List idx2token;
-
-/**
- * Parse the Vocabulary to JSON files
- * [PAD], [CLS], [SEP], [MASK], [UNK] are reserved tokens
- * @param jsonFile

[incubator-mxnet] 01/07: wip

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-bert-qa-example
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 665ed7d3fec1fb1560b1985dd6ab2c092cf0d87c
Author: gigasquid 
AuthorDate: Fri Apr 12 16:34:08 2019 -0400

wip
---
 .../clojure-package/examples/bert-qa/.gitignore|  12 +++
 contrib/clojure-package/examples/bert-qa/README.md |  22 +
 .../examples/bert-qa/get_bert_data.sh  |  29 ++
 .../clojure-package/examples/bert-qa/project.clj   |  10 ++
 .../examples/bert-qa/src/bert_qa/core.clj  | 107 +
 .../examples/bert-qa/src/java}/BertDataParser.java |  10 +-
 .../examples/bert-qa/src/java}/BertQA.java |   7 +-
 .../examples/bert-qa/test/bert_qa/core_test.clj|   7 ++
 .../javaapi/infer/bert/BertDataParser.java |  10 +-
 .../mxnetexamples/javaapi/infer/bert/BertQA.java   |   1 +
 10 files changed, 202 insertions(+), 13 deletions(-)

diff --git a/contrib/clojure-package/examples/bert-qa/.gitignore 
b/contrib/clojure-package/examples/bert-qa/.gitignore
new file mode 100644
index 000..d18f225
--- /dev/null
+++ b/contrib/clojure-package/examples/bert-qa/.gitignore
@@ -0,0 +1,12 @@
+/target
+/classes
+/checkouts
+profiles.clj
+pom.xml
+pom.xml.asc
+*.jar
+*.class
+/.lein-*
+/.nrepl-port
+.hgignore
+.hg/
diff --git a/contrib/clojure-package/examples/bert-qa/README.md 
b/contrib/clojure-package/examples/bert-qa/README.md
new file mode 100644
index 000..fc21bdd
--- /dev/null
+++ b/contrib/clojure-package/examples/bert-qa/README.md
@@ -0,0 +1,22 @@
+# bert-qa
+
+A Clojure library designed to ... well, that part is up to you.
+
+## Usage
+
+FIXME
+
+## License
+
+Copyright © 2019 FIXME
+
+This program and the accompanying materials are made available under the
+terms of the Eclipse Public License 2.0 which is available at
+http://www.eclipse.org/legal/epl-2.0.
+
+This Source Code may also be made available under the following Secondary
+Licenses when the conditions for such availability set forth in the Eclipse
+Public License, v. 2.0 are satisfied: GNU General Public License as published 
by
+the Free Software Foundation, either version 2 of the License, or (at your
+option) any later version, with the GNU Classpath Exception which is available
+at https://www.gnu.org/software/classpath/license.html.
diff --git a/contrib/clojure-package/examples/bert-qa/get_bert_data.sh 
b/contrib/clojure-package/examples/bert-qa/get_bert_data.sh
new file mode 100755
index 000..603194a
--- /dev/null
+++ b/contrib/clojure-package/examples/bert-qa/get_bert_data.sh
@@ -0,0 +1,29 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+data_path=model
+
+if [ ! -d "$data_path" ]; then
+  mkdir -p "$data_path"
+  curl 
https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/BertQA/vocab.json
 -o $data_path/vocab.json
+  curl 
https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/BertQA/static_bert_qa-0002.params
 -o $data_path/static_bert_qa-0002.params
+  curl 
https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/BertQA/static_bert_qa-symbol.json
 -o $data_path/static_bert_qa-symbol.json
+fi
diff --git a/contrib/clojure-package/examples/bert-qa/project.clj 
b/contrib/clojure-package/examples/bert-qa/project.clj
new file mode 100644
index 000..328d040
--- /dev/null
+++ b/contrib/clojure-package/examples/bert-qa/project.clj
@@ -0,0 +1,10 @@
+(defproject bert-qa "0.1.0-SNAPSHOT"
+  :description "BERT QA Example"
+  :plugins [[lein-cljfmt "0.5.7"]]
+  :dependencies [[org.clojure/clojure "1.9.0"]
+ [org.apache.mxnet.contrib.clojure/clojure-mxnet 
"1.5.0-SNAPSHOT"]
+ [com.google.code.gson/gson "2.8.5"]
+ [cheshire "5.8.1"]]
+  :pedantic? :skip
+  :java-source-paths ["src/java"]
+  :repl-options {:init-ns bert-qa.core})
diff --git a/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
new file mode 100644
index 000..941e6

[incubator-mxnet] 07/07: move input to edn file and rearrange things

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-bert-qa-example
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 03d2f4f7b3d95995d9b1fd4f542d7400570a971a
Author: gigasquid 
AuthorDate: Sat Apr 13 11:59:06 2019 -0400

move input to edn file and rearrange things
---
 contrib/clojure-package/examples/bert-qa/README.md |  79 +---
 .../examples/bert-qa/squad-samples.edn |  17 
 .../examples/bert-qa/src/bert_qa/core.clj  | 104 +++--
 3 files changed, 137 insertions(+), 63 deletions(-)

diff --git a/contrib/clojure-package/examples/bert-qa/README.md 
b/contrib/clojure-package/examples/bert-qa/README.md
index fc21bdd..a61e270 100644
--- a/contrib/clojure-package/examples/bert-qa/README.md
+++ b/contrib/clojure-package/examples/bert-qa/README.md
@@ -1,22 +1,73 @@
 # bert-qa
 
-A Clojure library designed to ... well, that part is up to you.
+**This example was based off of the Java API one. It shows how to do inference 
with a pre-trained BERT network that is trained on Questions and Answers using 
the [SQuAD Dataset](https://rajpurkar.github.io/SQuAD-explorer/)**
 
-## Usage
+The pretrained model was created using GluonNLP and then exported to the MXNet 
symbol format. You can find more information in the background section below.
 
-FIXME
+In this tutorial, we will walk through the BERT QA model trained by MXNet. 
+Users can provide a question with a paragraph contains answer to the model and
+the model will be able to find the best answer from the answer paragraph.
 
-## License
+Example:
 
-Copyright © 2019 FIXME
+```
+{:input-answer "Steam engines are external combustion engines, where the 
working fluid is separate from the combustion products. Non-combustion heat 
sources such as solar power, nuclear power or geothermal energy may be used. 
The ideal thermodynamic cycle used to analyze this process is called the 
Rankine cycle. In the cycle, water is heated and transforms into steam within a 
boiler operating at a high pressure. When expanded through pistons or turbines, 
mechanical work is done. The redu [...]
+  :input-question "Along with geothermal and nuclear, what is a notable 
non-combustion heat source?"
+  :ground-truth-answers ["solar"
+ "solar power"
+ "solar power, nuclear power or geothermal 
energysolar"]}
+```
 
-This program and the accompanying materials are made available under the
-terms of the Eclipse Public License 2.0 which is available at
-http://www.eclipse.org/legal/epl-2.0.
+The prediction in this case would be `solar power`
+
+## Setup Guide
+
+### Step 1: Download the model
+
+For this tutorial, you can get the model and vocabulary by running following 
bash file. This script will use `wget` to download these artifacts from AWS S3.
+
+From the `scala-package/examples/scripts/infer/bert/` folder run:
+
+```bash
+./get_bert_data.sh
+```
+
+Some sample questions and answers are provide in the `squad-sample.edn` file. 
Some are taken directly from the SQuAD dataset and one was just made up. Feel 
free to edit the file and add your own!
+
+
+## To run
+
+* `lein install` in the root of the main project directory
+* cd into this project directory and do `lein run`. This will execute the cpu 
version.
+
+`lein run :cpu` - to run with cpu
+`lein run :gpu` - to run with gpu
+
+## Background
+
+To learn more about how BERT works in MXNet, please follow this [MXNet Gluon 
tutorial on NLP using 
BERT](https://medium.com/apache-mxnet/gluon-nlp-bert-6a489bdd3340).
+
+The model was extracted from MXNet GluonNLP with static length settings.
+
+[Download link for the script](https://gluon-nlp.mxnet.io/_downloads/bert.zip)
+
+The original description can be found in the [MXNet GluonNLP model 
zoo](https://gluon-nlp.mxnet.io/model_zoo/bert/index.html#bert-base-on-squad-1-1).
+```bash
+python static_finetune_squad.py --optimizer adam --accumulate 2 --batch_size 6 
--lr 3e-5 --epochs 2 --gpu 0 --export
+
+```
+This script will generate `json` and `param` fles that are the standard MXNet 
model files.
+By default, this model are using `bert_12_768_12` model with extra layers for 
QA jobs.
+
+After that, to be able to use it in Java, we need to export the dictionary 
from the script to parse the text
+to actual indexes. Please add the following lines after [this 
line](https://github.com/dmlc/gluon-nlp/blob/master/scripts/bert/staticbert/static_finetune_squad.py#L262).
+```python
+import json
+json_str = vocab.to_json()
+f = open("vocab.json", "w")
+f.write(json_str)
+f.close()
+```
+This would export the token vocabulary in json format.
+Once you have these three files, you will be able to run this example without 
problems.
 
-This Source Code may also be made available under the following Secondary
-Licenses when the conditions for s

[incubator-mxnet] 02/07: prediction

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-bert-qa-example
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 9a15c06d52636cd96a8a3b768dfae6e5527cf043
Author: gigasquid 
AuthorDate: Fri Apr 12 17:17:15 2019 -0400

prediction
---
 .../examples/bert-qa/src/bert_qa/core.clj  | 26 +-
 1 file changed, 21 insertions(+), 5 deletions(-)

diff --git a/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
index 941e6aa..1876a83 100644
--- a/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
+++ b/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
@@ -7,7 +7,8 @@
 [org.apache.clojure-mxnet.dtype :as dtype]
 [org.apache.clojure-mxnet.context :as context]
 [org.apache.clojure-mxnet.layout :as layout]
-[org.apache.clojure-mxnet.ndarray :as ndarray])
+[org.apache.clojure-mxnet.ndarray :as ndarray]
+[org.apache.clojure-mxnet.infer :as infer])
   (:import (bert BertDataParser)
(bert BertQA)))
 
@@ -81,10 +82,25 @@
 input-batch [(ndarray/array indexes [1 seq-length] {:context ctx})
  (ndarray/array token-types [1 seq-length] {:context ctx})
  (ndarray/array [valid-length] [1] {:context ctx})]
-
-
-]
-input-batch)
+input-descs [{:name "data0"
+  :shape [1 seq-length]
+  :dtype dtype/FLOAT32
+  :layout layout/NT}
+ {:name "data1"
+  :shape [1 seq-length]
+  :dtype dtype/FLOAT32
+  :layout layout/NT}
+ {:name "data2"
+  :shape [1]
+  :dtype dtype/FLOAT32
+  :layout layout/N}]
+factory (infer/model-factory model-path-prefix input-descs)
+predictor (infer/create-predictor
+   factory
+   {:contexts [ctx]
+:epoch 2})
+result (first (infer/predict-with-ndarray predictor input-batch))]
+result)
   
   )
 

[incubator-mxnet] 06/07: computational complexity

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-bert-qa-example
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 3cdf9898f29520454a6b55d17ff6114a594adac6
Author: gigasquid 
AuthorDate: Fri Apr 12 20:04:18 2019 -0400

computational complexity
---
 contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj 
b/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
index e44ad40..079f227 100644
--- a/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
+++ b/contrib/clojure-package/examples/bert-qa/src/bert_qa/core.clj
@@ -17,9 +17,9 @@
 (def model-vocab "model/vocab.json")
 ;; the input question
 #_(def input-q "When did BBC Japan start broadcasting?")
-(def input-q "When did Herve serve as a Byzantine general?")
+  (def input-q "What branch of theoretical computer science deals with broadly 
classifying computational problems by difficulty and class of relationship?")
 ;;; the input answer
-(def input-a "One of the first Norman mercenaries to serve as a Byzantine 
general was Hervé in the 1050s. By then however, there were already Norman 
mercenaries serving as far away as Trebizond and Georgia. They were based at 
Malatya and Edessa, under the Byzantine duke of Antioch, Isaac Komnenos. In the 
1060s, Robert Crispin led the Normans of Edessa against the Turks. Roussel de 
Bailleul even tried to carve out an independent state in Asia Minor with 
support from the local population,  [...]
+(def input-a "Computational complexity theory is a branch of the theory of 
computation in theoretical computer science that focuses on classifying 
computational problems according to their inherent difficulty, and relating 
those classes to each other. A computational problem is understood to be a task 
that is in principle amenable to being solved by a computer, which is 
equivalent to stating that the problem may be solved by mechanical application 
of mathematical steps, such as an algorithm.")
 #_(def input-a (str "BBC Japan was a general entertainment Channel.\n"
   " Which operated between December 2004 and April 2006.\n"
   "It ceased operations after its Japanese distributor 
folded."))
@@ -74,8 +74,8 @@
 (first))]
 (println "start-idx" start-idx "end-idx" end-idx)
 (if (> end-idx start-idx)
-  (subvec tokens (dec start-idx) (inc end-idx))
-  (subvec tokens (dec end-idx) (inc end-idx)) )
+  (subvec tokens start-idx (inc end-idx))
+  (subvec tokens end-idx (inc end-idx)) )
 ))
 
 (defn infer [ctx]
@@ -135,5 +135,4 @@
 
   (infer :cpu)
 
-  (get (:token2idx (get-vocab)) "[UNK]")
 )

[incubator-mxnet] branch clojure-bert-qa-example created (now 03d2f4f)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-bert-qa-example
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at 03d2f4f  move input to edn file and rearrange things

This branch includes the following new commits:

 new 665ed7d  wip
 new 9a15c06  prediction
 new 37e7b84  qa example working
 new 1106bac  cleaning up example
 new 459c8d9  try another question
 new 3cdf989  computational complexity
 new 03d2f4f  move input to edn file and rearrange things

The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-mxnet] branch master updated: [Clojure] enhance draw bounding box (#14567)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 0284236  [Clojure] enhance draw bounding box (#14567)
0284236 is described below

commit 02842368a89ffabf0eab3c90928feefc7aed463a
Author: Carin Meier 
AuthorDate: Thu Apr 11 19:01:55 2019 -0400

[Clojure] enhance draw bounding box (#14567)

* add test for drawing bounding box

* Uses the core image drawing bounding box functionality for the object 
detection example.

Adjust the specs and names to make it easier to run with object detection 
and clojure draw bounding box example

* feedback from @chouffe

* refactor to be 3 top predictions instead of 5 to make the images less 
crowded
---
 .../examples/infer/objectdetector/project.clj  |  1 -
 .../infer/objectdetector/src/infer/draw.clj| 44 
 .../src/infer/objectdetector_example.clj   | 59 +-
 .../test/infer/objectdetector_example_test.clj |  6 +--
 contrib/clojure-package/integration-tests.sh   |  2 +-
 .../src/org/apache/clojure_mxnet/image.clj | 26 +-
 .../test/org/apache/clojure_mxnet/image_test.clj   | 16 +-
 7 files changed, 66 insertions(+), 88 deletions(-)

diff --git a/contrib/clojure-package/examples/infer/objectdetector/project.clj 
b/contrib/clojure-package/examples/infer/objectdetector/project.clj
index cdd9a89..da01797 100644
--- a/contrib/clojure-package/examples/infer/objectdetector/project.clj
+++ b/contrib/clojure-package/examples/infer/objectdetector/project.clj
@@ -22,7 +22,6 @@
   :aliases {"run-detector" ["run" "--" "-m" 
"models/resnet50_ssd/resnet50_ssd_model" "-i" "images/dog.jpg" "-d" "images/"]}
   :dependencies [[org.clojure/clojure "1.9.0"]
  [org.clojure/tools.cli "0.4.1"]
- [origami "4.0.0-3"]
  [org.apache.mxnet.contrib.clojure/clojure-mxnet 
"1.5.0-SNAPSHOT"]]
   :main ^:skip-aot infer.objectdetector-example
   :profiles {:uberjar {:aot :all}})
diff --git 
a/contrib/clojure-package/examples/infer/objectdetector/src/infer/draw.clj 
b/contrib/clojure-package/examples/infer/objectdetector/src/infer/draw.clj
deleted file mode 100644
index d29b34b..000
--- a/contrib/clojure-package/examples/infer/objectdetector/src/infer/draw.clj
+++ /dev/null
@@ -1,44 +0,0 @@
-;; Licensed to the Apache Software Foundation (ASF) under one or more
-;; contributor license agreements.  See the NOTICE file distributed with
-;; this work for additional information regarding copyright ownership.
-;; The ASF licenses this file to You under the Apache License, Version 2.0
-;; (the "License"); you may not use this file except in compliance with
-;; the License.  You may obtain a copy of the License at
-;;
-;;http://www.apache.org/licenses/LICENSE-2.0
-;;
-;; Unless required by applicable law or agreed to in writing, software
-;; distributed under the License is distributed on an "AS IS" BASIS,
-;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-;; See the License for the specific language governing permissions and
-;; limitations under the License.
-;;
-
-(ns infer.draw
-  (:require
-   [opencv4.colors.rgb :as rgb]
-   [opencv4.core :refer [FONT_HERSHEY_PLAIN imread imwrite new-point put-text! 
rectangle]]))
-
-(defn black-boxes! [img results]
-  (doseq [{confidence :confidence label :label top-left :top-left bottom-right 
:bottom-right} results]
-(let [w (.width img)
-  h (.height img)
-  top-left-p (new-point (int (* w (first top-left))) (int (* h (second 
top-left
-  bottom-right-p (new-point (int (* w (first bottom-right))) (int (* h 
(second bottom-right]
-  (if (< 15 confidence)
-(do
-  (rectangle img top-left-p bottom-right-p rgb/white 1)
-  (put-text! img
- (str label "[" confidence "% ]")
- top-left-p
- FONT_HERSHEY_PLAIN
- 1.0
- rgb/white 1)
-  img)
-
-(defn draw-bounds [image results output-dir]
-  (let [out-file (str output-dir "/" (.getName (clojure.java.io/as-file 
image)))]
-(-> image
-(imread)
-(black-boxes! results)
-(imwrite out-file
\ No newline at end of file
diff --git 
a/contrib/clojure-package/examples/infer/objectdetector/src/infer/objectdetector_example.clj
 
b/contrib/clojure-package/examples/infer/objectdetector/src/infer/objectdetector_example.clj
index 9331798..65d822f 100644
--- 
a/contrib/clojure-package/examples/infer/objectdetector/src/infer/objectdetector_example.clj
+++ 
b/contr

[incubator-mxnet] branch clojure-enhance-bounding-box deleted (was 5e18d8e)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-enhance-bounding-box
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


 was 5e18d8e  refactor to be 3 top predictions instead of 5 to make the 
images less crowded

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

[incubator-mxnet] branch clojure-enhance-bounding-box created (now 5e18d8e)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-enhance-bounding-box
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at 5e18d8e  refactor to be 3 top predictions instead of 5 to make the 
images less crowded

No new revisions were added by this update.

[incubator-mxnet] branch master updated: fix tests (#14565)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 5f19362  fix tests (#14565)
5f19362 is described below

commit 5f19362c2edb5c70576dc1b6341548715f8f430a
Author: Carin Meier 
AuthorDate: Fri Mar 29 13:26:15 2019 -0400

fix tests (#14565)
---
 .../clojure-package/test/dev/generator_test.clj| 28 ++
 contrib/clojure-package/test/good-test-ndarray.clj |  1 +
 .../test/org/apache/clojure_mxnet/module_test.clj  | 11 -
 3 files changed, 18 insertions(+), 22 deletions(-)

diff --git a/contrib/clojure-package/test/dev/generator_test.clj 
b/contrib/clojure-package/test/dev/generator_test.clj
index 7551bc1..05b4a74 100644
--- a/contrib/clojure-package/test/dev/generator_test.clj
+++ b/contrib/clojure-package/test/dev/generator_test.clj
@@ -86,19 +86,17 @@
 (is (= "LRN" (-> lrn-info vals ffirst :name str)
 
 (deftest test-symbol-vector-args
-  ;; FIXME
-  #_(is (= `(if (clojure.core/map? kwargs-map-or-vec-or-sym)
+  (is (= '(if (clojure.core/map? kwargs-map-or-vec-or-sym)
 (util/empty-list)
 (util/coerce-param
-  kwargs-map-or-vec-or-sym
-  #{"scala.collection.Seq"}))
+ kwargs-map-or-vec-or-sym
+ #{"scala.collection.Seq"}))
  (gen/symbol-vector-args
 
 (deftest test-symbol-map-args
-  ;; FIXME
-  #_(is (= `(if (clojure.core/map? kwargs-map-or-vec-or-sym)
+  (is (= '(if (clojure.core/map? kwargs-map-or-vec-or-sym)
 (org.apache.clojure-mxnet.util/convert-symbol-map
-  kwargs-map-or-vec-or-sym)
+ kwargs-map-or-vec-or-sym)
 nil)
  (gen/symbol-map-args
 
@@ -160,14 +158,13 @@
:exception-types [],
:flags #{:public}}]}
 function-name (symbol "div")]
-;; FIXME
-#_(is (= '(([sym sym-or-Object]
+(is (= '(([sym sym-or-object]
   (util/coerce-return
-(.$div
-  sym
-  (util/nil-or-coerce-param
-sym-or-Object
-#{"org.apache.mxnet.Symbol" "java.lang.Object"})
+   (.$div
+sym
+(util/nil-or-coerce-param
+ sym-or-object
+ #{"org.apache.mxnet.Symbol" "java.lang.Object"})
(gen/gen-symbol-function-arity op-name op-values function-name)
 
 (deftest test-gen-ndarray-function-arity
@@ -210,5 +207,4 @@
fname)
   good-contents (slurp "test/good-test-ndarray.clj")
   contents (slurp fname)]
-  ;; FIXME
-  #_(is (= good-contents contents)
+  (is (= good-contents contents)
diff --git a/contrib/clojure-package/test/good-test-ndarray.clj 
b/contrib/clojure-package/test/good-test-ndarray.clj
index 5e7131a..b048a81 100644
--- a/contrib/clojure-package/test/good-test-ndarray.clj
+++ b/contrib/clojure-package/test/good-test-ndarray.clj
@@ -35,3 +35,4 @@
  ndarray-or-double-or-float
  #{"org.apache.mxnet.MX_PRIMITIVES$MX_PRIMITIVE_TYPE"
"org.apache.mxnet.NDArray"})
+
diff --git 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/module_test.clj 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/module_test.clj
index 44b984b..e03c438 100644
--- a/contrib/clojure-package/test/org/apache/clojure_mxnet/module_test.clj
+++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/module_test.clj
@@ -246,6 +246,7 @@
 d-shape1 [10 3 64 64]
 d-shape2 [10 3 32 32]
 l-shape [10]
+
 mod (m/module my-sym {:data-names ["data1" "data2"]})
 data-batch {:data [(ndarray/random-uniform 0 9 (str (mx-shape/->shape 
d-shape1)))
(ndarray/random-uniform 5 15 (str (mx-shape/->shape 
d-shape2)))]
@@ -280,9 +281,8 @@
 :index nil
 :pad 0}]
   (-> mod
-  (m/forward data-batch))
-  ;; FIXME
-  #_(is (= [(first l-shape) num-class]
+  (m/forward data-batch-2))
+  (is (= [(first l-shape) num-class]
  (-> mod
  (m/outputs-merged)
  (first)
@@ -301,9 +301,8 @@
 :index nil
 :pad 0}]
   (-> mod
-  (m/forward data-batch))
-  ;; FIXME
-  #_(is (= [(first l-shape) num-class]
+  (m/forward data-batch-2))
+  (is (= [(first l-shape) num-class]
  (-> mod
  (m/outputs-merged)
  (first)

[incubator-mxnet] branch master updated: [clojure][image] add draw-bounding-box interop (#14533)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 4d04238  [clojure][image] add draw-bounding-box interop (#14533)
4d04238 is described below

commit 4d04238c4a087a87e089f9c627cd64870598bcfb
Author: Arthur Caillau 
AuthorDate: Fri Mar 29 16:42:05 2019 +0100

[clojure][image] add draw-bounding-box interop (#14533)

* [clojure][image] add draw-bounding-box interop

* [clojure][image] add more specs and type conversion
---
 .../src/org/apache/clojure_mxnet/image.clj | 66 ++
 1 file changed, 66 insertions(+)

diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/image.clj 
b/contrib/clojure-package/src/org/apache/clojure_mxnet/image.clj
index 7fb603e..e2e98c4 100644
--- a/contrib/clojure-package/src/org/apache/clojure_mxnet/image.clj
+++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/image.clj
@@ -23,6 +23,7 @@
 [org.apache.clojure-mxnet.util :as util]
 [clojure.spec.alpha :as s])
   (:import (org.apache.mxnet Image NDArray)
+   (java.awt.image BufferedImage)
(java.io InputStream)))
 
 ;; Flags for conversion of images
@@ -199,3 +200,68 @@
   [input]
   (util/validate! ::to-image-ndarray input "Invalid input array")
   (Image/toImage input))
+
+(s/def ::buffered-image #(instance? BufferedImage %))
+(s/def ::xmin integer?)
+(s/def ::xmax integer?)
+(s/def ::ymin integer?)
+(s/def ::ymax integer?)
+(s/def ::coordinate (s/keys :req-un [::xmin ::xmax ::ymin ::ymax]))
+(s/def ::coordinates (s/coll-of ::coordinate))
+(s/def ::names (s/nilable (s/coll-of string?)))
+(s/def ::stroke (s/and integer? pos?))
+(s/def ::font-size-mult (s/and float? pos?))
+(s/def ::transparency (s/and float? #(<= 0.0 % 1.0)))
+(s/def ::coordinates-names
+  (fn [[coordinates names]] (= (count coordinates) (count names
+
+(defn- convert-coordinate
+  "Convert bounding box coordinate to Scala correct types."
+  [{:keys [xmin xmax ymin ymax]}]
+  {:xmin (int xmin)
+   :xmax (int xmax)
+   :ymin (int ymin)
+   :ymax (int ymax)})
+
+(defn draw-bounding-box!
+  "Draw bounding boxes on `buffered-image` and Mutate the input image.
+  `buffered-image`: BufferedImage
+  `coordinates`: collection of {:xmin int :xmax int :ymin int :ymax int}
+  `font-size-mult`: positive float - Font size multiplier
+  `names`: collection of strings - List of names for the bounding boxes
+  `stroke`: positive integer - thickness of the bounding box
+  `transparency`: float in (0.0, 1.0) - Transparency of the bounding box
+  returns: Modified `buffered-image`
+  Ex:
+(draw-bounding-box! img [{:xmin 0 :xmax 100 :ymin 0 :ymax 100}])
+(draw-bounding-box! [{:xmin 190 :xmax 850 :ymin 50 :ymax 450}
+ {:xmin 200 :xmax 350 :ymin 440 :ymax 530}]
+{:stroke 2
+ :names [\"pug\" \"cookie\"]
+ :transparency 0.8
+ :font-size-mult 2.0})"
+  ([buffered-image coordinates]
+   (draw-bounding-box! buffered-image coordinates {}))
+  ([buffered-image coordinates
+{:keys [names stroke font-size-mult transparency]
+ :or {stroke 3 font-size-mult 1.0 transparency 1.0}
+ :as opts}]
+  (util/validate! ::buffered-image buffered-image "Invalid input image")
+  (util/validate! ::coordinates coordinates "Invalid input coordinates")
+  (util/validate! ::names names "Invalid input names")
+  (util/validate! ::stroke stroke "Invalid input stroke")
+  (util/validate! ::font-size-mult font-size-mult "Invalid input 
font-size-mult")
+  (util/validate! ::transparency transparency "Invalid input transparency")
+  (when (pos? (count names))
+(util/validate!  ::coordinates-names [coordinates names] "Invalid number 
of names"))
+  (Image/drawBoundingBox
+buffered-image
+(->> coordinates
+ (map convert-coordinate)
+ (map util/convert-map)
+ (into-array))
+(util/->option (into-array names))
+(util/->option (int stroke))
+(util/->option (float font-size-mult))
+(util/->option (float transparency)))
+  buffered-image))

[incubator-mxnet] branch master updated: Chouffe/clojure fix tests (#14531)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 9f5dfbf  Chouffe/clojure fix tests (#14531)
9f5dfbf is described below

commit 9f5dfbf778a02855086d1ccd713cf551ca1b05c3
Author: Arthur Caillau 
AuthorDate: Fri Mar 29 16:01:26 2019 +0100

Chouffe/clojure fix tests (#14531)

* fix ndarray-test namespace

* fix symbol-test

* fix operator_test

* fix imageclassifier_test

* fix rest of test files and add fixme pragmas

* fix util-test

* [clojure][tests] remove keyword->snake-case duplicate
---
 .../src/org/apache/clojure_mxnet/util.clj  |  17 +++-
 .../clojure-package/test/dev/generator_test.clj| 103 +++--
 contrib/clojure-package/test/good-test-ndarray.clj |   1 -
 .../org/apache/clojure_mxnet/executor_test.clj |   8 +-
 .../clojure_mxnet/infer/imageclassifier_test.clj   |  12 +--
 .../test/org/apache/clojure_mxnet/module_test.clj  |  35 ++-
 .../test/org/apache/clojure_mxnet/ndarray_test.clj |  12 +--
 .../org/apache/clojure_mxnet/operator_test.clj |  12 +--
 .../test/org/apache/clojure_mxnet/symbol_test.clj  |  10 +-
 .../test/org/apache/clojure_mxnet/util_test.clj|   4 +-
 10 files changed, 126 insertions(+), 88 deletions(-)

diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj 
b/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj
index 7eb1426..89ac1cd 100644
--- a/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj
+++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj
@@ -74,8 +74,17 @@
 (defn option->value [opt]
   ($/view opt))
 
-(defn keyword->snake-case [vals]
-  (mapv (fn [v] (if (keyword? v) (string/replace (name v) "-" "_") v)) vals))
+(defn keyword->snake-case
+  "Transforms a keyword `kw` into a snake-case string.
+  `kw`: keyword
+  returns: string
+  Ex:
+(keyword->snake-case :foo-bar) ;\"foo_bar\"
+(keyword->snake-case :foo) ;\"foo\""
+  [kw]
+  (if (keyword? kw)
+(string/replace (name kw) "-" "_")
+kw))
 
 (defn convert-tuple [param]
   (apply $/tuple param))
@@ -111,8 +120,8 @@
 (empty-map)
 (apply $/immutable-map (->> param
 (into [])
-flatten
-keyword->snake-case
+(flatten)
+(mapv keyword->snake-case)
 
 (defn convert-symbol-map [param]
   (convert-map (tuple-convert-by-param-name param)))
diff --git a/contrib/clojure-package/test/dev/generator_test.clj 
b/contrib/clojure-package/test/dev/generator_test.clj
index a3ec338..7551bc1 100644
--- a/contrib/clojure-package/test/dev/generator_test.clj
+++ b/contrib/clojure-package/test/dev/generator_test.clj
@@ -86,18 +86,21 @@
 (is (= "LRN" (-> lrn-info vals ffirst :name str)
 
 (deftest test-symbol-vector-args
-  (is (= `(if (clojure.core/map? kwargs-map-or-vec-or-sym)
+  ;; FIXME
+  #_(is (= `(if (clojure.core/map? kwargs-map-or-vec-or-sym)
 (util/empty-list)
 (util/coerce-param
- kwargs-map-or-vec-or-sym
- #{"scala.collection.Seq"}))) (gen/symbol-vector-args)))
+  kwargs-map-or-vec-or-sym
+  #{"scala.collection.Seq"}))
+ (gen/symbol-vector-args
 
 (deftest test-symbol-map-args
-  (is (= `(if (clojure.core/map? kwargs-map-or-vec-or-sym)
+  ;; FIXME
+  #_(is (= `(if (clojure.core/map? kwargs-map-or-vec-or-sym)
 (org.apache.clojure-mxnet.util/convert-symbol-map
- kwargs-map-or-vec-or-sym)
-nil))
-  (gen/symbol-map-args)))
+  kwargs-map-or-vec-or-sym)
+nil)
+ (gen/symbol-map-args
 
 (deftest test-add-symbol-arities
   (let [params (map symbol ["sym-name" "kwargs-map" "symbol-list" 
"kwargs-map-1"])
@@ -112,36 +115,36 @@
ar1))
 (is (= '([sym-name kwargs-map-or-vec-or-sym]
  (foo
-  sym-name
-  nil
-  (if
-   (clojure.core/map? kwargs-map-or-vec-or-sym)
-(util/empty-list)
-(util/coerce-param
- kwargs-map-or-vec-or-sym
- #{"scala.collection.Seq"}))
-  (if
-   (clojure.core/map? kwargs-map-or-vec-or-sym)
-(org.apache.clojure-mxnet.util/convert-symbol-map
- kwargs-map-or-vec-or-sym)
-nil
-ar2)
+   sym-name
+   nil
+   (if
+ (clojure.core/map? kwar

[incubator-mxnet] branch master updated: [clojure]: add comp-metric based on CompositeEvalMetric (#14553)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new b20f08b  [clojure]: add comp-metric based on CompositeEvalMetric 
(#14553)
b20f08b is described below

commit b20f08b17578186424e4b7bbfc44d643d9404296
Author: Anatoly 
AuthorDate: Fri Mar 29 10:56:45 2019 -0400

[clojure]: add comp-metric based on CompositeEvalMetric (#14553)

* [clojure]: add comp-metric based on CompositeEvalMetric

* [clojure]: add comp-metric test
---
 .../src/org/apache/clojure_mxnet/eval_metric.clj   | 18 ++
 .../test/org/apache/clojure_mxnet/eval_metric_test.clj |  9 +
 2 files changed, 23 insertions(+), 4 deletions(-)

diff --git 
a/contrib/clojure-package/src/org/apache/clojure_mxnet/eval_metric.clj 
b/contrib/clojure-package/src/org/apache/clojure_mxnet/eval_metric.clj
index 1946103..f1fe2d1 100644
--- a/contrib/clojure-package/src/org/apache/clojure_mxnet/eval_metric.clj
+++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/eval_metric.clj
@@ -18,7 +18,7 @@
 (ns org.apache.clojure-mxnet.eval-metric
   (:refer-clojure :exclude [get update])
   (:require [org.apache.clojure-mxnet.util :as util])
-  (:import (org.apache.mxnet Accuracy TopKAccuracy F1 Perplexity MAE MSE RMSE 
CustomMetric)))
+  (:import (org.apache.mxnet Accuracy TopKAccuracy F1 Perplexity MAE MSE RMSE 
CustomMetric CompositeEvalMetric)))
 
 (defn accuracy
   "Basic Accuracy Metric"
@@ -74,11 +74,21 @@
   [f-eval mname]
   `(new CustomMetric (util/scala-fn ~f-eval) ~mname))
 
+(defn comp-metric
+  "Create a metric instance composed out of several metrics"
+  [metrics]
+  (let [cm (CompositeEvalMetric.)]
+(doseq [m metrics] (.add cm m))
+cm))
+
 (defn get
-  "Get the values of the metric in a vector form (name and value)"
+  "Get the values of the metric in as a map of {name value} pairs"
   [metric]
-  (let [[[mname] [mvalue]] (util/tuple->vec (.get metric))]
-[mname mvalue]))
+  (let [m (apply zipmap (-> (.get metric)
+util/tuple->vec))]
+(if-not (instance? CompositeEvalMetric metric)
+  (first m)
+  m)))
 
 (defn reset
   "clear the internal statistics to an initial state"
diff --git 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/eval_metric_test.clj 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/eval_metric_test.clj
index d6da2ec..1f4dba3 100644
--- a/contrib/clojure-package/test/org/apache/clojure_mxnet/eval_metric_test.clj
+++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/eval_metric_test.clj
@@ -57,3 +57,12 @@
   "my-metric")]
 (eval-metric/update metric [(ndarray/ones [2])] [(ndarray/ones [2])])
 (is (= ["my-metric" 0.0] (eval-metric/get metric)
+
+(deftest test-comp-metric
+  (let [metric (eval-metric/comp-metric [(eval-metric/accuracy)
+ (eval-metric/f1)
+ (eval-metric/top-k-accuracy 2)])]
+(eval-metric/update metric [(ndarray/ones [2])] [(ndarray/ones [2 3])])
+(is (= {"accuracy" 0.0
+"f1" 0.0
+"top_k_accuracy" 1.0} (eval-metric/get metric)

[incubator-mxnet] branch master updated: [Clojure] Helper function for n-dim vector to ndarray (#14305)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new a4b9802  [Clojure] Helper function for n-dim vector to ndarray (#14305)
a4b9802 is described below

commit a4b98024b951fca91df3188a06141ea3e3411015
Author: Kedar Bellare 
AuthorDate: Mon Mar 11 06:39:34 2019 -0700

[Clojure] Helper function for n-dim vector to ndarray (#14305)

* [Clojure] Helper function for n-dim vector to ndarray

* More tests, specs and rename method

* Address comments

* Allow every number type
---
 .../src/org/apache/clojure_mxnet/ndarray.clj   | 21 +
 .../src/org/apache/clojure_mxnet/util.clj  | 20 
 .../test/org/apache/clojure_mxnet/ndarray_test.clj | 12 ++
 .../test/org/apache/clojure_mxnet/util_test.clj| 27 ++
 4 files changed, 80 insertions(+)

diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/ndarray.clj 
b/contrib/clojure-package/src/org/apache/clojure_mxnet/ndarray.clj
index 151e18b..9caa00d 100644
--- a/contrib/clojure-package/src/org/apache/clojure_mxnet/ndarray.clj
+++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/ndarray.clj
@@ -94,6 +94,27 @@
   ([start stop]
(arange start stop {})))
 
+(defn ->ndarray
+  "Creates a new NDArray based on the given n-dimenstional vector
+   of numbers.
+`nd-vec`: n-dimensional vector with numbers.
+`opts-map` {
+   `ctx`: Context of the output ndarray, will use default context if 
unspecified.
+}
+returns: `ndarray` with the given values and matching the shape of the 
input vector.
+   Ex:
+(->ndarray [5.0 -4.0])
+(->ndarray [5 -4] {:ctx (context/cpu)})
+(->ndarray [[1 2 3] [4 5 6]])
+(->ndarray [[[1.0] [2.0]]]"
+  ([nd-vec {:keys [ctx]
+:or {ctx (mx-context/default-context)}
+:as opts}]
+   (array (vec (clojure.core/flatten nd-vec))
+  (util/nd-seq-shape nd-vec)
+  {:ctx ctx}))
+  ([nd-vec] (->ndarray nd-vec {})))
+
 (defn slice
   "Return a sliced NDArray that shares memory with current one."
   ([ndarray i]
diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj 
b/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj
index 6b5f507..7eb1426 100644
--- a/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj
+++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj
@@ -218,6 +218,26 @@
 (throw (ex-info error-msg
 (s/explain-data spec value)
 
+(s/def ::non-empty-seq (s/and sequential? not-empty))
+(defn to-array-nd
+  "Converts any N-D sequential structure to an array
+   with the same dimensions."
+  [nd-seq]
+  (validate! ::non-empty-seq nd-seq "Invalid N-D sequence")
+  (if (sequential? (first nd-seq))
+(to-array (mapv to-array-nd nd-seq))
+(to-array nd-seq)))
+
+(defn nd-seq-shape
+  "Computes the shape of a n-dimensional sequential structure"
+  [nd-seq]
+  (validate! ::non-empty-seq nd-seq "Invalid N-D sequence")
+  (loop [s nd-seq
+ shape [(count s)]]
+(if (sequential? (first s))
+  (recur (first s) (conj shape (count (first s
+  shape)))
+
 (defn map->scala-tuple-seq
   "* Convert a map to a scala-Seq of scala-Tubple.
* Should also work if a seq of seq of 2 things passed.
diff --git 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/ndarray_test.clj 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/ndarray_test.clj
index a9ae296..ee7c16b 100644
--- a/contrib/clojure-package/test/org/apache/clojure_mxnet/ndarray_test.clj
+++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/ndarray_test.clj
@@ -146,6 +146,18 @@
 (is (= [0.0 0.0 0.5 0.5 1.0 1.0 1.5 1.5 2.0 2.0 2.5 2.5 3.0 3.0 3.5 3.5 
4.0 4.0 4.5 4.5]
(->vec (ndarray/arange start stop {:step step :repeat repeat}))
 
+(deftest test->ndarray
+  (let [nda1 (ndarray/->ndarray [5.0 -4.0])
+nda2 (ndarray/->ndarray [[1 2 3]
+ [4 5 6]])
+nda3 (ndarray/->ndarray [[[7.0] [8.0]]])]
+(is (= [5.0 -4.0] (->vec nda1)))
+(is (= [2] (mx-shape/->vec (shape nda1
+(is (= [1.0 2.0 3.0 4.0 5.0 6.0] (->vec nda2)))
+(is (= [2 3] (mx-shape/->vec (shape nda2
+(is (= [7.0 8.0] (->vec nda3)))
+(is (= [1 2 1] (mx-shape/->vec (shape nda3))
+
 (deftest test-power
   (let [nda (ndarray/array [3 5] [2 1])]
 
diff --git 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/util_test.clj 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/util_test.clj
index 4ed7d38..15c4859 100644
--- a/contrib/clojure-package/test/org/apache/clojure_mxnet/util_test.clj
+++ b/contrib/clojur

[incubator-mxnet] branch master updated: [clojure-package][wip] add `->nd-vec` function in `ndarray.clj` (#14308)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 8be97d7  [clojure-package][wip] add `->nd-vec` function in 
`ndarray.clj` (#14308)
8be97d7 is described below

commit 8be97d7a79f9ea9815e41956e5f15ddcf25026b6
Author: Arthur Caillau 
AuthorDate: Mon Mar 11 00:46:50 2019 +0100

[clojure-package][wip] add `->nd-vec` function in `ndarray.clj` (#14308)

* [clojure-package][wip] add `->nd-vec` function in `ndarray.clj`

* WIP
* Unit tests need to be added

* [clojure-package][ndarray] add unit tests for `->nd-vec` util fn
---
 .../src/org/apache/clojure_mxnet/ndarray.clj   | 58 +++---
 .../test/org/apache/clojure_mxnet/ndarray_test.clj | 12 +
 2 files changed, 64 insertions(+), 6 deletions(-)

diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/ndarray.clj 
b/contrib/clojure-package/src/org/apache/clojure_mxnet/ndarray.clj
index 651bdcb..151e18b 100644
--- a/contrib/clojure-package/src/org/apache/clojure_mxnet/ndarray.clj
+++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/ndarray.clj
@@ -16,15 +16,18 @@
 ;;
 
 (ns org.apache.clojure-mxnet.ndarray
+  "NDArray API for Clojure package."
   (:refer-clojure :exclude [* - + > >= < <= / cast concat flatten identity 
load max
 min repeat reverse set sort take to-array empty 
shuffle
 ref])
-  (:require [org.apache.clojure-mxnet.base :as base]
-[org.apache.clojure-mxnet.context :as mx-context]
-[org.apache.clojure-mxnet.shape :as mx-shape]
-[org.apache.clojure-mxnet.util :as util]
-[clojure.reflect :as r]
-[t6.from-scala.core :refer [$] :as $])
+  (:require
+[clojure.spec.alpha :as s]
+
+[org.apache.clojure-mxnet.base :as base]
+[org.apache.clojure-mxnet.context :as mx-context]
+[org.apache.clojure-mxnet.shape :as mx-shape]
+[org.apache.clojure-mxnet.util :as util]
+[t6.from-scala.core :refer [$] :as $])
   (:import (org.apache.mxnet NDArray)))
 
 ;; loads the generated functions into the namespace
@@ -167,3 +170,46 @@
 
 (defn shape-vec [ndarray]
   (mx-shape/->vec (shape ndarray)))
+
+(s/def ::ndarray #(instance? NDArray %))
+(s/def ::vector vector?)
+(s/def ::sequential sequential?)
+(s/def ::shape-vec-match-vec
+  (fn [[v vec-shape]] (= (count v) (reduce clojure.core/* 1 vec-shape
+
+(s/fdef vec->nd-vec
+:args (s/cat :v ::sequential :shape-vec ::sequential)
+:ret ::vector)
+
+(defn- vec->nd-vec
+  "Convert a vector `v` into a n-dimensional vector given the `shape-vec`
+   Ex:
+(vec->nd-vec [1 2 3] [1 1 3])   ;[[[1 2 3]]]
+(vec->nd-vec [1 2 3 4 5 6] [2 3 1]) ;[[[1] [2] [3]] [[4] [5] [6]]]
+(vec->nd-vec [1 2 3 4 5 6] [1 2 3]) ;[[[1 2 3]] [4 5 6]]]
+(vec->nd-vec [1 2 3 4 5 6] [3 1 2]) ;[[[1 2]] [[3 4]] [[5 6]]]
+(vec->nd-vec [1 2 3 4 5 6] [3 2])   ;[[1 2] [3 4] [5 6]]"
+  [v [s1 & ss :as shape-vec]]
+  (util/validate! ::sequential v "Invalid input vector `v`")
+  (util/validate! ::sequential shape-vec "Invalid input vector `shape-vec`")
+  (util/validate! ::shape-vec-match-vec
+  [v shape-vec]
+  "Mismatch between vector `v` and vector `shape-vec`")
+  (if-not (seq ss)
+(vec v)
+(->> v
+ (partition (clojure.core// (count v) s1))
+ vec
+ (mapv #(vec->nd-vec % ss)
+
+(s/fdef ->nd-vec :args (s/cat :ndarray ::ndarray) :ret ::vector)
+
+(defn ->nd-vec
+  "Convert an ndarray `ndarray` into a n-dimensional Clojure vector.
+  Ex:
+(->nd-vec (array [1] [1 1 1]))   ;[[[1.0]]]
+(->nd-vec (array [1 2 3] [3 1 1]))   ;[[[1.0]] [[2.0]] [[3.0]]]
+(->nd-vec (array [1 2 3 4 5 6]) [3 1 2]) ;[[[1.0 2.0]] [[3.0 4.0]] [[5.0 
6.0]]]"
+  [ndarray]
+  (util/validate! ::ndarray ndarray "Invalid input array")
+  (vec->nd-vec (->vec ndarray) (shape-vec ndarray)))
diff --git 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/ndarray_test.clj 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/ndarray_test.clj
index 9ffd3ab..a9ae296 100644
--- a/contrib/clojure-package/test/org/apache/clojure_mxnet/ndarray_test.clj
+++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/ndarray_test.clj
@@ -473,3 +473,15 @@
 (is (= [2 2] (ndarray/->int-vec nda)))
 (is (= [2.0 2.0] (ndarray/->double-vec nda)))
 (is (= [(byte 2) (byte 2)] (ndarray/->byte-vec nda)
+
+(deftest test->nd-vec
+  (is (= [[[1.0]]]
+ (ndarray/->nd-vec (ndarray/array [1] [1 1 1]
+  (is (= [[[1.0]] [[2.0]] [[3.0]]]
+ (ndarray/->nd-vec (ndarray/array [1 2 3] [3 1 1]))

[incubator-mxnet] 01/01: temporarily disable integration tests that depend on origami repo due to network connectivity issues

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch disable-clojure-origami-tests
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git

commit 1cdda9340c56807ef278eb463fdcba815a6bc6c4
Author: gigasquid 
AuthorDate: Sat Mar 9 16:07:08 2019 -0500

temporarily disable integration tests that depend on origami repo due to 
network connectivity issues
---
 contrib/clojure-package/integration-tests.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/clojure-package/integration-tests.sh 
b/contrib/clojure-package/integration-tests.sh
index 3f80ea5..5ae26e8 100755
--- a/contrib/clojure-package/integration-tests.sh
+++ b/contrib/clojure-package/integration-tests.sh
@@ -26,7 +26,7 @@ lein install
 # then run through the examples 
 EXAMPLES_HOME=${MXNET_HOME}/contrib/clojure-package/examples
 # use AWK pattern for blacklisting
-TEST_CASES=`find ${EXAMPLES_HOME} -name test | awk 
'!/dontselect1|cnn-text-classification/'`
+TEST_CASES=`find ${EXAMPLES_HOME} -name test | awk 
'!/dontselect1|cnn-text-classification|gan|neural-style|infer|pre-trained-models/'`
 for i in $TEST_CASES ; do
  cd ${i} && lein test
 done

[incubator-mxnet] branch disable-clojure-origami-tests created (now 1cdda93)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch disable-clojure-origami-tests
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


  at 1cdda93  temporarily disable integration tests that depend on origami 
repo due to network connectivity issues

This branch includes the following new commits:

 new 1cdda93  temporarily disable integration tests that depend on origami 
repo due to network connectivity issues

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[incubator-mxnet] branch master updated: [clojure-package] fix docstrings in `normal.clj` (#14295)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 0f88f61  [clojure-package] fix docstrings in `normal.clj` (#14295)
0f88f61 is described below

commit 0f88f61379bd5f59fff6b825be1507d020bf2b7e
Author: Arthur Caillau 
AuthorDate: Tue Mar 5 17:53:38 2019 +0100

[clojure-package] fix docstrings in `normal.clj` (#14295)

* Fixed documentation string in `normal` function
* Added spec to catch `high` < `low` in `uniform`
* Added spec to catch `scale` <= 0 in `normal`
* Added unit tests
---
 .../src/org/apache/clojure_mxnet/random.clj| 70 +-
 .../test/org/apache/clojure_mxnet/random_test.clj  |  4 +-
 2 files changed, 45 insertions(+), 29 deletions(-)

diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/random.clj 
b/contrib/clojure-package/src/org/apache/clojure_mxnet/random.clj
index 0ec2039..1261e65 100644
--- a/contrib/clojure-package/src/org/apache/clojure_mxnet/random.clj
+++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/random.clj
@@ -16,70 +16,84 @@
 ;;
 
 (ns org.apache.clojure-mxnet.random
+  "Random Number interface of mxnet."
   (:require
-   [org.apache.clojure-mxnet.shape :as mx-shape]
-   [org.apache.clojure-mxnet.context :as context]
[clojure.spec.alpha :as s]
+   [org.apache.clojure-mxnet.context :as context]
+   [org.apache.clojure-mxnet.shape :as mx-shape]
[org.apache.clojure-mxnet.util :as util])
   (:import (org.apache.mxnet Context Random)))
 
 (s/def ::low number?)
 (s/def ::high number?)
+(s/def ::low-high (fn [[low high]] (<= low high)))
 (s/def ::shape-vec (s/coll-of pos-int? :kind vector?))
 (s/def ::ctx #(instance? Context %))
 (s/def ::uniform-opts (s/keys :opt-un [::ctx]))
 
 (defn uniform
-  "Generate uniform distribution in [low, high) with shape.
-low: The lower bound of distribution.
-high: The upper bound of distribution.
-shape-vec: vector shape of the ndarray generated.
-opts-map {
-  ctx: Context of output ndarray, will use default context if not 
specified.
-  out: Output place holder}
-returns: The result ndarray with generated result./"
+  "Generate uniform distribution in [`low`, `high`) with shape.
+`low`: The lower bound of distribution.
+`high`: The upper bound of distribution.
+`shape-vec`: vector shape of the ndarray generated.
+`opts-map` {
+  `ctx`: Context of output ndarray, will use default context if not 
specified.
+  `out`: Output place holder}
+returns: The result ndarray with generated result.
+   Ex:
+ (uniform 0 1 [1 10])
+ (uniform -10 10 [100 100])"
   ([low high shape-vec {:keys [ctx out] :as opts}]
-   (util/validate! ::uniform-opts opts "Incorrect random uniform parameters")
+   (util/validate! ::uniform-opts opts  "Incorrect random uniform parameters")
(util/validate! ::low low  "Incorrect random uniform parameter")
(util/validate! ::high high  "Incorrect random uniform parameters")
+   (util/validate! ::low-high [low high]  "Incorrect random uniform 
parameters")
(util/validate! ::shape-vec shape-vec  "Incorrect random uniform 
parameters")
(Random/uniform (float low) (float high) (mx-shape/->shape shape-vec) ctx 
out))
   ([low high shape-vec]
(uniform low high shape-vec {})))
 
 (s/def ::loc number?)
-(s/def ::scale number?)
+(s/def ::scale (s/and number? pos?))
 (s/def ::normal-opts (s/keys :opt-un [::ctx]))
 
 (defn normal
-  "Generate normal(Gaussian) distribution N(mean, stdvar^^2) with shape.
-loc: The standard deviation of the normal distribution
-scale: The upper bound of distribution.
-shape-vec: vector shape of the ndarray generated.
-opts-map {
-  ctx: Context of output ndarray, will use default context if not 
specified.
-  out: Output place holder}
-returns: The result ndarray with generated result./"
+  "Generate normal (Gaussian) distribution N(mean, stdvar^^2) with shape.
+`loc`: Mean (centre) of the distribution.
+`scale`: Standard deviation (spread or width) of the distribution.
+`shape-vec`: vector shape of the ndarray generated.
+`opts-map` {
+  `ctx`: Context of output ndarray, will use default context if not 
specified.
+  `out`: Output place holder}
+returns: The result ndarray with generated result.
+   Ex:
+ (normal 0 1 [10 10])
+ (normal -5 4 [2 3])"
   ([loc scale shape-vec {:keys [ctx out] :as opts}]
(util/validate! ::normal-opts opts  "Incorrect random normal parameters")
(util/validate! ::loc loc  "Incorrect random normal parameters")
(util/validate! ::scale scale  "Incorrect random normal parameters")
(util/val

[incubator-mxnet] branch master updated: [clojure-package] improve docstrings in `image.clj` (#14307)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 21f67bd  [clojure-package] improve docstrings in `image.clj` (#14307)
21f67bd is described below

commit 21f67bdb5564dccda7d5d681871302994d76b5dc
Author: Arthur Caillau 
AuthorDate: Tue Mar 5 01:13:51 2019 +0100

[clojure-package] improve docstrings in `image.clj` (#14307)
---
 .../src/org/apache/clojure_mxnet/image.clj | 76 --
 1 file changed, 69 insertions(+), 7 deletions(-)

diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/image.clj 
b/contrib/clojure-package/src/org/apache/clojure_mxnet/image.clj
index e2e87ed..7fb603e 100644
--- a/contrib/clojure-package/src/org/apache/clojure_mxnet/image.clj
+++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/image.clj
@@ -16,6 +16,7 @@
 ;;
 
 (ns org.apache.clojure-mxnet.image
+  "Image API of Clojure package."
   (:require [t6.from-scala.core :refer [$ $$] :as $]
 [org.apache.clojure-mxnet.dtype :as dtype]
 [org.apache.clojure-mxnet.ndarray :as ndarray]
@@ -37,7 +38,18 @@
   (s/keys :opt-un [::color-flag ::to-rgb ::output]))
 
 (defn decode-image
-  "Decodes an image from an input stream"
+  "Decodes an image from an input stream with OpenCV
+`input-stream`: `InputStream` - Contains the binary encoded image
+`color-flag`: 0 or 1 - Convert decoded image to grayscale (0) or color (1)
+`to-rgb`: boolean - Whether to convert decoded image to mxnet's default RGB
+format (instead of opencv's default BGR)
+`output`: nil or `NDArray`
+returns: `NDArray` with dtype uint8
+
+  Ex:
+(decode-image input-stream)
+(decode-image input-stream {:color-flag 1})
+(decode-image input-stream {:color-flag 0 :output nd})"
   ([input-stream {:keys [color-flag to-rgb output]
   :or {color-flag COLOR to-rgb true output nil}
   :as opts}]
@@ -54,7 +66,19 @@
   (s/or :none nil? :some ::to-rgb))
 
 (defn read-image
-  "Reads an image file and returns an ndarray"
+  "Reads an image file and returns an ndarray with OpenCV. It returns image in
+   RGB by default instead of OpenCV's default BGR.
+`filename`: string - Name of the image file to be loaded
+`color-flag`: 0 or 1 - Convert decoded image to grayscale (0) or color (1)
+`to-rgb`: boolean - Whether to convert decoded image to mxnet's default RGB
+format (instead of opencv's default BGR)
+`output`: nil or `NDArray`
+returns: `NDArray` with dtype uint8
+
+   Ex:
+ (read-image \"cat.jpg\")
+ (read-image \"cat.jpg\" {:color-flag 0})
+ (read-image \"cat.jpg\" {:color-flag 1 :output nd})"
   ([filename {:keys [color-flag to-rgb output]
   :or {color-flag nil to-rgb nil output nil}
   :as opts}]
@@ -74,7 +98,17 @@
 (s/def ::optional-int (s/or :none nil? :some int?))
 
 (defn resize-image
-  "Resizes the image array to (width, height)"
+  "Resizes the image array to (width, height)
+   `input`: `NDArray` - source image in NDArray
+   `w`: int - Width of resized image
+   `h`: int - Height of resized image
+   `interpolation`: Interpolation method. Default is INTER_LINEAR
+   `ouput`: nil or `NDArray`
+   returns: `NDArray`
+
+   Ex:
+ (resize-image nd-img 300 300)
+ (resize-image nd-img 28 28 {:output nd})"
   ([input w h {:keys [interpolation output]
:or {interpolation nil output nil}
:as opts}]
@@ -88,7 +122,21 @@
(resize-image input w h {})))
 
 (defn apply-border
-  "Pad image border"
+  "Pad image border with OpenCV.
+   `input`: `NDArray` - source image in NDArray
+   `top`: int - Top margin
+   `bottom`: int - Bottom margin
+   `left`: int - Left margin
+   `right`: int - Right margin
+   `fill-type`: nil or Filling type - Default BORDER_CONSTANT
+   `value`: nil or double - Deprecated, use `values` instead
+   `values`: Fill with value(RGB or gray), up to 4 channels
+   `output`: nil or `NDArray`
+   returns: `NDArray`
+
+   Ex:
+ (apply-border img-nd 1 1 1 1)
+ (apply-border img-nd 3 3 0 0)"
   ([input top bottom left right
 {:keys [fill-type value values output]
  :or {fill-type nil value nil values nil output nil}
@@ -109,7 +157,17 @@
(apply-border input top bottom left right {})))
 
 (defn fixed-crop
-  "Return a fixed crop of the image"
+  "Return a fixed crop of the image.
+   `input`: `NDArray` - Source image in NDArray
+   `x0`: int - Starting x point
+   `y0`: int - Starting y point
+   `w`: int - Width of the image
+   `h`: int - Height of the image
+   returns: cropped `NDArray`
+
+   Ex:
+ (fixed-crop nd-img 0 0 28 28)
+ (fixed-crop nd-img 10 0 1

[incubator-mxnet] branch master updated: upgrade codox to work with lein 2.9.0 (#14133)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 85d3fa3  upgrade codox to work with lein 2.9.0 (#14133)
85d3fa3 is described below

commit 85d3fa34901c8c31815aa59ae5e125e3c6feea9b
Author: Carin Meier 
AuthorDate: Wed Feb 13 18:44:55 2019 -0500

upgrade codox to work with lein 2.9.0 (#14133)
---
 contrib/clojure-package/project.clj | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/clojure-package/project.clj 
b/contrib/clojure-package/project.clj
index 61d39e2..e2b999d 100644
--- a/contrib/clojure-package/project.clj
+++ b/contrib/clojure-package/project.clj
@@ -36,7 +36,7 @@
  [org.apache.logging.log4j/log4j-api "2.8.1"]
  [org.slf4j/slf4j-log4j12 "1.7.25" :exclusions 
[org.slf4j/slf4j-api]]]
   :pedantic? :skip
-  :plugins [[lein-codox "0.10.3" :exclusions [org.clojure/clojure]]
+  :plugins [[lein-codox "0.10.6" :exclusions [org.clojure/clojure]]
 [lein-cloverage "1.0.10" :exclusions [org.clojure/clojure]]
 [lein-cljfmt "0.5.7"]]
   :codox {:namespaces [#"^org\.apache\.clojure-mxnet\.(?!gen).*"]}

[incubator-mxnet] branch master updated: [Clojure] Add resource scope to clojure package (#13993)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new fbc5723  [Clojure] Add resource scope to clojure package (#13993)
fbc5723 is described below

commit fbc57238ada275549b1370feb763c0ae7b81d8a2
Author: Carin Meier 
AuthorDate: Mon Feb 4 21:01:08 2019 -0500

[Clojure] Add resource scope to clojure package (#13993)

* Add resource scope to clojure package

* add rat

* fix integration test

* feedback from @benkamphaus
- move from defs to atoms to make the tests a bit better

* adding alias with-do and with-let
more tests

* another test

* Add examples in docstring

* refactor example and test to use resource-scope/with-let

* fix tests and problem with laziness
now they work as expected!

* refactor to be a bit more modular

* remove comments
---
 .../src/imclassification/train_mnist.clj   |  65 -
 .../test/imclassification/train_mnist_test.clj |  13 +-
 .../org/apache/clojure_mxnet/resource_scope.clj|  53 
 .../src/org/apache/clojure_mxnet/util.clj  |   6 +
 .../apache/clojure_mxnet/resource_scope_test.clj   | 146 +
 .../test/org/apache/clojure_mxnet/util_test.clj|   7 +
 6 files changed, 252 insertions(+), 38 deletions(-)

diff --git 
a/contrib/clojure-package/examples/imclassification/src/imclassification/train_mnist.clj
 
b/contrib/clojure-package/examples/imclassification/src/imclassification/train_mnist.clj
index e61e9eb..164b5f2 100644
--- 
a/contrib/clojure-package/examples/imclassification/src/imclassification/train_mnist.clj
+++ 
b/contrib/clojure-package/examples/imclassification/src/imclassification/train_mnist.clj
@@ -25,7 +25,8 @@
 [org.apache.clojure-mxnet.kvstore :as kvstore]
 [org.apache.clojure-mxnet.kvstore-server :as kvstore-server]
 [org.apache.clojure-mxnet.optimizer :as optimizer]
-[org.apache.clojure-mxnet.eval-metric :as eval-metric])
+[org.apache.clojure-mxnet.eval-metric :as eval-metric]
+[org.apache.clojure-mxnet.resource-scope :as resource-scope])
   (:gen-class))
 
 (def data-dir "data/") ;; the data directory to store the mnist data
@@ -51,28 +52,6 @@
 (when-not (.exists (io/file (str data-dir "train-images-idx3-ubyte")))
   (sh "../../scripts/get_mnist_data.sh"))
 
-;;; Load the MNIST datasets
-(defonce train-data (mx-io/mnist-iter {:image (str data-dir 
"train-images-idx3-ubyte")
-   :label (str data-dir 
"train-labels-idx1-ubyte")
-   :label-name "softmax_label"
-   :input-shape [784]
-   :batch-size batch-size
-   :shuffle true
-   :flat true
-   :silent false
-   :seed 10
-   :num-parts num-workers
-   :part-index 0}))
-
-(defonce test-data (mx-io/mnist-iter {:image (str data-dir 
"t10k-images-idx3-ubyte")
-  :label (str data-dir 
"t10k-labels-idx1-ubyte")
-  :input-shape [784]
-  :batch-size batch-size
-  :flat true
-  :silent false
-  :num-parts num-workers
-  :part-index 0}))
-
 (defn get-symbol []
   (as-> (sym/variable "data") data
 (sym/fully-connected "fc1" {:data data :num-hidden 128})
@@ -82,7 +61,31 @@
 (sym/fully-connected "fc3" {:data data :num-hidden 10})
 (sym/softmax-output "softmax" {:data data})))
 
-(defn start 
+
+(defn train-data []
+  (mx-io/mnist-iter {:image (str data-dir "train-images-idx3-ubyte")
+ :label (str data-dir "train-labels-idx1-ubyte")
+ :label-name "softmax_label"
+ :input-shape [784]
+ :batch-size batch-size
+ :shuffle true
+ :flat true
+ :silent false
+ :seed 10
+ :num-parts num-workers
+ :part-index 0}))
+
+(defn eval-data []
+  (mx-io/mnist-iter {:image (str data-dir "t10k-images-idx3-ubyte")
+ :label (str data-dir "t10k-labels-idx1-ubyte")
+ :input-shape [784]
+

[incubator-mxnet] branch master updated: rewrote the concat test to avoid flaky failures (#14049)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new ac962cd  rewrote the concat test to avoid flaky failures (#14049)
ac962cd is described below

commit ac962cd3efee4be01bc0a000948e86fbe1aa02fd
Author: Carin Meier 
AuthorDate: Mon Feb 4 19:23:40 2019 -0500

rewrote the concat test to avoid flaky failures (#14049)

ran 1 times with no failures
---
 .../org/apache/clojure_mxnet/operator_test.clj | 35 ++
 1 file changed, 9 insertions(+), 26 deletions(-)

diff --git 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/operator_test.clj 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/operator_test.clj
index c97711b..3b97190 100644
--- a/contrib/clojure-package/test/org/apache/clojure_mxnet/operator_test.clj
+++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/operator_test.clj
@@ -51,33 +51,16 @@
 (is (= out-grad grad))
 
 (deftest test-concat
-  (let [shape-vecs [[2 2] [3 2]]
-x (sym/variable "x")
-y (sym/variable "y")
-out (sym/concat "conc" nil [x y] {:dim 0})
-arr (mapv #(ndarray/empty %) shape-vecs)
-arr-np (mapv #(ndarray/copy %) arr)
-arr-grad (map #(ndarray/empty %) shape-vecs)
-arg-names (sym/list-arguments out)
-grad-map (zipmap arg-names arr-grad)
-args (sym/list-arguments out)
-[arg-shapes out-shapes aux-shapes] (sym/infer-shape out (zipmap args 
shape-vecs))
-out-shape-vec (first out-shapes)
-out-grad (ndarray/empty out-shape-vec)
-exec1 (sym/bind out (context/default-context) arr grad-map)
-out1 (-> (executor/forward exec1)
+  (let [a (sym/variable "a")
+  b (sym/variable "b")
+  c (sym/concat "conc" nil [a b] {:dim 0})
+  exec (sym/bind c (context/default-context) {"a" (ndarray/array [1 2] [2 
1])
+  "b" (ndarray/array [3 4] [2 
1])})
+  output (-> (executor/forward exec)
  (executor/outputs)
- (first))
-ret (ndarray/concatenate arr)]
-(is (= out1 ret))
-
-;;backward
-(ndarray/copy-to out1 out-grad)
-(ndarray/+= out-grad 1)
-(executor/backward exec1 out-grad)
-(let [grads arr-grad
-  np-grads arr-np]
-  (is (= grads (mapv #(ndarray/+ % 1) np-grads))
+ (first))]
+(is (= [1.0 2.0 3.0 4.0] (ndarray/->vec output)))
+(is (= [4 1] (ndarray/shape-vec output)
 
 (defn check-regression [model forward-fn backward-fn]
   (let [shape-vec [3 1]

[incubator-mxnet] branch master updated: Update project.clj file to use the snapshots repo to be able to pull (#13935)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 4c88f30  Update project.clj file to use the snapshots repo to be able 
to pull (#13935)
4c88f30 is described below

commit 4c88f3049695b9c751a6d1a228853f18b2733358
Author: Carin Meier 
AuthorDate: Tue Jan 22 12:31:20 2019 -0500

Update project.clj file to use the snapshots repo to be able to pull 
(#13935)

nightly Scala jar - also update readme
---
 contrib/clojure-package/README.md   | 14 +++---
 contrib/clojure-package/project.clj | 27 ---
 2 files changed, 11 insertions(+), 30 deletions(-)

diff --git a/contrib/clojure-package/README.md 
b/contrib/clojure-package/README.md
index ba6160a..840c15e 100644
--- a/contrib/clojure-package/README.md
+++ b/contrib/clojure-package/README.md
@@ -142,21 +142,13 @@ With this option, you will install a Git revision of the 
Clojure package source
 
 To run examples, you can now use `lein run` in any of the example directories, 
e.g., `examples/imclassification`. You can also specify the compute device, 
e.g., `lein run :cpu 2` (for 2 CPUs) or `lein run :gpu` (for 1 GPU).
 
- Experimental: Using Scala Snapshot Jars
-**Note:** Instead of a release tag, you can also use a development version of 
the Clojure package, e.g., Git `master`, together with the prebuilt Scala jar. 
There is a repo of nightly built snapshots of Scala jars. You can use them in 
your `project.clj` by adding a repository:
+ Using Scala Nightly Snapshot Jars
+**Note:** Instead of a release tag, you can also use a development version of 
the Clojure package, e.g., Git `master`, together with the prebuilt Scala jar. 
There is a repo of nightly built snapshots of Scala jars. You can use them in 
your `project.clj` by adding them as a dependency:
 
-```
-["snapshots" {:url 
"https://repository.apache.org/content/repositories/snapshots;
-  :snapshots true
-  :sign-releases false
-  :checksum :fail
-  :update :always
-  :releases {:checksum :fail :update :always}}]
-```
 
 Then you should be able to run with your dependency:
 
-[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "latest-version-SNAPSHOT"]
+[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu 
""]
 
 
 In that case, however, breakage can happen at any point, for instance when the 
Scala development version adds, changes or removes an interface and the Clojure 
development version moves along. If you really need the most recent version, 
you should consider [installation option 3](#option-3-everything-from-source).
diff --git a/contrib/clojure-package/project.clj 
b/contrib/clojure-package/project.clj
index c4428ce..61d39e2 100644
--- a/contrib/clojure-package/project.clj
+++ b/contrib/clojure-package/project.clj
@@ -23,10 +23,10 @@
   :dependencies [[org.clojure/clojure "1.9.0"]
  [t6/from-scala "0.3.0"]
 
- ;; Jars from Nexus
- ;[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.2.1"]
- ;[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu "1.2.1"]
- ;[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu "1.2.1"]
+ ;; To use with nightly snapshot
+ ;[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu 
""]
+ ;[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu 
""]
+ ;[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu 
"https://repository.apache.org/content/repositories/staging;
-  ;; If a repository contains releases only setting
-  ;; :snapshots to false will speed up 
dependencies.
-  :snapshots true
-  ;; Disable signing releases deployed to this 
repo.
-  ;; (Not recommended.)
-  :sign-releases false
-  ;; You can also set the policies for how to 
handle
-  ;; :checksum failures to :fail, :warn, or 
:ignore.
-  :checksum :fail
-  ;; How often should this repository be checked 
for
-  ;; snapshot updates? (:daily, :always, or :never)
-  :update :always
-  ;; You can also apply them to releases only:
-  :releases {:checksum :fail :update :always}}]])
+  :repositories [["staging" {:url 
"https://repository.apache.org/content/repositories/staging;  

[incubator-mxnet] branch master updated: #13624 clojure nightly tests (#13624)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new eebdd5f  #13624 clojure nightly tests (#13624)
eebdd5f is described below

commit eebdd5f644e953da76ac35a898992ef5c83d3f30
Author: Nicolas Modrzyk 
AuthorDate: Tue Jan 22 22:56:49 2019 +0900

#13624 clojure nightly tests (#13624)
---
 ci/docker/runtime_functions.sh |   8 ++
 ci/jenkins/Jenkins_steps.groovy|  13 +
 ci/jenkins/Jenkinsfile_unix_cpu|   1 +
 .../src/infer/imageclassifier_example.clj  |   2 ++
 .../infer/objectdetector/images/marcel.jpg | Bin 0 -> 116161 bytes
 .../src/infer/objectdetector_example.clj   |  32 ++---
 .../test/infer/objectdetector_example_test.clj |   5 +++-
 contrib/clojure-package/integration-tests.sh   |  18 +++-
 8 files changed, 61 insertions(+), 18 deletions(-)

diff --git a/ci/docker/runtime_functions.sh b/ci/docker/runtime_functions.sh
index 76fd2c6..4c70c1a 100755
--- a/ci/docker/runtime_functions.sh
+++ b/ci/docker/runtime_functions.sh
@@ -868,6 +868,14 @@ unittest_ubuntu_cpu_clojure() {
 ./contrib/clojure-package/ci-test.sh
 }
 
+unittest_ubuntu_cpu_clojure_integration() {
+set -ex
+make scalapkg USE_OPENCV=1 USE_BLAS=openblas USE_DIST_KVSTORE=1 
ENABLE_TESTCOVERAGE=1
+make scalainstall USE_OPENCV=1 USE_BLAS=openblas USE_DIST_KVSTORE=1 
ENABLE_TESTCOVERAGE=1
+./contrib/clojure-package/integration-tests.sh
+}
+
+
 unittest_ubuntu_cpugpu_perl() {
 set -ex
 ./perl-package/test.sh
diff --git a/ci/jenkins/Jenkins_steps.groovy b/ci/jenkins/Jenkins_steps.groovy
index 079a384..9db3588 100644
--- a/ci/jenkins/Jenkins_steps.groovy
+++ b/ci/jenkins/Jenkins_steps.groovy
@@ -887,6 +887,19 @@ def test_unix_clojure_cpu() {
 }]
 }
 
+def test_unix_clojure_integration_cpu() {
+return ['Clojure: CPU Integration': {
+  node(NODE_LINUX_CPU) {
+ws('workspace/ut-clojure-integration-cpu') {
+  timeout(time: max_time, unit: 'MINUTES') {
+utils.unpack_and_init('cpu', mx_lib, true)
+utils.docker_run('ubuntu_cpu', 
'unittest_ubuntu_cpu_clojure_integration', false)
+  }
+}
+  }
+}]
+}
+
 def test_unix_r_cpu() {
 return ['R: CPU': {
   node(NODE_LINUX_CPU) {
diff --git a/ci/jenkins/Jenkinsfile_unix_cpu b/ci/jenkins/Jenkinsfile_unix_cpu
index 2cf2c35..9446348 100644
--- a/ci/jenkins/Jenkinsfile_unix_cpu
+++ b/ci/jenkins/Jenkinsfile_unix_cpu
@@ -52,6 +52,7 @@ core_logic: {
 custom_steps.test_unix_scala_cpu(),
 custom_steps.test_unix_scala_mkldnn_cpu(),
 custom_steps.test_unix_clojure_cpu(),
+custom_steps.test_unix_clojure_integration_cpu(),
 custom_steps.test_unix_perl_cpu(),
 custom_steps.test_unix_r_cpu(),
 custom_steps.test_unix_julia07_cpu(),
diff --git 
a/contrib/clojure-package/examples/infer/imageclassifier/src/infer/imageclassifier_example.clj
 
b/contrib/clojure-package/examples/infer/imageclassifier/src/infer/imageclassifier_example.clj
index 6994b4f..bc8b82e 100644
--- 
a/contrib/clojure-package/examples/infer/imageclassifier/src/infer/imageclassifier_example.clj
+++ 
b/contrib/clojure-package/examples/infer/imageclassifier/src/infer/imageclassifier_example.clj
@@ -74,6 +74,8 @@
 image-file-batches (->> input-dir
 io/file
 file-seq
+sort
+reverse
 (filter #(.isFile %))
 (filter #(re-matches #".*\.jpg$" (.getPath %)))
 (mapv #(.getPath %))
diff --git 
a/contrib/clojure-package/examples/infer/objectdetector/images/marcel.jpg 
b/contrib/clojure-package/examples/infer/objectdetector/images/marcel.jpg
new file mode 100644
index 000..1bf7387
Binary files /dev/null and 
b/contrib/clojure-package/examples/infer/objectdetector/images/marcel.jpg differ
diff --git 
a/contrib/clojure-package/examples/infer/objectdetector/src/infer/objectdetector_example.clj
 
b/contrib/clojure-package/examples/infer/objectdetector/src/infer/objectdetector_example.clj
index 6010ddc..9331798 100644
--- 
a/contrib/clojure-package/examples/infer/objectdetector/src/infer/objectdetector_example.clj
+++ 
b/contrib/clojure-package/examples/infer/objectdetector/src/infer/objectdetector_example.clj
@@ -78,32 +78,44 @@
 
 (defn detect-single-image
   "Detect objects in a single image and print top-5 predictions"
-  [detector input-image output-dir]
+  ([detector input-dir] (detect-single-image detector input-dir "results"))
+  ([detector input-image output-dir]
+(.mkdir (io/file output-dir))
   (let [image (infer/load-image

[incubator-mxnet] branch clojure-resource-scope deleted (was 0a5e57a)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a change to branch clojure-resource-scope
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.


 was 0a5e57a  add resource scope to imclassification example

This change permanently discards the following revisions:

 discard 0a5e57a  add resource scope to imclassification example
 discard f8c0768  clean up and add tests
 discard 0f525b8  wip

[incubator-mxnet] branch clojure-resource-scope updated: add resource scope to imclassification example

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-resource-scope
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/clojure-resource-scope by this 
push:
 new 0a5e57a  add resource scope to imclassification example
0a5e57a is described below

commit 0a5e57a9817a17c81a6c730165c9e403d595c68a
Author: gigasquid 
AuthorDate: Sun Jan 20 14:33:27 2019 -0500

add resource scope to imclassification example
---
 .../src/imclassification/train_mnist.clj   | 65 +++---
 .../org/apache/clojure_mxnet/resource_scope.clj| 11 +---
 2 files changed, 32 insertions(+), 44 deletions(-)

diff --git 
a/contrib/clojure-package/examples/imclassification/src/imclassification/train_mnist.clj
 
b/contrib/clojure-package/examples/imclassification/src/imclassification/train_mnist.clj
index e61e9eb..017e19b 100644
--- 
a/contrib/clojure-package/examples/imclassification/src/imclassification/train_mnist.clj
+++ 
b/contrib/clojure-package/examples/imclassification/src/imclassification/train_mnist.clj
@@ -25,7 +25,8 @@
 [org.apache.clojure-mxnet.kvstore :as kvstore]
 [org.apache.clojure-mxnet.kvstore-server :as kvstore-server]
 [org.apache.clojure-mxnet.optimizer :as optimizer]
-[org.apache.clojure-mxnet.eval-metric :as eval-metric])
+[org.apache.clojure-mxnet.eval-metric :as eval-metric]
+[org.apache.clojure-mxnet.resource-scope :as resource-scope])
   (:gen-class))
 
 (def data-dir "data/") ;; the data directory to store the mnist data
@@ -51,28 +52,6 @@
 (when-not (.exists (io/file (str data-dir "train-images-idx3-ubyte")))
   (sh "../../scripts/get_mnist_data.sh"))
 
-;;; Load the MNIST datasets
-(defonce train-data (mx-io/mnist-iter {:image (str data-dir 
"train-images-idx3-ubyte")
-   :label (str data-dir 
"train-labels-idx1-ubyte")
-   :label-name "softmax_label"
-   :input-shape [784]
-   :batch-size batch-size
-   :shuffle true
-   :flat true
-   :silent false
-   :seed 10
-   :num-parts num-workers
-   :part-index 0}))
-
-(defonce test-data (mx-io/mnist-iter {:image (str data-dir 
"t10k-images-idx3-ubyte")
-  :label (str data-dir 
"t10k-labels-idx1-ubyte")
-  :input-shape [784]
-  :batch-size batch-size
-  :flat true
-  :silent false
-  :num-parts num-workers
-  :part-index 0}))
-
 (defn get-symbol []
   (as-> (sym/variable "data") data
 (sym/fully-connected "fc1" {:data data :num-hidden 128})
@@ -82,7 +61,7 @@
 (sym/fully-connected "fc3" {:data data :num-hidden 10})
 (sym/softmax-output "softmax" {:data data})))
 
-(defn start 
+(defn start
   ([devs] (start devs num-epoch))
   ([devs _num-epoch]
   (when scheduler-host
@@ -96,16 +75,34 @@
 (do
   (println "Starting Training of MNIST ")
   (println "Running with context devices of" devs)
-  (let [_mod (m/module (get-symbol) {:contexts devs})]
-(m/fit _mod {:train-data train-data
-:eval-data test-data
-:num-epoch _num-epoch
-:fit-params (m/fit-params {:kvstore kvstore
-   :optimizer optimizer
-   :eval-metric eval-metric})})
-(println "Finish fit")
-_mod
-)
+  (resource-scope/using
+   (let [_mod (m/module (get-symbol) {:contexts devs})]
+ (m/fit _mod {:train-data (mx-io/mnist-iter {:image (str data-dir 
"train-images-idx3-ubyte")
+ :label (str data-dir 
"train-labels-idx1-ubyte")
+ :label-name 
"softmax_label"
+ :input-shape [784]
+ :batch-size batch-size
+ :shuffle true
+ :flat true
+ :silent false
+ :seed 10
+  

[incubator-mxnet] branch clojure-resource-scope updated: clean up and add tests

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch clojure-resource-scope
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/clojure-resource-scope by this 
push:
 new f8c0768  clean up and add tests
f8c0768 is described below

commit f8c07688954af72d1b2fa0aa14c6b9277a06488d
Author: gigasquid 
AuthorDate: Sun Jan 20 13:45:12 2019 -0500

clean up and add tests
---
 .../org/apache/clojure_mxnet/resource_scope.clj| 22 --
 .../src/org/apache/clojure_mxnet/util.clj  |  6 ++
 .../apache/clojure_mxnet/resource_scope_test.clj   | 19 +++
 .../test/org/apache/clojure_mxnet/util_test.clj|  7 +++
 4 files changed, 36 insertions(+), 18 deletions(-)

diff --git 
a/contrib/clojure-package/src/org/apache/clojure_mxnet/resource_scope.clj 
b/contrib/clojure-package/src/org/apache/clojure_mxnet/resource_scope.clj
index 75c7f76..e7dea84 100644
--- a/contrib/clojure-package/src/org/apache/clojure_mxnet/resource_scope.clj
+++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/resource_scope.clj
@@ -1,16 +1,10 @@
 (ns org.apache.clojure-mxnet.resource-scope
-  (:require [org.apache.clojure-mxnet.util :as util]
-[t6.from-scala.core :refer [$ $$] :as $]
-[org.apache.clojure-mxnet.ndarray :as ndarray])
+  (:require [org.apache.clojure-mxnet.util :as util])
   (:import (org.apache.mxnet ResourceScope)))
 
-(require '[clojure.reflect :as r])
-
-(defmacro forms->scala-fn
-  "Creates a scala fn of zero args from forms"
-  [& forms]
-  `($/fn []
- (do ~@forms)))
+;; Captures all Native Resources created using the ResourceScope and
+;; at the end of the body, de allocates all the Native resources by calling 
close on them.
+;;This method will not deAllocate NativeResources returned from the forms.
 
 (defmacro using
   "Uses a Resource Scope for all forms"
@@ -25,11 +19,3 @@
   z {:just-a-test (def temp-z (ndarray/ones [3 3]))}
   y (ndarray/+ temp-x 1)]
   y
-
-(comment
-  (.isDisposed temp-x)
-  (.isDisposed temp-y)
-  (.isDisposed temp-z)
-  (.isDisposed return-val)
-  (ndarray/->vec return-val))
-
diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj 
b/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj
index 43970c0..6b5f507 100644
--- a/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj
+++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj
@@ -239,3 +239,9 @@
(apply $/immutable-list))
   ;; pass-through
   map-or-tuple-seq)))
+
+(defmacro forms->scala-fn
+  "Creates a scala fn of zero args from forms"
+  [& forms]
+  `($/fn []
+ (do ~@forms)))
diff --git 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/resource_scope_test.clj 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/resource_scope_test.clj
new file mode 100644
index 000..dd1024c
--- /dev/null
+++ 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/resource_scope_test.clj
@@ -0,0 +1,19 @@
+(ns org.apache.clojure-mxnet.resource-scope-test
+  (:require [org.apache.clojure-mxnet.ndarray :as ndarray]
+[org.apache.clojure-mxnet.resource-scope :as resource-scope]
+[clojure.test :refer :all]))
+
+(deftest test-resource-scope-with-ndarray
+  (let [x (ndarray/ones [2 2])
+return-val (resource-scope/using
+(def temp-x (ndarray/ones [3 1]))
+(def temp-y (ndarray/ones [3 1]))
+(let [z {:just-a-test (def temp-z (ndarray/ones [3 3]))}
+  y (ndarray/+ temp-x 1)]
+  y))]
+(is (true? (.isDisposed temp-x)))
+(is (true? (.isDisposed temp-y)))
+(is (true? (.isDisposed temp-z)))
+(is (false? (.isDisposed return-val)))
+(is (false? (.isDisposed x)))
+(is (= [2.0 2.0 2.0] (ndarray/->vec return-val)
diff --git 
a/contrib/clojure-package/test/org/apache/clojure_mxnet/util_test.clj 
b/contrib/clojure-package/test/org/apache/clojure_mxnet/util_test.clj
index c26f83d..4ed7d38 100644
--- a/contrib/clojure-package/test/org/apache/clojure_mxnet/util_test.clj
+++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/util_test.clj
@@ -226,3 +226,10 @@
   (let [nda (util/map->scala-tuple-seq {:a-b (ndarray/ones [1 2])})]
 (is (= "a_b" (._1 (.head nda
 (is (= [1.0 1.0] (ndarray/->vec (._2 (.head nda)))
+
+(deftest test-forms->scala-fn
+  (let [scala-fn (util/forms->scala-fn
+  (def x 1)
+  (def y 2)
+  {:x x :y y})]
+(is (= {:x 1 :y 2} (.apply scala-fn)

[incubator-mxnet] branch master updated: #13813 examples with opencv4/origami (#13813)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 754ff76   #13813 examples with opencv4/origami (#13813)
754ff76 is described below

commit 754ff7669b4b3d481a80d734d79921e0c5d58bce
Author: Nicolas Modrzyk 
AuthorDate: Thu Jan 17 07:28:25 2019 +0900

 #13813 examples with opencv4/origami (#13813)
---
 contrib/clojure-package/examples/gan/.gitignore|   1 +
 contrib/clojure-package/examples/gan/project.clj   |   7 +-
 .../clojure-package/examples/gan/src/gan/viz.clj   |  56 +++--
 .../examples/infer/objectdetector/.gitignore   |   1 +
 .../examples/infer/objectdetector/README.md|   3 +
 .../examples/infer/objectdetector/project.clj  |   3 +
 .../infer/objectdetector/src/infer/draw.clj|  44 +++
 .../src/infer/objectdetector_example.clj   |  71 +++-
 .../examples/neural-style/project.clj  |   6 +-
 .../neural-style/src/neural_style/core.clj | 127 +
 .../neural-style/test/neural_style/vgg_19_test.clj |  11 +-
 .../examples/pre-trained-models/README.md  |  10 ++
 .../examples/pre-trained-models/project.clj|   5 +-
 .../src/pre_trained_models/predict_image.clj   |  33 +++---
 14 files changed, 201 insertions(+), 177 deletions(-)

diff --git a/contrib/clojure-package/examples/gan/.gitignore 
b/contrib/clojure-package/examples/gan/.gitignore
index c53038e..ea80131 100644
--- a/contrib/clojure-package/examples/gan/.gitignore
+++ b/contrib/clojure-package/examples/gan/.gitignore
@@ -9,3 +9,4 @@ pom.xml.asc
 /.nrepl-port
 .hgignore
 .hg/
+results
\ No newline at end of file
diff --git a/contrib/clojure-package/examples/gan/project.clj 
b/contrib/clojure-package/examples/gan/project.clj
index a326f7a..4393987 100644
--- a/contrib/clojure-package/examples/gan/project.clj
+++ b/contrib/clojure-package/examples/gan/project.clj
@@ -15,11 +15,12 @@
 ;; limitations under the License.
 ;;
 
-(defproject gan "0.1.0-SNAPSHOT"
+(defproject gan-origami "0.1.0-SNAPSHOT"
   :description "GAN MNIST with MXNet"
   :plugins [[lein-cljfmt "0.5.7"]]
+  :repositories [["vendredi" {:url 
"https://repository.hellonico.info/repository/hellonico/"}]]
   :dependencies [[org.clojure/clojure "1.9.0"]
  [org.apache.mxnet.contrib.clojure/clojure-mxnet 
"1.5.0-SNAPSHOT"]
- [org.openpnp/opencv "3.4.2-1"]
+ [origami "4.0.0-3"]
  ]
-  :main gan.gan-mnist)
+  :main gan.gan-mnist)
\ No newline at end of file
diff --git a/contrib/clojure-package/examples/gan/src/gan/viz.clj 
b/contrib/clojure-package/examples/gan/src/gan/viz.clj
index 67f7880..08da53c 100644
--- a/contrib/clojure-package/examples/gan/src/gan/viz.clj
+++ b/contrib/clojure-package/examples/gan/src/gan/viz.clj
@@ -18,14 +18,9 @@
 (ns gan.viz
   (:require [org.apache.clojure-mxnet.ndarray :as ndarray]
 [org.apache.clojure-mxnet.shape :as mx-shape]
-[org.apache.clojure-mxnet.io :as mx-io])
-  (:import (nu.pattern OpenCV)
-   (org.opencv.core Core CvType Mat Size)
-   (org.opencv.imgproc Imgproc)
-   (org.opencv.imgcodecs Imgcodecs)))
-
-;;; Viz stuff
-(OpenCV/loadShared)
+[org.apache.clojure-mxnet.io :as mx-io]
+[opencv4.utils :as cvu]
+[opencv4.core :as cv :refer [CV_8UC1 new-matofbyte flip! imwrite 
new-size hconcat! vconcat! new-mat merge!]]))
 
 (defn clip [x]
   (->> x
@@ -37,29 +32,11 @@
(mapv #(.byteValue %
 
 (defn get-img [raw-data channels height width flip]
-  (let [totals (* height width)
-img (if (> channels 1)
-  ;; rgb image
-  (let [[ra ga ba] (byte-array (partition totals raw-data))
-rr (new Mat height width (CvType/CV_8U))
-gg (new Mat height width (CvType/CV_8U))
-bb (new Mat height width (CvType/CV_8U))
-result (new Mat)]
-(.put rr (int 0) (int 0) ra)
-(.put gg (int 0) (int 0) ga)
-(.put bb (int 0) (int 0) ba)
-(Core/merge (java.util.ArrayList. [bb gg rr]) result)
-result)
+  (let [img (if (> channels 1)
+  (throw (Exception. "Image with 3 channels (RGB) not supported"))
   ;; gray image
-  (let [result (new Mat height width (CvType/CV_8U))
-_ (.put result (int 0) (int 0) (byte-array raw-data))]
-result))]
-(do
-  (if flip
-(let [result (new Mat)
-  _ (Core/flip img result (int 0))]
-  result)
-img
+  (cv/>> (new-mat height width CV_8UC1) (byte-ar

[incubator-mxnet] branch master updated: Modifying clojure CNN text classification example (#13865)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 0e57930  Modifying clojure CNN text classification example (#13865)
0e57930 is described below

commit 0e57930011085cedf59ffe040729ea037ceeece3
Author: Kedar Bellare 
AuthorDate: Sun Jan 13 09:18:32 2019 -0800

Modifying clojure CNN text classification example (#13865)

* Modifying clojure CNN text classification example

* Small fixes

* Another minor fix
---
 .../examples/cnn-text-classification/README.md |  38 +++-
 .../src/cnn_text_classification/classifier.clj |  45 +++--
 .../src/cnn_text_classification/data_helper.clj| 195 ++---
 .../cnn_text_classification/classifier_test.clj|  52 +++---
 4 files changed, 218 insertions(+), 112 deletions(-)

diff --git a/contrib/clojure-package/examples/cnn-text-classification/README.md 
b/contrib/clojure-package/examples/cnn-text-classification/README.md
index 86a8abb..19bb913 100644
--- a/contrib/clojure-package/examples/cnn-text-classification/README.md
+++ b/contrib/clojure-package/examples/cnn-text-classification/README.md
@@ -3,19 +3,19 @@
 An example of text classification using CNN
 
 To use you must download the MR polarity dataset and put it in the path 
specified in the mr-dataset-path
-The dataset can be obtained here: 
[https://github.com/yoonkim/CNN_sentence](https://github.com/yoonkim/CNN_sentence).
 The two files `rt-polarity.neg`
+The dataset can be obtained here: 
[CNN_sentence](https://github.com/yoonkim/CNN_sentence). The two files 
`rt-polarity.neg`
 and `rt-polarity.pos` must be put in a directory. For example, 
`data/mr-data/rt-polarity.neg`.
 
 You also must download the glove word embeddings. The suggested one to use is 
the smaller 50 dimension one
-`glove.6B.50d.txt` which is contained in the download file here 
[https://nlp.stanford.edu/projects/glove/](https://nlp.stanford.edu/projects/glove/)
+`glove.6B.50d.txt` which is contained in the download file here: 
[GloVe](https://nlp.stanford.edu/projects/glove/)
 
 ## Usage
 
 You can run through the repl with
-`(train-convnet {:embedding-size 50 :batch-size 100 :test-size 100 :num-epoch 
10 :max-examples 1000})`
+`(train-convnet {:embedding-size 50 :batch-size 100 :test-size 100 :num-epoch 
10 :max-examples 1000 :pretrained-embedding :glove})`
 
 or
-`JVM_OPTS="Xmx1g" lein run` (cpu)
+`JVM_OPTS="-Xmx1g" lein run` (cpu)
 
 You can control the devices you run on by doing:
 
@@ -24,10 +24,36 @@ You can control the devices you run on by doing:
 `lein run :gpu 2` - This will run on 2 gpu devices
 
 
-The max-examples only loads 1000 each of the dataset to keep the time and 
memory down. To run all the examples, 
-change the main to be (train-convnet {:embedding-size 50 :batch-size 100 
:test-size 1000 :num-epoch 10)
+The max-examples only loads 1000 each of the dataset to keep the time and 
memory down. To run all the examples,
+change the main to be (train-convnet {:embedding-size 50 :batch-size 100 
:test-size 1000 :num-epoch 10 :pretrained-embedding :glove})
 
 and then run
 
 - `lein uberjar`
 - `java -Xms1024m -Xmx2048m -jar 
target/cnn-text-classification-0.1.0-SNAPSHOT-standalone.jar`
+
+## Usage with word2vec
+
+You can also use word2vec embeddings in order to train the text classification 
model.
+Before training, you will need to download 
[GoogleNews-vectors-negative300.bin](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing)
 first.
+Once you've downloaded the embeddings (which are in a gzipped format),
+you'll need to unzip them and place them in the `contrib/clojure-package/data` 
directory.
+
+Then you can run training on a subset of examples through the repl using:
+```
+(train-convnet {:embedding-size 300 :batch-size 100 :test-size 100 :num-epoch 
10 :max-examples 1000 :pretrained-embedding :word2vec})
+```
+Note that loading word2vec embeddings consumes memory and takes some time.
+
+You can also train them using `JVM_OPTS="-Xmx8g" lein run` once you've modified
+the parameters to `train-convnet` (see above) in 
`src/cnn_text_classification/classifier.clj`.
+In order to run training with word2vec on the complete data set, you will need 
to run:
+```
+(train-convnet {:embedding-size 300 :batch-size 100 :test-size 1000 :num-epoch 
10 :pretrained-embedding :word2vec})
+```
+You should be able to achieve an accuracy of `~0.78` using the parameters 
above.
+
+## Usage with learned embeddings
+
+Lastly, similar to the python CNN text classification example, you can learn 
the embeddings based on training data.
+This can be achieved by setting `:pretrained-embedding nil` (or omitting that 
parameter altogether).
diff --git 
a/contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classificat

[incubator-mxnet] branch master updated: [Clojure] package infer tweaks (#13864)

[cmeier]

This is an automated email from the ASF dual-hosted git repository.

cmeier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new c2110ad  [Clojure] package infer tweaks (#13864)
c2110ad is described below

commit c2110ada6d43f10710d181c0deb0673fe6d829b2
Author: Carin Meier 
AuthorDate: Sat Jan 12 19:01:17 2019 -0500

[Clojure] package infer tweaks (#13864)

* change object detection prediction to be a map

* change predictions to a map for image-classifiers

* change return types of the classifiers to be a map
- add tests for base classifier and with-ndarray as well

* tweak return types and inputs for predict
- add test for plain predict

* updated infer-classify examples

* adjust the infer/object detections tests

* tweak predictor test

* Feedback from @kedarbellare review

* put scaling back in

* put back predict so it can handle multiple inputs

* restore original functions signatures (remove first)
---
 .../src/infer/imageclassifier_example.clj  |  19 ++-
 .../test/infer/imageclassifier_example_test.clj|  25 ++--
 .../src/infer/objectdetector_example.clj   |  25 ++--
 .../test/infer/objectdetector_example_test.clj |  24 ++--
 .../predictor/src/infer/predictor_example.clj  |   4 +-
 .../src/org/apache/clojure_mxnet/infer.clj | 137 -
 .../clojure_mxnet/infer/imageclassifier_test.clj   |  96 +++
 .../clojure_mxnet/infer/objectdetector_test.clj|  47 ---
 .../apache/clojure_mxnet/infer/predictor_test.clj  |  24 +++-
 9 files changed, 250 insertions(+), 151 deletions(-)

diff --git 
a/contrib/clojure-package/examples/infer/imageclassifier/src/infer/imageclassifier_example.clj
 
b/contrib/clojure-package/examples/infer/imageclassifier/src/infer/imageclassifier_example.clj
index 4ec7ff7..6994b4f 100644
--- 
a/contrib/clojure-package/examples/infer/imageclassifier/src/infer/imageclassifier_example.clj
+++ 
b/contrib/clojure-package/examples/infer/imageclassifier/src/infer/imageclassifier_example.clj
@@ -55,8 +55,8 @@
   "Print image classifier predictions for the given input file"
   [predictions]
   (println (apply str (repeat 80 "=")))
-  (doseq [[label probability] predictions]
-(println (format "Class: %s Probability=%.8f" label probability)))
+  (doseq [p predictions]
+(println p))
   (println (apply str (repeat 80 "="
 
 (defn classify-single-image
@@ -64,8 +64,8 @@
   [classifier input-image]
   (let [image (infer/load-image-from-file input-image)
 topk 5
-[predictions] (infer/classify-image classifier image topk)]
-predictions))
+predictions (infer/classify-image classifier image topk)]
+[predictions]))
 
 (defn classify-images-in-dir
   "Classify all jpg images in the directory"
@@ -78,12 +78,10 @@
 (filter #(re-matches #".*\.jpg$" (.getPath %)))
 (mapv #(.getPath %))
 (partition-all batch-size))]
-(apply
- concat
- (for [image-files image-file-batches]
-   (let [image-batch (infer/load-image-paths image-files)
- topk 5]
- (infer/classify-image-batch classifier image-batch topk))
+(apply concat (for [image-files image-file-batches]
+(let [image-batch (infer/load-image-paths image-files)
+  topk 5]
+  (infer/classify-image-batch classifier image-batch 
topk))
 
 (defn run-classifier
   "Runs an image classifier based on options provided"
@@ -98,6 +96,7 @@
 factory {:contexts [(context/default-context)]})]
 (println "Classifying a single image")
 (print-predictions (classify-single-image classifier input-image))
+(println "\n")
 (println "Classifying images in a directory")
 (doseq [predictions (classify-images-in-dir classifier input-dir)]
   (print-predictions predictions
diff --git 
a/contrib/clojure-package/examples/infer/imageclassifier/test/infer/imageclassifier_example_test.clj
 
b/contrib/clojure-package/examples/infer/imageclassifier/test/infer/imageclassifier_example_test.clj
index 5b3e08d..4b71f84 100644
--- 
a/contrib/clojure-package/examples/infer/imageclassifier/test/infer/imageclassifier_example_test.clj
+++ 
b/contrib/clojure-package/examples/infer/imageclassifier/test/infer/imageclassifier_example_test.clj
@@ -43,27 +43,16 @@
 
 (deftest test-single-classification
   (let [classifier (create-classifier)
-predictions (classify-single-image classifier image-file)]
+[[predictions]] (classify-single-image classifier image-file)]
 (is (some? predictions))
 (is (= 5 (co