[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package
URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197869647
##
File path:
contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/classifier.clj
##
@@ -0,0 +1,112 @@
+;;
+;; Licensed to the Apache Software Foundation (ASF) under one or more
+;; contributor license agreements. See the NOTICE file distributed with
+;; this work for additional information regarding copyright ownership.
+;; The ASF licenses this file to You under the Apache License, Version 2.0
+;; (the "License"); you may not use this file except in compliance with
+;; the License. You may obtain a copy of the License at
+;;
+;;http://www.apache.org/licenses/LICENSE-2.0
+;;
+;; Unless required by applicable law or agreed to in writing, software
+;; distributed under the License is distributed on an "AS IS" BASIS,
+;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+;; See the License for the specific language governing permissions and
+;; limitations under the License.
+;;
+
+(ns cnn-text-classification.classifier
+ (:require [cnn-text-classification.data-helper :as data-helper]
+[org.apache.clojure-mxnet.eval-metric :as eval-metric]
+[org.apache.clojure-mxnet.io :as mx-io]
+[org.apache.clojure-mxnet.module :as m]
+[org.apache.clojure-mxnet.ndarray :as ndarray]
+[org.apache.clojure-mxnet.optimizer :as optimizer]
+[org.apache.clojure-mxnet.symbol :as sym]
+[org.apache.clojure-mxnet.context :as context])
+ (:gen-class))
+
+(def mr-dataset-path "data/mr-data") ;; the MR polarity dataset path
+(def glove-file-path "data/glove/glove.6B.50d.txt")
+
+(defn shuffle-data [test-num {:keys [data label sentence-count sentence-size
embedding-size]}]
+ (println "Shuffling the data and splitting into training and test sets")
+ (println {:sentence-count sentence-count
+:sentence-size sentence-size
+:embedding-size embedding-size})
+ (let [shuffled (shuffle (map (fn [d l] [d l]) data label))
+train-num (- (count shuffled) test-num)
+training (into [] (take train-num shuffled))
+test (into [] (drop train-num shuffled))]
+{:training {:data (ndarray/array (into [] (flatten (mapv (fn [v] (first
v)) training)))
+ [train-num 1 sentence-size
embedding-size]) ;; has to be channel x y
+:label (ndarray/array (into [] (flatten (mapv (fn [v] (last v)
) training)))
+ [train-num])}
+ :test {:data (ndarray/array (into [] (flatten (mapv (fn [v] (first v))
test)))
+ [test-num 1 sentence-size embedding-size])
;; has to be channel x y
+:label (ndarray/array (into [] (flatten (mapv (fn [v] (last v) )
test)))
+ [test-num])}}))
+
+;;; convnet with multiple filter sizes
+;; from Convolutional Neural Networks for Sentence Classification by Yoon Kim
+(defn get-multi-filter-convnet [num-embed sentence-size batch-size]
+ (let [filter-list [3 4 5]
+num-filter 100
+num-label 2
+dropout 0.5
+input-x (sym/variable "data")
+polled-outputs (mapv (fn [filter-size]
+ (as-> (sym/convolution {:data input-x
Review comment:
I think the alignment is messed up here, I expect:
`
(as-> exp
name
form1
form2)
`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package
URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197870642
##
File path:
contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/classifier.clj
##
@@ -0,0 +1,112 @@
+;;
+;; Licensed to the Apache Software Foundation (ASF) under one or more
+;; contributor license agreements. See the NOTICE file distributed with
+;; this work for additional information regarding copyright ownership.
+;; The ASF licenses this file to You under the Apache License, Version 2.0
+;; (the "License"); you may not use this file except in compliance with
+;; the License. You may obtain a copy of the License at
+;;
+;;http://www.apache.org/licenses/LICENSE-2.0
+;;
+;; Unless required by applicable law or agreed to in writing, software
+;; distributed under the License is distributed on an "AS IS" BASIS,
+;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+;; See the License for the specific language governing permissions and
+;; limitations under the License.
+;;
+
+(ns cnn-text-classification.classifier
+ (:require [cnn-text-classification.data-helper :as data-helper]
+[org.apache.clojure-mxnet.eval-metric :as eval-metric]
+[org.apache.clojure-mxnet.io :as mx-io]
+[org.apache.clojure-mxnet.module :as m]
+[org.apache.clojure-mxnet.ndarray :as ndarray]
+[org.apache.clojure-mxnet.optimizer :as optimizer]
+[org.apache.clojure-mxnet.symbol :as sym]
+[org.apache.clojure-mxnet.context :as context])
+ (:gen-class))
+
+(def mr-dataset-path "data/mr-data") ;; the MR polarity dataset path
+(def glove-file-path "data/glove/glove.6B.50d.txt")
+
+(defn shuffle-data [test-num {:keys [data label sentence-count sentence-size
embedding-size]}]
+ (println "Shuffling the data and splitting into training and test sets")
+ (println {:sentence-count sentence-count
+:sentence-size sentence-size
+:embedding-size embedding-size})
+ (let [shuffled (shuffle (map (fn [d l] [d l]) data label))
+train-num (- (count shuffled) test-num)
+training (into [] (take train-num shuffled))
+test (into [] (drop train-num shuffled))]
+{:training {:data (ndarray/array (into [] (flatten (mapv (fn [v] (first
v)) training)))
+ [train-num 1 sentence-size
embedding-size]) ;; has to be channel x y
+:label (ndarray/array (into [] (flatten (mapv (fn [v] (last v)
) training)))
+ [train-num])}
+ :test {:data (ndarray/array (into [] (flatten (mapv (fn [v] (first v))
test)))
+ [test-num 1 sentence-size embedding-size])
;; has to be channel x y
+:label (ndarray/array (into [] (flatten (mapv (fn [v] (last v) )
test)))
+ [test-num])}}))
+
+;;; convnet with multiple filter sizes
+;; from Convolutional Neural Networks for Sentence Classification by Yoon Kim
+(defn get-multi-filter-convnet [num-embed sentence-size batch-size]
+ (let [filter-list [3 4 5]
+num-filter 100
+num-label 2
+dropout 0.5
+input-x (sym/variable "data")
+polled-outputs (mapv (fn [filter-size]
+ (as-> (sym/convolution {:data input-x
+ :kernel [filter-size
num-embed]
+ :num-filter
num-filter}) data
+ (sym/activation {:data data :act-type "relu"})
+ (sym/pooling {:data data
+ :pool-type "max"
Review comment:
Is it worth having separate const namespace?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package
URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197871096
##
File path:
contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/classifier.clj
##
@@ -0,0 +1,112 @@
+;;
+;; Licensed to the Apache Software Foundation (ASF) under one or more
+;; contributor license agreements. See the NOTICE file distributed with
+;; this work for additional information regarding copyright ownership.
+;; The ASF licenses this file to You under the Apache License, Version 2.0
+;; (the "License"); you may not use this file except in compliance with
+;; the License. You may obtain a copy of the License at
+;;
+;;http://www.apache.org/licenses/LICENSE-2.0
+;;
+;; Unless required by applicable law or agreed to in writing, software
+;; distributed under the License is distributed on an "AS IS" BASIS,
+;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+;; See the License for the specific language governing permissions and
+;; limitations under the License.
+;;
+
+(ns cnn-text-classification.classifier
+ (:require [cnn-text-classification.data-helper :as data-helper]
+[org.apache.clojure-mxnet.eval-metric :as eval-metric]
+[org.apache.clojure-mxnet.io :as mx-io]
+[org.apache.clojure-mxnet.module :as m]
+[org.apache.clojure-mxnet.ndarray :as ndarray]
+[org.apache.clojure-mxnet.optimizer :as optimizer]
+[org.apache.clojure-mxnet.symbol :as sym]
+[org.apache.clojure-mxnet.context :as context])
+ (:gen-class))
+
+(def mr-dataset-path "data/mr-data") ;; the MR polarity dataset path
+(def glove-file-path "data/glove/glove.6B.50d.txt")
+
+(defn shuffle-data [test-num {:keys [data label sentence-count sentence-size
embedding-size]}]
+ (println "Shuffling the data and splitting into training and test sets")
+ (println {:sentence-count sentence-count
+:sentence-size sentence-size
+:embedding-size embedding-size})
+ (let [shuffled (shuffle (map (fn [d l] [d l]) data label))
+train-num (- (count shuffled) test-num)
+training (into [] (take train-num shuffled))
+test (into [] (drop train-num shuffled))]
+{:training {:data (ndarray/array (into [] (flatten (mapv (fn [v] (first
v)) training)))
+ [train-num 1 sentence-size
embedding-size]) ;; has to be channel x y
+:label (ndarray/array (into [] (flatten (mapv (fn [v] (last v)
) training)))
+ [train-num])}
+ :test {:data (ndarray/array (into [] (flatten (mapv (fn [v] (first v))
test)))
+ [test-num 1 sentence-size embedding-size])
;; has to be channel x y
+:label (ndarray/array (into [] (flatten (mapv (fn [v] (last v) )
test)))
+ [test-num])}}))
+
+;;; convnet with multiple filter sizes
+;; from Convolutional Neural Networks for Sentence Classification by Yoon Kim
+(defn get-multi-filter-convnet [num-embed sentence-size batch-size]
+ (let [filter-list [3 4 5]
+num-filter 100
+num-label 2
+dropout 0.5
+input-x (sym/variable "data")
+polled-outputs (mapv (fn [filter-size]
+ (as-> (sym/convolution {:data input-x
+ :kernel [filter-size
num-embed]
+ :num-filter
num-filter}) data
+ (sym/activation {:data data :act-type "relu"})
+ (sym/pooling {:data data
+ :pool-type "max"
+ :kernel [(inc (- sentence-size
filter-size)) 1]
+ :stride [1 1]})))
+ filter-list)
+total-filters (* num-filter (count filter-list))
+concat (sym/concat "concat" nil polled-outputs {:dim 1})
+hpool (sym/reshape "hpool" {:data concat :target-shape [batch-size
total-filters]})
+hdrop (if (> dropout 0) (sym/dropout "hdrop" {:data hpool :p dropout})
hpool)
+fc (sym/fully-connected "fc1" {:data hdrop :num-hidden num-label})]
+(sym/softmax-output "softmax" {:data fc})))
+
+(defn train-convnet [{:keys [devs embedding-size batch-size test-size
num-epoch max-examples]}]
+ (let [glove (data-helper/load-glove glove-file-path) ;; you can also use
word2vec
+ms-dataset (data-helper/load-ms-with-embeddings mr-dataset-path
embedding-size glove max-examples)
+sentence-size (:sentence-size ms-dataset)
+shuffled (shuffle-data test-size ms-dataset)
+train-data (mx-io/ndarray-iter [(get-in shuffled [:training :data])]
+ {:label[(get-in
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197856261 ## File path: contrib/clojure-package/README.md ## @@ -0,0 +1,203 @@ +# Clojure MXNet + +A clojure package to the MXNet Deep Learning library + +## Introduction + +MXNet is a first class, modern deep learning library that AWS has officially picked as its chosen library. It supports multiple languages on a first class basis and is incubating as an Apache project. + +The motivation for creating a Clojure package is to be able to open the deep learning library to the Clojure ecosystem and build bridges for future development and innovation for the community. It provides all the needed tools including low level and high level apis, dynamic graphs, and things like GAN and natural language support. + +For high leverage, the Clojure package has been built on the existing Scala package using interop. This has allowed rapid development and close parity with the Scala functionality. This also leaves the door open to directly developing code against the jni-bindings with Clojure in the future in an incremental fashion, using the test suites as a refactoring guide. + +## Current State and Plans + +The Clojure package is nearing the end of its first development milestone which is to achieve a close parity with the Scala package and to potentially be included into the main project for official Clojure language support. + +What is needed now is alpha testing on both OSX and Linux to discover any bugs, rough edges, and generally harden it before an official PR is opened on the main project. + +Help with this effort is greatly appreciated and contributors will be recognized in the project README. + +Testing instructions can be found in the Testing.md + +## Getting Started + +The following systems are supported: + +- OSX cpu +- Linux cpu +- Linux gpu + +There are two ways of getting going. The first way is the easiest and that is to use the pre-built jars from Maven. The second way is to build from source. In both cases, you will need to load the prereqs and dependencies, (like opencv). + +It's been tested on AWS Deep Learning AMI and OSX High Sierra 10.13.4 + + +### Prerequisites + +**If you are using the AWS Deep Learning Ubuntu or Linux AMI you should be good to go without doing anything on this step.** + + +Follow the instructions from https://mxnet.incubator.apache.org/install/osx_setup.html or https://mxnet.incubator.apache.org/install/ubuntu_setup.html +about _Prepare Environment for GPU Installation_ +and _Install MXNet dependencies_ + + + Cloning the repo and running from source + +To use the prebuilt jars, you will need to replace the native version of the line in the project dependencies with your configuration. + +`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu "1.2.0"]` +or +`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu "1.2.0"]` +or +`[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.2.0"]` + + +```clojure + +(ns tutorial.ndarray + (:require [org.apache.clojure-mxnet.ndarray :as ndarray] +[org.apache.clojure-mxnet.context :as context])) + +;;Create NDArray +(def a (ndarray/zeros [100 50])) ;;all zero arrray of dimension 100 x 50 +(def b (ndarray/ones [256 32 128 1])) ;; all one array of dimension +(def c (ndarray/array [1 2 3 4 5 6] [2 3])) ;; array with contents of a shape 2 x 3 + +;;; There are also ways to convert to a vec or get the shape as an object or vec +(ndarray/->vec c) ;=> [1.0 2.0 3.0 4.0 5.0 6.0] +``` + +See the examples/tutorial section for more. + + +The jars from maven with the needed MXNet native binaries in it. On startup, the native libraries are extracted from the jar and copied into a temporary location on your path. On termination, they are deleted. + +If you want details on the flags (opencv verison and cuda version of the jars), they are documented here https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process + + +### Build from MXNET Source + +Checkout the latest sha from the main package + +`git clone --recursive https://github.com/dmlc/mxnet ~/mxnet` +`cd ~/mxnet` + + +`git checkout tags/1.2.0 -b release-1.2.0` + +`git submodule update --init --recursive` + +Sometimes it useful to use this script to clean hard +https://gist.github.com/nicktoumpelis/11214362 + + +Go here to do the base package installation https://mxnet.incubator.apache.org/install/index.html + + Run `make scalapkg` then `make scalainstall` + +then replace the correct jar for your architecture in the project.clj, example `[ml.dmlc.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.0.1-SNAPSHOT"]` + + Test your installation + +To test your installation, you should run `lein test`. This will run the test suite (CPU) for the clojure package. + + + Generation of NDArray and Symbol apis + +The bulk of the ndarray and symbol apis are generated via java reflection
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package
URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197857831
##
File path: contrib/clojure-package/examples/cnn-text-classification/README.md
##
@@ -0,0 +1,38 @@
+# cnn-text-classification
+
+An example of text classification using CNN
+
+To use you must download the MR polarity dataset and put it in the path
specified in the mr-dataset-path
+The dataset can be obtained here:
[https://github.com/yoonkim/CNN_sentence](https://github.com/yoonkim/CNN_sentence).
The two files `rt-polarity.neg`
+and `rt-polarity.pos` must be put in a directory. For example,
`data/mr-data/rt-polarity.neg`.
+
+You also must download the glove word embeddings. The suggested one to use is
the smaller 50 dimension one
+`glove.6B.50d.txt` which is contained in the download file here
[https://nlp.stanford.edu/projects/glove/](https://nlp.stanford.edu/projects/glove/)
+
+## Usage
+
+You can run through the repl with
+`(train-convnet {:embedding-size 50 :batch-size 100 :test-size 100 :num-epoch
10 :max-examples 1000})`
+
+or
+`JVM_OPTS="Xmx1g" lein run` (cpu)
+
+You can control the devices you run on by doing:
+
+`lein run :cpu 2` - This will run on 2 cpu devices
+`lein run :gpu 1` - This will run on 1 gpu device
+`lein run :gpu 2` - This will run on 2 gpu devices
+
+
+The max-examples only loads 1000 each of the dataset to keep the time and
memory down. To run all the examples,
+change the main to be (train-convnet {:embedding-size 50 :batch-size 100
:test-size 1000 :num-epoch 10)
+
+and then run
+
+- `lein uberjar`
+- `java -Xms1024m -Xmx2048m -jar
target/cnn-text-classification-0.1.0-SNAPSHOT-standalone.jar`
+
+
Review comment:
Extra lines here
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197856623 ## File path: contrib/clojure-package/README.md ## @@ -0,0 +1,203 @@ +# Clojure MXNet + +A clojure package to the MXNet Deep Learning library + +## Introduction + +MXNet is a first class, modern deep learning library that AWS has officially picked as its chosen library. It supports multiple languages on a first class basis and is incubating as an Apache project. + +The motivation for creating a Clojure package is to be able to open the deep learning library to the Clojure ecosystem and build bridges for future development and innovation for the community. It provides all the needed tools including low level and high level apis, dynamic graphs, and things like GAN and natural language support. + +For high leverage, the Clojure package has been built on the existing Scala package using interop. This has allowed rapid development and close parity with the Scala functionality. This also leaves the door open to directly developing code against the jni-bindings with Clojure in the future in an incremental fashion, using the test suites as a refactoring guide. + +## Current State and Plans + +The Clojure package is nearing the end of its first development milestone which is to achieve a close parity with the Scala package and to potentially be included into the main project for official Clojure language support. + +What is needed now is alpha testing on both OSX and Linux to discover any bugs, rough edges, and generally harden it before an official PR is opened on the main project. + +Help with this effort is greatly appreciated and contributors will be recognized in the project README. + +Testing instructions can be found in the Testing.md + +## Getting Started + +The following systems are supported: + +- OSX cpu +- Linux cpu +- Linux gpu + +There are two ways of getting going. The first way is the easiest and that is to use the pre-built jars from Maven. The second way is to build from source. In both cases, you will need to load the prereqs and dependencies, (like opencv). + +It's been tested on AWS Deep Learning AMI and OSX High Sierra 10.13.4 + + +### Prerequisites + +**If you are using the AWS Deep Learning Ubuntu or Linux AMI you should be good to go without doing anything on this step.** + + +Follow the instructions from https://mxnet.incubator.apache.org/install/osx_setup.html or https://mxnet.incubator.apache.org/install/ubuntu_setup.html +about _Prepare Environment for GPU Installation_ +and _Install MXNet dependencies_ + + + Cloning the repo and running from source + +To use the prebuilt jars, you will need to replace the native version of the line in the project dependencies with your configuration. + +`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu "1.2.0"]` +or +`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu "1.2.0"]` +or +`[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.2.0"]` + + +```clojure + +(ns tutorial.ndarray + (:require [org.apache.clojure-mxnet.ndarray :as ndarray] +[org.apache.clojure-mxnet.context :as context])) + +;;Create NDArray +(def a (ndarray/zeros [100 50])) ;;all zero arrray of dimension 100 x 50 +(def b (ndarray/ones [256 32 128 1])) ;; all one array of dimension +(def c (ndarray/array [1 2 3 4 5 6] [2 3])) ;; array with contents of a shape 2 x 3 + +;;; There are also ways to convert to a vec or get the shape as an object or vec +(ndarray/->vec c) ;=> [1.0 2.0 3.0 4.0 5.0 6.0] +``` + +See the examples/tutorial section for more. + + +The jars from maven with the needed MXNet native binaries in it. On startup, the native libraries are extracted from the jar and copied into a temporary location on your path. On termination, they are deleted. + +If you want details on the flags (opencv verison and cuda version of the jars), they are documented here https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process + + +### Build from MXNET Source + +Checkout the latest sha from the main package + +`git clone --recursive https://github.com/dmlc/mxnet ~/mxnet` +`cd ~/mxnet` + + +`git checkout tags/1.2.0 -b release-1.2.0` + +`git submodule update --init --recursive` + +Sometimes it useful to use this script to clean hard +https://gist.github.com/nicktoumpelis/11214362 + + +Go here to do the base package installation https://mxnet.incubator.apache.org/install/index.html + + Run `make scalapkg` then `make scalainstall` + +then replace the correct jar for your architecture in the project.clj, example `[ml.dmlc.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.0.1-SNAPSHOT"]` + + Test your installation + +To test your installation, you should run `lein test`. This will run the test suite (CPU) for the clojure package. + + + Generation of NDArray and Symbol apis + +The bulk of the ndarray and symbol apis are generated via java reflection
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197852862 ## File path: .gitignore ## @@ -170,3 +170,5 @@ tests/mxnet_unit_tests # generated wrappers for ccache cc cxx +contrib/clojure-package/test/test-ndarray.clj +contrib/clojure-package/test/test-symbol.clj Review comment: newline? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r194591348 ## File path: contrib/clojure-package/scripts/get_cifar_data.sh ## @@ -0,0 +1,38 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + + +set -e Review comment: I'd add -vx options to bash scripts in addition to -e This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package
URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197870886
##
File path:
contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/classifier.clj
##
@@ -0,0 +1,112 @@
+;;
+;; Licensed to the Apache Software Foundation (ASF) under one or more
+;; contributor license agreements. See the NOTICE file distributed with
+;; this work for additional information regarding copyright ownership.
+;; The ASF licenses this file to You under the Apache License, Version 2.0
+;; (the "License"); you may not use this file except in compliance with
+;; the License. You may obtain a copy of the License at
+;;
+;;http://www.apache.org/licenses/LICENSE-2.0
+;;
+;; Unless required by applicable law or agreed to in writing, software
+;; distributed under the License is distributed on an "AS IS" BASIS,
+;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+;; See the License for the specific language governing permissions and
+;; limitations under the License.
+;;
+
+(ns cnn-text-classification.classifier
+ (:require [cnn-text-classification.data-helper :as data-helper]
+[org.apache.clojure-mxnet.eval-metric :as eval-metric]
+[org.apache.clojure-mxnet.io :as mx-io]
+[org.apache.clojure-mxnet.module :as m]
+[org.apache.clojure-mxnet.ndarray :as ndarray]
+[org.apache.clojure-mxnet.optimizer :as optimizer]
+[org.apache.clojure-mxnet.symbol :as sym]
+[org.apache.clojure-mxnet.context :as context])
+ (:gen-class))
+
+(def mr-dataset-path "data/mr-data") ;; the MR polarity dataset path
+(def glove-file-path "data/glove/glove.6B.50d.txt")
+
+(defn shuffle-data [test-num {:keys [data label sentence-count sentence-size
embedding-size]}]
+ (println "Shuffling the data and splitting into training and test sets")
+ (println {:sentence-count sentence-count
+:sentence-size sentence-size
+:embedding-size embedding-size})
+ (let [shuffled (shuffle (map (fn [d l] [d l]) data label))
+train-num (- (count shuffled) test-num)
+training (into [] (take train-num shuffled))
+test (into [] (drop train-num shuffled))]
+{:training {:data (ndarray/array (into [] (flatten (mapv (fn [v] (first
v)) training)))
+ [train-num 1 sentence-size
embedding-size]) ;; has to be channel x y
+:label (ndarray/array (into [] (flatten (mapv (fn [v] (last v)
) training)))
+ [train-num])}
+ :test {:data (ndarray/array (into [] (flatten (mapv (fn [v] (first v))
test)))
+ [test-num 1 sentence-size embedding-size])
;; has to be channel x y
+:label (ndarray/array (into [] (flatten (mapv (fn [v] (last v) )
test)))
+ [test-num])}}))
+
+;;; convnet with multiple filter sizes
+;; from Convolutional Neural Networks for Sentence Classification by Yoon Kim
+(defn get-multi-filter-convnet [num-embed sentence-size batch-size]
+ (let [filter-list [3 4 5]
+num-filter 100
+num-label 2
+dropout 0.5
+input-x (sym/variable "data")
+polled-outputs (mapv (fn [filter-size]
+ (as-> (sym/convolution {:data input-x
+ :kernel [filter-size
num-embed]
+ :num-filter
num-filter}) data
+ (sym/activation {:data data :act-type "relu"})
+ (sym/pooling {:data data
+ :pool-type "max"
+ :kernel [(inc (- sentence-size
filter-size)) 1]
+ :stride [1 1]})))
+ filter-list)
+total-filters (* num-filter (count filter-list))
+concat (sym/concat "concat" nil polled-outputs {:dim 1})
+hpool (sym/reshape "hpool" {:data concat :target-shape [batch-size
total-filters]})
+hdrop (if (> dropout 0) (sym/dropout "hdrop" {:data hpool :p dropout})
hpool)
Review comment:
`(> dropout 0)` -> `(pos? dropout)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197853479 ## File path: contrib/clojure-package/Testing.md ## @@ -0,0 +1,23 @@ +## Help with Testing Review comment: nitpick: can the filename be all lowercase or all uppercase? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package
URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197869011
##
File path:
contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/classifier.clj
##
@@ -0,0 +1,112 @@
+;;
+;; Licensed to the Apache Software Foundation (ASF) under one or more
+;; contributor license agreements. See the NOTICE file distributed with
+;; this work for additional information regarding copyright ownership.
+;; The ASF licenses this file to You under the Apache License, Version 2.0
+;; (the "License"); you may not use this file except in compliance with
+;; the License. You may obtain a copy of the License at
+;;
+;;http://www.apache.org/licenses/LICENSE-2.0
+;;
+;; Unless required by applicable law or agreed to in writing, software
+;; distributed under the License is distributed on an "AS IS" BASIS,
+;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+;; See the License for the specific language governing permissions and
+;; limitations under the License.
+;;
+
+(ns cnn-text-classification.classifier
+ (:require [cnn-text-classification.data-helper :as data-helper]
+[org.apache.clojure-mxnet.eval-metric :as eval-metric]
+[org.apache.clojure-mxnet.io :as mx-io]
+[org.apache.clojure-mxnet.module :as m]
+[org.apache.clojure-mxnet.ndarray :as ndarray]
+[org.apache.clojure-mxnet.optimizer :as optimizer]
+[org.apache.clojure-mxnet.symbol :as sym]
+[org.apache.clojure-mxnet.context :as context])
+ (:gen-class))
+
+(def mr-dataset-path "data/mr-data") ;; the MR polarity dataset path
+(def glove-file-path "data/glove/glove.6B.50d.txt")
+
+(defn shuffle-data [test-num {:keys [data label sentence-count sentence-size
embedding-size]}]
+ (println "Shuffling the data and splitting into training and test sets")
+ (println {:sentence-count sentence-count
+:sentence-size sentence-size
+:embedding-size embedding-size})
+ (let [shuffled (shuffle (map (fn [d l] [d l]) data label))
+train-num (- (count shuffled) test-num)
+training (into [] (take train-num shuffled))
+test (into [] (drop train-num shuffled))]
+{:training {:data (ndarray/array (into [] (flatten (mapv (fn [v] (first
v)) training)))
+ [train-num 1 sentence-size
embedding-size]) ;; has to be channel x y
+:label (ndarray/array (into [] (flatten (mapv (fn [v] (last v)
) training)))
+ [train-num])}
+ :test {:data (ndarray/array (into [] (flatten (mapv (fn [v] (first v))
test)))
+ [test-num 1 sentence-size embedding-size])
;; has to be channel x y
+:label (ndarray/array (into [] (flatten (mapv (fn [v] (last v) )
test)))
+ [test-num])}}))
+
+;;; convnet with multiple filter sizes
+;; from Convolutional Neural Networks for Sentence Classification by Yoon Kim
+(defn get-multi-filter-convnet [num-embed sentence-size batch-size]
+ (let [filter-list [3 4 5]
+num-filter 100
+num-label 2
+dropout 0.5
+input-x (sym/variable "data")
+polled-outputs (mapv (fn [filter-size]
Review comment:
Can we extract named fn that we use in mapv?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r194590775 ## File path: contrib/clojure-package/.gitignore ## @@ -0,0 +1,40 @@ +/target +/classes +/checkouts +pom.xml +pom.xml.asc +*.jar +*.class +/.lein-* +/.nrepl-port +.hgignore +.hg/ +data/* +model/* +*~ +*.params +*.states +*.json +examples/module/data/* +examples/module/target/* +examples/rnn/data/char_lstm.zip +examples/rnn/data/obama.txt +examples/pre-trained-models/caltech-256/caltech-256-60-train.rec +examples/pre-trained-models/caltech-256/caltech-256-60-val.rec +examples/pre-trained-models/model/synset.txt +examples/pre-trained-models/test-image.jpg +examples/imclassification/data/* +examples/gan/data/* +examples/gan/results/* +examples/cnn-text-classification/data/glove/* +examples/cnn-text-classification/data/mr-data/* +examples/multi-label/data/mnist.zip +examples/multi-label/data/t10k-images-idx3-ubyte +examples/multi-label/data/t10k-labels-idx1-ubyte +examples/multi-label/data/train-images-idx3-ubyte +examples/multi-label/data/train-labels-idx1-ubyte +examples/visualization/test-vis/* +examples/visualization/test-vis.pdf +.DS_Store +src/.DS_Store +src/org/.DS_Store Review comment: Newline? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package
URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197868019
##
File path:
contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/classifier.clj
##
@@ -0,0 +1,112 @@
+;;
+;; Licensed to the Apache Software Foundation (ASF) under one or more
+;; contributor license agreements. See the NOTICE file distributed with
+;; this work for additional information regarding copyright ownership.
+;; The ASF licenses this file to You under the Apache License, Version 2.0
+;; (the "License"); you may not use this file except in compliance with
+;; the License. You may obtain a copy of the License at
+;;
+;;http://www.apache.org/licenses/LICENSE-2.0
+;;
+;; Unless required by applicable law or agreed to in writing, software
+;; distributed under the License is distributed on an "AS IS" BASIS,
+;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+;; See the License for the specific language governing permissions and
+;; limitations under the License.
+;;
+
+(ns cnn-text-classification.classifier
+ (:require [cnn-text-classification.data-helper :as data-helper]
+[org.apache.clojure-mxnet.eval-metric :as eval-metric]
+[org.apache.clojure-mxnet.io :as mx-io]
+[org.apache.clojure-mxnet.module :as m]
+[org.apache.clojure-mxnet.ndarray :as ndarray]
+[org.apache.clojure-mxnet.optimizer :as optimizer]
+[org.apache.clojure-mxnet.symbol :as sym]
+[org.apache.clojure-mxnet.context :as context])
+ (:gen-class))
+
+(def mr-dataset-path "data/mr-data") ;; the MR polarity dataset path
+(def glove-file-path "data/glove/glove.6B.50d.txt")
+
+(defn shuffle-data [test-num {:keys [data label sentence-count sentence-size
embedding-size]}]
+ (println "Shuffling the data and splitting into training and test sets")
+ (println {:sentence-count sentence-count
+:sentence-size sentence-size
+:embedding-size embedding-size})
+ (let [shuffled (shuffle (map (fn [d l] [d l]) data label))
+train-num (- (count shuffled) test-num)
+training (into [] (take train-num shuffled))
+test (into [] (drop train-num shuffled))]
+{:training {:data (ndarray/array (into [] (flatten (mapv (fn [v] (first
v)) training)))
Review comment:
You don't need fn here, `(mapv first training)` would be fine.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197854812 ## File path: contrib/clojure-package/README.md ## @@ -0,0 +1,203 @@ +# Clojure MXNet + +A clojure package to the MXNet Deep Learning library + +## Introduction + +MXNet is a first class, modern deep learning library that AWS has officially picked as its chosen library. It supports multiple languages on a first class basis and is incubating as an Apache project. + +The motivation for creating a Clojure package is to be able to open the deep learning library to the Clojure ecosystem and build bridges for future development and innovation for the community. It provides all the needed tools including low level and high level apis, dynamic graphs, and things like GAN and natural language support. + +For high leverage, the Clojure package has been built on the existing Scala package using interop. This has allowed rapid development and close parity with the Scala functionality. This also leaves the door open to directly developing code against the jni-bindings with Clojure in the future in an incremental fashion, using the test suites as a refactoring guide. + +## Current State and Plans + +The Clojure package is nearing the end of its first development milestone which is to achieve a close parity with the Scala package and to potentially be included into the main project for official Clojure language support. + +What is needed now is alpha testing on both OSX and Linux to discover any bugs, rough edges, and generally harden it before an official PR is opened on the main project. + +Help with this effort is greatly appreciated and contributors will be recognized in the project README. + +Testing instructions can be found in the Testing.md + +## Getting Started + +The following systems are supported: + +- OSX cpu +- Linux cpu +- Linux gpu + +There are two ways of getting going. The first way is the easiest and that is to use the pre-built jars from Maven. The second way is to build from source. In both cases, you will need to load the prereqs and dependencies, (like opencv). + +It's been tested on AWS Deep Learning AMI and OSX High Sierra 10.13.4 + + +### Prerequisites + +**If you are using the AWS Deep Learning Ubuntu or Linux AMI you should be good to go without doing anything on this step.** + + +Follow the instructions from https://mxnet.incubator.apache.org/install/osx_setup.html or https://mxnet.incubator.apache.org/install/ubuntu_setup.html +about _Prepare Environment for GPU Installation_ +and _Install MXNet dependencies_ + + + Cloning the repo and running from source + +To use the prebuilt jars, you will need to replace the native version of the line in the project dependencies with your configuration. + +`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu "1.2.0"]` +or +`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu "1.2.0"]` +or +`[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.2.0"]` + + +```clojure + +(ns tutorial.ndarray + (:require [org.apache.clojure-mxnet.ndarray :as ndarray] +[org.apache.clojure-mxnet.context :as context])) + +;;Create NDArray +(def a (ndarray/zeros [100 50])) ;;all zero arrray of dimension 100 x 50 +(def b (ndarray/ones [256 32 128 1])) ;; all one array of dimension +(def c (ndarray/array [1 2 3 4 5 6] [2 3])) ;; array with contents of a shape 2 x 3 + +;;; There are also ways to convert to a vec or get the shape as an object or vec +(ndarray/->vec c) ;=> [1.0 2.0 3.0 4.0 5.0 6.0] +``` + +See the examples/tutorial section for more. + + +The jars from maven with the needed MXNet native binaries in it. On startup, the native libraries are extracted from the jar and copied into a temporary location on your path. On termination, they are deleted. + +If you want details on the flags (opencv verison and cuda version of the jars), they are documented here https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process + + +### Build from MXNET Source + +Checkout the latest sha from the main package + +`git clone --recursive https://github.com/dmlc/mxnet ~/mxnet` +`cd ~/mxnet` + + +`git checkout tags/1.2.0 -b release-1.2.0` + +`git submodule update --init --recursive` + +Sometimes it useful to use this script to clean hard +https://gist.github.com/nicktoumpelis/11214362 + + +Go here to do the base package installation https://mxnet.incubator.apache.org/install/index.html + + Run `make scalapkg` then `make scalainstall` + +then replace the correct jar for your architecture in the project.clj, example `[ml.dmlc.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.0.1-SNAPSHOT"]` + + Test your installation + +To test your installation, you should run `lein test`. This will run the test suite (CPU) for the clojure package. + + + Generation of NDArray and Symbol apis + +The bulk of the ndarray and symbol apis are generated via java reflection
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package
URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197866693
##
File path:
contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/classifier.clj
##
@@ -0,0 +1,112 @@
+;;
+;; Licensed to the Apache Software Foundation (ASF) under one or more
+;; contributor license agreements. See the NOTICE file distributed with
+;; this work for additional information regarding copyright ownership.
+;; The ASF licenses this file to You under the Apache License, Version 2.0
+;; (the "License"); you may not use this file except in compliance with
+;; the License. You may obtain a copy of the License at
+;;
+;;http://www.apache.org/licenses/LICENSE-2.0
+;;
+;; Unless required by applicable law or agreed to in writing, software
+;; distributed under the License is distributed on an "AS IS" BASIS,
+;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+;; See the License for the specific language governing permissions and
+;; limitations under the License.
+;;
+
+(ns cnn-text-classification.classifier
+ (:require [cnn-text-classification.data-helper :as data-helper]
+[org.apache.clojure-mxnet.eval-metric :as eval-metric]
+[org.apache.clojure-mxnet.io :as mx-io]
+[org.apache.clojure-mxnet.module :as m]
+[org.apache.clojure-mxnet.ndarray :as ndarray]
+[org.apache.clojure-mxnet.optimizer :as optimizer]
+[org.apache.clojure-mxnet.symbol :as sym]
+[org.apache.clojure-mxnet.context :as context])
+ (:gen-class))
+
+(def mr-dataset-path "data/mr-data") ;; the MR polarity dataset path
+(def glove-file-path "data/glove/glove.6B.50d.txt")
+
+(defn shuffle-data [test-num {:keys [data label sentence-count sentence-size
embedding-size]}]
+ (println "Shuffling the data and splitting into training and test sets")
+ (println {:sentence-count sentence-count
+:sentence-size sentence-size
+:embedding-size embedding-size})
+ (let [shuffled (shuffle (map (fn [d l] [d l]) data label))
+train-num (- (count shuffled) test-num)
+training (into [] (take train-num shuffled))
Review comment:
Might be worth having specific coding convention. I would suggest
https://github.com/bbatsov/clojure-style-guide
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197855350 ## File path: contrib/clojure-package/README.md ## @@ -0,0 +1,203 @@ +# Clojure MXNet + +A clojure package to the MXNet Deep Learning library + +## Introduction + +MXNet is a first class, modern deep learning library that AWS has officially picked as its chosen library. It supports multiple languages on a first class basis and is incubating as an Apache project. + +The motivation for creating a Clojure package is to be able to open the deep learning library to the Clojure ecosystem and build bridges for future development and innovation for the community. It provides all the needed tools including low level and high level apis, dynamic graphs, and things like GAN and natural language support. + +For high leverage, the Clojure package has been built on the existing Scala package using interop. This has allowed rapid development and close parity with the Scala functionality. This also leaves the door open to directly developing code against the jni-bindings with Clojure in the future in an incremental fashion, using the test suites as a refactoring guide. + +## Current State and Plans + +The Clojure package is nearing the end of its first development milestone which is to achieve a close parity with the Scala package and to potentially be included into the main project for official Clojure language support. + +What is needed now is alpha testing on both OSX and Linux to discover any bugs, rough edges, and generally harden it before an official PR is opened on the main project. + +Help with this effort is greatly appreciated and contributors will be recognized in the project README. + +Testing instructions can be found in the Testing.md + +## Getting Started + +The following systems are supported: + +- OSX cpu +- Linux cpu +- Linux gpu + +There are two ways of getting going. The first way is the easiest and that is to use the pre-built jars from Maven. The second way is to build from source. In both cases, you will need to load the prereqs and dependencies, (like opencv). + +It's been tested on AWS Deep Learning AMI and OSX High Sierra 10.13.4 + + +### Prerequisites + +**If you are using the AWS Deep Learning Ubuntu or Linux AMI you should be good to go without doing anything on this step.** + + +Follow the instructions from https://mxnet.incubator.apache.org/install/osx_setup.html or https://mxnet.incubator.apache.org/install/ubuntu_setup.html +about _Prepare Environment for GPU Installation_ +and _Install MXNet dependencies_ + + + Cloning the repo and running from source + +To use the prebuilt jars, you will need to replace the native version of the line in the project dependencies with your configuration. + +`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu "1.2.0"]` +or +`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu "1.2.0"]` +or +`[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.2.0"]` + + +```clojure + +(ns tutorial.ndarray + (:require [org.apache.clojure-mxnet.ndarray :as ndarray] +[org.apache.clojure-mxnet.context :as context])) + +;;Create NDArray +(def a (ndarray/zeros [100 50])) ;;all zero arrray of dimension 100 x 50 +(def b (ndarray/ones [256 32 128 1])) ;; all one array of dimension +(def c (ndarray/array [1 2 3 4 5 6] [2 3])) ;; array with contents of a shape 2 x 3 + +;;; There are also ways to convert to a vec or get the shape as an object or vec +(ndarray/->vec c) ;=> [1.0 2.0 3.0 4.0 5.0 6.0] +``` + +See the examples/tutorial section for more. + + +The jars from maven with the needed MXNet native binaries in it. On startup, the native libraries are extracted from the jar and copied into a temporary location on your path. On termination, they are deleted. + +If you want details on the flags (opencv verison and cuda version of the jars), they are documented here https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process + + +### Build from MXNET Source + +Checkout the latest sha from the main package + +`git clone --recursive https://github.com/dmlc/mxnet ~/mxnet` +`cd ~/mxnet` + + +`git checkout tags/1.2.0 -b release-1.2.0` + +`git submodule update --init --recursive` + +Sometimes it useful to use this script to clean hard +https://gist.github.com/nicktoumpelis/11214362 + + +Go here to do the base package installation https://mxnet.incubator.apache.org/install/index.html + + Run `make scalapkg` then `make scalainstall` + +then replace the correct jar for your architecture in the project.clj, example `[ml.dmlc.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.0.1-SNAPSHOT"]` + + Test your installation + +To test your installation, you should run `lein test`. This will run the test suite (CPU) for the clojure package. + + + Generation of NDArray and Symbol apis + +The bulk of the ndarray and symbol apis are generated via java reflection
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197854610 ## File path: contrib/clojure-package/README.md ## @@ -0,0 +1,203 @@ +# Clojure MXNet + +A clojure package to the MXNet Deep Learning library + +## Introduction + +MXNet is a first class, modern deep learning library that AWS has officially picked as its chosen library. It supports multiple languages on a first class basis and is incubating as an Apache project. + +The motivation for creating a Clojure package is to be able to open the deep learning library to the Clojure ecosystem and build bridges for future development and innovation for the community. It provides all the needed tools including low level and high level apis, dynamic graphs, and things like GAN and natural language support. + +For high leverage, the Clojure package has been built on the existing Scala package using interop. This has allowed rapid development and close parity with the Scala functionality. This also leaves the door open to directly developing code against the jni-bindings with Clojure in the future in an incremental fashion, using the test suites as a refactoring guide. + +## Current State and Plans + +The Clojure package is nearing the end of its first development milestone which is to achieve a close parity with the Scala package and to potentially be included into the main project for official Clojure language support. + +What is needed now is alpha testing on both OSX and Linux to discover any bugs, rough edges, and generally harden it before an official PR is opened on the main project. + +Help with this effort is greatly appreciated and contributors will be recognized in the project README. + +Testing instructions can be found in the Testing.md + +## Getting Started + +The following systems are supported: + +- OSX cpu +- Linux cpu +- Linux gpu + +There are two ways of getting going. The first way is the easiest and that is to use the pre-built jars from Maven. The second way is to build from source. In both cases, you will need to load the prereqs and dependencies, (like opencv). + +It's been tested on AWS Deep Learning AMI and OSX High Sierra 10.13.4 + + +### Prerequisites + +**If you are using the AWS Deep Learning Ubuntu or Linux AMI you should be good to go without doing anything on this step.** + + +Follow the instructions from https://mxnet.incubator.apache.org/install/osx_setup.html or https://mxnet.incubator.apache.org/install/ubuntu_setup.html +about _Prepare Environment for GPU Installation_ +and _Install MXNet dependencies_ + + + Cloning the repo and running from source + +To use the prebuilt jars, you will need to replace the native version of the line in the project dependencies with your configuration. + +`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu "1.2.0"]` +or +`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu "1.2.0"]` +or +`[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.2.0"]` + + +```clojure + +(ns tutorial.ndarray + (:require [org.apache.clojure-mxnet.ndarray :as ndarray] +[org.apache.clojure-mxnet.context :as context])) + +;;Create NDArray +(def a (ndarray/zeros [100 50])) ;;all zero arrray of dimension 100 x 50 +(def b (ndarray/ones [256 32 128 1])) ;; all one array of dimension +(def c (ndarray/array [1 2 3 4 5 6] [2 3])) ;; array with contents of a shape 2 x 3 + +;;; There are also ways to convert to a vec or get the shape as an object or vec +(ndarray/->vec c) ;=> [1.0 2.0 3.0 4.0 5.0 6.0] +``` + +See the examples/tutorial section for more. + + +The jars from maven with the needed MXNet native binaries in it. On startup, the native libraries are extracted from the jar and copied into a temporary location on your path. On termination, they are deleted. + +If you want details on the flags (opencv verison and cuda version of the jars), they are documented here https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process + + +### Build from MXNET Source + +Checkout the latest sha from the main package + +`git clone --recursive https://github.com/dmlc/mxnet ~/mxnet` +`cd ~/mxnet` + + +`git checkout tags/1.2.0 -b release-1.2.0` + +`git submodule update --init --recursive` + +Sometimes it useful to use this script to clean hard +https://gist.github.com/nicktoumpelis/11214362 + + +Go here to do the base package installation https://mxnet.incubator.apache.org/install/index.html + + Run `make scalapkg` then `make scalainstall` + +then replace the correct jar for your architecture in the project.clj, example `[ml.dmlc.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.0.1-SNAPSHOT"]` + + Test your installation + +To test your installation, you should run `lein test`. This will run the test suite (CPU) for the clojure package. + + + Generation of NDArray and Symbol apis + +The bulk of the ndarray and symbol apis are generated via java reflection
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package
URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197868359
##
File path:
contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/classifier.clj
##
@@ -0,0 +1,112 @@
+;;
+;; Licensed to the Apache Software Foundation (ASF) under one or more
+;; contributor license agreements. See the NOTICE file distributed with
+;; this work for additional information regarding copyright ownership.
+;; The ASF licenses this file to You under the Apache License, Version 2.0
+;; (the "License"); you may not use this file except in compliance with
+;; the License. You may obtain a copy of the License at
+;;
+;;http://www.apache.org/licenses/LICENSE-2.0
+;;
+;; Unless required by applicable law or agreed to in writing, software
+;; distributed under the License is distributed on an "AS IS" BASIS,
+;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+;; See the License for the specific language governing permissions and
+;; limitations under the License.
+;;
+
+(ns cnn-text-classification.classifier
+ (:require [cnn-text-classification.data-helper :as data-helper]
+[org.apache.clojure-mxnet.eval-metric :as eval-metric]
+[org.apache.clojure-mxnet.io :as mx-io]
+[org.apache.clojure-mxnet.module :as m]
+[org.apache.clojure-mxnet.ndarray :as ndarray]
+[org.apache.clojure-mxnet.optimizer :as optimizer]
+[org.apache.clojure-mxnet.symbol :as sym]
+[org.apache.clojure-mxnet.context :as context])
+ (:gen-class))
+
+(def mr-dataset-path "data/mr-data") ;; the MR polarity dataset path
+(def glove-file-path "data/glove/glove.6B.50d.txt")
+
+(defn shuffle-data [test-num {:keys [data label sentence-count sentence-size
embedding-size]}]
+ (println "Shuffling the data and splitting into training and test sets")
+ (println {:sentence-count sentence-count
+:sentence-size sentence-size
+:embedding-size embedding-size})
+ (let [shuffled (shuffle (map (fn [d l] [d l]) data label))
+train-num (- (count shuffled) test-num)
+training (into [] (take train-num shuffled))
+test (into [] (drop train-num shuffled))]
+{:training {:data (ndarray/array (into [] (flatten (mapv (fn [v] (first
v)) training)))
Review comment:
And I think we should be generally clear on coding preferences whether `#()`
form is preferable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r194593837 ## File path: contrib/clojure-package/src/dev/generator.clj ## @@ -0,0 +1,329 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;;http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns dev.generator + (:require [t6.from-scala.core :as $] Review comment: Dollar sign is overloaded here, can we use `scala` instead. Alternatively :refer decode-scala-symbol and drop the namespace in usages. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kurman commented on a change in pull request #11205: Clojure Contrib Package
kurman commented on a change in pull request #11205: Clojure Contrib Package URL: https://github.com/apache/incubator-mxnet/pull/11205#discussion_r197857545 ## File path: contrib/clojure-package/examples/cnn-text-classification/README.md ## @@ -0,0 +1,38 @@ +# cnn-text-classification + +An example of text classification using CNN + +To use you must download the MR polarity dataset and put it in the path specified in the mr-dataset-path +The dataset can be obtained here: [https://github.com/yoonkim/CNN_sentence](https://github.com/yoonkim/CNN_sentence). The two files `rt-polarity.neg` +and `rt-polarity.pos` must be put in a directory. For example, `data/mr-data/rt-polarity.neg`. + +You also must download the glove word embeddings. The suggested one to use is the smaller 50 dimension one Review comment: Curious, why Glove over Word2vec? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
