Author: pat
Date: Tue Apr 21 00:16:09 2015
New Revision: 1675008
URL: http://svn.apache.org/r1675008
Log:
CMS commit to mahout by pat
Modified:
mahout/site/mahout_cms/trunk/content/users/environment/how-to-build-an-app.mdtext
Modified:
mahout/site/mahout_cms/trunk/content/users/environment/how-to-build-an-app.mdtext
URL:
http://svn.apache.org/viewvc/mahout/site/mahout_cms/trunk/content/users/environment/how-to-build-an-app.mdtext?rev=1675008&r1=1675007&r2=1675008&view=diff
==============================================================================
---
mahout/site/mahout_cms/trunk/content/users/environment/how-to-build-an-app.mdtext
(original)
+++
mahout/site/mahout_cms/trunk/content/users/environment/how-to-build-an-app.mdtext
Tue Apr 21 00:16:09 2015
@@ -1,47 +1,168 @@
-#Multiple Indicator Creation
+#How to create and App using Mahout
-This is an example of how to create more that two indictors from more than two
user interaction types with Mahout. We will use very simple hand created
example data for one might see in an ecommerce application. The application
records three interactions for item-purchase, item-detail-view, and
category-preference (search for or click on a category).
+This is an example of how to create a simple app using Mahout as a Library.
The source is available on Github in the [3-input-cooc
project](https://github.com/pferrel/3-input-cooc) with more explanation about
what it does. For this tutorial we'll concentrate on how to create an app.
-*spark-itemsimilarity* will handle two inputs but here we have three and
rather than running *spark-itemsimilarity* twice we will create our own app to
do it.
+This example is for reading three interactions types and creating indicators
for them using cooccurrence and cross-cooccurrence. The indicators will be
written to text files in a format ready for search engine indexing in
recommender.
##Setup
In order to build and run the CooccurrenceDriver you need to install the
following:
* Install the Java 7 JDK from Oracle. Mac users look here: [Java SE
Development Kit
7u72](http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html).
-* Install sbt (simple build tool) 0.13.x for <a
href="Installing-sbt-on-Mac.html">Mac</a>, <a
href="Installing-sbt-on-Windows.html">Windows</a>,
-<a href="Installing-sbt-on-Linux.html">Linux</a>, or
-<a href="Manual-Installation.html">manual installation</a>.
+* Install sbt (simple build tool) 0.13.x for
[Mac](http://www.scala-sbt.org/release/tutorial/Installing-sbt-on-Mac.html),[Linux](http://www.scala-sbt.org/release/tutorial/Installing-sbt-on-Linux.html)
or [manual
instalation](http://www.scala-sbt.org/release/tutorial/Manual-Installation.html).
* Install [Mahout](http://mahout.apache.org/general/downloads.html). Don't
forget to setup MAHOUT_HOME and MAHOUT_LOCAL
-##Build
-Building the examples from project's root folder:
+##Application
+Using Mahout as a library in an application will require a little Scala code.
We have an App trait in Scala so we'll create an object, which inherits from
```App```
- $ sbt pack
+```
+object CooccurrenceDriver extends App {
+}
+```
+This will look a little different than Java since ```App``` does delayed
initialization, which causes the main body to be executed when the App is
launched, just as in Java you would create a CooccurrenceDriver.main.
+
+Before we can execute something on Spark we'll need to create a context. We
could use raw Spark calls here but default values are setup for a Ma // strip
off names, which only takes and array of IndexedDatasets
+ val indicatorMatrices = SimilarityAnalysis.cooccurrencesIDSs(actions.map(a
=> a._2))
+hout context.
+
+```
+implicit val mc = mahoutSparkContext(masterUrl = "local", appName =
"2-input-cooc")
+```
+We need to read in three files containing different interaction types. The
files will each be read into a Mahout IndexedDataset. This allows us to
preserve application-specific user and item IDs throughout the calculations.
+
+For example, here is data/purchase.csv:
+
+```
+u1,iphone
+u1,ipad
+u2,nexus
+u2,galaxy
+u3,surface
+u4,iphone
+u4,galaxy
+
+```
+Mahout has a helper function that reads the text delimited in
SparkEngine.indexedDatasetDFSReadElements. The function reads single elements
in a distributed way to create the IndexedDataset.
+
+Notice we read in all datasets before we adjust the number of rows in them to
match the total number of users in the data. This is so the math works out even
if some users took one action but not another.
+
+```
+/**
+ * Read files of element tuples and create IndexedDatasets one per action.
These share a userID BiMap but have
+ * their own itemID BiMaps
+ */
+def readActions(actionInput: Array[(String, String)]): Array[(String,
IndexedDataset)] = {
+ var actions = Array[(String, IndexedDataset)]()
+
+ val userDictionary: BiMap[String, Int] = HashBiMap.create()
+
+ // The first action named in the sequence is the "primary" action and
+ // begins to fill up the user dictionary
+ for ( actionDescription <- actionInput ) {// grab the path to actions
+ val action: IndexedDataset = SparkEngine.indexedDatasetDFSReadElements(
+ actionDescription._2,
+ schema = DefaultIndexedDatasetElementReadSchema,
+ existingRowIDs = userDictionary)
+ userDictionary.putAll(action.rowIDs)
+ // put the name in the tuple with the indexedDataset
+ actions = actions :+ (actionDescription._1, action)
+ }
+
+ // After all actions are read in the userDictonary will contain every user
seen,
+ // even if they may not have taken all actions . Now we adjust the row rank
of
+ // all IndxedDataset's to have this number of rows
+ // Note: this is very important or the cooccurrence calc may fail
+ val numUsers = userDictionary.size() // one more than the cardinality
+
+ val resizedNameActionPairs = actions.map { a =>
+ //resize the matrix by, in effect by adding empty rows
+ val resizedMatrix = a._2.create(a._2.matrix, userDictionary,
a._2.columnIDs).newRowCardinality(numUsers)
+ (a._1, resizedMatrix) // return the Tuple of (name, IndexedDataset)
+ }
+ resizedNameActionPairs // return the array of Tuples
+}
+
+```
+
+Now that we have the data read in we can perform the cooccurrence calculation.
+
+```
+// strip off names, which only takes and array of IndexedDatasets
+val indicatorMatrices = SimilarityAnalysis.cooccurrencesIDSs(actions.map(a =>
a._2))
+
+```
+
+All we need to do now is write the indicators.
+
+```
+// zip a pair of arrays into an array of pairs, reattaching the action names
+val indicatorDescriptions = actions.map(a => a._1).zip(indicatorMatrices)
+writeIndicators(indicatorDescriptions)
+```
+
+The ```writeIndicators``` method uses the default write function
```dfsWrite```.
+
+```
+/**
+ * Write indicatorMatrices to the output dir in the default format
+ */
+def writeIndicators( indicators: Array[(String, IndexedDataset)]) = {
+ for (indicator <- indicators ) {
+ val indicatorDir = OutputPath + indicator._1
+ indicator._2.dfsWrite(
+ indicatorDir, // do we have to remove the last $ char?
+ // omit LLR strengths and format for search engine indexing
+ IndexedDatasetWriteBooleanSchema)
+ }
+}
+
+```
-This will automatically set up some launcher scripts for the driver. To run
execute
+See the Github project for the full source. Now we create a build.sbt to build
the example.
- $ target/pack/bin/cooc
-
-The driver will execute in Spark standalone mode one the provided sample data
and output log information including various information about the input data.
The output will be in /path/to/3-input-cooc/data/indicators/*indicator-type*
+```
+name := "cooccurrence-driver"
-##CooccurrenceDriver
+organization := "com.finderbots"
-This driver takes three actions in three separate input files. The input is in
tuple form (user-id,item-id) one per line. It calculates all cooccurrence and
cross-cooccurrence indicators. The sample actions are trivial hand made
examples with somewhat intuitive data.
+version := "0.1"
-Actions:
+scalaVersion := "2.10.4"
- 1. **Purchase**: user purchases
- 2. **View**: user product details views
- 3. **Category**: user preference for category tags
-
-Indicators:
+val sparkVersion = "1.1.1"
+
+libraryDependencies ++= Seq(
+ "log4j" % "log4j" % "1.2.17",
+ // Mahout's Spark code
+ "commons-io" % "commons-io" % "2.4",
+ "org.apache.mahout" % "mahout-math-scala_2.10" % "0.10.0",
+ "org.apache.mahout" % "mahout-spark_2.10" % "0.10.0",
+ "org.apache.mahout" % "mahout-math" % "0.10.0",
+ "org.apache.mahout" % "mahout-hdfs" % "0.10.0",
+ // Google collections, AKA Guava
+ "com.google.guava" % "guava" % "16.0")
+
+resolvers += "typesafe repo" at " http://repo.typesafe.com/typesafe/releases/";
- 1. **Purchase cooccurrence**: may be interpretted as a list if similar items
for each item. Similar in terms of which users purchased them.
- 2. **View cross-cooccurrence**: may be interpretted as a list of similar
items in terms of which users viewed the item where the view led to a purchase.
- 3. **Category cross-cooccurrence**: may be interpretted as a list of similar
categories in terms of which users preferred the category and this led to a
purchase.
+resolvers += Resolver.mavenLocal
-##Data
-Mahout has reader traits that will read text delimited files. Input for
*spark-itemsimilarity* and this CooccurrenceDriver are tuples of
(user-id,item-id) with one line per tuple. The inputs for CooccurrenceDriver
are files but in *spark-itemsimilarity* they may be directories of "part-xxxxx"
files. These can be found in the ```data``` directory.
+packSettings
+
+packMain := Map(
+ "cooc" -> "CooccurrenceDriver"
+)
+
+```
+
+##Build
+Building the examples from project's root folder:
+
+ $ sbt pack
+
+This will automatically set up some launcher scripts for the driver. To run
execute
+
+ $ target/pack/bin/cooc
+
+The driver will execute in Spark standalone mode and put the data in
/path/to/3-input-cooc/data/indicators/*indicator-type*
##Using a Debugger
To build and run this example in a debugger like IntelliJ IDEA. Install from
the IntelliJ site and add the Scala plugin.
@@ -58,3 +179,9 @@ Now choose "Application" in the left pan
After setting breakpoints you are now ready to debug the configuration. Go to
the Run->Debug... menu and pick your configuration. This will execute using a
local standalone instance of Spark.
+
+##The Mahout Shell
+
+For small script-like apps you may wish to use the Mahout shell. It is a Scala
REPL type interactive shell built on the Spark shell with Mahout-Samsara
extensions.
+
+For the shell you won't need the context, since it is created when the shell
is launched. To control the configuration of Mahout and Spark we set
environment variables.
\ No newline at end of file
