This is an automated email from the ASF dual-hosted git repository.
tbonelee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new 1aeed09a00 [ZEPPELIN-6508] Remove outdated Mahout interpreter helper
and documentation
1aeed09a00 is described below
commit 1aeed09a00f5185a38b2dd9d468df54ef68b7237
Author: Jongyoul Lee <[email protected]>
AuthorDate: Sun Aug 23 17:57:20 2026 +0900
[ZEPPELIN-6508] Remove outdated Mahout interpreter helper and documentation
### What is this PR for?
Remove the obsolete Mahout integration artifacts from Zeppelin.
The existing `mahout.md` page does not describe a standalone Zeppelin
interpreter module. It relies on a Python 2 helper that clones the Spark and
Flink interpreter settings, then adds Mahout 0.12.2 artifacts built for Scala
2.10. The accompanying documentation and tutorial are limited to Spark 1.5/1.6
and no longer match Zeppelin's supported runtime stack.
This PR removes the helper, its documentation and navigation entries, and
the bundled Mahout tutorial notebook. Apache Mahout's current Qumat Python
packages use a separate Python integration path and are intentionally outside
the scope of this cleanup PR.
### What type of PR is it?
Improvement
### Todos
* [x] Remove the legacy Mahout helper script
* [x] Remove the obsolete Mahout interpreter documentation and navigation
entries
* [x] Remove the bundled Mahout tutorial notebook
* [x] Verify that no Mahout or Qumat references remain in the repository
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-6508
### How should this be tested?
* `bundle exec jekyll build --safe` in the documented Ruby 4.0.6 Docker
environment
* `ruby check_external_resources.rb _site`
* `git diff --check`
* Verify that the generated site contains no Mahout page or navigation entry
* Verify that the repository contains no remaining Mahout or Qumat
references
### Screenshots (if appropriate)
Not applicable.
### Questions:
* Does the license files need to update? No.
* Is there breaking changes for older versions? The PR removes unsupported
legacy documentation, a helper script, and a tutorial notebook; no interpreter
module or supported runtime behavior is changed.
* Does this needs documentation? This PR removes the obsolete documentation
and all links to it.
Closes #5430 from jongyoul/codex/ZEPPELIN-6508-remove-mahout-integration.
Signed-off-by: ChanHo Lee <[email protected]>
---
docs/_includes/themes/zeppelin/_navigation.html | 1 -
docs/index.md | 1 -
docs/interpreter/mahout.md | 186 -----
.../Using Mahout_2BYEZ5EVK.zpln | 887 ---------------------
scripts/mahout/add_mahout_interpreters.py | 290 -------
5 files changed, 1365 deletions(-)
diff --git a/docs/_includes/themes/zeppelin/_navigation.html
b/docs/_includes/themes/zeppelin/_navigation.html
index 9554b48a75..82eb3d6d49 100644
--- a/docs/_includes/themes/zeppelin/_navigation.html
+++ b/docs/_includes/themes/zeppelin/_navigation.html
@@ -148,7 +148,6 @@
<li><a href="{{BASE_PATH}}/interpreter/java.html">Java</a></li>
<li><a
href="{{BASE_PATH}}/interpreter/jupyter.html">Jupyter</a></li>
<li><a href="{{BASE_PATH}}/interpreter/livy.html">Livy</a></li>
- <li><a
href="{{BASE_PATH}}/interpreter/mahout.html">Mahout</a></li>
<li><a
href="{{BASE_PATH}}/interpreter/markdown.html">Markdown</a></li>
<li><a
href="{{BASE_PATH}}/interpreter/mongodb.html">MongoDB</a></li>
<li><a
href="{{BASE_PATH}}/interpreter/neo4j.html">Neo4j</a></li>
diff --git a/docs/index.md b/docs/index.md
index 1393c0550d..feca2b3693 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -147,7 +147,6 @@ limitations under the License.
* [JDBC](./interpreter/jdbc.html)
* [Jupyter](./interpreter/jupyter.html)
* [Livy](./interpreter/livy.html)
- * [Mahout](./interpreter/mahout.html)
* [Markdown](./interpreter/markdown.html)
* [MongoDB](./interpreter/mongodb.html)
* [Neo4j](./interpreter/neo4j.html)
diff --git a/docs/interpreter/mahout.md b/docs/interpreter/mahout.md
deleted file mode 100644
index 0baa8987ea..0000000000
--- a/docs/interpreter/mahout.md
+++ /dev/null
@@ -1,186 +0,0 @@
----
-layout: page
-title: "Mahout Interpreter for Apache Zeppelin"
-description: "Apache Mahout provides a unified API (the R-Like Scala DSL) for
quickly creating machine learning algorithms on a variety of engines."
-group: interpreter
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-{% include JB/setup %}
-
-# Apache Mahout Interpreter for Apache Zeppelin
-
-<div id="toc"></div>
-
-## Installation
-
-Apache Mahout is a collection of packages that enable machine learning and
matrix algebra on underlying engines such as Apache Flink or Apache Spark. A
convenience script for creating and configuring two Mahout enabled interpreters
exists. The `%sparkMahout` and `%flinkMahout` interpreters do not exist by
default but can be easily created using this script.
-
-### Easy Installation
-To quickly and easily get up and running using Apache Mahout, run the
following command from the top-level directory of the Zeppelin install:
-
-```bash
-python scripts/mahout/add_mahout.py
-```
-
-This will create the `%sparkMahout` and `%flinkMahout` interpreters, and
restart Zeppelin.
-
-### Advanced Installation
-
-The `add_mahout.py` script contains several command line arguments for
advanced users.
-
- <table class="table-configuration">
- <tr>
- <th>Argument</th>
- <th>Description</th>
- <th>Example</th>
- </tr>
- <tr>
- <td markdown="span">`--zeppelin_home`</td>
- <td markdown="span">This is the path to the Zeppelin installation. This
flag is not needed if the script is run from the top-level installation
directory or from the `zeppelin/scripts/mahout` directory.</td>
- <td markdown="span">`/path/to/zeppelin`</td>
- </tr>
- <tr>
- <td markdown="span">`--mahout_home`</td>
- <td markdown="span">If the user has already installed Mahout, this flag
can set the path to `MAHOUT_HOME`. If this is set, downloading Mahout will be
skipped.</td>
- <td markdown="span">`/path/to/mahout_home`</td>
- </tr>
- <tr>
- <td markdown="span">`--restart_later`</td>
- <td>Restarting is necessary for updates to take effect. By default the
script will restart Zeppelin for you. Restart will be skipped if this flag is
set.</td>
- <td>NA</td>
- </tr>
- <tr>
- <td markdown="span">`--force_download`</td>
- <td>This flag will force the script to re-download the binary even if it
already exists. This is useful for previously failed downloads.</td>
- <td>NA</td>
- </tr>
- <tr>
- <td markdown="span">`--overwrite_existing`</td>
- <td markdown="span">This flag will force the script to overwrite
existing `%sparkMahout` and `%flinkMahout` interpreters. Useful when you want
to just start over.</td>
- <td>NA</td>
- </tr>
-</table>
-
-__NOTE 1:__ Apache Mahout at this time only supports Spark 1.5 and Spark 1.6
and Scala 2.10. If the user is using another version of Spark (e.g. 2.0), the
`%sparkMahout` will likely not work. The `%flinkMahout` interpreter will still
work and the user is encouraged to develop with that engine as the code can be
ported via copy and paste, as is evidenced by the tutorial notebook.
-
-__NOTE 2:__ If using Apache Flink in cluster mode, the following libraries
will also need to be copied to `${FLINK_HOME}/lib`
-- mahout-math-0.12.2.jar
-- mahout-math-scala_2.10-0.12.2.jar
-- mahout-flink_2.10-0.12.2.jar
-- mahout-hdfs-0.12.2.jar
--
[com.google.guava:guava:14.0.1](http://central.maven.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.jar)
-
-## Overview
-
-The [Apache Mahout](http://mahout.apache.org/)™ project's goal is to build an
environment for quickly creating scalable performant machine learning
applications.
-
-Apache Mahout software provides three major features:
-
-- A simple and extensible programming environment and framework for building
scalable algorithms
-- A wide variety of premade algorithms for Scala + Apache Spark, H2O, Apache
Flink
-- Samsara, a vector math experimentation environment with R-like syntax which
works at scale
-
-In other words:
-
-*Apache Mahout provides a unified API for quickly creating machine learning
algorithms on a variety of engines.*
-
-## How to use
-
-When starting a session with Apache Mahout, depending on which engine you are
using (Spark or Flink), a few imports must be made and a _Distributed Context_
must be declared. Copy and paste the following code and run once to get
started.
-
-### Flink
-
-```scala
-%flinkMahout
-
-import org.apache.flink.api.scala._
-import org.apache.mahout.math.drm._
-import org.apache.mahout.math.drm.RLikeDrmOps._
-import org.apache.mahout.flinkbindings._
-import org.apache.mahout.math._
-import scalabindings._
-import RLikeOps._
-
-implicit val ctx = new FlinkDistributedContext(benv)
-```
-
-### Spark
-```scala
-%sparkMahout
-
-import org.apache.mahout.math._
-import org.apache.mahout.math.scalabindings._
-import org.apache.mahout.math.drm._
-import org.apache.mahout.math.scalabindings.RLikeOps._
-import org.apache.mahout.math.drm.RLikeDrmOps._
-import org.apache.mahout.sparkbindings._
-
-implicit val sdc: org.apache.mahout.sparkbindings.SparkDistributedContext =
sc2sdc(sc)
-```
-
-### Same Code, Different Engines
-
-After importing and setting up the distributed context, the Mahout R-Like DSL
is consistent across engines. The following code will run in both
`%flinkMahout` and `%sparkMahout`
-
-```scala
-val drmData = drmParallelize(dense(
- (2, 2, 10.5, 10, 29.509541), // Apple Cinnamon Cheerios
- (1, 2, 12, 12, 18.042851), // Cap'n'Crunch
- (1, 1, 12, 13, 22.736446), // Cocoa Puffs
- (2, 1, 11, 13, 32.207582), // Froot Loops
- (1, 2, 12, 11, 21.871292), // Honey Graham Ohs
- (2, 1, 16, 8, 36.187559), // Wheaties Honey Gold
- (6, 2, 17, 1, 50.764999), // Cheerios
- (3, 2, 13, 7, 40.400208), // Clusters
- (3, 3, 13, 4, 45.811716)), numPartitions = 2)
-
-drmData.collect(::, 0 until 4)
-
-val drmX = drmData(::, 0 until 4)
-val y = drmData.collect(::, 4)
-val drmXtX = drmX.t %*% drmX
-val drmXty = drmX.t %*% y
-
-
-val XtX = drmXtX.collect
-val Xty = drmXty.collect(::, 0)
-val beta = solve(XtX, Xty)
-```
-
-## Leveraging Resource Pools for Visualization
-
-Resource Pools are a powerful Zeppelin feature that lets us share information
between interpreters. A fun trick is to take the output of our work in Mahout
and analyze it in other languages.
-
-### Setting up a Resource Pool in Flink
-
-In Spark based interpreters resource pools are accessed via the
ZeppelinContext API. To put and get things from the resource pool one can be
done simple
-
-```scala
-val myVal = 1
-z.put("foo", myVal)
-val myFetchedVal = z.get("foo")
-```
-
-To add this functionality to a Flink based interpreter we declare the following
-
-```scala
-%flinkMahout
-
-import org.apache.zeppelin.interpreter.InterpreterContext
-
-val z = InterpreterContext.get().getResourcePool()
-```
-
-Now we can access the resource pool in a consistent manner from the
`%flinkMahout` interpreter.
diff --git a/notebook/Miscellaneous Tutorial/Using Mahout_2BYEZ5EVK.zpln
b/notebook/Miscellaneous Tutorial/Using Mahout_2BYEZ5EVK.zpln
deleted file mode 100644
index aab04dfc8c..0000000000
--- a/notebook/Miscellaneous Tutorial/Using Mahout_2BYEZ5EVK.zpln
+++ /dev/null
@@ -1,887 +0,0 @@
-{
- "paragraphs": [
- {
- "text": "%md\n\n### The [Apache Mahout](http://mahout.apache.org/)™
project\u0027s goal is to build an environment for quickly creating scalable
performant machine learning applications.\n\n#### Apache Mahout software
provides three major features:\n\n- A simple and extensible programming
environment and framework for building scalable algorithms\n- A wide variety of
premade algorithms for Scala + Apache Spark, H2O, Apache Flink\n- Samsara, a
vector math experimentation environment [...]
- "dateUpdated": "Sep 28, 2016 10:01:52 AM",
- "config": {
- "colWidth": 12.0,
- "enabled": true,
- "editorMode": "ace/mode/scala",
- "editorHide": true,
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475013396125_39313566",
- "id": "20160927-155636_1798325301",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003ch3\u003eThe \u003ca
href\u003d\"http://mahout.apache.org/\"\u003eApache Mahout\u003c/a\u003e™
project\u0027s goal is to build an environment for quickly creating scalable
performant machine learning applications.\u003c/h3\u003e\n\u003ch4\u003eApache
Mahout software provides three major
features:\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eA simple and extensible
programming environment and framework for building scalable
algorithms\u003c/li\u003e\n\u003cli\u0 [...]
- }
- ]
- },
- "dateCreated": "Sep 27, 2016 3:56:36 AM",
- "dateStarted": "Sep 27, 2016 4:02:55 AM",
- "dateFinished": "Sep 27, 2016 4:02:55 AM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%md\n\n#### \"Installing\" the Apache Mahout dependencies and
configuring a new Spark and Flink interpreter\n\nThe following two paragraphs
are convenience paragraphs. You **only need to run them once** to create two
new interpreters `%spark.mahout` and `%flink.mahout`. These are intended for
users who don\u0027t have Apache Mahout already installed. They assume you
started Apache Zeppelin from the top level directory or from the bin. You can
tell which one is you by weat [...]
- "dateUpdated": "Sep 27, 2016 4:31:15 AM",
- "config": {
- "colWidth": 12.0,
- "enabled": true,
- "editorMode": "ace/mode/scala",
- "editorHide": true,
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475015019489_-1704057033",
- "id": "20160927-162339_341514150",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003ch4\u003e\u0026ldquo;Installing\u0026rdquo; the
Apache Mahout dependencies and configuring a new Spark and Flink
interpreter\u003c/h4\u003e\n\u003cp\u003eThe following two paragraphs are
convenience paragraphs. You \u003cstrong\u003eonly need to run them
once\u003c/strong\u003e to create two new interpreters
\u003ccode\u003e%spark.mahout\u003c/code\u003e and
\u003ccode\u003e%flink.mahout\u003c/code\u003e. These are intended for users
who don\u0027t have Apache M [...]
- }
- ]
- },
- "dateCreated": "Sep 27, 2016 4:23:39 AM",
- "dateStarted": "Sep 27, 2016 4:31:12 AM",
- "dateFinished": "Sep 27, 2016 4:31:13 AM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "title": "Convenience Paragraph if you started Zeppelin by
\u0027./zeppelin-daemon.sh start\u0027",
- "text": "%sh\n\npython ../scripts/mahout/add_mahout.py",
- "dateUpdated": "Dec 17, 2016 3:41:45 PM",
- "config": {
- "colWidth": 12.0,
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- },
- "enabled": true,
- "editorMode": "ace/mode/sh",
- "title": true,
- "results": {},
- "editorSetting": {
- "language": "sh",
- "editOnDblClick": false
- }
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475014957043_-748248820",
- "id": "20160927-162237_1864782562",
- "dateCreated": "Sep 27, 2016 4:22:37 AM",
- "status": "READY",
- "progressUpdateIntervalMs": 500
- },
- {
- "title": "Convenience Paragraph if you started Zeppelin by
\u0027bin/zeppelin-daemon.sh start\u0027",
- "text": "%sh\npython scripts/mahout/add_mahout_interpreters.py",
- "dateUpdated": "Dec 17, 2016 3:41:46 PM",
- "config": {
- "colWidth": 12.0,
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- },
- "enabled": true,
- "editorMode": "ace/mode/sh",
- "title": true,
- "results": {},
- "editorSetting": {
- "language": "sh",
- "editOnDblClick": false
- }
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475018789604_-139338572",
- "id": "20160927-172629_1189436716",
- "dateCreated": "Sep 27, 2016 5:26:29 AM",
- "status": "READY",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%md\n\nAfter the interpreters are created you will need to
\u0027bind\u0027 them by clicking on the little gear in the top right corner,
scrolling to the top, and clicking on `mahoutFlink` and `mahoutSpark` so that
they are highlighted in blue.\n\n#### Running Mahout code\n\nYou will need to
import certain libraries, and declare the _Mahout Distributed Context_ when you
first start your notebook using the interpreters. \n\nIf using Apache Flink the
code you need to run is: [...]
- "dateUpdated": "Sep 28, 2016 12:36:44 PM",
- "config": {
- "colWidth": 12.0,
- "enabled": true,
- "editorMode": "ace/mode/markdown",
- "editorHide": true,
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475014730618_1513783554",
- "id": "20160927-161850_1560940440",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cp\u003eAfter the interpreters are created you will
need to \u0027bind\u0027 them by clicking on the little gear in the top right
corner, scrolling to the top, and clicking on
\u003ccode\u003emahoutFlink\u003c/code\u003e and
\u003ccode\u003emahoutSpark\u003c/code\u003e so that they are highlighted in
blue.\u003c/p\u003e\n\u003ch4\u003eRunning Mahout
code\u003c/h4\u003e\n\u003cp\u003eYou will need to import certain libraries,
and declare the \u003cem\u003eMahout [...]
- }
- ]
- },
- "dateCreated": "Sep 27, 2016 4:18:50 AM",
- "dateStarted": "Sep 28, 2016 10:17:05 AM",
- "dateFinished": "Sep 28, 2016 10:17:06 AM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%flinkMahout\n\n// Imports and creating the distributed
context, similar but not exactly the same
///////////////////////////////////////////\nimport
org.apache.flink.api.scala._\nimport org.apache.mahout.math.drm._\nimport
org.apache.mahout.math.drm.RLikeDrmOps._\nimport
org.apache.mahout.flinkbindings._\nimport org.apache.mahout.math._\nimport
scalabindings._\nimport RLikeOps._\n\n\nimplicit val ctx \u003d new
FlinkDistributedContext(benv)\n\n// CODE IS EXACTLY THE SAME [...]
- "dateUpdated": "Sep 28, 2016 1:41:59 PM",
- "config": {
- "colWidth": 6.0,
- "enabled": true,
- "editorMode": "ace/mode/markdown",
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475015779325_-1869239670",
- "id": "20160927-163619_899520006",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "import org.apache.flink.api.scala._\nimport
org.apache.mahout.math.drm._\nimport
org.apache.mahout.math.drm.RLikeDrmOps._\nimport
org.apache.mahout.flinkbindings._\nimport org.apache.mahout.math._\nimport
scalabindings._\nimport RLikeOps._\nctx:
org.apache.mahout.flinkbindings.FlinkDistributedContext \u003d
org.apache.mahout.flinkbindings.FlinkDistributedContext@4452b0a5\nwarning:
Class it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap not found - continuing
with a st [...]
- }
- ]
- },
- "dateCreated": "Sep 27, 2016 4:36:19 AM",
- "dateStarted": "Sep 28, 2016 1:41:59 PM",
- "dateFinished": "Sep 28, 2016 1:42:25 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%sparkMahout\n\n// Imports and creating the distributed
context, similar but not exactly the same
///////////////////////////////////////////\n\nimport
org.apache.mahout.math._\nimport org.apache.mahout.math.scalabindings._\nimport
org.apache.mahout.math.drm._\nimport
org.apache.mahout.math.scalabindings.RLikeOps._\nimport
org.apache.mahout.math.drm.RLikeDrmOps._\nimport
org.apache.mahout.sparkbindings._\n\nimplicit val sdc:
org.apache.mahout.sparkbindings.SparkDistributed [...]
- "dateUpdated": "Sep 28, 2016 1:45:09 PM",
- "config": {
- "colWidth": 6.0,
- "enabled": true,
- "editorMode": "ace/mode/scala",
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475016737629_-774084480",
- "id": "20160927-165217_1266863511",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "\nimport org.apache.mahout.math._\n\nimport
org.apache.mahout.math.scalabindings._\n\nimport
org.apache.mahout.math.drm._\n\nimport
org.apache.mahout.math.scalabindings.RLikeOps._\n\nimport
org.apache.mahout.math.drm.RLikeDrmOps._\n\nimport
org.apache.mahout.sparkbindings._\n\nsdc:
org.apache.mahout.sparkbindings.SparkDistributedContext \u003d
org.apache.mahout.sparkbindings.SparkDistributedContext@32c46474\n\ndrmData:
org.apache.mahout.math.drm.CheckpointedDrm[Int] [...]
- }
- ]
- },
- "dateCreated": "Sep 27, 2016 4:52:17 AM",
- "dateStarted": "Sep 28, 2016 1:45:09 PM",
- "dateFinished": "Sep 28, 2016 1:45:23 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "title": "Use Resource Pools with Zeppelin",
- "text": "%md\n\n### Taking advantage of Zeppelin Resource Pools\n\nOne
of the major motivations for integrating Apache Mahout with Apache Zeppelin was
the many benefits that come from leveraging the resource pools. A resource
pool is a block of memory that can be acccessed by all interpreters and is
useful for sharing small variables between the interpreters. \n\nThe Spark
interpreter has a simple interface for accessing the ResourcePools, the Flink
interface is less documented bu [...]
- "dateUpdated": "Sep 27, 2016 5:55:31 AM",
- "config": {
- "colWidth": 12.0,
- "enabled": true,
- "title": true,
- "editorMode": "ace/mode/markdown",
- "editorHide": true,
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475016792277_-1100474141",
- "id": "20160927-165312_1668894932",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003ch3\u003eTaking advantage of Zeppelin Resource
Pools\u003c/h3\u003e\n\u003cp\u003eOne of the major motivations for integrating
Apache Mahout with Apache Zeppelin was the many benefits that come from
leveraging the resource pools. A resource pool is a block of memory that can
be acccessed by all interpreters and is useful for sharing small variables
between the interpreters.\u003c/p\u003e\n\u003cp\u003eThe Spark interpreter has
a simple interface for accessing t [...]
- }
- ]
- },
- "dateCreated": "Sep 27, 2016 4:53:12 AM",
- "dateStarted": "Sep 27, 2016 5:40:35 AM",
- "dateFinished": "Sep 27, 2016 5:40:36 AM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "title": "Flink ResourcePools",
- "text": "%flinkMahout\n\nimport
org.apache.zeppelin.interpreter.InterpreterContext\n\nval resourcePool \u003d
InterpreterContext.get().getResourcePool()\n\nresourcePool.put(\"flinkBeta\",
beta.asFormatString)\n",
- "dateUpdated": "Sep 28, 2016 1:42:35 PM",
- "config": {
- "colWidth": 6.0,
- "enabled": true,
- "editorMode": "ace/mode/scala",
- "title": true,
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475019635571_-1705373112",
- "id": "20160927-174035_1591078106",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "import
org.apache.zeppelin.interpreter.InterpreterContext\nresourcePool:
org.apache.zeppelin.resource.ResourcePool \u003d
org.apache.zeppelin.resource.DistributedResourcePool@3fdd93cc\n"
- }
- ]
- },
- "dateCreated": "Sep 27, 2016 5:40:35 AM",
- "dateStarted": "Sep 28, 2016 1:42:35 PM",
- "dateFinished": "Sep 28, 2016 1:42:36 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "title": "Spark ResourcePools",
- "text": "%sparkMahout\n\n\n\n\nz.put(\"sparkBeta\",
beta.asFormatString)",
- "dateUpdated": "Sep 28, 2016 1:45:35 PM",
- "config": {
- "colWidth": 6.0,
- "enabled": true,
- "editorMode": "ace/mode/scala",
- "title": true,
- "results": []
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475019751650_-1885234738",
- "id": "20160927-174231_1288588876",
- "results": {
- "code": "SUCCESS",
- "msg": []
- },
- "dateCreated": "Sep 27, 2016 5:42:31 AM",
- "dateStarted": "Sep 28, 2016 1:45:35 PM",
- "dateFinished": "Sep 28, 2016 1:45:36 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "title": "Collect Results in Python and Evaluate Differences",
- "text": "%spark.pyspark\n\nimport ast\n\nflinkBetaDict \u003d
ast.literal_eval(z.get(\"flinkBeta\"))\nsparkBetaDict \u003d
ast.literal_eval(z.get(\"sparkBeta\"))\n\nprint \"----------------- differences
between betas calulated in Flink and Spark-----------------\"\nfor i in
range(0,4):\n print \"beta\", i, \": \" , flinkBetaDict[i] -
sparkBetaDict[i]",
- "dateUpdated": "Sep 28, 2016 1:45:37 PM",
- "config": {
- "colWidth": 12.0,
- "enabled": true,
- "editorMode": "ace/mode/python",
- "title": true,
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475020470280_1661203311",
- "id": "20160927-175430_1451783515",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "----------------- differences between betas calulated in
Flink and Spark-----------------\nbeta 0 : -5.24025267623e-14\nbeta 1 :
-2.44249065418e-14\nbeta 2 : 2.28705943073e-14\nbeta 3 : -1.47104550763e-14\n"
- }
- ]
- },
- "dateCreated": "Sep 27, 2016 5:54:30 AM",
- "dateStarted": "Sep 28, 2016 1:45:38 PM",
- "dateFinished": "Sep 28, 2016 1:45:38 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%md\n\n## Plotting Mahout with R\n\nThe following examples show
how we can leverage R to plot our results from Mahout\n",
- "dateUpdated": "Sep 28, 2016 12:34:33 PM",
- "config": {
- "colWidth": 12.0,
- "enabled": true,
- "editorMode": "ace/mode/markdown",
- "editorHide": true,
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475087633007_-566041383",
- "id": "20160928-123353_147363530",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003ch2\u003ePlotting Mahout with
R\u003c/h2\u003e\n\u003cp\u003eThe following examples show how we can leverage
R to plot our results from Mahout\u003c/p\u003e\n"
- }
- ]
- },
- "dateCreated": "Sep 28, 2016 12:33:53 PM",
- "dateStarted": "Sep 28, 2016 12:34:30 PM",
- "dateFinished": "Sep 28, 2016 12:34:30 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%flinkMahout\nval mxRnd \u003d
Matrices.symmetricUniformView(5000, 2, 1234)\nval drmRand \u003d
drmParallelize(mxRnd)\n\n\nval drmSin \u003d drmRand.mapBlock() {case (keys,
block) \u003d\u003e \n val blockB \u003d block.like()\n for (i \u003c- 0
until block.nrow) {\n blockB(i, 0) \u003d block(i, 0) \n blockB(i, 1)
\u003d Math.sin((block(i, 0) * 8))\n }\n keys -\u003e
blockB\n}\n\nresourcePool.put(\"flinkSinDrm\", drm.drmSampleToTSV(drmSin,
0.85))",
- "dateUpdated": "Sep 28, 2016 1:52:44 PM",
- "config": {
- "colWidth": 6.0,
- "enabled": true,
- "editorMode": "ace/mode/scala",
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 284.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475020580886_2102494975",
- "id": "20160927-175620_816809523",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "mxRnd: org.apache.mahout.math.Matrix \u003d \n{\n 0
\u003d\u003e\t{0:0.4586377101191827,1:0.07261898163580698}\n 1
\u003d\u003e\t{0:0.48977896201757654,1:0.2695201068510176}\n 2
\u003d\u003e\t{0:0.33215452109376786,1:0.2148377346657124}\n 3
\u003d\u003e\t{0:0.4497098649240723,1:0.4331127334380502}\n 4
\u003d\u003e\t{0:-0.03782634247193647,1:-0.32353833540588983}\n 5
\u003d\u003e\t{0:0.15137106418749705,1:0.422446220403861}\n 6
\u003d\u003e\t{0:0.2714115385692545,1:-0 [...]
- }
- ]
- },
- "dateCreated": "Sep 27, 2016 5:56:20 AM",
- "dateStarted": "Sep 28, 2016 1:42:42 PM",
- "dateFinished": "Sep 28, 2016 1:42:52 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%sparkMahout\nval mxRnd \u003d
Matrices.symmetricUniformView(5000, 2, 1234)\nval drmRand \u003d
drmParallelize(mxRnd)\n\n\nval drmSin \u003d drmRand.mapBlock() {case (keys,
block) \u003d\u003e \n val blockB \u003d block.like()\n for (i \u003c- 0
until block.nrow) {\n blockB(i, 0) \u003d block(i, 0) \n blockB(i, 1)
\u003d Math.sin((block(i, 0) * 8))\n }\n keys -\u003e
blockB\n}\n\nz.put(\"sparkSinDrm\",
org.apache.mahout.math.drm.drmSampleToTSV(drmSin, 0.85))\n",
- "dateUpdated": "Sep 27, 2016 6:38:39 AM",
- "config": {
- "colWidth": 6.0,
- "enabled": true,
- "editorMode": "ace/mode/scala",
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475021390512_-2030189316",
- "id": "20160927-180950_1754833838",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "\n\n\n\n\n\n\n\n\n\n\n\n\nmxRnd:
org.apache.mahout.math.Matrix \u003d \n{\n 0
\u003d\u003e\t{0:0.4586377101191827,1:0.07261898163580698}\n 1
\u003d\u003e\t{0:0.48977896201757654,1:0.2695201068510176}\n 2
\u003d\u003e\t{0:0.33215452109376786,1:0.2148377346657124}\n 3
\u003d\u003e\t{0:0.4497098649240723,1:0.4331127334380502}\n 4
\u003d\u003e\t{0:-0.03782634247193647,1:-0.32353833540588983}\n 5
\u003d\u003e\t{0:0.15137106418749705,1:0.422446220403861}\n 6 \u003d\u003e\t
[...]
- }
- ]
- },
- "dateCreated": "Sep 27, 2016 6:09:50 AM",
- "dateStarted": "Sep 27, 2016 6:38:39 AM",
- "dateFinished": "Sep 27, 2016 6:38:40 AM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%spark.r {\"imageWidth\":
\"400px\"}\n\nlibrary(\"ggplot2\")\n\nflinkSinStr \u003d
z.get(\"flinkSinDrm\")\nsparkSinStr \u003d z.get(\"sparkSinDrm\")\n\nflinkData
\u003c- read.table(text\u003d flinkSinStr, sep\u003d\"\\t\",
header\u003dFALSE)\nsparkData \u003c- read.table(text\u003d sparkSinStr,
sep\u003d\"\\t\", header\u003dFALSE)\n\nplot(flinkData, col\u003d\"red\")\n#
Graph trucks with red dashed line and square points\npoints(sparkData,
col\u003d\"blue\")\n\n# Create a [...]
- "dateUpdated": "Sep 28, 2016 1:52:26 PM",
- "config": {
- "colWidth": 6.0,
- "enabled": true,
- "editorMode": "ace/mode/r",
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475021654999_1062405375",
- "id": "20160927-181414_1420533932",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cp\u003e\u003cimg
src\u003d\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAfgAAAH4CAIAAAApSmgoAAAACXBIWXMAAAsSAAALEgHS3X78AAAgAElEQVR4nOzdeVzP9wMH8Nf32yWVTrocRblCMYyJ3JJE5JpjrplrZnMzc5v9xmy2sa25bZght6LccuScEnKVSiqlEqX6/P74tCILUd9332+v58Mffb7yeb++n/Tq3fvz+X6+CkmSQEREmkspOgAREZUsFj0RkYZj0RMRaTgWPRGRhmPRExFpOBY9EZGGY9ETEWk4Fj0RkYZj0RMRaTgWPRGRhmPRExFpOBY9EZGGY9ETEWk4Fj0RkYZj0RMRaTgWPRGRhmPRExFpOBY9EZGGY9ETEWk4Fj0RkYZj0RMRaTgWPRGRhmPRExFpOBa9xnr8+PHcuXNdXFwMDQ0tLCz
[...]
- }
- ]
- },
- "dateCreated": "Sep 27, 2016 6:14:14 AM",
- "dateStarted": "Sep 27, 2016 6:42:20 AM",
- "dateFinished": "Sep 27, 2016 6:42:20 AM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "title": "Create a Gaussian Matrix",
- "text": "%flinkMahout\n\nval mxRnd3d \u003d
Matrices.symmetricUniformView(5000, 3, 1234)\nval drmRand3d \u003d
drmParallelize(mxRnd3d)\n\nval drmGauss \u003d drmRand3d.mapBlock() {case
(keys, block) \u003d\u003e\n val blockB \u003d block.like()\n for (i \u003c-
0 until block.nrow) {\n val x: Double \u003d block(i, 0)\n val y: Double
\u003d block(i, 1)\n val z: Double \u003d block(i, 2)\n\n blockB(i, 0)
\u003d x\n blockB(i, 1) \u003d y\n blockB(i, 2) \u003d Math. [...]
- "dateUpdated": "Sep 28, 2016 1:53:22 PM",
- "config": {
- "colWidth": 6.0,
- "enabled": true,
- "editorMode": "ace/mode/scala",
- "tableHide": true,
- "title": true,
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475021740078_127388926",
- "id": "20160927-181540_1706054053",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "mxRnd3d: org.apache.mahout.math.Matrix \u003d \n{\n 0
\u003d\u003e\t{0:0.4586377101191827,1:0.07261898163580698,2:-0.4120814898385057}\n
1
\u003d\u003e\t{0:0.48977896201757654,1:0.2695201068510176,2:0.2035624121801051}\n
2
\u003d\u003e\t{0:0.33215452109376786,1:0.2148377346657124,2:0.22923597484837382}\n
3
\u003d\u003e\t{0:0.4497098649240723,1:0.4331127334380502,2:-0.26063522630725094}\n
4
\u003d\u003e\t{0:-0.03782634247193647,1:-0.32353833540588983,2:-0.442325626678
[...]
- }
- ]
- },
- "dateCreated": "Sep 27, 2016 6:15:40 AM",
- "dateStarted": "Sep 28, 2016 1:50:54 PM",
- "dateFinished": "Sep 28, 2016 1:51:00 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%spark.r {\"imageWidth\":
\"400px\"}\n\nlibrary(scatterplot3d)\n\n\nflinkGaussStr \u003d
z.get(\"flinkGaussDrm\")\nflinkData \u003c- read.table(text\u003d
flinkGaussStr, sep\u003d\"\\t\", header\u003dFALSE)\n\nscatterplot3d(flinkData,
color\u003d\"green\")\n\n",
- "dateUpdated": "Sep 28, 2016 1:54:56 PM",
- "config": {
- "colWidth": 6.0,
- "enabled": true,
- "editorMode": "ace/mode/r",
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475023444293_-1038534869",
- "id": "20160927-184404_773885252",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cp\u003e\u003cimg
src\u003d\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAfgAAAH4CAIAAAApSmgoAAAACXBIWXMAAAsSAAALEgHS3X78AAAgAElEQVR4nOydd3iUVdr/P9MnmcmkV0IKJYQeivQOIqzYUJCigA2wIpZdFCvuWteyFrAXBARcG4KigogK0hQpIi10Qnqv0+7fH2edX17ffXdXNzAwnM/FxRUmT2bOPE/4Pmfuc5/v1yAiaDQajSZ0MQZ7ABqNRqM5uWih12g0mhBHC71Go9GEOFroNRqNJsTRQq/RaDQhjhZ6jUajCXG00Gs0Gk2Io4Veo9FoQhwt9BqNRhPiaKHXaDSaEEcLvUaj0YQ4Wug1Go0mxNFCr9FoNCGOFnqNRqMJcbTQazQaTYijhV6j0WhCHC30Go1GE+JooddoNJoQRwu9RqPRhDh
[...]
- }
- ]
- },
- "dateCreated": "Sep 27, 2016 6:44:04 AM",
- "dateStarted": "Sep 28, 2016 1:52:10 PM",
- "dateFinished": "Sep 28, 2016 1:52:10 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%md\n\n**NOTE** To install `scatterplot3d` on Ubuntu
use:\n\n```sh\nsudo apt-get install r-cran-scatterplot3d\n```\n\n",
- "dateUpdated": "Sep 28, 2016 1:54:37 PM",
- "config": {
- "colWidth": 6.0,
- "enabled": true,
- "editorMode": "ace/mode/markdown",
- "editorHide": true,
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {},
- "map": {
- "baseMapType": "Streets",
- "isOnline": true,
- "pinCols": []
- }
- }
- }
- ]
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475091302527_1223653372",
- "id": "20160928-133502_1743267136",
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cp\u003e\u003cstrong\u003eNOTE\u003c/strong\u003e To
install \u003ccode\u003escatterplot3d\u003c/code\u003e on Ubuntu
use:\u003c/p\u003e\n\u003cpre\u003e\u003ccode class\u003d\"sh\"\u003esudo
apt-get install r-cran-scatterplot3d\n\u003c/code\u003e\u003c/pre\u003e\n"
- }
- ]
- },
- "dateCreated": "Sep 28, 2016 1:35:02 AM",
- "dateStarted": "Sep 28, 2016 1:54:32 PM",
- "dateFinished": "Sep 28, 2016 1:54:33 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%md\n",
- "dateUpdated": "Sep 28, 2016 1:54:32 PM",
- "config": {},
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1475092472681_-955530461",
- "id": "20160928-135432_2099340527",
- "dateCreated": "Sep 28, 2016 1:54:32 PM",
- "status": "READY",
- "progressUpdateIntervalMs": 500
- }
- ],
- "name": "Using Mahout",
- "id": "2BYEZ5EVK",
- "angularObjects": {
- "2C6WUGPNH:shared_process": [],
- "2C4A8RJNB:shared_process": [],
- "2C4DTK2ZT:shared_process": [],
- "2C6XKJWBR:shared_process": [],
- "2C6AHZPMK:shared_process": [],
- "2C5SU66WQ:shared_process": [],
- "2C6AMJ98Q:shared_process": [],
- "2C4AJZK72:shared_process": [],
- "2C3STPSD7:shared_process": [],
- "2C4FJN9CK:shared_process": [],
- "2C3CW6JBY:shared_process": [],
- "2C5UPQX6Q:shared_process": [],
- "2C5873KN4:shared_process": [],
- "2C5719XN4:shared_process": [],
- "2C52DE5G3:shared_process": [],
- "2C4G28E63:shared_process": [],
- "2C6CU96BC:shared_process": [],
- "2C49A6WY3:shared_process": [],
- "2C3NE73HG:shared_process": []
- },
- "config": {},
- "info": {}
-}
diff --git a/scripts/mahout/add_mahout_interpreters.py
b/scripts/mahout/add_mahout_interpreters.py
deleted file mode 100644
index 307364db00..0000000000
--- a/scripts/mahout/add_mahout_interpreters.py
+++ /dev/null
@@ -1,290 +0,0 @@
-# /**
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements. See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership. The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License. You may obtain a copy of the License at
-# *
-# * http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
-
-import argparse
-import json
-
-from os.path import isfile
-from os import getcwd
-
-from subprocess import call, check_call
-
-
-#######################################################################################################################
-# I put these here so it will (hopeully) be easy(er) to bump versions /
maintain
-# If there is demand, we could easily make parts or all comand line arguments
as well
-#######################################################################################################################
-tar_name = "apache-mahout-distribution-0.12.2.tar.gz"
-mahout_bin_url = "http://apache.osuosl.org/mahout/0.12.2/%s"; % tar_name
-mahout_version = "0.12.2"
-
-parser = argparse.ArgumentParser()
-
-parser.add_argument("--force_download", help="force download Apache Mahout",
action="store_true")
-parser.add_argument("--restart_later", help="force download Apache Mahout",
action="store_true")
-parser.add_argument("--zeppelin_home", help="path to ZEPPELIN_HOME")
-parser.add_argument("--mahout_home", help="path to MAHOUT_HOME, use this if
you have already installed Apache Mahout")
-parser.add_argument("--overwrite_existing", help="if %sparkMahout or
%flinkMahout exist, delete them and create new ones. Otherwise Fail.",
action="store_true")
-
-args = parser.parse_args()
-
-
-
-class ZeppelinTerpWrangler:
- def __init__(self, interpreter_json_path):
- self.interpreter_json_path = interpreter_json_path
-
- def _getTerpID(self, terpName):
- terp_id = None
- for k, v in self.interpreter_json['interpreterSettings'].iteritems():
- if v['name'] == terpName:
- terp_id = k
- break
-
- return terp_id
-
- def _terpExists(self, terpName):
- terp_id = self._getTerpID(terpName)
- if terp_id == None:
- return False
- return True
-
- def createTerp(self, original_terp_name, new_terp_name,
overwrite_existing=True ):
-
- new_terp_id = new_terp_name
- if self._terpExists(new_terp_name):
- print "Found existing '%s' interpreter..." % new_terp_name
- if overwrite_existing:
- print "deleting %s from interpreter.json" %new_terp_name
- del
self.interpreter_json['interpreterSettings'][self._getTerpID(new_terp_name)]
- else:
- print "exiting program."
- exit(1)
-
- orig_terp_id = self._getTerpID(original_terp_name)
-
- from copy import deepcopy
- self.interpreter_json['interpreterSettings'][new_terp_id] = deepcopy(
- self.interpreter_json['interpreterSettings'][orig_terp_id])
- self.interpreter_json['interpreterSettings'][new_terp_id]['name'] =
new_terp_name
- self.interpreter_json['interpreterSettings'][new_terp_id]['id'] =
new_terp_id
- print "created new interpreter '%s' from interpreter '%s" %
(new_terp_name, original_terp_name)
-
- def _readTerpJson(self):
- with open(self.interpreter_json_path) as f:
- self.interpreter_json = json.load(f)
-
- def _writeTerpJson(self):
- with open(self.interpreter_json_path, 'wb') as f:
- json.dump(self.interpreter_json, f, sort_keys=True, indent=4)
-
- def _updateTerpProp(self, terpName, property, value):
- terp_id = self._getTerpID(terpName)
-
self.interpreter_json['interpreterSettings'][terp_id]['properties'][property] =
value
-
- def _addTerpDep(self, terpName="", dep="", exclusions=None):
- if self.interpreter_json == {}:
- print "no interpreter.json loaded, reading last one downloaded"
- self._readTerpJson()
- terp_id = self._getTerpID(terpName)
- deps =
self.interpreter_json['interpreterSettings'][terp_id]['dependencies']
-
- dep_dict = {
- u'groupArtifactVersion': dep,
- u'local': False
-
- }
- if exclusions != None:
- dep_dict["exclusions"] = exclusions
- deps.append(dep_dict)
-
- ## Remove Duplicate Dependencies
- seen = set()
- new_deps = list()
- for d in deps:
- t = d.items()
- if t[0] not in seen:
- seen.add(t[0])
- new_deps.append(d)
-
- self.interpreter_json['interpreterSettings'][terp_id]['dependencies']
= new_deps
-
- def addMahoutConfig(self, terpName, mahout_home, mahout_version =
"0.12.2"):
-
- print "updating '%s' with Apache Mahout dependencies and settings" %
terpName
-
- terpDeps = ["%s/mahout-math-%s.jar" % (mahout_home, mahout_version),
- "%s/mahout-math-scala_2.10-%s.jar" % (mahout_home,
mahout_version)]
-
- if "spark" in terpName.lower():
- configs = {
- "spark.kryo.referenceTracking": "false",
- "spark.kryo.registrator":
"org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator",
- "spark.kryoserializer.buffer": "32k",
- "spark.kryoserializer.buffer.max": "600m",
- "spark.serializer":
"org.apache.spark.serializer.KryoSerializer"
- }
- terpDeps.append('%s/mahout-spark_2.10-%s-dependency-reduced.jar' %
(mahout_home, mahout_version))
- terpDeps.append("%s/mahout-spark_2.10-%s.jar" % (mahout_home,
mahout_version))
- terpDeps.append("%s/mahout-spark-shell_2.10-%s.jar" %
(mahout_home, mahout_version))
-
- if "flink" in terpName.lower():
- configs = {
- "taskmanager.numberOfTaskSlots" : "12"
- }
- addlDeps = [
- "%s/mahout-flink_2.10-%s.jar" % (mahout_home, mahout_version),
- "%s/mahout-hdfs-%s.jar" % (mahout_home, mahout_version),
- "com.google.guava:guava:14.0.1"
- #"%s/guava-14.0.1.jar" % mahout_home ## reuired in lib dir if
running against cluster
- ]
- for t in addlDeps:
- terpDeps.append(t)
-
- for k, v in configs.iteritems():
- self._updateTerpProp(terpName, k, v)
-
- for t in terpDeps:
- self._addTerpDep(terpName, t)
-
-#######################################################################################################################
-# Need to be sure we know where Zeppelin Top directory is so we can edit conf
files
-#
-#######################################################################################################################
-
-def valid_zeppelin_home(path):
- return isfile(path + "/bin/zeppelin-daemon.sh")
-
-if args.zeppelin_home == None:
- zeppelin_home = getcwd()
- if (zeppelin_home.split("/")[-1] == "bin") and
(isfile("zeppelin-daemon.sh")):
- print "we're in the zeppelin/bin"
- zeppelin_home = "/".join(zeppelin_home.split("/")[:-1])
- print "--zeppelin_home not specified, using %s" % zeppelin_home
-else:
- zeppelin_home = args.zeppelin_home
-
-
-if not valid_zeppelin_home(zeppelin_home):
- print "%s does not appear to be a valid ZEPPELIN_HOME - e.g. the top level
directory of the ZEPPELIN install" % zeppelin_home
- exit(1)
-else:
- print "ZEPPELIN_HOME validated"
-
-interpreter_json_path = zeppelin_home + "/conf/interpreter.json"
-
-if not isfile(interpreter_json_path):
- print "interpreter.json doesn't exist. Checking weather Zeppelin is
running."
- status = call(["bin/zeppelin-daemon.sh", 'status'], cwd=zeppelin_home)
- if status == 1:
- print "Zeppelin doesn't appear to be running- it is possible that
Zeppelin has never been run (interpreter.json is created when Zeppelin is run)"
- print "I'm going to try to start Zeppelin to create interpreter.json"
- call(["bin/zeppelin-daemon.sh", 'start'], cwd=zeppelin_home)
- from time import sleep
- sleep(3)
- else:
- print "We're in the correct top-level directory, Zeppelin appears to
be running, but there is no 'interpreter.json'. \
- \nThis is a confusing case. Please try restarting Zeppelin, but if
that doesn't work reach out on the mailing list."
-
-if isfile(interpreter_json_path):
- z = ZeppelinTerpWrangler(interpreter_json_path)
-else:
- print "'interpreter.json' not found in %s/conf" % args.zeppelin_home
- exit(1)
-
-#######################################################################################################################
-# If --mahout_home not set, download and untar Mahout in to ZEPPELIN_HOME
-# Set MAHOUT_HOME to ZEPPELIN_HOME/<mahout_untar_dir>
-#######################################################################################################################
-
-def download_mahout():
- if args.force_download:
- print "--force_download: OK, deleting existing tar if it exists."
- call(["rm", "%s/%s" % (zeppelin_home, tar_name)])
- return True
- elif isfile("%s/%s" % (zeppelin_home, tar_name)):
- print "%s found, skipping download" % tar_name
- return False
- elif args.mahout_home:
- print "--mahout_home set, skipping download"
- return False
- else:
- return True
-
-if download_mahout():
- check_call(['wget', mahout_bin_url], cwd= zeppelin_home)
- check_call(['tar', 'xzf', tar_name], cwd= zeppelin_home)
-
-
-
-if args.mahout_home:
- mahout_home = args.mahout_home
-else:
- mahout_home = zeppelin_home + "/" + ".".join(tar_name.split(".")[:-2])
-
-#######################################################################################################################
-# Create new interpreters
-#######################################################################################################################
-
-z._readTerpJson()
-z.createTerp("spark", "sparkMahout", args.overwrite_existing)
-z.createTerp("flink", "flinkMahout", args.overwrite_existing)
-z.addMahoutConfig("sparkMahout", mahout_home, mahout_version)
-z.addMahoutConfig("flinkMahout", mahout_home, mahout_version)
-z._writeTerpJson()
-
-#######################################################################################################################
-# Add "export MAHOUT_HOME=... to conf/zeppelin-env.sh
-# Create if doesn't exist.
-#######################################################################################################################
-
-mahout_home_str = '\nexport MAHOUT_HOME=%s\n' % (mahout_home)
-
-zeppelin_env_sh_path = '%s/conf/zeppelin-env.sh' % zeppelin_home
-if isfile(zeppelin_env_sh_path):
- with open(zeppelin_env_sh_path, 'rb') as f:
- zeppelin_env_sh = f.readlines()
- if any(["export MAHOUT_HOME=" in line for line in zeppelin_env_sh]):
- print "'export MAHOUT_HOME=...' already exists in zeppelin_env.sh, not
appending"
- else:
- print "appending '%s' to conf/zeppelin-env.sh" % mahout_home_str
- with open(zeppelin_env_sh_path, 'a') as f:
- f.write(mahout_home_str)
-else:
- print "appending '%s' to conf/zeppelin-env.sh" % mahout_home_str
- with open(zeppelin_env_sh_path, 'wb') as f:
- f.write(mahout_home_str)
-
-
-#######################################################################################################################
-# You have to restart Apache Zeppelin for new terps to show up... do this for
user unless the specified otherwise
-#
-#######################################################################################################################
-if not args.restart_later:
- print "restarting Apache Zeppelin to load new interpreters..."
- check_call(["bin/zeppelin-daemon.sh", 'restart'], cwd= zeppelin_home)
-else:
- print "--restart_later flag detected: remember to restart Zeppelin to see
new Mahout interpreters!!"
-
-#######################################################################################################################
-# Good bye
-#######################################################################################################################
-
-print
"---------------------------------------------------------------------------------------------------------------"
-print "all done! Thanks for using Apache Mahout"
-print "bye"
