[
https://issues.apache.org/jira/browse/MAHOUT-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966097#comment-13966097
]
Dmitriy Lyubimov commented on MAHOUT-1489:
------------------------------------------
Ok. this is way too complicated for me. I did a quick hack \[1] which seems to
work at least in local mode . Works like a charm. Here is the session dump
(need just to start o.a.m.sparkbindings.shell.Main class from idea). I also
filtered out most of spark debug messages that are enabled by default:
{code:title="Mahout Spark Shell session"}
14/04/10 17:52:27 INFO spark.HttpServer: Starting HTTP Server
14/04/10 17:52:27 INFO server.Server: jetty-7.6.8.v20121106
14/04/10 17:52:27 INFO server.AbstractConnector: Started
[email protected]:60204
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 0.9.0
/_/
Using Scala version 2.10.3 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51)
Type in expressions to have them evaluated.
Type :help for more information.
Created spark context..
Spark context available as sc.
scala> val a = dense((1,2,3),(3,4,5))
a: org.apache.mahout.math.DenseMatrix =
{
0 => {0:1.0,1:2.0,2:3.0}
1 => {0:3.0,1:4.0,2:5.0}
}
scala> val drmA = drmParallelize(a)
drmA: org.apache.mahout.sparkbindings.drm.CheckpointedDrm[Int] =
org.apache.mahout.sparkbindings.drm.CheckpointedDrmBase@791b95a1
scala> val drmAtA = drmA.t %*% drmA
drmAtA: org.apache.mahout.sparkbindings.drm.DrmLike[Int] =
OpAB(OpAt(org.apache.mahout.sparkbindings.drm.CheckpointedDrmBase@791b95a1),org.apache.mahout.sparkbindings.drm.CheckpointedDrmBase@791b95a1)
scala> drmAtA.collect
res0: org.apache.mahout.math.Matrix =
{
0 => {0:10.0,1:14.0,2:18.0}
1 => {0:14.0,1:20.0,2:26.0}
2 => {0:18.0,1:26.0,2:34.0}
}
scala>
{code}
I suggest you to fork my branch, take it as a basis. It basically works, now
the stuff that needs to happen is to verify it in distributed mode and modify
mahout shell script to set up proper paths etc. to launch it w.r.t "mahout
shell <master>" command. Should be simple an uneventful enough now.
\[1\]: https://github.com/dlyubimov/mahout-commits/tree/shell
> Interactive Scala & Spark Bindings Shell & Script processor
> -----------------------------------------------------------
>
> Key: MAHOUT-1489
> URL: https://issues.apache.org/jira/browse/MAHOUT-1489
> Project: Mahout
> Issue Type: New Feature
> Affects Versions: 1.0
> Reporter: Saikat Kanjilal
> Assignee: Dmitriy Lyubimov
> Fix For: 1.0
>
>
> Build an interactive shell /scripting (just like spark shell). Something very
> similar in R interactive/script runner mode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
