[
https://issues.apache.org/jira/browse/MAHOUT-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14001041#comment-14001041
]
Anand Avati commented on MAHOUT-1544:
-------------------------------------
[~ssc] - I investigated how Spark is achieving the task of exposing user
defined data types and closures to the server. It turns out, Spark is using a
heavily refactored and modified version of the Scala 2.10 REPL. Those
modifications are now part of Scala 2.11 (in the form of two options:
Yclass-based-repl and Yrepl-outdir). So if Mahout were to use Scala 2.11 (which
in turn depends on Spark to be available in 2.11 Scala) then the proposal in
this JIRA can be achieved with much more simplicity (It also turns out that
Spark's REPL itself can be made much simpler when moved to Scala 2.11, for the
same reasons). However right now I'm trying to get Spark's dependencies in
Scala 2.11.
So in any case, this proposal can be achieved at the earliest with Spark 1.1
(Spark devs are considering Scala 2.11 support in version 1.1). To do it any
earlier, we would need to inherit a refactored and modified version of Scala
REPL, just like how Spark is doing - just not worth the effort.
What's the right "process" here now? Close this JIRA? Or leave it open with a
decreased priority?
> make Mahout DSL shell depend dynamically on Spark
> -------------------------------------------------
>
> Key: MAHOUT-1544
> URL: https://issues.apache.org/jira/browse/MAHOUT-1544
> Project: Mahout
> Issue Type: Improvement
> Reporter: Anand Avati
> Fix For: 1.0
>
> Attachments: 0001-spark-shell-rename-to-shell.patch,
> 0002-shell-make-dependency-on-Spark-optional-and-dynamic.patch,
> 0002-shell-make-dependency-on-Spark-optional-and-dynamic.patch,
> 0002-shell-make-dependency-on-Spark-optional-and-dynamic.patch
>
>
> Today the Mahout's scala shell depends on spark.
> Create a cleaner separation between the shell and Spark. For e.g, the in core
> scalabindings and operators do not need Spark. So make Spark a runtime
> "addon" to the shell. Similarly in the future new distributed backend engines
> can transparently (dynamically) be available through the DSL shell.
> The new shell works, looks and feels exactly like the shell before, but has a
> cleaner modular architecture.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
