[
https://issues.apache.org/jira/browse/MAHOUT-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14619697#comment-14619697
]
ASF GitHub Bot commented on MAHOUT-1653:
----------------------------------------
Github user dlyubimov commented on a diff in the pull request:
https://github.com/apache/mahout/pull/146#discussion_r34216367
--- Diff:
spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala
---
@@ -48,55 +83,87 @@ class MahoutSparkILoop extends SparkILoop {
conf.set("spark.executor.memory", "1g")
- sparkContext = mahoutSparkContext(
+ sdc = mahoutSparkContext(
masterUrl = master,
appName = "Mahout Spark Shell",
customJars = jars,
sparkConf = conf
)
- echo("Created spark context..")
+ _interp.sparkContext = sdc
+
+ echoToShell("Created spark context..")
sparkContext
}
+ // this is technically not part of Spark's explicitly defined Developer
API though
+ // nothing in the SparkILoopInit.scala file is marked as such.
override def initializeSpark() {
- intp.beQuietDuring {
- command("""
- @transient implicit val sdc:
org.apache.mahout.math.drm.DistributedContext =
- new org.apache.mahout.sparkbindings.SparkDistributedContext(
- org.apache.spark.repl.Main.interp.createSparkContext())
+ _interp.beQuietDuring {
+
+ // get the spark context, at the same time create and store a mahout
distributed context.
+ _interp.interpret("""
+ @transient val sc = {
--- End diff --
i honestly don't know if spark declared it as implicit. if it does not, we
don't have to either i suppose.
the reason mahout context is implicit is because all top-level routines
like drmParallelize(...) look for it.
On Wed, Jul 8, 2015 at 1:35 PM, Andrew Palumbo <[email protected]>
wrote:
> In
>
spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala
> <https://github.com/apache/mahout/pull/146#discussion_r34195169>:
>
> >
> > - @transient implicit val sdc:
org.apache.mahout.math.drm.DistributedContext =
> > - new
org.apache.mahout.sparkbindings.SparkDistributedContext(
> > - org.apache.spark.repl.Main.interp.createSparkContext())
> > + _interp.beQuietDuring {
> > +
> > + // get the spark context, at the same time create and store a
mahout distributed context.
> > + _interp.interpret("""
> > + @transient val sc = {
>
> @dlyubimov <https://github.com/dlyubimov> I've made the necessary changes
> clean up the the redundant creation of a SparkDistributedContext. Thanks
> alot for the input.
>
> You mentioned is that we could declare the SparkContext as implicit val
> sc = .... Is there a reason that it should be implicit?
>
> I've left it as val sc =... for now since that is the way Spark declares
> is in this method. Thx.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/mahout/pull/146/files#r34195169>.
>
> Spark 1.3
> ---------
>
> Key: MAHOUT-1653
> URL: https://issues.apache.org/jira/browse/MAHOUT-1653
> Project: Mahout
> Issue Type: Dependency upgrade
> Reporter: Andrew Musselman
> Assignee: Andrew Palumbo
> Fix For: 0.11.0
>
>
> Support Spark 1.3
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
