[jira] [Commented] (TOREE-398) Toree - R : Error in getSparkSession(): SparkSession not initialized when using 'spark'

2017-03-22 Thread Mario Briggs (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937724#comment-15937724
 ] 

Mario Briggs commented on TOREE-398:


[~mariusvniekerk] any thoughts on above, since i see your PR handled 2.0 
support in R Interpreter

> Toree - R : Error in getSparkSession(): SparkSession not initialized when 
> using 'spark'
> ---
>
> Key: TOREE-398
> URL: https://issues.apache.org/jira/browse/TOREE-398
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.2.0
>Reporter: Mario Briggs
>
> Using Toree R interpreter with Spark 2.0.
> After kernel is initialized, 'spark' var is available 
> >>
> spark
> Out[19]:
> Java ref type org.apache.spark.sql.SparkSession id 33
> <<
> However trying to use it , ends up with this
> >>
> localDF <- data.frame(name=c("a", "b", "c"), age=c(19, 23, 18))
> df <- createDataFrame(spark, localDF)
> Out[20]:
> Message: Error in getSparkSession(): SparkSession not initialized
> <<
> Trying to do sparkR.session(), leads to eror that only one SparkContext can 
> be created
> Seem zepellin hit same issue and fixed - 
> https://issues.apache.org/jira/browse/ZEPPELIN-1260
> I see 2.0 fixes are around this line - 
> https://github.com/apache/incubator-toree/blob/master/sparkr-interpreter/src/main/resources/kernelR/sparkr_runner.R#L122



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

2017-03-22 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937248#comment-15937248
 ] 

ASF GitHub Bot commented on TOREE-395:
--

GitHub user kevin-bates reopened a pull request:

https://github.com/apache/incubator-toree/pull/116

[TOREE-395] disable cell result output from console

This issue - discussed in 
[TOREE-395](https://issues.apache.org/jira/browse/TOREE-395) (and somewhat in 
[TOREE-380](https://issues.apache.org/jira/browse/TOREE-380)) - essentially 
takes the appropriate portion of PR #104 and delivers it here.  This change 
makes the scala interpreter behave like other interpreters (and kernels) by not 
issuing the cell results to the console (where it is typically captured by the 
Jupyter logging) and could contain sensitive information.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kevin-bates/incubator-toree 
remove-results-from-output

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/116.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #116


commit aee12381ce23fe7d399aa6488486a0f750027915
Author: Kevin Bates 
Date:   2017-03-21T21:45:43Z

[TOREE-395] disable cell result output

commit e610241a7b4caeb3d013e5c195c9030ce521102f
Author: Kevin Bates 
Date:   2017-03-21T23:51:52Z

preserve list of multiOutputStream

commit 9dee732b34140b01aaed81eb04e12fbe2571bacc
Author: Kevin Bates 
Date:   2017-03-22T21:17:47Z

remove multiOutputStream relative to cell results output




> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

2017-03-22 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937247#comment-15937247
 ] 

ASF GitHub Bot commented on TOREE-395:
--

Github user kevin-bates closed the pull request at:

https://github.com/apache/incubator-toree/pull/116


> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

2017-03-22 Thread Kevin Bates (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937098#comment-15937098
 ] 

Kevin Bates commented on TOREE-395:
---

I tend to agree and will take a look.  A little worried that it will trickle 
across multiple fronts, but that may be a premature comment.

> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

2017-03-22 Thread Kevin Bates (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936992#comment-15936992
 ] 

Kevin Bates commented on TOREE-395:
---

Thank you for your input [~mariusvniekerk].  I think this gives us consensus.  
Since the PR is in this state already, I'd say its ready to merge unless there 
are further comments.

> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

2017-03-22 Thread Marius Van Niekerk (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936967#comment-15936967
 ] 

Marius Van Niekerk commented on TOREE-395:
--

I'm perfectly fine with it being removed completely.  The degree of pollution 
that spark by default puts in the jupyter log without changing your log4j is 
already intense.

> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (TOREE-398) Toree - R : Error in getSparkSession(): SparkSession not initialized when using 'spark'

2017-03-22 Thread Mario Briggs (JIRA) 
Mario Briggs created TOREE-398:
--

 Summary: Toree - R : Error in getSparkSession(): SparkSession not 
initialized when using 'spark'
 Key: TOREE-398
 URL: https://issues.apache.org/jira/browse/TOREE-398
 Project: TOREE
  Issue Type: Bug
Affects Versions: 0.2.0
Reporter: Mario Briggs


Using Toree R interpreter with Spark 2.0.

After kernel is initialized, 'spark' var is available 
>>
spark
Out[19]:
Java ref type org.apache.spark.sql.SparkSession id 33
<<

However trying to use it , ends up with this
>>
localDF <- data.frame(name=c("a", "b", "c"), age=c(19, 23, 18))
df <- createDataFrame(spark, localDF)
Out[20]:
Message: Error in getSparkSession(): SparkSession not initialized
<<

Trying to do sparkR.session(), leads to eror that only one SparkContext can be 
created

Seem zepellin hit same issue and fixed - 
https://issues.apache.org/jira/browse/ZEPPELIN-1260

I see 2.0 fixes are around this line - 
https://github.com/apache/incubator-toree/blob/master/sparkr-interpreter/src/main/resources/kernelR/sparkr_runner.R#L122



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-397) Avoid hard coding deploy mode property (spark.submit.deployMode)

2017-03-22 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936871#comment-15936871
 ] 

ASF GitHub Bot commented on TOREE-397:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/117

[TOREE-397] Avoid hardcoding the deploy mode property

By checking if a proper spark.submit.deployMode property
is available in the sparkConf enables choosing the deploy
mode by using system properties such as $SPARK_OPTS.

This change continues to use client mode if no configuration
was provided.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree toree-397

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/117.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #117


commit fd887d80fd4241ea3be77699d27227287f366a95
Author: Luciano Resende 
Date:   2017-03-22T18:26:26Z

[TOREE-397] Avoid hardcoding the deploy mode property

By checking if a proper spark.submit.deployMode property
is available in the sparkConf enables choosing the deploy
mode by using system properties such as $SPARK_OPTS.

This change continues to use client mode if no configuration
was provided.




> Avoid hard coding deploy mode property (spark.submit.deployMode)
> 
>
> Key: TOREE-397
> URL: https://issues.apache.org/jira/browse/TOREE-397
> Project: TOREE
>  Issue Type: Sub-task
>Affects Versions: 0.1.0, 0.2.0
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (TOREE-397) Avoid hard coding deploy mode property (spark.submit.deployMode)

2017-03-22 Thread Luciano Resende (JIRA) 
Luciano Resende created TOREE-397:
-

 Summary: Avoid hard coding deploy mode property 
(spark.submit.deployMode)
 Key: TOREE-397
 URL: https://issues.apache.org/jira/browse/TOREE-397
 Project: TOREE
  Issue Type: Sub-task
Affects Versions: 0.1.0, 0.2.0
Reporter: Luciano Resende
Assignee: Luciano Resende






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (TOREE-369) Enable Toree to run in Yarn cluster mode

2017-03-22 Thread Luciano Resende (JIRA) 

 [ 
https://issues.apache.org/jira/browse/TOREE-369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luciano Resende reassigned TOREE-369:
-

Assignee: Luciano Resende

> Enable Toree to run in Yarn cluster mode
> 
>
> Key: TOREE-369
> URL: https://issues.apache.org/jira/browse/TOREE-369
> Project: TOREE
>  Issue Type: Improvement
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-380) Interpreters should be allowed to send results other than text/plain.

2017-03-22 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936608#comment-15936608
 ] 

ASF GitHub Bot commented on TOREE-380:
--

Github user kevin-bates commented on a diff in the pull request:

https://github.com/apache/incubator-toree/pull/104#discussion_r107460553
  
--- Diff: 
scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
 ---
@@ -18,37 +18,45 @@
 package org.apache.toree.kernel.interpreter.scala
 
 import java.io.ByteArrayOutputStream
-import java.net.{URL, URLClassLoader}
-import java.nio.charset.Charset
 import java.util.concurrent.ExecutionException
 
 import com.typesafe.config.{Config, ConfigFactory}
 import org.apache.spark.SparkContext
-import org.apache.spark.sql.SparkSession
 import org.apache.spark.repl.Main
+import org.apache.spark.sql.SparkSession
 
 import org.apache.toree.interpreter._
-import org.apache.toree.kernel.api.{KernelLike, KernelOptions}
+import org.apache.toree.kernel.api.KernelLike
 import org.apache.toree.utils.{MultiOutputStream, TaskManager}
 import org.slf4j.LoggerFactory
 import org.apache.toree.kernel.BuildInfo
 
 import scala.annotation.tailrec
+import scala.collection.mutable
 import scala.concurrent.{Await, Future}
 import scala.language.reflectiveCalls
 import scala.tools.nsc.Settings
 import scala.tools.nsc.interpreter.{IR, OutputStream}
 import scala.tools.nsc.util.ClassPath
-import scala.util.{Try => UtilTry}
+
+import org.apache.toree.kernel.protocol.v5.MIMEType
+import org.apache.toree.magic.MagicOutput
+import vegas.DSL.ExtendedUnitSpecBuilder
+import vegas.render.StaticHTMLRenderer
+
 
 class ScalaInterpreter(private val config:Config = ConfigFactory.load) 
extends Interpreter with ScalaInterpreterSpecific {
+  import ScalaInterpreter._
+
+   private var kernel: KernelLike = _
+
protected val logger = LoggerFactory.getLogger(this.getClass.getName)
 
protected val _thisClassloader = this.getClass.getClassLoader
 
protected val lastResultOut = new ByteArrayOutputStream()
 
-   protected val multiOutputStream = MultiOutputStream(List(Console.out, 
lastResultOut))
+   protected val multiOutputStream = lastResultOut
--- End diff --

At a minimum, this needs to remain a MultiOutputStream (of one item) else 
it will produce a compilation error in 
[here](https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/test/scala/integration/interpreter/scala/AddExternalJarMagicSpecForIntegration.scala#L41).
  Removal of Console.out is further discussed in #116.


> Interpreters should be allowed to send results other than text/plain.
> -
>
> Key: TOREE-380
> URL: https://issues.apache.org/jira/browse/TOREE-380
> Project: TOREE
>  Issue Type: Improvement
>Reporter: Ryan Blue
>
> Jupyter allows kernels to send results using different content types, but 
> this isn't allowed by Toree for its interpreters. This prevents custom 
> display logic. The basic problem is that {{ExecuteOutput}} is a {{String}} 
> and not a {{Map[String, String]}} like {{CellMagicOutput}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

2017-03-22 Thread Ryan Blue (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936567#comment-15936567
 ] 

Ryan Blue commented on TOREE-395:
-

I don't think there is a reason to keep it. When using Toree with Jupyter 
notebooks, the user doesn't see the second copy of results. That opens up the 
possibility to print sensitive data as was already pointed out, but, more 
importantly, it pollutes the notebook server log with results that aren't 
useful when you go to look at it. Is there a use case for notebooks that you 
can think of that makes this a useful feature?

The other environment to think about is Jupyter console. When you use Toree 
with the console, everything gets printed as cell output (which my PR 
addresses) and is printed to stdout. That it produces double output is the main 
reason why we disabled it.

> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-396) SparkR interpeter does not connect to Spark 2.1

2017-03-22 Thread Chip Senkbeil (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936457#comment-15936457
 ] 

Chip Senkbeil commented on TOREE-396:
-

Thanks for pointing this out. Looks like the timeout was previously set to 6000.

> SparkR interpeter does not connect to Spark 2.1
> ---
>
> Key: TOREE-396
> URL: https://issues.apache.org/jira/browse/TOREE-396
> Project: TOREE
>  Issue Type: Bug
>Reporter: Mario Briggs
>
> I was trying the SparkR interpreter and trying to connect to a Spark 2.1 
> cluster. I get the following failure when execute any cell
> "Failed to connect JVM: Error in socketConnection(host = hostname, port = 
> port, server = FALSE, : argument "timeout" is missing, with no default
> Calls: sparkR.connect ... tryCatch -> tryCatchList -> tryCatchOne -> 
> 
> "
> Looks like in Spark2.1, the 
> [timeout|https://github.com/apache/spark/blob/branch-2.1/R/pkg/R/client.R#L22]
>  param to sparkRConnect no longer has a default value. 
> So this needs to fixed in 
> [here|https://github.com/apache/incubator-toree/blob/master/sparkr-interpreter/src/main/resources/kernelR/sparkr_runner.R#L65]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

2017-03-22 Thread Chip Senkbeil (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936446#comment-15936446
 ] 

Chip Senkbeil commented on TOREE-395:
-

I commented on the PR, but yeah I'm in favor of leaving it as an option with 
the default condition being disabled UNLESS one of you has a reason you believe 
that Console.out should be removed entirely.

> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

2017-03-22 Thread Kevin Bates (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936434#comment-15936434
 ] 

Kevin Bates edited comment on TOREE-395 at 3/22/17 2:48 PM:


[~rdblue], [~senkwich] feels removing the ability to capture cell results is a 
bit extreme - so I think we should reach an agreement here.  How about I revert 
back to my original proposal by making inclusion of Console.out conditional 
based on debug tracing?  Then, when you've got your WIP PR in a more complete 
state, we could revisit its removal - since perhaps the cell capture is part of 
the pluggable display apparatus.

My primary stance is that, by default, we NOT capture cell results and that 
enabling such capture essentially requires local filesystem access (so I'd 
prefer the debug condition over a command line condition).

Thoughts?  Comments?


was (Author: kbates):
[~rdblue] Chip feels removing the ability to capture cell results is a bit 
extreme - so I think we should reach an agreement here.  How about I revert 
back to my original proposal by making inclusion of Console.out conditional 
based on debug tracing?  Then, when you've got your WIP PR in a more complete 
state, we could revisit its removal - since perhaps the cell capture is part of 
the pluggable display apparatus.

My primary stance is that, by default, we NOT capture cell results and that 
enabling such capture essentially requires local filesystem access (so I'd 
prefer the debug condition over a command line condition).

Thoughts?  Comments?

> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

2017-03-22 Thread Kevin Bates (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936434#comment-15936434
 ] 

Kevin Bates commented on TOREE-395:
---

[~rdblue] Chip feels removing the ability to capture cell results is a bit 
extreme - so I think we should reach an agreement here.  How about I revert 
back to my original proposal by making inclusion of Console.out conditional 
based on debug tracing?  Then, when you've got your WIP PR in a more complete 
state, we could revisit its removal - since perhaps the cell capture is part of 
the pluggable display apparatus.

My primary stance is that, by default, we NOT capture cell results and that 
enabling such capture essentially requires local filesystem access (so I'd 
prefer the debug condition over a command line condition).

Thoughts?  Comments?

> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (TOREE-396) SparkR interpeter does not connect to Spark 2.1

2017-03-22 Thread Mario Briggs (JIRA) 
Mario Briggs created TOREE-396:
--

 Summary: SparkR interpeter does not connect to Spark 2.1
 Key: TOREE-396
 URL: https://issues.apache.org/jira/browse/TOREE-396
 Project: TOREE
  Issue Type: Bug
Reporter: Mario Briggs


I was trying the SparkR interpreter and trying to connect to a Spark 2.1 
cluster. I get the following failure when execute any cell

"Failed to connect JVM: Error in socketConnection(host = hostname, port = port, 
server = FALSE, : argument "timeout" is missing, with no default
Calls: sparkR.connect ... tryCatch -> tryCatchList -> tryCatchOne -> 
"
Looks like in Spark2.1, the 
[timeout|https://github.com/apache/spark/blob/branch-2.1/R/pkg/R/client.R#L22] 
param to sparkRConnect no longer has a default value. 
So this needs to fixed in 
[here|https://github.com/apache/incubator-toree/blob/master/sparkr-interpreter/src/main/resources/kernelR/sparkr_runner.R#L65]





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)