[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[ https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936434#comment-15936434 ] Kevin Bates commented on TOREE-395: --- [~rdblue] Chip feels removing the ability to capture cell results is a bit extreme - so I think we should reach an agreement here. How about I revert back to my original proposal by making inclusion of Console.out conditional based on debug tracing? Then, when you've got your WIP PR in a more complete state, we could revisit its removal - since perhaps the cell capture is part of the pluggable display apparatus. My primary stance is that, by default, we NOT capture cell results and that enabling such capture essentially requires local filesystem access (so I'd prefer the debug condition over a command line condition). Thoughts? Comments? > Provide a way to disable automatic printing of results in Toree Scala > - > > Key: TOREE-395 > URL: https://issues.apache.org/jira/browse/TOREE-395 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Kun Liu > > Scala REPL supports a flag, ":silent" to enable/disable automatic printing of > results (those messages begin with "res"). This is also supported in Spark > shell. But for Toree Scala kernel, this flag is not supported. Thus the > result of a cell would be always printed. > But if there is any logging mechanism, the results would be recorded in a log > file, while this may not be desirable. For instance, a user may not want any > sensitive data logged when running "dataRDD.take(5)" in a cell. > Also found the possible source codes for this: > https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala > There are three methods with silent: Boolean = false as parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[ https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936434#comment-15936434 ] Kevin Bates edited comment on TOREE-395 at 3/22/17 2:48 PM: [~rdblue], [~senkwich] feels removing the ability to capture cell results is a bit extreme - so I think we should reach an agreement here. How about I revert back to my original proposal by making inclusion of Console.out conditional based on debug tracing? Then, when you've got your WIP PR in a more complete state, we could revisit its removal - since perhaps the cell capture is part of the pluggable display apparatus. My primary stance is that, by default, we NOT capture cell results and that enabling such capture essentially requires local filesystem access (so I'd prefer the debug condition over a command line condition). Thoughts? Comments? was (Author: kbates): [~rdblue] Chip feels removing the ability to capture cell results is a bit extreme - so I think we should reach an agreement here. How about I revert back to my original proposal by making inclusion of Console.out conditional based on debug tracing? Then, when you've got your WIP PR in a more complete state, we could revisit its removal - since perhaps the cell capture is part of the pluggable display apparatus. My primary stance is that, by default, we NOT capture cell results and that enabling such capture essentially requires local filesystem access (so I'd prefer the debug condition over a command line condition). Thoughts? Comments? > Provide a way to disable automatic printing of results in Toree Scala > - > > Key: TOREE-395 > URL: https://issues.apache.org/jira/browse/TOREE-395 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Kun Liu > > Scala REPL supports a flag, ":silent" to enable/disable automatic printing of > results (those messages begin with "res"). This is also supported in Spark > shell. But for Toree Scala kernel, this flag is not supported. Thus the > result of a cell would be always printed. > But if there is any logging mechanism, the results would be recorded in a log > file, while this may not be desirable. For instance, a user may not want any > sensitive data logged when running "dataRDD.take(5)" in a cell. > Also found the possible source codes for this: > https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala > There are three methods with silent: Boolean = false as parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[ https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937098#comment-15937098 ] Kevin Bates commented on TOREE-395: --- I tend to agree and will take a look. A little worried that it will trickle across multiple fronts, but that may be a premature comment. > Provide a way to disable automatic printing of results in Toree Scala > - > > Key: TOREE-395 > URL: https://issues.apache.org/jira/browse/TOREE-395 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Kun Liu > > Scala REPL supports a flag, ":silent" to enable/disable automatic printing of > results (those messages begin with "res"). This is also supported in Spark > shell. But for Toree Scala kernel, this flag is not supported. Thus the > result of a cell would be always printed. > But if there is any logging mechanism, the results would be recorded in a log > file, while this may not be desirable. For instance, a user may not want any > sensitive data logged when running "dataRDD.take(5)" in a cell. > Also found the possible source codes for this: > https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala > There are three methods with silent: Boolean = false as parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[ https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936992#comment-15936992 ] Kevin Bates commented on TOREE-395: --- Thank you for your input [~mariusvniekerk]. I think this gives us consensus. Since the PR is in this state already, I'd say its ready to merge unless there are further comments. > Provide a way to disable automatic printing of results in Toree Scala > - > > Key: TOREE-395 > URL: https://issues.apache.org/jira/browse/TOREE-395 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Kun Liu > > Scala REPL supports a flag, ":silent" to enable/disable automatic printing of > results (those messages begin with "res"). This is also supported in Spark > shell. But for Toree Scala kernel, this flag is not supported. Thus the > result of a cell would be always printed. > But if there is any logging mechanism, the results would be recorded in a log > file, while this may not be desirable. For instance, a user may not want any > sensitive data logged when running "dataRDD.take(5)" in a cell. > Also found the possible source codes for this: > https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala > There are three methods with silent: Boolean = false as parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933729#comment-15933729
]
Kevin Bates edited comment on TOREE-395 at 3/20/17 11:20 PM:
-
Changing the initialization of {{multiOutputStream}} in ScalInterpreter.scala
from:
{{protected val multiOutputStream = MultiOutputStream(List(Console.out,
lastResultOut))}}
to
protected val multiOutputStream = MultiOutputStream(List(new
ConditionalOutputStream(Console.out, logger.isDebugEnabled), lastResultOut))
appears to do the trick. My concern is that it would require developers to
enable debug in order to see the output they see today. However, I also feel
its best to error on the side of not logging potentially sensitive information
by default.
If we took this approach, we could add the following lines to log4j.properties
so that only the scala interpreter could be enabled for debug.
{{# Change the following to DEBUG to see interpreter results in log}}
{{log4j.logger.org.apache.toree.kernel.interpreter.scala=INFO}}
Another option is to add an interpreter argument to indicate that scala
interpreter results should be produced (or should be silent - depending on what
the default should be).
was (Author: kbates):
Changing the initialization of {{multiOutputStream}} in ScalInterpreter.scala
from:
{{protected val multiOutputStream = MultiOutputStream(List(Console.out,
lastResultOut))}}
to
{{protected val multiOutputStream = MultiOutputStream(List(new
ConditionalOutputStream(Console.out, logger.isDebugEnabled), lastResultOut))}}
appears to do the trick. My concern is that it would require developers to
enable debug in order to see the output they see today. However, I also feel
its best to error on the side of not logging potentially sensitive information
by default.
Another option is to add an interpreter argument to indicate that scala
interpreter results should be produced (or should be silent - depending on what
the default should be).
> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of
> results (those messages begin with "res"). This is also supported in Spark
> shell. But for Toree Scala kernel, this flag is not supported. Thus the
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log
> file, while this may not be desirable. For instance, a user may not want any
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933109#comment-15933109
]
Kevin Bates commented on TOREE-395:
---
I agree that this could be considered a security issue - particularly if the
logged output is on a different server than the notebook (e.g., using JKG with
nb2kg).
This appears to be a function of how the output stream is initialized.
Currently it's setup using both {{Console.out}} and {{ByteArrayOutputStream}}.
However, if I remove {{Console.out}} from the initialization of
{{multiOutputStream}} in ScalaInterpreter.scala I still get the results in the
notebook but do not see the results in the output logs.
I would propose that we change the initialization of {{multiOutputStream}} to
only add {{Console.out}} if the logger reflects that debug is enabled or add a
specific option that prevents such logged output.
Comments?
> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of
> results (those messages begin with "res"). This is also supported in Spark
> shell. But for Toree Scala kernel, this flag is not supported. Thus the
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log
> file, while this may not be desirable. For instance, a user may not want any
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933729#comment-15933729
]
Kevin Bates edited comment on TOREE-395 at 3/21/17 3:20 PM:
Changing the initialization of {{multiOutputStream}} in ScalInterpreter.scala
from:
{{protected val multiOutputStream = MultiOutputStream(List(Console.out,
lastResultOut))}}
to
protected val multiOutputStream = MultiOutputStream(List(new
ConditionalOutputStream(Console.out, logger.isDebugEnabled), lastResultOut))
appears to do the trick. My concern is that it would require developers to
enable debug in order to see the output they see today. However, I also feel
its best to error on the side of not logging potentially sensitive information
by default.
If we took this approach, we could add the following lines to log4j.properties
so that only the scala interpreter could be enabled for debug.
{{# Change the following to DEBUG to see interpreter results in log}}
{{log4j.logger.org.apache.toree.kernel.interpreter.scala=INFO}}
Other options are to add an interpreter argument to indicate that scala
interpreter results should be produced (or should be silent - depending on what
the default should be) or remove the code to post to the console altogether
(and match the other interpreters).
was (Author: kbates):
Changing the initialization of {{multiOutputStream}} in ScalInterpreter.scala
from:
{{protected val multiOutputStream = MultiOutputStream(List(Console.out,
lastResultOut))}}
to
protected val multiOutputStream = MultiOutputStream(List(new
ConditionalOutputStream(Console.out, logger.isDebugEnabled), lastResultOut))
appears to do the trick. My concern is that it would require developers to
enable debug in order to see the output they see today. However, I also feel
its best to error on the side of not logging potentially sensitive information
by default.
If we took this approach, we could add the following lines to log4j.properties
so that only the scala interpreter could be enabled for debug.
{{# Change the following to DEBUG to see interpreter results in log}}
{{log4j.logger.org.apache.toree.kernel.interpreter.scala=INFO}}
Another option is to add an interpreter argument to indicate that scala
interpreter results should be produced (or should be silent - depending on what
the default should be).
> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of
> results (those messages begin with "res"). This is also supported in Spark
> shell. But for Toree Scala kernel, this flag is not supported. Thus the
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log
> file, while this may not be desirable. For instance, a user may not want any
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[ https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934841#comment-15934841 ] Kevin Bates commented on TOREE-395: --- This is great news Ryan - thanks for the update. I apologize for not taking a closer look at the referenced PR. The more I thought about things and realized that the other interpreters (Toree and otherwise) didn't produce this output, the more I leaned to removing this capability entirely. When do you anticipate this PR getting completed? > Provide a way to disable automatic printing of results in Toree Scala > - > > Key: TOREE-395 > URL: https://issues.apache.org/jira/browse/TOREE-395 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Kun Liu > > Scala REPL supports a flag, ":silent" to enable/disable automatic printing of > results (those messages begin with "res"). This is also supported in Spark > shell. But for Toree Scala kernel, this flag is not supported. Thus the > result of a cell would be always printed. > But if there is any logging mechanism, the results would be recorded in a log > file, while this may not be desirable. For instance, a user may not want any > sensitive data logged when running "dataRDD.take(5)" in a cell. > Also found the possible source codes for this: > https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala > There are three methods with silent: Boolean = false as parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-437) Cell interrupts do not occur in background
Kevin Bates created TOREE-437: - Summary: Cell interrupts do not occur in background Key: TOREE-437 URL: https://issues.apache.org/jira/browse/TOREE-437 Project: TOREE Issue Type: Improvement Components: Kernel Affects Versions: 0.2.0 Reporter: Kevin Bates Assignee: Kevin Bates Whenever Toree is run in the background - either directly from the shell or indirectly from the Jupyter stack that is started in the background - cell interrupts (via ctrl-C or SIGINT from parent) are not received - resulting in the inability to interrupt long-running cells. This can be most simply demonstrated by invoking run.sh into the background (e.g., run.sh &) then issue ctrl-C (or `kill -2 `) to no avail. This is related to [TOREE-33] but only pertains to cell interrupt functionality since complete life-cycle management is assumed by the parent for the shutdown (double ctrl-C) scenario. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TOREE-437) Cell interrupts do not occur in background
[ https://issues.apache.org/jira/browse/TOREE-437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Bates updated TOREE-437: -- Description: Whenever Toree is run in the background - either directly from the shell or indirectly from the Jupyter stack that is started in the background - cell interrupts (via ctrl-C or SIGINT from parent) are not received - resulting in the inability to interrupt long-running cells. This can be most simply demonstrated by invoking run.sh into the background (e.g., run.sh &) then issue ctrl-C (or `kill -2 `) to no avail. This is related to TOREE-33 but only pertains to cell interrupt functionality since complete life-cycle management is assumed by the parent for the shutdown (double ctrl-C) scenario. was: Whenever Toree is run in the background - either directly from the shell or indirectly from the Jupyter stack that is started in the background - cell interrupts (via ctrl-C or SIGINT from parent) are not received - resulting in the inability to interrupt long-running cells. This can be most simply demonstrated by invoking run.sh into the background (e.g., run.sh &) then issue ctrl-C (or `kill -2 `) to no avail. This is related to [TOREE-33] but only pertains to cell interrupt functionality since complete life-cycle management is assumed by the parent for the shutdown (double ctrl-C) scenario. > Cell interrupts do not occur in background > -- > > Key: TOREE-437 > URL: https://issues.apache.org/jira/browse/TOREE-437 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Kevin Bates >Assignee: Kevin Bates > Labels: usability > > Whenever Toree is run in the background - either directly from the shell or > indirectly from the Jupyter stack that is started in the background - cell > interrupts (via ctrl-C or SIGINT from parent) are not received - resulting in > the inability to interrupt long-running cells. > This can be most simply demonstrated by invoking run.sh into the background > (e.g., run.sh &) then issue ctrl-C (or `kill -2 `) to no avail. > This is related to TOREE-33 but only pertains to cell interrupt functionality > since complete life-cycle management is assumed by the parent for the > shutdown (double ctrl-C) scenario. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TOREE-456) Convert env variable TOREE_ALTERNATE_SIGINT to command-line option
[
https://issues.apache.org/jira/browse/TOREE-456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Bates updated TOREE-456:
--
Description:
The recently added support (TOREE-437) for an alternate interrupt signal was
implemented using the environment variable {{TOREE_ALTERNATE_SIGINT}}.
However, we've recently learned that support for {{SPARK_YARN_USER_ENV}} is
being removed in Spark 2.2 and its via that variable that
{{TOREE_ALTERNATE_SIGINT}} is conveyed to the Toree kernel when launched as a
Yarn cluster application (_cluster mode_).
As a result, and based on similar discussion on TOREE-443, it would be better
to convey the need for an alternate interrupt signal via a command line option.
This way, all launch modes of Toree could set this option in the same manner -
via {{___TOREE_OPTS___}} in {{kernel.json}}, for example.
The command-line option should be something like:
{{--alternate-sigint=}}
with a description of something like:
{{Specifies the signal to use instead of SIGINT for interrupting a long-running
cell. The value is a string and does not include the SIG prefix. Use of USR2
is recommended.}}
was:
The recently added support (TOREE-437) for an alternate interrupt signal was
implemented using the environment variable {{TOREE_ALTERNATE_SIGINT}}.
However, we've recently learned that support for {{SPARK_YARN_USER_ENV}} is
being removed in Spark 2.2 and its via that variable that
{{TOREE_ALTERNATE_SIGINT}} is conveyed to the Toree kernel when launched as a
Yarn cluster application (_cluster mode_).
As a result, and based on similar discussion on TOREE-443, it would be better
to convey the need for an alternate interrupt signal via a command line option.
This way, all launch modes of Toree could set this option in the same manner -
via {{__TOREE_OPTS__}} in {{kernel.json}}, for example.
The command-line option should be something like:
{{--alternate-sigint=}}
with a description of something like:
{{Specifies the signal to use instead of SIGINT for interrupting a long-running
cell. The value is a string and does not include the SIG prefix. Use of USR2
is recommended.}}
> Convert env variable TOREE_ALTERNATE_SIGINT to command-line option
> --
>
> Key: TOREE-456
> URL: https://issues.apache.org/jira/browse/TOREE-456
> Project: TOREE
> Issue Type: Improvement
> Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Kevin Bates
>Assignee: Sanjay Saxena
>Priority: Minor
>
> The recently added support (TOREE-437) for an alternate interrupt signal was
> implemented using the environment variable {{TOREE_ALTERNATE_SIGINT}}.
> However, we've recently learned that support for {{SPARK_YARN_USER_ENV}} is
> being removed in Spark 2.2 and its via that variable that
> {{TOREE_ALTERNATE_SIGINT}} is conveyed to the Toree kernel when launched as a
> Yarn cluster application (_cluster mode_).
> As a result, and based on similar discussion on TOREE-443, it would be better
> to convey the need for an alternate interrupt signal via a command line
> option. This way, all launch modes of Toree could set this option in the
> same manner - via {{___TOREE_OPTS___}} in {{kernel.json}}, for example.
> The command-line option should be something like:
> {{--alternate-sigint=}}
> with a description of something like:
> {{Specifies the signal to use instead of SIGINT for interrupting a
> long-running cell. The value is a string and does not include the SIG
> prefix. Use of USR2 is recommended.}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TOREE-497) Fail to executing empty cell
[
https://issues.apache.org/jira/browse/TOREE-497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791773#comment-16791773
]
Kevin Bates commented on TOREE-497:
---
Thank you - interesting. So the notebook front end must prevent empty cells
from being sent in the first place.
And you're using Toree as s Scala kernel?
> Fail to executing empty cell
>
>
> Key: TOREE-497
> URL: https://issues.apache.org/jira/browse/TOREE-497
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
>Reporter: Manu Zhang
>Priority: Major
>
> Executing empty cell would throw
> {code:java}
> java.lang.ClassNotFoundException: java.lang.String cannot be cast to
> scala.collection.immutable.Map
> {code}
> at {{ExecuteRequestRelay$packageFutureResponse}}
> It seems {{Map}} is expected for {{ExecuteResult}} but a {{String}} is
> returned when executing empty cell.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TOREE-497) Fail to executing empty cell
[
https://issues.apache.org/jira/browse/TOREE-497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790830#comment-16790830
]
Kevin Bates commented on TOREE-497:
---
I'm unable to reproduce this using Notebook/Lab. However, it doesn't appear as
though its even sending an empty cell's content for execution.
How are you producing this issue? If not via Notebook, can you please describe
your client application? If you are using Notebook, please provide version
information and whether you're using a jupyter gateway with NB2KG or not.
> Fail to executing empty cell
>
>
> Key: TOREE-497
> URL: https://issues.apache.org/jira/browse/TOREE-497
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
>Reporter: Manu Zhang
>Priority: Major
>
> Executing empty cell would throw
> {code:java}
> java.lang.ClassNotFoundException: java.lang.String cannot be cast to
> scala.collection.immutable.Map
> {code}
> at {{ExecuteRequestRelay$packageFutureResponse}}
> It seems {{Map}} is expected for {{ExecuteResult}} but a {{String}} is
> returned when executing empty cell.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (TOREE-503) Cannot Use Scala Enumerations as Method Arguments
[
https://issues.apache.org/jira/browse/TOREE-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913444#comment-16913444
]
Kevin Bates edited comment on TOREE-503 at 8/22/19 5:00 PM:
Hi Alexander - thank you for opening the issue.
I can reproduce this on Spark 2.3 systems. However, running Toree against
Spark 2.4 does *not* reproduce the issue. I see that emr-5.10 uses Spark 2.2.
Are you in a position to try against
[emr-5.20+|https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-5x.html]
(which uses Spark 2.4)?
Interestingly enough, the issue doesn't reproduce in any version of the
spark-shell, but I suspect there could be some subtlety where the Spark 2.4
REPL happens to address this.
was (Author: kbates):
Hi Alexander - thank you opening the issue.
I can reproduce this on Spark 2.3 systems. However, running Toree against
Spark 2.4 does *not* reproduce the issue. I see that emr-5.10 uses Spark 2.2.
Are you in a position to try against
[emr-5.20+|https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-5x.html]
(which uses Spark 2.4)?
Interestingly enough, the issue doesn't reproduce in any version of the
spark-shell, but I suspect there could be some subtlety where the Spark 2.4
REPL happens to address this.
> Cannot Use Scala Enumerations as Method Arguments
> -
>
> Key: TOREE-503
> URL: https://issues.apache.org/jira/browse/TOREE-503
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
> Environment: Amazon EMR Release emr-5.10.0, JupyterLab 0.35.4
>Reporter: Alexander Gunter
>Priority: Major
>
> Under common circumstances, cells containing method calls where an argument
> is an Enumeration will fail to compile when using the Toree kernel in
> JupyterLab. The circumstances relate to the scoping between notebook cells.
> The arrangement below will fail to compile, producing the shown output.
>
> _CELL 1:_
> {code:java}
> object ExampleEnum extends Enumeration {
> type ExampleEnum = Value
> val A, B, C = Value
> }{code}
> {code:java}
> defined object ExampleEnum
> {code}
>
> _CELL 2:_
> {code:java}
> def exampleMethod(enum: ExampleEnum.Value): Unit = {
> println(enum.toString)
> }
> exampleMethod(ExampleEnum.A){code}
> {code:java}
> exampleMethod: (enum: ExampleEnum.Value)Unit
> A
> {code}
>
> _CELL 3:_
> {code:java}
> exampleMethod(ExampleEnum.A){code}
> {code:java}
> Name: Compile Error
> Message: :30: error: type mismatch;
> found : ExampleEnum.Value
> required: ExampleEnum.Value
>exampleMethod(ExampleEnum.A)
> ^
> StackTrace:
> {code}
>
> Note that exampleMethod() works as expected when called inside the same cell
> as its definition (demo'd in Cell 2). It also works as expected in all cells
> if it is instead defined in Cell 1 alongside ExampleEnum (i.e. if you merge
> Cell1 and Cell 2).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (TOREE-504) Version missing from shutdown_reply header
Kevin Bates created TOREE-504:
-
Summary: Version missing from shutdown_reply header
Key: TOREE-504
URL: https://issues.apache.org/jira/browse/TOREE-504
Project: TOREE
Issue Type: Bug
Components: Kernel
Affects Versions: 0.3.0
Reporter: Kevin Bates
The 5.0 Jupyter message protocol adds a {{version}} field to [message
headers|https://jupyter-client.readthedocs.io/en/latest/messaging.html#general-message-format].
However, shutdown_reply headers are not setting the protocol version. As a
result, clients that adhere to the protocol can have issues when {{version}} is
present in the response header as an empty string, leading to noise and
possible incorrect behaviors following Toree's shutdown.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (TOREE-503) Cannot Use Scala Enumerations as Method Arguments
[
https://issues.apache.org/jira/browse/TOREE-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926762#comment-16926762
]
Kevin Bates commented on TOREE-503:
---
Yeah, it must be something very subtle (and difficult to locate) since the pre
2.4 Spark REPLs don't reproduce the issue.
Closing now. We can reopen should some more concrete clues come up that point
to Toree.
> Cannot Use Scala Enumerations as Method Arguments
> -
>
> Key: TOREE-503
> URL: https://issues.apache.org/jira/browse/TOREE-503
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
> Environment: Amazon EMR Release emr-5.10.0, JupyterLab 0.35.4
>Reporter: Alexander Gunter
>Priority: Major
>
> Under common circumstances, cells containing method calls where an argument
> is an Enumeration will fail to compile when using the Toree kernel in
> JupyterLab. The circumstances relate to the scoping between notebook cells.
> The arrangement below will fail to compile, producing the shown output.
>
> _CELL 1:_
> {code:java}
> object ExampleEnum extends Enumeration {
> type ExampleEnum = Value
> val A, B, C = Value
> }{code}
> {code:java}
> defined object ExampleEnum
> {code}
>
> _CELL 2:_
> {code:java}
> def exampleMethod(enum: ExampleEnum.Value): Unit = {
> println(enum.toString)
> }
> exampleMethod(ExampleEnum.A){code}
> {code:java}
> exampleMethod: (enum: ExampleEnum.Value)Unit
> A
> {code}
>
> _CELL 3:_
> {code:java}
> exampleMethod(ExampleEnum.A){code}
> {code:java}
> Name: Compile Error
> Message: :30: error: type mismatch;
> found : ExampleEnum.Value
> required: ExampleEnum.Value
>exampleMethod(ExampleEnum.A)
> ^
> StackTrace:
> {code}
>
> Note that exampleMethod() works as expected when called inside the same cell
> as its definition (demo'd in Cell 2). It also works as expected in all cells
> if it is instead defined in Cell 1 alongside ExampleEnum (i.e. if you merge
> Cell1 and Cell 2).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Closed] (TOREE-503) Cannot Use Scala Enumerations as Method Arguments
[
https://issues.apache.org/jira/browse/TOREE-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Bates closed TOREE-503.
-
Fix Version/s: 0.3.0
Assignee: Kevin Bates
Resolution: Workaround
This appears to only be an issue if Toree is used on pre Spark 2.4 systems. It
cannot be reproduced on Spark 2.4+. Since the same version of Toree is used in
both cases, this implies this is probably something in Spark or some subtle
variation related to the version of Spark.
I chose 'Workaround' because this issue can be worked around if the method and
call to the method are in the same cell (for Spark < 2.4). The other
workaround is to upgrade to Spark 2.4, in which case the locality requirement
is not applicable.
> Cannot Use Scala Enumerations as Method Arguments
> -
>
> Key: TOREE-503
> URL: https://issues.apache.org/jira/browse/TOREE-503
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
> Environment: Amazon EMR Release emr-5.10.0, JupyterLab 0.35.4
>Reporter: Alexander Gunter
>Assignee: Kevin Bates
>Priority: Major
> Fix For: 0.3.0
>
>
> Under common circumstances, cells containing method calls where an argument
> is an Enumeration will fail to compile when using the Toree kernel in
> JupyterLab. The circumstances relate to the scoping between notebook cells.
> The arrangement below will fail to compile, producing the shown output.
>
> _CELL 1:_
> {code:java}
> object ExampleEnum extends Enumeration {
> type ExampleEnum = Value
> val A, B, C = Value
> }{code}
> {code:java}
> defined object ExampleEnum
> {code}
>
> _CELL 2:_
> {code:java}
> def exampleMethod(enum: ExampleEnum.Value): Unit = {
> println(enum.toString)
> }
> exampleMethod(ExampleEnum.A){code}
> {code:java}
> exampleMethod: (enum: ExampleEnum.Value)Unit
> A
> {code}
>
> _CELL 3:_
> {code:java}
> exampleMethod(ExampleEnum.A){code}
> {code:java}
> Name: Compile Error
> Message: :30: error: type mismatch;
> found : ExampleEnum.Value
> required: ExampleEnum.Value
>exampleMethod(ExampleEnum.A)
> ^
> StackTrace:
> {code}
>
> Note that exampleMethod() works as expected when called inside the same cell
> as its definition (demo'd in Cell 2). It also works as expected in all cells
> if it is instead defined in Cell 1 alongside ExampleEnum (i.e. if you merge
> Cell1 and Cell 2).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (TOREE-503) Cannot Use Scala Enumerations as Method Arguments
[
https://issues.apache.org/jira/browse/TOREE-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925958#comment-16925958
]
Kevin Bates commented on TOREE-503:
---
Hi Alexander - I guess I'm inclined to close this issue because you can define
the method in the same cell as its called. Once you upgrade to Spark 2.4, that
shouldn't be necessary. In addition, my thought is that this is more about
Spark than Toree since the same version of Toree produces different behaviors -
depending on the version of Spark.
Just want to check with you first.
> Cannot Use Scala Enumerations as Method Arguments
> -
>
> Key: TOREE-503
> URL: https://issues.apache.org/jira/browse/TOREE-503
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
> Environment: Amazon EMR Release emr-5.10.0, JupyterLab 0.35.4
>Reporter: Alexander Gunter
>Priority: Major
>
> Under common circumstances, cells containing method calls where an argument
> is an Enumeration will fail to compile when using the Toree kernel in
> JupyterLab. The circumstances relate to the scoping between notebook cells.
> The arrangement below will fail to compile, producing the shown output.
>
> _CELL 1:_
> {code:java}
> object ExampleEnum extends Enumeration {
> type ExampleEnum = Value
> val A, B, C = Value
> }{code}
> {code:java}
> defined object ExampleEnum
> {code}
>
> _CELL 2:_
> {code:java}
> def exampleMethod(enum: ExampleEnum.Value): Unit = {
> println(enum.toString)
> }
> exampleMethod(ExampleEnum.A){code}
> {code:java}
> exampleMethod: (enum: ExampleEnum.Value)Unit
> A
> {code}
>
> _CELL 3:_
> {code:java}
> exampleMethod(ExampleEnum.A){code}
> {code:java}
> Name: Compile Error
> Message: :30: error: type mismatch;
> found : ExampleEnum.Value
> required: ExampleEnum.Value
>exampleMethod(ExampleEnum.A)
> ^
> StackTrace:
> {code}
>
> Note that exampleMethod() works as expected when called inside the same cell
> as its definition (demo'd in Cell 2). It also works as expected in all cells
> if it is instead defined in Cell 1 alongside ExampleEnum (i.e. if you merge
> Cell1 and Cell 2).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (TOREE-504) Version missing from shutdown_reply header
[
https://issues.apache.org/jira/browse/TOREE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933543#comment-16933543
]
Kevin Bates commented on TOREE-504:
---
[~lresende] - that's probably a good idea. I had added code in jupyter to
still tolerate the missing version since IRKernel had the same issue (as others
might as well), but yes, it would help with Toree's "compliance".
> Version missing from shutdown_reply header
> --
>
> Key: TOREE-504
> URL: https://issues.apache.org/jira/browse/TOREE-504
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
>Reporter: Kevin Bates
>Assignee: Kevin Bates
>Priority: Major
> Fix For: 0.4.0
>
>
> The 5.0 Jupyter message protocol adds a {{version}} field to [message
> headers|https://jupyter-client.readthedocs.io/en/latest/messaging.html#general-message-format].
> However, shutdown_reply headers are not setting the protocol version. As a
> result, clients that adhere to the protocol can have issues when {{version}}
> is present in the response header as an empty string, leading to noise and
> possible incorrect behaviors following Toree's shutdown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TOREE-421) KernelSecurityManager doesn't allow users to create their own thread groups
[
https://issues.apache.org/jira/browse/TOREE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955516#comment-16955516
]
Kevin Bates commented on TOREE-421:
---
bq. enable different thread groups properly
I don't think anything has to happen to enable proper thread group creation
other than NOT preventing its creation via the SecurityManager.
Although I'd prefer to not have to support an additional CLI option, I'm
wondering if a boolean option such as {{--prevent-new-thread-groups}} that
defaults to {{false}} would be beneficial. This would open up thread group
creation by default, but any applications that require their prevention could
then add the CLI option. I realize this could be considered a "regression",
but I suspect most won't be needing this, and applications that require its
prevention have probably already been "trained" to not attempt thread group
creation in the first place (due to its previous prevention).
If others feel we don't need a CLI option, we can simply remove this portion of
the SecurityManager and move on.
Thoughts?
> KernelSecurityManager doesn't allow users to create their own thread groups
> ---
>
> Key: TOREE-421
> URL: https://issues.apache.org/jira/browse/TOREE-421
> Project: TOREE
> Issue Type: Bug
>Reporter: Piyush Narang
>Priority: Major
>
> I'm trying to run a Spark Scala job using Toree and I'm running into some
> issues as the code in our job calls into one of our libraries which tries to
> create threads in its own ThreadGroup:
> https://github.com/twitter/util/blob/develop/util-core/src/main/scala/com/twitter/concurrent/NamedPoolThreadFactory.scala#L28
> This seems to cause this check in Toree's KernelSecurityManager to trip:
> https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/security/KernelSecurityManager.scala#L121
> Stack looks like:
> {code}
> Name: java.lang.SecurityException
> Message: Not allowed to modify ThreadGroups!
> StackTrace: at
> org.apache.toree.security.KernelSecurityManager.checkAccess(KernelSecurityManager.scala:114)
> at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
> at java.lang.Thread.init(Thread.java:394)
> at java.lang.Thread.init(Thread.java:349)
> at java.lang.Thread.(Thread.java:599)
> at
> com.twitter.concurrent.NamedPoolThreadFactory.newThread(NamedPoolThreadFactory.scala:32)
> ...
> {code}
> Here's a simple repro:
> {code}
> println(Thread.currentThread().getThreadGroup) // default thread group
> val group: ThreadGroup = new
> ThreadGroup(Thread.currentThread().getThreadGroup(), "name")
> val hello = new Thread(group, new Runnable {
> def run() {
> println("hello world")
> }
> })
> println(hello.getThreadGroup)
> hello.start
> {code}
> Any suggestions for working around this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TOREE-421) KernelSecurityManager doesn't allow users to create their own thread groups
[
https://issues.apache.org/jira/browse/TOREE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953155#comment-16953155
]
Kevin Bates commented on TOREE-421:
---
Thank you for your post Chip! I had been looking at this yesterday and had
come to the same conclusion prior to seeing your response. That said, its
tremendously helpful to have insight into the history of that logic in the
first place - thank you.
Since we know, by virtue of the SecurityManager's existing enforcement, that no
one is leveraging this "functionality", I'm inclined to unconditionally allow
the creation of new thread groups rather than only via a CLI option. IMHO, the
latter still seems inhibiting, although "safe".
Do others have a strong opinion on this - particularly in allowing different
thread groups other than the one Toree is running in? From my understanding,
any new thread groups will "inherit" the attributes (priorities, etc.) from
Toree's.
> KernelSecurityManager doesn't allow users to create their own thread groups
> ---
>
> Key: TOREE-421
> URL: https://issues.apache.org/jira/browse/TOREE-421
> Project: TOREE
> Issue Type: Bug
>Reporter: Piyush Narang
>Priority: Major
>
> I'm trying to run a Spark Scala job using Toree and I'm running into some
> issues as the code in our job calls into one of our libraries which tries to
> create threads in its own ThreadGroup:
> https://github.com/twitter/util/blob/develop/util-core/src/main/scala/com/twitter/concurrent/NamedPoolThreadFactory.scala#L28
> This seems to cause this check in Toree's KernelSecurityManager to trip:
> https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/security/KernelSecurityManager.scala#L121
> Stack looks like:
> {code}
> Name: java.lang.SecurityException
> Message: Not allowed to modify ThreadGroups!
> StackTrace: at
> org.apache.toree.security.KernelSecurityManager.checkAccess(KernelSecurityManager.scala:114)
> at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
> at java.lang.Thread.init(Thread.java:394)
> at java.lang.Thread.init(Thread.java:349)
> at java.lang.Thread.(Thread.java:599)
> at
> com.twitter.concurrent.NamedPoolThreadFactory.newThread(NamedPoolThreadFactory.scala:32)
> ...
> {code}
> Here's a simple repro:
> {code}
> println(Thread.currentThread().getThreadGroup) // default thread group
> val group: ThreadGroup = new
> ThreadGroup(Thread.currentThread().getThreadGroup(), "name")
> val hello = new Thread(group, new Runnable {
> def run() {
> println("hello world")
> }
> })
> println(hello.getThreadGroup)
> hello.start
> {code}
> Any suggestions for working around this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TOREE-421) KernelSecurityManager doesn't allow users to create their own thread groups
[
https://issues.apache.org/jira/browse/TOREE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953146#comment-16953146
]
Kevin Bates commented on TOREE-421:
---
Including Chip's response from the mailing list...
{quote}This was an old implementation geared towards allowing us to manage all
launched threads so we could interrupt/kill them on request.
I'm not sure how useful this feature has been to folks, so my thought is that
we either remove it entirely or make it a feature that can be enabled/disabled
through a CLI option.
[[~lresende]], do you have any thoughts about which direction you'd like to go
here?
{quote}
> KernelSecurityManager doesn't allow users to create their own thread groups
> ---
>
> Key: TOREE-421
> URL: https://issues.apache.org/jira/browse/TOREE-421
> Project: TOREE
> Issue Type: Bug
>Reporter: Piyush Narang
>Priority: Major
>
> I'm trying to run a Spark Scala job using Toree and I'm running into some
> issues as the code in our job calls into one of our libraries which tries to
> create threads in its own ThreadGroup:
> https://github.com/twitter/util/blob/develop/util-core/src/main/scala/com/twitter/concurrent/NamedPoolThreadFactory.scala#L28
> This seems to cause this check in Toree's KernelSecurityManager to trip:
> https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/security/KernelSecurityManager.scala#L121
> Stack looks like:
> {code}
> Name: java.lang.SecurityException
> Message: Not allowed to modify ThreadGroups!
> StackTrace: at
> org.apache.toree.security.KernelSecurityManager.checkAccess(KernelSecurityManager.scala:114)
> at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
> at java.lang.Thread.init(Thread.java:394)
> at java.lang.Thread.init(Thread.java:349)
> at java.lang.Thread.(Thread.java:599)
> at
> com.twitter.concurrent.NamedPoolThreadFactory.newThread(NamedPoolThreadFactory.scala:32)
> ...
> {code}
> Here's a simple repro:
> {code}
> println(Thread.currentThread().getThreadGroup) // default thread group
> val group: ThreadGroup = new
> ThreadGroup(Thread.currentThread().getThreadGroup(), "name")
> val hello = new Thread(group, new Runnable {
> def run() {
> println("hello world")
> }
> })
> println(hello.getThreadGroup)
> hello.start
> {code}
> Any suggestions for working around this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
