[
https://issues.apache.org/jira/browse/TOREE-188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713752#comment-17713752
]
Ricardo Gaspar commented on TOREE-188:
--------------------------------------
Yes, I'm also facing the same issue where the current magics overload all the
other existing magics (like those avaiable in spylon)
using scala-toree:
{code:java}
%lsmagic
Available line magics:
%lsmagic %showtypes %showoutput %adddeps %truncation %addjar
Available cell magics:
%%sql %%html %%javascript %%dataframe %%scala
Type %<magic_name> for usage info.
Available line magics:
%lsmagic %showtypes %showoutput %adddeps %truncation %addjar
Available cell magics:
%%sql %%html %%javascript %%dataframe %%scala
Type %<magic_name> for usage info. {code}
using spylon:
{code:java}
%lsmagic
Available line magics:
%activity %cd %connect_info %conversation %dot %download %edit %get
%help %html %include %install %install_magic %javascript %jigsaw %kernel
%kx %latex %load %ls %lsmagic %macro %magic %matplotlib %parallel
%plot %pmap %px %python %reload_magics %restart %run %scala %scheme
%set %shell
Available cell magics:
%%activity %%brain %%conversation %%debug %%dot %%file %%help %%html
%%init_spark %%javascript %%kx %%latex %%macro %%pipe %%processing %%px
%%python %%scala %%scheme %%shell %%show %%time %%tutor {code}
*Additionally* there's no way to configure spark properties, especially adding
jars to spark driver and executors and then import the functions. `%Addjar`
doesn't do that Whereas with spylon one can invoke `%init_spark` to load the
jars and it works fine when importing in other cells. e.g.
{code:java}
%%init_spark
launcher.jars = ["/home/rgaspar/myjar-1.5.1.jar"]{code}
> % and %% magics need to be evaluated outside of interpreters
> ------------------------------------------------------------
>
> Key: TOREE-188
> URL: https://issues.apache.org/jira/browse/TOREE-188
> Project: TOREE
> Issue Type: Bug
> Reporter: Chip Senkbeil
>
> In an old version of the Spark Kernel, we evaluated magics by running them
> through a separate actor than the interpreter actor. The "magic executor"
> actor would then load the magic and evaluate it.
> The current version translates magics like {{%AddJar
> http://example.com/myjar.jar}} into
> {{kernel.magics.addJar("http://example.com/myjar.jar";)}}. That code would
> then be executed on the interpreter.
> The problems that we have discovered are
> 1. This causes magics to be specific to Scala as the generated code is
> Scala-specific
> 2. Requires an interpreter to be started to evaluate the magic
> This causes problems for cases where we want to run magics in other languages
> or run magics that do not start a lazily-evaluated interpreter.
> One use case of this change is to be able to add jars to the Scala
> interpreter before it is started. It is impossible to do so via a magic when
> the magic itself needs an interpreter to be evaluated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
