[
https://issues.apache.org/jira/browse/TOREE-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897536#comment-15897536
]
Luciano Resende commented on TOREE-390:
---------------------------------------
Are you seeing this issue because you are using local[*] spark, that gets
initiated when toree gets initiated? Would you have the same issue if you just
connect to an existing Spark Cluster ?
The problem with that is that the kernel would have sent the 'idle' status
telling the consumer he is ready to process requests, while he is not. And when
requests are received, then we get all sorts of strange issues such as sc not
defined, or some weird scala issues etc.... which was recently in Toree before
the following fix:
https://github.com/apache/incubator-toree/commit/6ef0c8cec02a6b622e204ab0271321e0f8a75d38
> Lazily start Spark sessions
> ---------------------------
>
> Key: TOREE-390
> URL: https://issues.apache.org/jira/browse/TOREE-390
> Project: TOREE
> Issue Type: Improvement
> Reporter: Ryan Blue
>
> In our deployment, more than half of the startup time for a Toree notebook is
> taken by starting a Spark session and waiting for containers. Lazily starting
> Spark sessions helps the notebook environment feel faster, even if the user
> is waiting on Spark to start up because the time waiting for Spark to start
> is clearly Spark, not Toree, and is initiated by the user.
> Also, lazily starting a Spark session allows users to change settings that
> can't be changed in a Spark context. It also enables the same startup code
> that would be used in a spark-submit application:
> {code:lang=java}
> SparkSession.builder
> .config(...)
> .setAppName(...)
> .getOrCreate()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
