Hello, I've looked into this, and my conclusion is that everything
regarding Spark is kind of broken. I don't know how to tame it in a
way that it runs in such a less-privileged project in SMC.
Regarding Scala, the idea would be to load the spark libraries and
then start from there with the context. The problem I run into is,
that it starts to create a temporary file in a directory in /tmp.
That's not a good idea, because it's shared. I tried to set the
environment variable
_JAVA_OPTIONS=-Djava.io.tmpdir=/projects/<your project id>/tmp/
and you can check this via "echo $_JAVA_OPTIONS" in a terminal in SMC.
However, there is still:
10:37:52,139 |-ERROR in
ch.qos.logback.core.rolling.RollingFileAppender[FILE] -
openFile(/tmp/javatmp/jupyter-scala.log,true) call failed.
java.io.FileNotFoundException: /tmp/jav
atmp/jupyter-scala.log (Permission denied)
at java.io.FileNotFoundException:
/tmp/javatmp/jupyter-scala.log (Permission denied)
in
cat ~/.smc/jupyter/jupyter-notebook.log
The code to start up Spark in a Scala kernel would be something like this:
```
import java.io.File
var spark_jars = new File("/projects/spark/spark/jars/")
spark_jars.listFiles.filter(_.isFile).toList.map(f =>
classpath.addPath(f.toString()))
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
val conf = new SparkConf().setAppName("local").setMaster("local[1]")
val sc = new SparkContext(conf)
sc
```
If you have any insights into this, I would be happy if you share them
with us :-)
-- Harald
On Sat, Jan 21, 2017 at 12:45 PM, <[email protected]> wrote:
> Hi,
>
> I've seen kernels for Apache Spark with Python and SageMath, but I'd like to
> use SageMathCloud for Apache Spark with Scala. Is that kernel not included
> yet? Or, how should I run code in Scala and Spark?
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-cloud" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sage-cloud/473bcf2a-809a-4bd3-992d-d96dcdb346cf%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"sage-cloud" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-cloud/CAGG4CB4tvADH-p9e%2BJcFQK2rTaO4V_fsOu7sXWpetEg62CSCYQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
