Github user matyix closed the pull request at:
https://github.com/apache/zeppelin/pull/2637
---
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
@naveenkumargp one more thing. There is another alternative we are
experimenting - we are adding Kubernetes integration to Livy and you would be
able to use the Livy interpreter (no Zeppelin code
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
@naveenkumargp you are welcome if you find bugs or have feature requests
related to our codebase feel free to open a GH issue at our fork.
---
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
@naveenkumargp Not sure this is the best place discussing your problem ...
anyways:
The message `Unable to start SparkK8RemoteInterpreterManagedProcess: Spark
Driver not found.` means
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
@naveenkumargp Ideally, this should be merged (chances are unlikely if you
check the history of the PR, and in a few days this will conflict with the
master branch, so will be even more unlikely
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
@echarles usually when adding a dependency on UI that should be downloaded
to local-repo/spark folder, then those jars will be set in --jars param. Could
you please check your local-repo/spark
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
hello @echarles
- we're also running into this NPE problem, my colleague @sancyx already
commented on this issue: https://issues.apache.org/jira/browse/ZEPPELIN-2475
and he's going
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
@zjffdu @felixcheung added the requested changes/suggestions.
---
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
hello @echarles. Yes it works for us - we use this extensively every day on
k8s. ln our env deps (local deps downloaded to localRepo) are added to
spark-submit command line with `--jars`, provided
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
Hello @zjffdu - I rebased and pushed upstream, please have a look. Thanks
for your comment, I look forward supporting you to get this PR merged upstream
(now that the Spark on Kubernetres fork got
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
From the command line it seems that the default `SparkInterpreter` is
launched not the K8 specific one. Could you please check if
spark.submit.deployMode is set to `cluster` since thats the other
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
@echarles jars from Zeppelin local-repo are set for spark-submit in
`sparks.jars` parameter
---
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
The packages option for spark submit is described
[here](https://spark.apache.org/docs/latest/submitting-applications.html) and
it seems to work. Using this option seems to be better alternative
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
@echarles Currently there are two ways to add external dependencies: add a
new paragraph to the notebook using `spark.dep` interpreter and `z.load()`.
This works because it downloads
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
@echarles Added some nice to have features like a separate log4j config for
k8s (same as for yarn), updated the doc and now Iâm only checking the
`processId` label on driver pod so users can
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
Hello @echarles
You can set both `spark.app.name` and `spark.kubernetes.driver.pod.name`
from interpreter settings and they will be set for `spark-submit` however
youâre right
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
Thanks @echarles. Let me know if anything else is needed to get this merged.
---
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
@zjffdu Not sure, never followed that one but I'll check with the folks
over there as I have a few PR's on that repo as well.
---
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
@zjffdu Spark submit needs a resource staging server (RSS) to be specified
as a parameter, you should start RSS, get the address of RSS - as described in
the documentation which is part of the PR
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
Hello @echarles, thanks for the feedback, followed up on that and made the
suggested changes, please see below:
* use watcher instead of polling
* set driver prefix form config
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
@echarles @zjffdu @felixcheung
It is absolutely makes sense to keep this PR and make further work with it.
Just to re-empahsize, the goal is to enable Zeppelin to submit notebooks
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
@zjffdu @felixcheung I have updated the original PR description with design
considerations and a typical K8S cluster and Zeppelin flow showing how I am
using this feature/PR currently
Github user matyix commented on the issue:
https://github.com/apache/zeppelin/pull/2637
Hello @zjffdu @felixcheung- I have added docs about how to use/build this
PR as well. Also the Dockerfile has been removed, the documentation contains
details of using a third party (mine
GitHub user matyix opened a pull request:
https://github.com/apache/zeppelin/pull/2637
Add support to run Spark interpreter on a Kubernetes cluster
### What is this PR for?
The goal of this PR is to be able to execute Spark notebooks on Kubernetes
in cluster mode, so
24 matches
Mail list logo
