Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1831
Thanks @1ambda for the fix. Thanks @zjffdu for reviewing and verifying.
It looks good to me, while it provides more consistent behavior between
`PysparkInterpreter` and
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
**For reviewers**
Fixed to use `spark.jars` instead of `classpath`.
- classpath doesn't include submitted jars at this moment (i could get 7
days ago, but not now)
- it
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1831
Thanks, @1ambda Do you mind to create a spark ticket as well ? The behavior
inconsistency between different modes seems an issue of spark, we need to
clarify it with spark community.
---
If
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1831
Sorry, I miss your last reply. Do you mean yarn-client mode works for you
in spark ?
I use the following command to launch pyspark and get the error as
following:
Launch pyspark ( I
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
1. I read and replied before.
> Q. README shows that user can use --py-files
> A. Users cannot benefit from --packages. They need to download, find
location of all transitive deps and
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1831
As I said before why not using `--py-files`, I check the repository of
pyspark-cassandra.
https://github.com/TargetHolding/pyspark-cassandra
README shows that user can use
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
@zjffdu
> since it doesn't resolve the yarn-client mode
1. PySpark also doens't support extending PYTHONPATH in yarn-client.
2. You are saying this is not right fix
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1831
I still think this is not a correct fix since it doesn't resolve the
yarn-client mode which I believe most of users use this mode.
---
If your project is set up for it, you can reply to this
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
Any update on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1831
hmm, it works in local mode but doesn't work in yarn-client mode. Could you
try yarn-client mode ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
@zjffdu
I'v just created gist to show `--packages` option download
pyspark-cassandra.
https://gist.github.com/1ambda/5caf92753ea2f95ada11b1c13945d261
```
downloading
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
@zjffdu Thanks for review :)
Then, How can I load
[pyspark-cassandra](https://github.com/TargetHolding/pyspark-cassandra#with-spark-packages)
for pyspark?
---
If your project is set up
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1831
right, I'm a bit concern if this would be the right fix for the issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1831
@1ambda Spark doesn't support specifying python packages throught
`--packages`, the correct usage is to use `--py-files`. Although this PR could
resolve your issue, but the issue here is not due
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
@astroshim Thanks for review!
It's the expected behavior. If spark submit doens't be properly loaded,
spark interpreter will die without errors.
---
If your project is set up for it, you
Github user astroshim commented on the issue:
https://github.com/apache/zeppelin/pull/1831
In my test, I got a ` INFO [2017-01-02 09:08:12,358] ({Exec Default
Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:164) -
Interpreter process exited 0` error when i try to
16 matches
Mail list logo
