Maziyar PANAHI created ZEPPELIN-3991:
----------------------------------------
Summary: Fail to bootstrap PySpark in yarn client mode
Key: ZEPPELIN-3991
URL: https://issues.apache.org/jira/browse/ZEPPELIN-3991
Project: Zeppelin
Issue Type: Bug
Components: pySpark, python-interpreter
Affects Versions: 0.9.0
Environment: Cloudera/CDH 6.1
Spark 2.4
Hadoop 3.0
*zeppein-env.sh*
{code:java}
export PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3
export
PYSPARK_DRIVER_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3
export PYSPARKPYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3
export PYSPARKDRIVERPYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3
{code}
Reporter: Maziyar PANAHI
Hi,
PySpark fails with following error in Zeppelin 0.9.0:
{code:java}
ERROR [2019-02-06 22:23:40,599] ({FIFOScheduler-Worker-1} Job.java[run]:174) -
Job failed
org.apache.zeppelin.interpreter.InterpreterException: Fail to bootstrap pyspark
at
org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:124)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:593)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:502)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at
org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:121)
at
org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Fail to run bootstrap script:
python/zeppelin_pyspark.py
at
org.apache.zeppelin.python.PythonInterpreter.bootstrapInterpreter(PythonInterpreter.java:581)
at
org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:122)
... 9 more
{code}
Full logs of YARN:
[https://gist.github.com/maziyarpanahi/3380e230246271217a2feb4512f5d665]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
