[jira] [Created] (ZEPPELIN-1883) Can't import packages requested by SPARK_SUBMIT_OPTION in pyspark

[Hoon Park (JIRA)]

Hoon Park created ZEPPELIN-1883:
-----------------------------------

             Summary: Can't import packages requested by SPARK_SUBMIT_OPTION in 
pyspark
                 Key: ZEPPELIN-1883
                 URL: https://issues.apache.org/jira/browse/ZEPPELIN-1883
             Project: Zeppelin
          Issue Type: Bug
          Components: pySpark
            Reporter: Hoon Park
             Fix For: 0.7.0


Zeppelin pyspark can't import submitted packages by {{SPARK_SUBMIT_OPTION}}. 
For example, 

{code}
// conf/zeppelin-env.sh
...

export SPARK_HOME="~/github/apache-spark/1.6.2-bin-hadoop2.6"
export SPARK_SUBMIT_OPTIONS="--packages 
com.datastax.spark:spark-cassandra-connector_2.10:1.6.2,TargetHolding:pyspark-cassandra:0.3.5
 --exclude-packages org.slf4j:slf4j-api"

...
{code}

And then try import that pyspark cassandra module in zeppelin pyspark 
interpreter

{code}
import pyspark_cassandra


Traceback (most recent call last):
  File 
"/var/folders/lr/8g9y625n5j39rz6qhkg8s6640000gn/T/zeppelin_pyspark-5266742863961917074.py",
 line 267, in <module>
    raise Exception(traceback.format_exc())
Exception: Traceback (most recent call last):
  File 
"/var/folders/lr/8g9y625n5j39rz6qhkg8s6640000gn/T/zeppelin_pyspark-5266742863961917074.py",
 line 265, in <module>
    exec(code)
  File "<stdin>", line 1, in <module>
ImportError: No module named pyspark_cassandra
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)