That does not work for me:
%pyspark
import pyspark_cassandra
(<type 'exceptions.ImportError'>, ImportError('No module named
pyspark_cassandra',), <traceback object at 0x7f9d16537290>)
Here is my conf/zeppelin-env.sh:
PSCC="/path/to/pyspark-cassandra/target"
export
ZEPPELIN_JAVA_OPTS="-Dspark.cassandra.connection.host=cassie
-Dspark.jars=$PSCC/pyspark_cassandra-0.1.4.jar"
export PYTHONPATH="$PSCC/pyspark_cassandra-0.1.3-py2.7.egg"
Interestingly, the logs indicate that they both loaded successfully:
Added JAR
file:/path/to/pyspark-cassandra/target/pyspark_cassandra-0.1.4.jar at
http://192.168.1.132:53081/jars/pyspark_cassandra-0.1.4.jar
Copying /path/to/pyspark-cassandra/target/pyspark_cassandra-0.1.4-py2.7.egg
to
/tmp/spark-11dd4dc8-6a78-4e9a-8073-d2ad2497f8ba/spark-8a841bb9-33c9-42a8-ac53-aab1e024d7d2/pyspark_cassandra-0.1.4-py2.7.egg
Added file
/path/to/pyspark-cassandra/target/pyspark_cassandra-0.1.4-py2.7.egg at
http://192.168.1.132:53081/files/pyspark_cassandra-0.1.4-py2.7.egg
On Tue, May 19, 2015 at 11:39 AM Jonathan Haddad <[email protected]> wrote:
> export PYTHONPATH=/path/to/module.egg I think is what I was doing.
>
> On Tue, May 19, 2015 at 11:27 AM Marty B <[email protected]> wrote:
>
>> Jon,
>>
>> How are you loading the Python module?
>>
>> The only way I could get it to load was to modify
>> spark/src/main/resources/python/zeppelin_pyspark.py and add it to the
>> SparkContext.
>>
>>
>> On Tue, May 19, 2015 at 10:43 AM Jonathan Haddad <[email protected]>
>> wrote:
>>
>>> Yep, I believe that was the one.
>>>
>>> On Tue, May 19, 2015 at 10:41 AM Marty B <[email protected]> wrote:
>>>
>>>> Jon,
>>>>
>>>> Is this the error message you see?
>>>>
>>>> (<class 'py4j.protocol.Py4JJavaError'>, Py4JJavaError(u'An error
>>>> occurred while calling o77.loadClass.\n', JavaObject id=o78), <traceback
>>>> object at 0x7f8949d124d0>)
>>>>
>>>> On Tue, May 19, 2015 at 9:49 AM Jonathan Haddad <[email protected]>
>>>> wrote:
>>>>
>>>>> I believe was able to make this part work by pointing PYTHONPATH to
>>>>> the egg. The part I couldn't get working was the driver class path, etc.
>>>>> I'm not really a Java guy so it's a lot of stumbling around for me.
>>>>>
>>>>> On Tue, May 19, 2015 at 9:46 AM Marty B <[email protected]> wrote:
>>>>>
>>>>>> The problem as I see it is that there appears to be no way to request
>>>>>> Zeppelin to load the .egg file (Python module containing
>>>>>> PySpark-Cassandra
>>>>>> interface).
>>>>>>
>>>>>> I opened ZEPPELIN-71 for this, but have seen no progress other than
>>>>>> acknowledgement.
>>>>>>
>>>>>> Perhaps additional feedback in the JIRA ticket would help the
>>>>>> Zeppelin team with prioritization. :-)
>>>>>>
>>>>>> I hope to find time over the upcoming holiday weekend (US) to look
>>>>>> further into fixing it - my first attempt failed.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, May 19, 2015 at 9:21 AM Jonathan Haddad <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> What I meant was I couldn't figure out how to get the Cassandra
>>>>>>> connector to work in pyspark. I had asked previously (including the
>>>>>>> flags
>>>>>>> I pass to pyspark) and gotten no response.
>>>>>>>
>>>>>>> If you've got a recommendation as to how to make the equivilent of
>>>>>>> this work in Zeppelin I'd love to give it a try:
>>>>>>>
>>>>>>> PYSPARK_ROOT=/Users/jhaddad/dev/pyspark-cassandra/target
>>>>>>> export PYSPARK_DRIVER_PYTHON=ipython
>>>>>>> PYSPARK_DRIVER_PYTHON_OPTS="notebook"
>>>>>>>
>>>>>>> pyspark \
>>>>>>> --jars ${PYSPARK_ROOT}/pyspark_cassandra-0.1.4.jar \
>>>>>>> --driver-class-path ${PYSPARK_ROOT}/pyspark_cassandra-0.1.4.jar \
>>>>>>> --py-files ${PYSPARK_ROOT}/pyspark_cassandra-0.1.4-py2.7.egg \
>>>>>>> --conf spark.cassandra.connection.host=127.0.0.1 \
>>>>>>> --master spark://127.0.0.1:7077 \
>>>>>>>
>>>>>>> Jon
>>>>>>>
>>>>>>> On Tue, May 19, 2015 at 4:30 AM moon soo Lee <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> First, thanks for trying Zeppelin.
>>>>>>>>
>>>>>>>> You guys mean pyspark is not working?
>>>>>>>> I'm developing zeppelin on OSX, so Zeppelin should work on it.
>>>>>>>>
>>>>>>>> If you can try latest code and post some log files, it'll be
>>>>>>>> helpful to investigate the problems.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> moon
>>>>>>>> On 2015년 5월 12일 (화) at 오후 4:39 Jonathan Haddad <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> How did you get it to work? I tried and failed miserably.
>>>>>>>>>
>>>>>>>>> Jon
>>>>>>>>>
>>>>>>>>> On May 12, 2015, at 12:25 AM, Tyler Mitchell <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>> I've been using Python with Spark 1.3.0 and copy of Zeppelin
>>>>>>>>> from today, but I'm getting some weirdness. Before digging into it
>>>>>>>>> further
>>>>>>>>> I also tried Spark 1.3.1 but got other strange things (output of
>>>>>>>>> notes not
>>>>>>>>> showing anything).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So curious, what versions are recommended or preferred? I'm
>>>>>>>>> trying on OSX so far.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Tyler
>>>>>>>>>
>>>>>>>>>
