Thank you Luciano! I should look into enterprise gateway some more it
looks very useful.
Yes, I was missing something. Turns out is a sparkmagic thing.. (from
the sparkmagic github readme)
Since all code is run on a remote driver through Livy, all structured data
> must be serialized to JSON and parsed by the Sparkmagic library so that it
> can be manipulated and visualized on the client side. In practice this
> means that you must use Python for client-side data manipulation in
> %%local mode.
For future travelers: note I placed the following example pie chart at the
end of a sparkmagic notebook to render a pie-chart. My next steps will be
to fill the pie chart from queries to the hiveContext..
%%local
import matplotlib.pyplot as plt
%matplotlib inline
# Data to plot
labels = 'Python', 'C++', 'Ruby', 'Java'
sizes = [215, 130, 245, 210]
colors = ['gold', 'yellowgreen', 'lightcoral', 'lightskyblue']
explode = (0.1, 0, 0, 0) # explode 1st slice
# Plot
plt.pie(sizes, explode=explode, labels=labels, colors=colors,
autopct='%1.1f%%', shadow=True, startangle=140)
plt.axis('equal')
plt.show()
On Monday, February 5, 2018 at 3:02:57 PM UTC-8, Luciano Resende wrote:
>
> What you might be missing is to pass some of these configurations to Spark.
>
> We have some configuration examples on the Jupyter Enterprise Gateway
> documentation:
>
> http://jupyter-enterprise-gateway.readthedocs.io/en/latest/getting-started-client-mode.html
>
> Also note that, if you are in a distributed Spark environment, the issue
> might also be that the necessary library is not available on the machine
> the work is running, compared to your local env where you might only have
> the Spark driver running.
>
> Please let us know if this is not the case.
>
> On Mon, Feb 5, 2018 at 12:31 PM, Tim Harsch <[email protected]
> <javascript:>> wrote:
>
>> This seems like it should be pretty basic, but I'm having a hard time
>> installing a module to the pyspark notebook. I must be missing something...
>>
>>
>> If I choose a python3 notebook, the module (matplotlib) is already there and
>> works as expected. I notice that root has default python as
>> /opt/conda/bin/python (python 3.6.3). The jovyan user has python as
>> /usr/bin/python (python 2.7.12).
>>
>>
>> If I try to use the module matplotlib in the pyspark kernel, I get this:
>>
>> No module named matplotlib.pyplot
>> Traceback (most recent call last):
>> ImportError: No module named matplotlib.pyplot
>>
>>
>> I've tried several things including installing pip for /usr/bin/python, and
>> installing the matplotlib module there.. but still it is a problem.
>>
>>
>> I have a docker contanier that inherits from all-spark-notebook, so I can
>> modify the container OS if needed.
>>
>>
>> I look at the kernel.json and see:
>>
>> /usr/local/share/jupyter/kernels/pysparkkernel/kernel.json
>>
>>
>> {"argv":["python","-m","sparkmagic.kernels.pysparkkernel.pysparkkernel",
>> "-f", "{connection_file}"],
>> "display_name":"PySpark"
>> }
>>
>>
>> I assume that since python is not fully qualified that is picks up
>> /usr/bin/python from the path.
>>
>>
>> Any ideas?
>>
>>
>> Thanks,
>>
>> Tim
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Project Jupyter" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To post to this group, send email to [email protected]
>> <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/jupyter/853cd237-6888-40ac-88bf-0ae5b94c3094%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/jupyter/853cd237-6888-40ac-88bf-0ae5b94c3094%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
--
You received this message because you are subscribed to the Google Groups
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jupyter/e733f8ac-d8b4-4cc1-a63b-dd817b6ab884%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
