What you might be missing is to pass some of these configurations to Spark.
We have some configuration examples on the Jupyter Enterprise Gateway documentation: http://jupyter-enterprise-gateway.readthedocs.io/en/latest/getting-started-client-mode.html Also note that, if you are in a distributed Spark environment, the issue might also be that the necessary library is not available on the machine the work is running, compared to your local env where you might only have the Spark driver running. Please let us know if this is not the case. On Mon, Feb 5, 2018 at 12:31 PM, Tim Harsch <[email protected]> wrote: > This seems like it should be pretty basic, but I'm having a hard time > installing a module to the pyspark notebook. I must be missing something... > > > If I choose a python3 notebook, the module (matplotlib) is already there and > works as expected. I notice that root has default python as > /opt/conda/bin/python (python 3.6.3). The jovyan user has python as > /usr/bin/python (python 2.7.12). > > > If I try to use the module matplotlib in the pyspark kernel, I get this: > > No module named matplotlib.pyplot > Traceback (most recent call last): > ImportError: No module named matplotlib.pyplot > > > I've tried several things including installing pip for /usr/bin/python, and > installing the matplotlib module there.. but still it is a problem. > > > I have a docker contanier that inherits from all-spark-notebook, so I can > modify the container OS if needed. > > > I look at the kernel.json and see: > > /usr/local/share/jupyter/kernels/pysparkkernel/kernel.json > > > {"argv":["python","-m","sparkmagic.kernels.pysparkkernel.pysparkkernel", > "-f", "{connection_file}"], > "display_name":"PySpark" > } > > > I assume that since python is not fully qualified that is picks up > /usr/bin/python from the path. > > > Any ideas? > > > Thanks, > > Tim > > > -- > You received this message because you are subscribed to the Google Groups > "Project Jupyter" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/jupyter/853cd237-6888-40ac-88bf-0ae5b94c3094%40googlegroups.com > <https://groups.google.com/d/msgid/jupyter/853cd237-6888-40ac-88bf-0ae5b94c3094%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/ -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAGU5spf9m%3D5bv_O53cbiseheNHt_6n1bqJz0Mu2UYGU6K7RbvA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
