So the traceback indicates that it's trying to launch a command in a subprocess, and failing to find that command. Can you work out what command it's trying to launch, and where the file for that command is on your filesystem?
Thomas On 11 May 2017 at 13:40, David Arenburg <[email protected]> wrote: > Hi takowl, > > They indeed a bit different, Jupyter has: > > '/mnt/anaconda/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/bin:/root/bin' > > > While from console I get: > > > '/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/bin:/root/bin' > > > Though I've tried doing in > > import os > os.environ['PATH'] = '/usr/local/bin:/bin:/usr/bin: > /usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2- > user/.local/bin:/home/ec2-user/bin:/usr/local/bin:/root/bin' > from pyspark import SparkConf > SparkConf() > > In Jupyter but still getting the same error > > Thanks > > On Thursday, May 11, 2017 at 3:30:51 PM UTC+3, takowl wrote: >> >> Can you check your PATH environment variable in both the console and in >> Jupyter? From Python, you can do that like this: >> >> import os >> os.environ['PATH'] >> >> On 11 May 2017 at 13:24, David Arenburg <[email protected]> wrote: >> >>> Hello all, >>> >>> I've been trying to figure this out for a week with no success. >>> >>> I'm simply trying to initiate SparkContext in Jupyter but getting the >>> following error when running SparkConf() : >>> >>> >>> ``` >>> from pyspark import SparkConf >>> SparkConf() >>> >>> ``` >>> >>> ---------------------------------------------------------------------------OSError >>> Traceback (most recent call >>> last)<ipython-input-12-0c80a6a098f7> in <module>() 1 #import >>> statements 2 from pyspark import SparkConf----> 3 SparkConf() 4 >>> /root/david/spark/python/pyspark/conf.pyc in __init__(self, loadDefaults, >>> _jvm, _jconf) 102 else: 103 from pyspark.context >>> import SparkContext--> 104 SparkContext._ensure_initialized() >>> 105 _jvm = _jvm or SparkContext._jvm 106 >>> self._jconf = _jvm.SparkConf(loadDefaults) >>> /root/david/spark/python/pyspark/context.pyc in _ensure_initialized(cls, >>> instance, gateway) 241 with SparkContext._lock: 242 >>> if not SparkContext._gateway:--> 243 >>> SparkContext._gateway = gateway or launch_gateway() 244 >>> SparkContext._jvm = SparkContext._gateway.jvm 245 >>> /root/david/spark/python/pyspark/java_gateway.pyc in launch_gateway() >>> 74 def preexec_func(): 75 >>> signal.signal(signal.SIGINT, signal.SIG_IGN)---> 76 proc = >>> Popen(command, stdin=PIPE, preexec_fn=preexec_func, env=env) 77 >>> else: 78 # preexec_fn not supported on Windows >>> /mnt/anaconda/lib/python2.7/subprocess.pyc in __init__(self, args, bufsize, >>> executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, >>> universal_newlines, startupinfo, creationflags) 709 >>> p2cread, p2cwrite, 710 >>> c2pread, c2pwrite,--> 711 errread, >>> errwrite) 712 except Exception: 713 # Preserve >>> original exception in case os.close raises. >>> /mnt/anaconda/lib/python2.7/subprocess.pyc in _execute_child(self, args, >>> executable, preexec_fn, close_fds, cwd, env, universal_newlines, >>> startupinfo, creationflags, shell, to_close, p2cread, p2cwrite, c2pread, >>> c2pwrite, errread, errwrite) 1341 raise 1342 >>> child_exception = pickle.loads(data)-> 1343 >>> raise child_exception 1344 1345 >>> OSError: [Errno 2] No such file or directory >>> >>> >>> >>> This is my startJupyter.sh script which I use to lunch Jupyter >>> >>> #!/bin/bash >>> if ps -ef |grep $USER| grep python > /dev/null >>> then >>> echo "Jupyter is Running - Restarting" >>> echo "Killing jupyter-notebook process" >>> >>> running_id=$(ps -ef |grep $USER| grep python) >>> stringarray=($running_id) >>> echo ${stringarray[1]} >>> kill -9 ${stringarray[1]} >>> >>> export SPARK_HOME='/usr/lib/spark/' >>> export PYTHONPATH=$SPARK_HOME/python/ >>> :$PYTHONPATH:$SPARK_HOME/python/lib/py4j-0.9-src.zip >>> >>> #jupyter nbextension enable --py widgetsnbextension >>> /mnt/anaconda/bin/jupyter notebook & >>> >>> else >>> echo "Jupyter is Not Running" >>> echo "Starting Jupyter-NoteBook" >>> export SPARK_HOME='/usr/lib/spark/' >>> export PYTHONPATH=$SPARK_HOME/python/ >>> :$PYTHONPATH:$SPARK_HOME/python/lib/py4j-0.9-src.zip >>> >>> #jupyter nbextension enable --py widgetsnbextension >>> /mnt/anaconda/bin/jupyter notebook & >>> fi >>> >>> >>> When running the same in python in console (not in Jupyter), it works >>> fine >>> >>> Python 2.7.12 |Anaconda 4.2.0 (64-bit)| (default, Jul 2 2016, 17:42:40) >>> [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 >>> Type "help", "copyright", "credits" or "license" for more information. >>> Anaconda is brought to you by Continuum Analytics. >>> Please check out: http://continuum.io/thanks and https://anaconda.org >>> >>> from pyspark import SparkConf >>> >>> SparkConf() >>> Using Spark's default log4j profile: org/apache/spark/log4j-default >>> s.properties >>> Setting default log level to "WARN". >>> To adjust logging level use sc.setLogLevel(newLevel). >>> <pyspark.conf.SparkConf object at 0x7f482f78b6d0> >>> >>> >>> I've validated both the python version and the module path in console >>> and in Jupyter and they seem to match >>> >>> >>> import sys >>> >>> sys.version >>> '2.7.12 |Anaconda 4.2.0 (64-bit)| (default, Jul 2 2016, 17:42:40) >>> \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]' >>> >>> >>> import inspect >>> >>> import pyspark >>> >>> inspect.getfile(pyspark) >>> '/root/david/spark/python/pyspark/__init__.pyc' >>> >>> I can't think of anything else I could do wrong, please help >>> Thanks, David >>> >>> My specs: >>> >>> NAME="Amazon Linux AMI" >>> VERSION="2017.03" >>> ID="amzn" >>> ID_LIKE="rhel fedora" >>> VERSION_ID="2017.03" >>> PRETTY_NAME="Amazon Linux AMI 2017.03" >>> ANSI_COLOR="0;33" >>> CPE_NAME="cpe:/o:amazon:linux:2017.03:ga" >>> HOME_URL="http://aws.amazon.com/amazon-linux-ami/"; >>> Amazon Linux AMI release 2017.03 >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Project Jupyter" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/jupyter/a1924dad-a893-447f-8b79-735bb2ff0fb8%40googlegroups.com >>> <https://groups.google.com/d/msgid/jupyter/a1924dad-a893-447f-8b79-735bb2ff0fb8%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Project Jupyter" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/jupyter/9b4d9cc4-fad5-4efb-8530-703e78578628%40googlegroups.com > <https://groups.google.com/d/msgid/jupyter/9b4d9cc4-fad5-4efb-8530-703e78578628%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAOvn4qjE%3DGB2PAwo8WiUX__j61JEXBpakKK-96HFue%2BiJun9cg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
