I understand this has been discussed before, however, now able to sort it out with suggested solution, so I decided to post here again - maybe my case is unique? Thank you very much, it is been stuck here for a week now, any help is greatly appreciated.
Environment: I am deploying/configuring jupyterhub on another cluster by essentially following my successful implementation on a previous sandbox cluster, stuck here now due to the current environment is not clear to me. Here is it: spark: /opt/cloudera/parcels/CDH/lib/spark python: /usr/bin/python, 2.7.5 Kernel (python 2): (the env part was manually added by following given working example here #2116 <https://github.com/jupyterhub/jupyterhub/issues/2116>), by the way, the kernel was created long time ago under Jupyter, would that be an issue? How do I know which python I was using to create the kernel? { "display_name": "Python 2", "language": "python", "argv": [ "python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "env": { "HADOOP_CONF_DIR":"/etc/hive/conf", "PYSPARK_PYTHON":"/usr/bin/python", "SPARK_HOME": "/opt/cloudera/parcels/CDH/lib/spark", "WRAPPED_SPARK_HOME": "/opt/cloudera/parcels/CDH/lib/spark", "PYTHONPATH": "{{ app_packages_home }}/lib/python2.7/site-packages:{{ jupyter_extension_venv }}/lib/python2.7/site-packages:{{ spark_home }}/python:{{ spark_home }}/python/lib/py4j-0.10.4-src.zip", "PYTHONSTARTUP": "/opt/cloudera/parcels/CDH/lib/spark/python/pyspark/shell.py", "PYSPARK_SUBMIT_ARGS": "--master yarn-client --jars {{ spark_home }}/lib/spark-examples.jar pyspark-shell" } } Notebook: import sys,os os.environ["SPARK_HOME"] = '/opt/cloudera/parcels/CDH/lib/spark' os.environ['PYSPARK_PYTHON'] = '/usr/bin/python' os.environ['PYSPARK_DRIVER_PYTHON'] = '/usr/bin/python' os.environ['JAVA_HOME'] = '/usr/java/latest' sys.path.append('/usr/bin/python') sys.path.append('/opt/cloudera/parcels/CDH/lib/spark/python/lib/py4j-0.9-src.zip') import pyspark from pyspark import SparkContext, SparkConf conf = SparkConf() conf.setMaster('yarn-client') conf.setAppName('raymond - test') sc = SparkContext(conf = conf) Error: ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[main,5,main] java.util.NoSuchElementException: key not found: _PYSPARK_DRIVER_CALLBACK_HOST This issue is also posted https://github.com/jupyterhub/jupyterhub/issues/2158 Thank you very much. *------------------------------------------------* *Sincerely yours,* *Raymond* -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAKZNNLLUoH%3DF%2BOXoYyDi0uUP_ibxwjk3PyNEyOWkudn1eZqE%3Dg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
