Assuming you don't have your environment variables setup in your .bash_profile you would do it like this -
import os import sys spark_home = '/usr/local/spark' sys.path.insert(0, spark_home + "/python") sys.path.insert(0, os.path.join(spark_home, 'python/lib/py4j-0.10.1-src.zip')) #os.environ['PYSPARK_SUBMIT_ARGS'] = """--master spark://54.68.147.137:7077 pyspark-shell""" <---- where you can pass commands you would pass in launching pyspark directly from command line from pyspark import SparkContext, SparkConf from pyspark.sql import SparkSession conf = SparkConf()\ .setMaster("local[8]")\ .setAppName("Test") sc = SparkContext(conf=conf) spark = SparkSession.builder\ .config(conf=sc.getConf())\ .enableHiveSupport()\ .getOrCreate() Mind you this is for spark 2.0 and above -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Python-in-Jupyter-Notebook-tp28268p28274.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org