Assuming you don't have your environment variables setup in your
.bash_profile you would do it like this -
import os
import sys
spark_home = '/usr/local/spark'
sys.path.insert(0, spark_home + "/python")
sys.path.insert(0, os.path.join(spark_home,
'python/lib/py4j-0.10.1-src.zip'))
#os.environ['PYSPARK_SUBMIT_ARGS'] = """--master spark://54.68.147.137:7077
pyspark-shell""" <---- where you can pass commands you would pass in
launching pyspark directly from command line
from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession
conf = SparkConf()\
.setMaster("local[8]")\
.setAppName("Test")
sc = SparkContext(conf=conf)
spark = SparkSession.builder\
.config(conf=sc.getConf())\
.enableHiveSupport()\
.getOrCreate()
Mind you this is for spark 2.0 and above
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Python-in-Jupyter-Notebook-tp28268p28274.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
