Hi, the way I manage things is, download spark, and set SPARK_HOME and the import findspark and run findspark.init(). And everything else works just fine.
I have never tried pip install pyspark though. Regards, Gourav Sengupta On Wed, Apr 4, 2018 at 11:28 PM, Andy Davidson < a...@santacruzintegration.com> wrote: > I am having trouble setting up my python3 virtualenv. > > I created a virtualenv ‘spark-2.3.0’ Installed pyspark using pip how ever > I am not able to import pyspark.sql.functions. I get “unresolved import” > when I try to import col() and lit() > > from pyspark.sql.functions import * > > > I found if I download spark from apache and set SPARK_ROOT I can get my > juypter notebook to work. This is a very error prone work around. I am > having simiilar problem with my eclipse pyDev virtualenv > > Any suggestions would be greatly appreciated > > Andy > > > # pip show in virtualenv > > (spark-2.3.0) $ pip show pyspark > > Name: pyspark > > Version: 2.3.0 > > Summary: Apache Spark Python API > > Home-page: https://github.com/apache/spark/tree/master/python > > Author: Spark Developers > > Author-email: d...@spark.apache.org > > License: http://www.apache.org/licenses/LICENSE-2.0 > > Location: /Users/foo/workSpace/pythonEnv/spark-2.3.0/lib/ > python3.6/site-packages > > Requires: py4j > > (spark-2.3.0) $ > > (spark-2.3.0) $ ls ~/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/site- > packages/pyspark/sql/functions.py > > ~/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/site-packages/pyspark/sql/ > functions.py > > > # Jupyter Notebook > > Export SPARK_ROOT=~/workSpace/spark/spark-2.3.0-bin-hadoop2.7 > > > > Eclipse pyDev virtual ENV > > > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org >
