hi, how are you running your SPARK cluster (is it in local mode or distributed mode). Do you have pyspark installed in anaconda?
Regards, Gourav Sengupta On Mon, Mar 7, 2016 at 9:28 AM, [email protected] < [email protected]> wrote: > Hi all > I had following configuration in spark worker (spark-env.sh) > export PYTHON_HOME=/opt/soft/anaconda2 > export PYSPARK_PYTHON=$PYTHON_HOME/bin/python > > I'm try to run a simple test script on > pyspark --master yarn --queue spark --executor-cores 1 --num-executors 10 > from pyspark import SparkContext, SparkConf > df = sc.textFile("/user/hive/warehouse/hdptest.db/111"); > df.collect(); > sortedCount = df.map(lambda x: (float(x), 1)).sortByKey(); > sortedCount.collect(); > > However , i get following error : > > > > > > *org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in > stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1.0 > (TID 7, datanode-8): org.apache.spark.SparkException: Error from python > worker: /usr/bin/python: module pyspark.daemon not foundPYTHONPATH was: > /home/hadoop/spark/python/lib/pyspark.zip:/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip:/data/hadoop_local/usercache/hdptest/filecache/38/spark-assembly-1.4.1-hadoop2.5.2.jar:pyspark.zip:py4j-0.8.2.1-src.zip* > > It seem that PYSPARK_PYTHON doesn't work in spark worker , can someone > please help me to solve it ? > > Thanks~ > ------------------------------ > [email protected] >
