Hi all
I had following configuration in spark worker (spark-env.sh)
export PYTHON_HOME=/opt/soft/anaconda2
export PYSPARK_PYTHON=$PYTHON_HOME/bin/python

I'm try to run a simple test script on  pyspark --master yarn --queue spark 
--executor-cores 1  --num-executors 10
from pyspark import SparkContext, SparkConf
df = sc.textFile("/user/hive/warehouse/hdptest.db/111");
df.collect();
sortedCount = df.map(lambda x: (float(x), 1)).sortByKey();
sortedCount.collect();

However , i get following error  : 
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1.0 (TID 
7, datanode-8): org.apache.spark.SparkException: 
Error from python worker:
  /usr/bin/python: module pyspark.daemon not found
PYTHONPATH was:
  
/home/hadoop/spark/python/lib/pyspark.zip:/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip:/data/hadoop_local/usercache/hdptest/filecache/38/spark-assembly-1.4.1-hadoop2.5.2.jar:pyspark.zip:py4j-0.8.2.1-src.zip

It seem that PYSPARK_PYTHON doesn't work in spark worker  ,  can someone please 
help me to solve it ? 

Thanks~


[email protected]

Reply via email to