Hi all
I had following configuration in spark worker (spark-env.sh)
export PYTHON_HOME=/opt/soft/anaconda2
export PYSPARK_PYTHON=$PYTHON_HOME/bin/python
I'm try to run a simple test script on pyspark --master yarn --queue spark
--executor-cores 1 --num-executors 10
from pyspark import SparkContext, SparkConf
df = sc.textFile("/user/hive/warehouse/hdptest.db/111");
df.collect();
sortedCount = df.map(lambda x: (float(x), 1)).sortByKey();
sortedCount.collect();
However , i get following error :
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in
stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1.0 (TID
7, datanode-8): org.apache.spark.SparkException:
Error from python worker:
/usr/bin/python: module pyspark.daemon not found
PYTHONPATH was:
/home/hadoop/spark/python/lib/pyspark.zip:/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip:/data/hadoop_local/usercache/hdptest/filecache/38/spark-assembly-1.4.1-hadoop2.5.2.jar:pyspark.zip:py4j-0.8.2.1-src.zip
It seem that PYSPARK_PYTHON doesn't work in spark worker , can someone please
help me to solve it ?
Thanks~
[email protected]