How to write DataFrame to single parquet file instead of multiple files under a folder in spark?

For better viewing, please see https://stackoverflow.com/questions/52964167/how-to-write-dataframe-to-single-parquet-file-instead-of-multiple-files-under-a - I have a folder with files [![enter image description here][1]][1] I want to do some transform to each file and save to another fo

Re: [External Sender] Re: How to make pyspark use custom python?

I am sure, all writen as my first post. So this make me very confusing. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: How to make pyspark use custom python?

The whole content in `spark-env.sh` is ``` SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=10.104.85.78:2181,10.104.114.131:2181,10.135.2.132:2181 -Dspark.deploy.zookeeper.dir=/spark" PYSPARK_PYTHON="/usr/local/miniconda3/bin/python" ``` I ran `/usr/l

How to make pyspark use custom python?

For better looking , please see https://stackoverflow.com/questions/52178406/howto-make-pyspark-use-custom-python -- I am using zeppelin connect remote spark cluster. remote spark is usin