Hi all, Found the answer from the following link:
https://forums.databricks.com/questions/918/how-to-set-size-of-parquet-output-files.html I can successfully setup parquet block size with spark.hadoop.parquet.block.size. The following is the sample code: # init block_size = 512 * 1024 conf = SparkConf().setAppName("myapp").setMaster("spark://spark1:7077").set('spark.cores.max', 20).set("spark.executor.cores", 10).set("spark.executor.memory", "10g").set('spark.hadoop.parquet.block.size', str(block_size)).set("spark.hadoop.dfs.blocksize", str(block_size)).set("spark.hadoop.dfs.block.size", str(block_size)).set("spark.hadoop.dfs.namenode.fs-limits.min-block-size", str(131072)) sc = SparkContext(conf=conf) spark = SparkSession(sc) # create DataFrame df_txt = spark.createDataFrame([{'temp': "hello"}, {'temp': "world"}, {'temp': "!"}]) # save using DataFrameWriter, resulting 512k-block-size df_txt.write.mode('overwrite').format('parquet').save('hdfs://spark1/tmp/temp_with_df') -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
