[ https://issues.apache.org/jira/browse/SPARK-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503921#comment-14503921 ]
Sebastian YEPES FERNANDEZ commented on SPARK-6921: -------------------------------------------------- I can also validate this with v1.3.1 > Spark SQL API "saveAsParquetFile" will output tachyon file with different > block size > ------------------------------------------------------------------------------------ > > Key: SPARK-6921 > URL: https://issues.apache.org/jira/browse/SPARK-6921 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.3.0 > Reporter: zhangxiongfei > Priority: Blocker > > I run below code in Spark Shell to access parquet files in Tachyon. > 1.First,created a DataFrame by loading a bunch of Parquet Files in Tachyon > val ta3 > =sqlContext.parquetFile("tachyon://tachyonserver:19998/apps/tachyon/zhangxf/parquetAdClick-6p-256m"); > 2.Second, set the "fs.local.block.size" to 256M to make sure that block > size of output files in Tachyon is 256M. > sc.hadoopConfiguration.setLong("fs.local.block.size",268435456) > 3.Third,saved above DataFrame into Parquet files that is stored in Tachyon > > ta3.saveAsParquetFile("tachyon://tachyonserver:19998/apps/tachyon/zhangxf/parquetAdClick-6p-256m-test"); > After above code run successfully, the output parquet files were stored in > Tachyon,but these files have different block size,below is the information of > those files in the path > "tachyon://tachyonserver:19998/apps/tachyon/zhangxf/parquetAdClick-6p-256m-test": > File Name Size Block Size > In-Memory Pin Creation Time > _SUCCESS 0.00 B 256.00 MB 100% > NO 04-13-2015 17:48:23:519 > _common_metadata 1088.00 B 256.00 MB 100% NO > 04-13-2015 17:48:23:741 > _metadata 22.71 KB 256.00 MB 100% NO > 04-13-2015 17:48:23:646 > part-r-00001.parquet 177.19 MB 32.00 MB 100% NO > 04-13-2015 17:46:44:626 > part-r-00002.parquet 177.21 MB 32.00 MB 100% NO > 04-13-2015 17:46:44:636 > part-r-00003.parquet 177.02 MB 32.00 MB 100% NO > 04-13-2015 17:46:45:439 > part-r-00004.parquet 177.21 MB 32.00 MB 100% NO > 04-13-2015 17:46:44:845 > part-r-00005.parquet 177.40 MB 32.00 MB 100% NO > 04-13-2015 17:46:44:638 > part-r-00006.parquet 177.33 MB 32.00 MB 100% NO > 04-13-2015 17:46:44:648 > It seems that the API saveAsParquetFile does not distribute/broadcast the > hadoopconfiguration to executors like the other API such as > saveAsTextFile.The configutation "fs.local.block.size" only take effects on > Driver. > If I set that configuration before loading parquet files,the problem is gone. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org