[ https://issues.apache.org/jira/browse/SPARK-10954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-10954. ------------------------------- Resolution: Not A Problem > Parquet version in the "created_by" metadata field of Parquet files written > by Spark 1.5 and 1.6 is wrong > --------------------------------------------------------------------------------------------------------- > > Key: SPARK-10954 > URL: https://issues.apache.org/jira/browse/SPARK-10954 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.5.0, 1.5.1, 1.6.0 > Reporter: Cheng Lian > Assignee: Gayathri Murali > Priority: Minor > > We've upgraded to parquet-mr 1.7.0 in Spark 1.5, but the {{created_by}} field > still says 1.6.0. This issue can be reproduced by generating any Parquet file > with Spark 1.5, and then check the metadata with {{parquet-meta}} CLI tool: > {noformat} > $ parquet-meta /tmp/parquet/dec > file: > file:/tmp/parquet/dec/part-r-00000-f210e968-1be5-40bc-bcbc-007f935e6dc7.gz.parquet > creator: parquet-mr version 1.6.0 > extra: org.apache.spark.sql.parquet.row.metadata = > {"type":"struct","fields":[{"name":"dec","type":"decimal(20,2)","nullable":true,"metadata":{}}]} > file schema: spark_schema > --------------------------------------------------------------------------------------------------------------------------------------------------------- > dec: OPTIONAL FIXED_LEN_BYTE_ARRAY O:DECIMAL R:0 D:1 > row group 1: RC:10 TS:140 OFFSET:4 > --------------------------------------------------------------------------------------------------------------------------------------------------------- > dec: FIXED_LEN_BYTE_ARRAY GZIP DO:0 FPO:4 SZ:99/140/1.41 VC:10 > ENC:PLAIN,BIT_PACKED,RLE > {noformat} > Note that this field is written by parquet-mr rather than Spark. However, > writing Parquet files using parquet-mr 1.7.0 directly without Spark 1.5 only > shows {{parquet-mr}} without any version number. Files written by parquet-mr > 1.8.1 without Spark look fine though. > Currently this isn't a big issue. But parquet-mr 1.8 checks for this field to > workaround PARQUET-251. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org