[
https://issues.apache.org/jira/browse/SPARK-6504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Wendell updated SPARK-6504:
-----------------------------------
Component/s: SQL
> Cannot read Parquet files generated from different versions at once
> -------------------------------------------------------------------
>
> Key: SPARK-6504
> URL: https://issues.apache.org/jira/browse/SPARK-6504
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.2.1
> Reporter: Marius Soutier
>
> When trying to read Parquet files generated by Spark 1.1.1 and 1.2.1 at the
> same time via
> `sqlContext.parquetFile("fileFrom1.1.parqut,fileFrom1.2.parquet")` an
> exception occurs:
> could not merge metadata: key org.apache.spark.sql.parquet.row.metadata has
> conflicting values:
> [{"type":"struct","fields":[{"name":"date","type":"string","nullable":true,"metadata":{}},{"name":"account","type":"string","nullable":true,"metadata":{}},{"name":"impressions","type":"long","nullable":false,"metadata":{}},{"name":"cost","type":"double","nullable":false,"metadata":{}},{"name":"clicks","type":"long","nullable":false,"metadata":{}},{"name":"conversions","type":"long","nullable":false,"metadata":{}},{"name":"orderValue","type":"double","nullable":false,"metadata":{}}]},
> StructType(List(StructField(date,StringType,true),
> StructField(account,StringType,true),
> StructField(impressions,LongType,false), StructField(cost,DoubleType,false),
> StructField(clicks,LongType,false), StructField(conversions,LongType,false),
> StructField(orderValue,DoubleType,false)))]
> The Schema is exactly equal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
