Marius Soutier created SPARK-6504:
-------------------------------------
Summary: Cannot read Parquet files generated from different
versions at once
Key: SPARK-6504
URL: https://issues.apache.org/jira/browse/SPARK-6504
Project: Spark
Issue Type: Bug
Affects Versions: 1.2.1
Reporter: Marius Soutier
When trying to read Parquet files generated by Spark 1.1.1 and 1.2.1 at the
same time via
`sqlContext.parquetFile("fileFrom1.1.parqut,fileFrom1.2.parquet")` an exception
occurs:
could not merge metadata: key org.apache.spark.sql.parquet.row.metadata has
conflicting values:
[{"type":"struct","fields":[{"name":"date","type":"string","nullable":true,"metadata":{}},{"name":"account","type":"string","nullable":true,"metadata":{}},{"name":"impressions","type":"long","nullable":false,"metadata":{}},{"name":"cost","type":"double","nullable":false,"metadata":{}},{"name":"clicks","type":"long","nullable":false,"metadata":{}},{"name":"conversions","type":"long","nullable":false,"metadata":{}},{"name":"orderValue","type":"double","nullable":false,"metadata":{}}]},
StructType(List(StructField(date,StringType,true),
StructField(account,StringType,true), StructField(impressions,LongType,false),
StructField(cost,DoubleType,false), StructField(clicks,LongType,false),
StructField(conversions,LongType,false),
StructField(orderValue,DoubleType,false)))]
The Schema is exactly equal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
