Re: Upgrading to Spark 2.0.1 broke array in parquet DataFrame

If you can reproduce the issue with Spark 2.0.2 I'd suggest opening a JIRA. On Fri, Nov 4, 2016 at 5:11 PM, Sam Goodwin wrote: > I have a table with a few columns, some of which are arrays. Since > upgrading from Spark 1.6 to Spark 2.0.1, the array fields are always

Upgrading to Spark 2.0.1 broke array in parquet DataFrame

I have a table with a few columns, some of which are arrays. Since upgrading from Spark 1.6 to Spark 2.0.1, the array fields are always null when reading in a DataFrame. When writing the Parquet files, the schema of the column is specified as StructField("packageIds",ArrayType(StringType)) The