Can you show your code please?
On 28 Apr 2015 13:20, "sranga" <[email protected]> wrote:

> Hi
>
> I am getting the following error when persisting an RDD in parquet format
> to
> an S3 location. This is code that was working in the 1.2 version. The
> version that it is failing to work is 1.3.1.
> Any help is appreciated.
>
> Caused by: java.lang.AssertionError: assertion failed: Conflicting
> partition
> column names detected:
>         ArrayBuffer(batch_id)
> ArrayBuffer()
>         at scala.Predef$.assert(Predef.scala:179)
>         at
>
> org.apache.spark.sql.parquet.ParquetRelation2$.resolvePartitions(newParquet.scala:933)
>         at
>
> org.apache.spark.sql.parquet.ParquetRelation2$.parsePartitions(newParquet.scala:851)
>         at
>
> org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$7.apply(newParquet.scala:311)
>         at
>
> org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$7.apply(newParquet.scala:303)
>         at scala.Option.getOrElse(Option.scala:120)
>         at
>
> org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache.refresh(newParquet.scala:303)
>         at
> org.apache.spark.sql.parquet.ParquetRelation2.insert(newParquet.scala:692)
>         at
>
> org.apache.spark.sql.parquet.DefaultSource.createRelation(newParquet.scala:129)
>         at
> org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:240)
>         at org.apache.spark.sql.DataFrame.save(DataFrame.scala:1196)
>         at
> org.apache.spark.sql.DataFrame.saveAsParquetFile(DataFrame.scala:995)
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/1-3-1-Persisting-RDD-in-parquet-Conflicting-partition-column-names-tp22678.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>

Reply via email to