Which version of Joda are you using ?
Here is snippet of dependency:tree out w.r.t. Joda :
[INFO] +- org.apache.flume:flume-ng-core:jar:1.4.0:compile
...
[INFO] | +- joda-time:joda-time:jar:2.1:compile
FYI
On Wed, Apr 8, 2015 at 12:53 PM, Patrick Grandjean <p.r.grandj...@gmail.com>
wrote:
> Hi,
>
> I have an RDD with objects containing Joda's LocalDate. When trying to
> save the RDD as Parquet, I get an exception. Here is the code:
>
>
> -------------------------------------------------------------------------------------
> val sqlC = new org.apache.spark.sql.SQLContext(sc)
> import sqlC._
>
> myRDD.saveAsParquetFile("parquet")
>
> -------------------------------------------------------------------------------------
>
> The exception:
>
> Exception in thread "main" scala.MatchError: org.joda.time.LocalDate (of
> class scala.reflect.internal.Types$TypeRef$$anon$6)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:105)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:33)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:125)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:123)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:123)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:33)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:100)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:33)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$class.attributesFor(ScalaReflection.scala:94)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.attributesFor(ScalaReflection.scala:33)
> at
> org.apache.spark.sql.SQLContext.createSchemaRDD(SQLContext.scala:111)
>
> Is it possible to extend Spark with adapters in order to support new
> types? How to add support for Joda types?
>
> I am using spark 1.2.1 with cloudera 5.3.2
>
> Patrick.
>
