Hi all, I tested this with a Date outside a map and it works fine so I think the issue is simply for Dates inside Maps. I will create a Jira for this unless there are objections.
Best regards, Patrick On Thu, 28 Jun 2018, 11:53 Patrick McGloin, <mcgloin.patr...@gmail.com> wrote: > Consider the following test, which will fail on the final show: > > * case class *UnitTestCaseClassWithDateInsideMap(map: Map[Date, Int]) > > test(*"Test a Date as key in a Map"*) { > *val *map = *UnitTestCaseClassWithDateInsideMap*(*Map*(Date.*valueOf*( > *"2018-06-28"*) -> 1)) > *val *options = *Map*(*"timestampFormat" *-> *"yyyy/MM/dd HH:mm:ss.SSS"*, > *"dateFormat" *-> *"yyyy/MM/dd"*) > *val *schema = Encoders.*product* > [UnitTestCaseClassWithDateInsideMap].schema > > *val *mapDF = *Seq*(map).toDF() > *val *jsonDF = mapDF.select(*to_json*(*struct*(mapDF.columns.head, > mapDF.columns.tail:_*), options)) > jsonDF.show() > > *val *jsonString = jsonDF.map(_.getString(0)).collect().head > > *val *stringDF = *Seq*(jsonString).toDF(*"json"*) > *val *parsedDF = stringDF.select(*from_json*(*$"json"*, schema, options)) > parsedDF.show() > } > > > The result of the line "jsonDF.show()" is as follows: > > +---------------------------------------------------+ > |structstojson(named_struct(NamePlaceholder(), map))| > +---------------------------------------------------+ > | {"map":{"17710":1}}| > +---------------------------------------------------+ > > As can be seen the date is not formatted correctly. The error with > "parsedDF.show()" is: > > java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String > cannot be cast to java.lang.Integer > > I have tried adding the options to to_json / from_json but it hasn't > helped. Am I using the wrong options? > > Is there another way to do this? > > Best regards, > Patrick > This message has been sent by ABN AMRO Bank N.V., which has its seat at Gustav > Mahlerlaan 10 (1082 PP) Amsterdam, the Netherlands > <https://maps.google.com/?q=Gustav+Mahlerlaan+10+(1082+PP)+Amsterdam,+the+Netherlands&entry=gmail&source=g>, > and is registered in the Commercial Register of Amsterdam under number > 34334259. >