[ https://issues.apache.org/jira/browse/SPARK-24682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527767#comment-16527767 ]
Apache Spark commented on SPARK-24682: -------------------------------------- User 'patrickmcgloin' has created a pull request for this issue: https://github.com/apache/spark/pull/21671 > from_json / to_json do not handle java.sql.Date inside Maps correctly > --------------------------------------------------------------------- > > Key: SPARK-24682 > URL: https://issues.apache.org/jira/browse/SPARK-24682 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.3.0 > Reporter: Patrick McGloin > Priority: Minor > > When a date is one of the types inside a Map, the from_json and to_json > functions do not handle it correctly. A date would be persisted like this: > "17710". And the from_json tries to read it in it complains with the error: > java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot > be cast to java.lang.Integer > Consider the following test, which will fail on the final show: > {color:#000080} > *case class* {color}UnitTestCaseClassWithDateInsideMap(map: > {color:#009f82}Map{color}[Date, Int]) > test({color:#008000}*"Test a Date as key in a Map"*{color}) { > {color:#000080}*val* {color}map = > _UnitTestCaseClassWithDateInsideMap_({color:#800080}_Map_{color}(Date._valueOf_({color:#008000}*"2018-06-28"*{color}) > -> {color:#0000FF}1{color})) > {color:#000080}*val* {color}options = > {color:#800080}_Map_{color}({color:#008000}*"timestampFormat"* {color}-> > {color:#008000}*"yyyy/MM/dd HH:mm:ss.SSS"*{color}, > {color:#008000}*"dateFormat"* {color}-> {color:#008000}*"yyyy/MM/dd"*{color}) > {color:#000080}*val* {color}schema = > Encoders._product_[UnitTestCaseClassWithDateInsideMap].schema > {color:#000080}*val* {color}mapDF = {color:#800080}_Seq_{color}(map).toDF() > {color:#000080}*val* {color}jsonDF = > mapDF.select(_to_json_(_struct_(mapDF.columns.head, mapDF.columns.tail:_*), > options)) > jsonDF.show() > {color:#000080}*val* {color}jsonString = > jsonDF.map(_.getString({color:#0000FF}0{color})).collect().head > {color:#000080}*val* {color}stringDF = > {color:#800080}_Seq_{color}(jsonString).toDF({color:#008000}*"json"*{color}) > {color:#000080}*val* {color}parsedDF = > stringDF.select(_from_json_({color:#008000}*$"json"*{color}, schema, options)) > parsedDF.show() > } > The result of the line "jsonDF.show()" is as follows: > +---------------------------------------------------+ > |structstojson(named_struct(NamePlaceholder(), map))| > +---------------------------------------------------+ > | \{"map":{"17710":1}}| > +---------------------------------------------------+ > As can be seen the date is not formatted correctly. The error with > "parsedDF.show()" is: > java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot > be cast to java.lang.Integer > As you can see I have added in the options with date formats but that does > not have any impact. If I do the same tests with a Date outside a Map it > works as expected. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org