[
https://issues.apache.org/jira/browse/SPARK-24682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526687#comment-16526687
]
Patrick McGloin commented on SPARK-24682:
-----------------------------------------
I would like to work on this.
> from_json / to_json do not handle java.sql.Date inside Maps correctly
> ---------------------------------------------------------------------
>
> Key: SPARK-24682
> URL: https://issues.apache.org/jira/browse/SPARK-24682
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Patrick McGloin
> Priority: Minor
>
> When a date is one of the types inside a Map, the from_json and to_json
> functions do not handle it correctly. A date would be persisted like this:
> "17710". And the from_json tries to read it in it complains with the error:
> java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot
> be cast to java.lang.Integer
> Consider the following test, which will fail on the final show:
> {color:#000080}
> *case class* {color}UnitTestCaseClassWithDateInsideMap(map:
> {color:#009f82}Map{color}[Date, Int])
> test({color:#008000}*"Test a Date as key in a Map"*{color}) {
> {color:#000080}*val* {color}map =
> _UnitTestCaseClassWithDateInsideMap_({color:#800080}_Map_{color}(Date._valueOf_({color:#008000}*"2018-06-28"*{color})
> -> {color:#0000FF}1{color}))
> {color:#000080}*val* {color}options =
> {color:#800080}_Map_{color}({color:#008000}*"timestampFormat"* {color}->
> {color:#008000}*"yyyy/MM/dd HH:mm:ss.SSS"*{color},
> {color:#008000}*"dateFormat"* {color}-> {color:#008000}*"yyyy/MM/dd"*{color})
> {color:#000080}*val* {color}schema =
> Encoders._product_[UnitTestCaseClassWithDateInsideMap].schema
> {color:#000080}*val* {color}mapDF = {color:#800080}_Seq_{color}(map).toDF()
> {color:#000080}*val* {color}jsonDF =
> mapDF.select(_to_json_(_struct_(mapDF.columns.head, mapDF.columns.tail:_*),
> options))
> jsonDF.show()
> {color:#000080}*val* {color}jsonString =
> jsonDF.map(_.getString({color:#0000FF}0{color})).collect().head
> {color:#000080}*val* {color}stringDF =
> {color:#800080}_Seq_{color}(jsonString).toDF({color:#008000}*"json"*{color})
> {color:#000080}*val* {color}parsedDF =
> stringDF.select(_from_json_({color:#008000}*$"json"*{color}, schema, options))
> parsedDF.show()
> }
> The result of the line "jsonDF.show()" is as follows:
> +---------------------------------------------------+
> |structstojson(named_struct(NamePlaceholder(), map))|
> +---------------------------------------------------+
> | \{"map":{"17710":1}}|
> +---------------------------------------------------+
> As can be seen the date is not formatted correctly. The error with
> "parsedDF.show()" is:
> java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot
> be cast to java.lang.Integer
> As you can see I have added in the options with date formats but that does
> not have any impact. If I do the same tests with a Date outside a Map it
> works as expected.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
