[
https://issues.apache.org/jira/browse/SPARK-23043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-23043.
-------------------------------
Resolution: Fixed
Fix Version/s: 2.4.0
Resolved by https://github.com/apache/spark/pull/20233
> Upgrade json4s-jackson to 3.5.3
> -------------------------------
>
> Key: SPARK-23043
> URL: https://issues.apache.org/jira/browse/SPARK-23043
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 2.2.1
> Reporter: Takako Shimamoto
> Assignee: Takako Shimamoto
> Priority: Minor
> Fix For: 2.4.0
>
>
> Spark tried to upgrade json4s several times, but failed and still use a few
> years old version 3.2.11. I hope the situation will improve soon.
> h3. Migration from 3.2.11
> Changes that affect Spark are as follows.
> h4. scalap dependency
> Since 3.3, json4s has its own scalap fork and doesn't depend on scala-lang's
> scalap.
> https://github.com/json4s/json4s/issues/108
> Whereas Spark has added a scala-lang's scalap dependency in POM by the
> following issue.
> https://github.com/apache/spark/pull/480
> We need to remove scala-lang's scalap dependency for json4s 3.5.3 (same
> solution as in json4s).
> h4. The behavior of \ function has changed
> Please see https://github.com/json4s/json4s/pull/309
> h5. a single element
> {code}
> val json =
> """[ {
> | "jobId" : 0,
> | "status" : "SUCCEEDED"
> |} ]""".stripMargin
> val ast = JsonMethods.parse(json)
> {code}
> * version 3.2.11
> {code}
> ast \ "status" // => JString(SUCCEEDED)
> ast \\ "status" // => JString(SUCCEEDED)
> {code}
> * version 3.5.3
> {code}
> ast \ "status" // => JArray(List(JString(SUCCEEDED)))
> ast \\ "status" // => JString(SUCCEEDED)
> {code}
> h5. more than a single elements
> {code}
> val json =
> """[ {
> | "jobId" : 1,
> | "status" : "FAILED"
> |}, {
> | "jobId" : 0,
> | "status" : "SUCCEEDED"
> |} ]""".stripMargin
> val ast = JsonMethods.parse(json)
> {code}
> * version 3.2.11
> {code}
> ast \ "status" // => JArray(List(JString(FAILED), JString(SUCCEEDED)))
> ast \\ "status" // => JObject(List((status,JString(FAILED)),
> (status,JString(SUCCEEDED))))
> {code}
> * version 3.5.3
> {code}
> ast \ "status" // => JArray(List(JString(FAILED), JString(SUCCEEDED)))
> ast \\ "status" // => JObject(List((status,JString(FAILED)),
> (status,JString(SUCCEEDED))))
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
