EnricoMi commented on PR #40933:
URL: https://github.com/apache/spark/pull/40933#issuecomment-1535994835
Note that this breaks downstream projects that want to write json:
```scala
spark.read.json("file.json")
```
```
java.lang.NoClassDefFoundError:
com/fasterxml/jackson/core/StreamReadConstraints
at
org.apache.spark.sql.catalyst.json.JSONOptions.buildJsonFactory(JSONOptions.scala:195)
at
org.apache.spark.sql.catalyst.json.JsonInferSchema.$anonfun$infer$1(JsonInferSchema.scala:83)
...
Caused by: java.lang.ClassNotFoundException:
com.fasterxml.jackson.core.StreamReadConstraints
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 16 more
```
The reason is that spark-core depends on avro 1.11.1, which pulls in
jackson-core 2.12.7:
```
[INFO] +- org.apache.spark:spark-core_2.12:jar:3.5.0-SNAPSHOT:provided
[INFO] | +- org.apache.avro:avro:jar:1.11.1:provided
[INFO] | | \- com.fasterxml.jackson.core:jackson-core:jar:2.12.7:provided
```
Project avro has upgraded to jackson 2.15.0 a few days ago:
https://github.com/apache/avro/commit/3b6c6cc43d54ae56b51dacfd6d86d54a0733d57b
I think for this upgrade in Spark to work, the avro dependency of spark-core
has to be upgraded to their next release as well.
My project depends on spark-core and spark-sql only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
