Hi,
But in fact, it does have those packages.
----------------------------------------
D:\02_bigdata\spark-3.5.0-bin-hadoop3\jars
2023/09/09 10:08 75,567 jackson-annotations-2.15.2.jar
2023/09/09 10:08 549,207 jackson-core-2.15.2.jar
2023/09/09 10:08 232,248 jackson-core-asl-1.9.13.jar
2023/09/09 10:08 1,620,088 jackson-databind-2.15.2.jar
2023/09/09 10:08 54,630 jackson-dataformat-yaml-2.15.2.jar
2023/09/09 10:08 122,937 jackson-datatype-jsr310-2.15.2.jar
2023/09/09 10:08 780,664 jackson-mapper-asl-1.9.13.jar
2023/09/09 10:08 513,968 jackson-module-scala_2.12-2.15.2.jar
eabour
From: Bjørn Jørgensen
Date: 2023-11-02 16:40
To: eab...@163.com
CC: user @spark; Saar Barhoom; moshik.vitas
Subject: Re: jackson-databind version mismatch
[SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from
pre-built distribution
tor. 2. nov. 2023 kl. 09:15 skrev Bjørn Jørgensen <bjornjorgen...@gmail.com>:
In spark 3.5.0 removed jackson-core-asl and jackson-mapper-asl those are with
groupid org.codehaus.jackson.
Those others jackson-* are with groupid com.fasterxml.jackson.core
tor. 2. nov. 2023 kl. 01:43 skrev eab...@163.com <eab...@163.com>:
Hi，
Please check the versions of jar files starting with "jackson-". Make sure
all versions are consistent. jackson jar list in spark-3.3.0:
================
2022/06/10 04:37 75,714 jackson-annotations-2.13.3.jar
2022/06/10 04:37 374,895 jackson-core-2.13.3.jar
2022/06/10 04:37 232,248 jackson-core-asl-1.9.13.jar
2022/06/10 04:37 1,536,542 jackson-databind-2.13.3.jar
2022/06/10 04:37 52,020 jackson-dataformat-yaml-2.13.3.jar
2022/06/10 04:37 121,201 jackson-datatype-jsr310-2.13.3.jar
2022/06/10 04:37 780,664 jackson-mapper-asl-1.9.13.jar
2022/06/10 04:37 458,981 jackson-module-scala_2.12-2.13.3.jar
================
Spark 3.3.0 uses Jackson version 2.13.3, while Spark 3.5.0 uses Jackson version
2.15.2. I think you can remove the lower version of Jackson package to keep the
versions consistent.
eabour
From: moshik.vi...@veeva.com.INVALID
Date: 2023-11-01 15:03
To: user@spark.apache.org
CC: 'Saar Barhoom'
Subject: jackson-databind version mismatch
Hi Spark team,
On upgrading spark version from 3.2.1 to 3.4.1 got the following issue:
java.lang.NoSuchMethodError: 'com.fasterxml.jackson.core.JsonGenerator
com.fasterxml.jackson.databind.ObjectMapper.createGenerator(java.io.OutputStream,
com.fasterxml.jackson.core.JsonEncoding)'
at
org.apache.spark.util.JsonProtocol$.toJsonString(JsonProtocol.scala:75)
at
org.apache.spark.SparkThrowableHelper$.getMessage(SparkThrowableHelper.scala:74)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$7(SQLExecution.scala:127)
at scala.Option.map(Option.scala:230)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:125)
at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:195)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:103)
at
org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:4165)
at org.apache.spark.sql.Dataset.head(Dataset.scala:3161)
at org.apache.spark.sql.Dataset.take(Dataset.scala:3382)
at org.apache.spark.sql.Dataset.takeAsList(Dataset.scala:3405)
at
com.crossix.safemine.cloud.utils.DebugRDDLogger.showDataset(DebugRDDLogger.java:84)
at
com.crossix.safemine.cloud.components.statistics.spark.StatisticsTransformer.getFillRateCountsWithSparkQuery(StatisticsTransformer.java:122)
at
com.crossix.safemine.cloud.components.statistics.spark.StatisticsTransformer.calculateStatistics(StatisticsTransformer.java:61)
at
com.crossix.safemine.cloud.components.statistics.spark.SparkFileStatistics.execute(SparkFileStatistics.java:102)
at
com.crossix.safemine.cloud.StatisticsFlow.calculateAllStatistics(StatisticsFlow.java:146)
at
com.crossix.safemine.cloud.StatisticsFlow.runStatistics(StatisticsFlow.java:119)
at
com.crossix.safemine.cloud.StatisticsFlow.initialFileStatistics(StatisticsFlow.java:77)
at com.crossix.safemine.cloud.SMCFlow.process(SMCFlow.java:221)
at com.crossix.safemine.cloud.SMCFlow.execute(SMCFlow.java:132)
at com.crossix.safemine.cloud.SMCFlow.run(SMCFlow.java:91)
I see that that spark package contains the dependency:
com.fasterxml.jackson.core:jackson-databind:jar:2.10.5:compile
But jackson-databind 2.10.5 does not contain
ObjectMapper.createGenerator(java.io.OutputStream,
com.fasterxml.jackson.core.JsonEncoding)
It was added on 2.11.0
Trying to upgrade jackson-databind fails with:
com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.5
requires Jackson Databind version >= 2.10.0 and < 2.11.0
According to spark 3.3.0 release notes: "Upgrade Jackson to 2.13.3" but spark
package of 3.4.1 contains Jackson of 2.10.5
(https://spark.apache.org/releases/spark-release-3-3-0.html)
What am I missing?
--
Moshik Vitas
Senior Software Developer, Crossix
Veeva Systems
m: +972-54-5326-400
moshik.vi...@veeva.com
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
