rahil-c commented on code in PR #6151:
URL: https://github.com/apache/hudi/pull/6151#discussion_r927272053
##########
packaging/hudi-spark-bundle/pom.xml:
##########
@@ -95,6 +95,12 @@
<include>org.antlr:stringtemplate</include>
<include>org.apache.parquet:parquet-avro</include>
+
<include>com.fasterxml.jackson.core:jackson-annotations</include>
+ <include>com.fasterxml.jackson.core:jackson-core</include>
+
<include>com.fasterxml.jackson.core:jackson-databind</include>
+
<include>com.fasterxml.jackson.dataformat:jackson-dataformat-yaml</include>
+
<include>com.fasterxml.jackson.module:jackson-module-scala_${scala.binary.version}</include>
Review Comment:
When running the IT tests with the spark3 was running into this dependency
conflict below
```
Exception in thread "main" java.lang.NoSuchMethodError:
com.fasterxml.jackson.databind.JsonMappingException.<init>(Ljava/io/Closeable;Ljava/lang/String;)V
at
com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
at
com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:46)
at
com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
at
com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:718)
at
org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at
org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
at org.apache.spark.SparkContext.withScope(SparkContext.scala:792)
at org.apache.spark.SparkContext.parallelize(SparkContext.scala:809)
at
org.apache.spark.api.java.JavaSparkContext.parallelize(JavaSparkContext.scala:136)
at HoodieJavaApp.run(HoodieJavaApp.java:141)
at HoodieJavaApp.main(HoodieJavaApp.java:111)
```
so in hudi-spark pom we define the following depedency which should ideally
provide the class and not result in this class not found
```
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
<version>${fasterxml.jackson.module.scala.version}</version>
</dependency>
```
this jackson module scala contains several jackson dependencies like
jackson-data bind etc.
From the mvn logs however it seems it was not getting included in the bundle
in several areas and was being excluded. So in order to get past this conflict
added it in the bundle.
```
[INFO] Excluding
com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.10.0 from the
shaded jar.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
