ayushtkn commented on PR #342:
URL: https://github.com/apache/tez/pull/342#issuecomment-2558341632
hmm, Something from HDFS, some jackson-databind transitive dependency
missing or so. The entire trace is like
```
java.lang.NoClassDefFoundError:
com/fasterxml/jackson/core/util/JacksonFeature
at
com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:656)
at
com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:558)
at
org.apache.hadoop.hdfs.server.blockmanagement.SlowPeerTracker.<clinit>(SlowPeerTracker.java:78)
at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.initSlowPeerTracker(DatanodeManager.java:373)
at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.<init>(DatanodeManager.java:263)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:502)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:926)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:851)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1396)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:495)
at
org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:256)
at
org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1158)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1042)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:974)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:594)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:533)
at
org.apache.tez.tests.ExternalTezServiceTestHelper.<init>(ExternalTezServiceTestHelper.java:63)
at
org.apache.tez.tests.TestExternalTezServices.setup(TestExternalTezServices.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at
com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.ClassNotFoundException:
com.fasterxml.jackson.core.util.JacksonFeature
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 37 more
```
Looking at the dependency tree now, Jackson 2.10 is getting pulled in by Avro
```
ayushsaxena@ayushsaxena tez-ext-service-tests % mvn clean dependency:tree |
grep jackson-core:jar -B4
[INFO] | +- org.apache.commons:commons-configuration2:jar:2.10.1:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.12.0:compile
[INFO] | +- org.apache.commons:commons-text:jar:1.10.0:compile
[INFO] | +- org.apache.avro:avro:jar:1.9.2:compile
[INFO] | | \- com.fasterxml.jackson.core:jackson-core:jar:2.10.2:compile
```
which earlier was coming from Jackson-databind defined in hadoop-common
```
ayushsaxena@ayushsaxena tez-ext-service-tests % mvn clean dependency:tree |
grep jackson-core:jar -B4
[INFO] | | +- org.apache.kerby:kerby-asn1:jar:1.0.1:compile
[INFO] | | \- org.apache.kerby:kerby-util:jar:1.0.1:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.7.1:compile
[INFO] | | +-
com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile
[INFO] | | \- com.fasterxml.jackson.core:jackson-core:jar:2.12.7:compile
ayushsaxena@ayushsaxena tez-ext-service-tests %
```
Why it started happening, I am not very sure but there was some Avro related
change in Hadoop
https://github.com/apache/hadoop/pull/7007/files#diff-a74b4a65ab1f4aed61344787bb9654ba7e835154730653848c88ca11f9965dc0R301
Ideally I feel we should exclude Jackson coming from Avro in Hadoop itself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
