[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
[ https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812647#comment-17812647 ] Ayush Saxena commented on HADOOP-18716: --- Nopes, feel free to take over If you plan to chase > [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8 > - > > Key: HADOOP-18716 > URL: https://issues.apache.org/jira/browse/HADOOP-18716 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinay Devadiga >Priority: Critical > > Compiled Hadoop - Hadoop branch 3.3.3 > mvn clean install - DskipTests=True > Java_Home -> points to Java-8 > maven version - 3.8.8 (Quite latest) > > Ran various whole test suit on my private cloud environment - > Changed Java_Home to Java-17 > > mvn surefire:test > > Out of 22k tests - 2.5 k tests failed . -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
[ https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812565#comment-17812565 ] Bilwa S T commented on HADOOP-18716: [~ayushtkn] Are you working on this? > [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8 > - > > Key: HADOOP-18716 > URL: https://issues.apache.org/jira/browse/HADOOP-18716 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinay Devadiga >Priority: Critical > > Compiled Hadoop - Hadoop branch 3.3.3 > mvn clean install - DskipTests=True > Java_Home -> points to Java-8 > maven version - 3.8.8 (Quite latest) > > Ran various whole test suit on my private cloud environment - > Changed Java_Home to Java-17 > > mvn surefire:test > > Out of 22k tests - 2.5 k tests failed . -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
[ https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779174#comment-17779174 ] Dheeren Beborrtha commented on HADOOP-18716: Any target time to fix these ? Thx. > [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8 > - > > Key: HADOOP-18716 > URL: https://issues.apache.org/jira/browse/HADOOP-18716 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinay Devadiga >Priority: Critical > > Compiled Hadoop - Hadoop branch 3.3.3 > mvn clean install - DskipTests=True > Java_Home -> points to Java-8 > maven version - 3.8.8 (Quite latest) > > Ran various whole test suit on my private cloud environment - > Changed Java_Home to Java-17 > > mvn surefire:test > > Out of 22k tests - 2.5 k tests failed . -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
[ https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17731795#comment-17731795 ] Wei-Chiu Chuang commented on HADOOP-18716: -- Please add a target version. > [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8 > - > > Key: HADOOP-18716 > URL: https://issues.apache.org/jira/browse/HADOOP-18716 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinay Devadiga >Priority: Critical > > Compiled Hadoop - Hadoop branch 3.3.3 > mvn clean install - DskipTests=True > Java_Home -> points to Java-8 > maven version - 3.8.8 (Quite latest) > > Ran various whole test suit on my private cloud environment - > Changed Java_Home to Java-17 > > mvn surefire:test > > Out of 22k tests - 2.5 k tests failed . -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
[ https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720220#comment-17720220 ] Ayush Saxena commented on HADOOP-18716: --- Thanx [~vinay._.devadiga] for the pointers. Let me find some time and raise a PR with these findings. In case you find some time feel free to takeover. > [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8 > - > > Key: HADOOP-18716 > URL: https://issues.apache.org/jira/browse/HADOOP-18716 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinay Devadiga >Priority: Critical > > Compiled Hadoop - Hadoop branch 3.3.3 > mvn clean install - DskipTests=True > Java_Home -> points to Java-8 > maven version - 3.8.8 (Quite latest) > > Ran various whole test suit on my private cloud environment - > Changed Java_Home to Java-17 > > mvn surefire:test > > Out of 22k tests - 2.5 k tests failed . -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
[ https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720086#comment-17720086 ] Vinay Devadiga commented on HADOOP-18716: - Hi [~ayushtkn] i dont have band width to raise PR , but here are most of the fixes explanation here are there tested it out . Any body can raise PR . h3. Appendix #1.The first major error which we encountered was the following stack trace - java.lang.NoClassDefFoundError: Could not initialize class com.google.inject.internal.cglib.core.$ReflectUtils at com.google.inject.internal.cglib.reflect.$FastClassEmitter.(FastClassEmitter.java:67) at com.google.inject.internal.cglib.reflect.$FastClass$Generator.generateClass(FastClass.java:72) This error seems to be the lack of support of the current guice version 4.0 for jre 17 . so after upgrading the guice version to 5.1.0 reduced the {*}errors by the 300{*}. *Ref:* [https://github.com/google/guice/issues/1536] [https://github.com/google/guice/pull/1529/files] --4.0 ++5.1.0 -- 2 . The second major error which we encountered the child jvms spawned by the maven do not have major add-opens which were causing various failures . ``` 1. java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts,TestJerseyClient 2. java.io.IOException: 0.0.0.0:43529: Unexpected HTTP response: code=500 != 200, op=MKDIRS, message=Server Error 4. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 6. java.lang.Exception: Unexpected exception, expected but was 7. org.apache.hadoop.service.ServiceStateException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues 8. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.ExceptionInInitializerError ``` and many more To fix this we use the arg line child property of -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError We add the following - add-opens in the arg line and in the child projects --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.math=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED This fix brought down the errors by 2000. So the current count is {*}reduced to 500{*}. --- 3. There were various mockito exceptions of these following stack trace java.lang.NullPointerException: Cannot enter synchronized block because "this.lock" is null Mockito cannot mock this class: class com.sun.jersey.api.client.ClientResponse. If you're not sure why you're getting this error, please report to the mailing list. Java : 17 JVM vendor name : Amazon.com Inc. JVM vendor version : 17.0.6+10-LTS JVM name : OpenJDK 64-Bit Server VM JVM version : 17.0.6+10-LTS JVM info : mixed mode, sharing OS name : Linux OS version : 4.14.309-231.529.amzn2.x86_64 You are seeing this disclaimer because Mockito is configured to create inlined mocks. You can learn about inline mocks and their limitations under item #39 of the Mockito class javadoc. Underlying exception : org.mockito.exceptions.base.MockitoException: Could not modify all classes [class com.sun.jersey.api.client.ClientResponse, class java.lang.Object] etc .These errors got fix after updating the mockito-core version and its scope org.mockito mockito-core 2.28.2 3.11.2 provided Other mockito exceptions are failing with following stack trace - Caused by: java.lang.UnsatisfiedLinkError: jdk.internal.misc.Unsafe.registerNatives()V at jdk.internal.misc.Unsafe.registerNatives(Native Method) at jdk.internal.misc.Unsafe.(Unsafe.java:57) at java.base/java.lang.Class.forName0(Native Method) This is surely because we have not access to {{Unsafe}} . Ref: [https://github.com/jboss-javassist/javassist/issues/227] [https://github.com/mockito/mockito/issues/1207] [. https://github.com/mockito/mockito/issues/2164|https://github.com/mockito/mockito/issues/2164] This brought down *error failures to 200* . # There were some other minor errors like for [MetricsInvariantChecker
[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
[ https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715693#comment-17715693 ] Ayush Saxena commented on HADOOP-18716: --- [~vinay._.devadiga] should raise a PR to upgrade Guice. I can validate locally. We don't use github actions for builds. The most you can plan just for your PR may be somewhere here install & change to Java17 [https://github.com/apache/hadoop/blob/trunk/dev-support/docker/Dockerfile#L61] > [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8 > - > > Key: HADOOP-18716 > URL: https://issues.apache.org/jira/browse/HADOOP-18716 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinay Devadiga >Priority: Critical > > Compiled Hadoop - Hadoop branch 3.3.3 > mvn clean install - DskipTests=True > Java_Home -> points to Java-8 > maven version - 3.8.8 (Quite latest) > > Ran various whole test suit on my private cloud environment - > Changed Java_Home to Java-17 > > mvn surefire:test > > Out of 22k tests - 2.5 k tests failed . -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
[ https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715689#comment-17715689 ] Vinay Devadiga commented on HADOOP-18716: - Another one - {color:#16191f}ava.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts,TestJerseyClient org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices [ERROR] testTaskAttemptsSlash(org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts) Time elapsed: 0.001 s <<< ERROR! {color}{color:#00}java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttemptsat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)at org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:250)at org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:260)at org.junit.runners.BlockJUnit4ClassRunner$2.runReflectiveCall(BlockJUnit4ClassRunner.java:309)at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:306)at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63){color} > [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8 > - > > Key: HADOOP-18716 > URL: https://issues.apache.org/jira/browse/HADOOP-18716 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinay Devadiga >Priority: Critical > > Compiled Hadoop - Hadoop branch 3.3.3 > mvn clean install - DskipTests=True > Java_Home -> points to Java-8 > maven version - 3.8.8 (Quite latest) > > Ran various whole test suit on my private cloud environment - > Changed Java_Home to Java-17 > > mvn surefire:test > > Out of 22k tests - 2.5 k tests failed . -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
[ https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715687#comment-17715687 ] Vinay Devadiga commented on HADOOP-18716: - Here is the most common error stack trace . {color:#16191f}java.lang.NoClassDefFoundError: Could not initialize class com.google.inject.internal.cglib.core.$ReflectUtils - {color} |java.lang.NoClassDefFoundError: Could not initialize class com.google.inject.internal.cglib.core.$ReflectUtils| |at com.google.inject.internal.cglib.reflect.$FastClassEmitter.(FastClassEmitter.java:67)| |at com.google.inject.internal.cglib.reflect.$FastClass$Generator.generateClass(FastClass.java:72)| |at com.google.inject.internal.cglib.core.$DefaultGeneratorStrategy.generate(DefaultGeneratorStrategy.java:25)| |at com.google.inject.internal.cglib.core.$AbstractClassGenerator.create(AbstractClassGenerator.java:216)| |at com.google.inject.internal.cglib.reflect.$FastClass$Generator.create(FastClass.java:64)| |at com.google.inject.internal.BytecodeGen.newFastClass(BytecodeGen.java:204)| |at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.(ProviderMethod.java:256)| |at com.google.inject.internal.ProviderMethod.create(ProviderMethod.java:71)| |at com.google.inject.internal.ProviderMethodsModule.createProviderMethod(ProviderMethodsModule.java:275)| |at com.google.inject.internal.ProviderMethodsModule.getProviderMethods(ProviderMethodsModule.java:144)| |at com.google.inject.internal.ProviderMethodsModule.configure(ProviderMethodsModule.java:123)| |at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)| |at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:349)| |at com.google.inject.AbstractModule.install(AbstractModule.java:122)| |at com.google.inject.servlet.ServletModule.configure(ServletModule.java:52)| |at com.google.inject.AbstractModule.configure(AbstractModule.java:62)| |at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)| |at com.google.inject.spi.Elements.getElements(Elements.java:110)| |at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)| |at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)| |at com.google.inject.Guice.createInjector(Guice.java:96)| {color:#16191f}It seems there is no support for guice 4 for java 17. need to upgrade to guice 5 Test name -{*}org.apache.hadoop.yarn.webapp.view.TestInfoBlock{*} {color}[https://github.com/google/guice/issues/1536] [https://github.com/google/guice/pull/1529/files] [https://bugs.openjdk.org/browse/JDK-8267178] [https://groups.google.com/g/google-guice/c/0inCjcpWUwc]{color:#00}. {color} > [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8 > - > > Key: HADOOP-18716 > URL: https://issues.apache.org/jira/browse/HADOOP-18716 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinay Devadiga >Priority: Critical > > Compiled Hadoop - Hadoop branch 3.3.3 > mvn clean install - DskipTests=True > Java_Home -> points to Java-8 > maven version - 3.8.8 (Quite latest) > > Ran various whole test suit on my private cloud environment - > Changed Java_Home to Java-17 > > mvn surefire:test > > Out of 22k tests - 2.5 k tests failed . -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
[ https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715686#comment-17715686 ] Vinay Devadiga commented on HADOOP-18716: - Yes [~ayushtkn] , by the way is there , way to run unit test through github actions . Where I can change the java environment . > [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8 > - > > Key: HADOOP-18716 > URL: https://issues.apache.org/jira/browse/HADOOP-18716 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinay Devadiga >Priority: Critical > > Compiled Hadoop - Hadoop branch 3.3.3 > mvn clean install - DskipTests=True > Java_Home -> points to Java-8 > maven version - 3.8.8 (Quite latest) > > Ran various whole test suit on my private cloud environment - > Changed Java_Home to Java-17 > > mvn surefire:test > > Out of 22k tests - 2.5 k tests failed . -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
[ https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715648#comment-17715648 ] Ayush Saxena commented on HADOOP-18716: --- the branch targeted to trunk, we plan to chase JDK-17 for trunk only mostly. let me know if you plan to fix stuff, I can add you as contributor & assign or extend some help > [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8 > - > > Key: HADOOP-18716 > URL: https://issues.apache.org/jira/browse/HADOOP-18716 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinay Devadiga >Priority: Blocker > > Compiled Hadoop - Hadoop branch 3.3.3 > mvn clean install - DskipTests=True > Java_Home -> points to Java-8 > maven version - 3.8.8 (Quite latest) > > Ran various whole test suit on my private cloud environment - > Changed Java_Home to Java-17 > > mvn surefire:test > > Out of 22k tests - 2.5 k tests failed . -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org