[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8

2024-01-31 Thread Ayush Saxena (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812647#comment-17812647
 ] 

Ayush Saxena commented on HADOOP-18716:
---

Nopes, feel free to take over If you plan to chase

> [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
> -
>
> Key: HADOOP-18716
> URL: https://issues.apache.org/jira/browse/HADOOP-18716
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Vinay Devadiga
>Priority: Critical
>
> Compiled Hadoop - Hadoop branch 3.3.3
> mvn clean install - DskipTests=True
> Java_Home ->  points to Java-8
> maven version - 3.8.8 (Quite latest)
>  
> Ran various whole test suit on my private cloud environment -  
> Changed Java_Home to   Java-17 
>  
> mvn surefire:test 
>  
> Out of 22k tests - 2.5 k tests failed .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8

2024-01-30 Thread Bilwa S T (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812565#comment-17812565
 ] 

Bilwa S T commented on HADOOP-18716:


[~ayushtkn] Are you working on this?

> [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
> -
>
> Key: HADOOP-18716
> URL: https://issues.apache.org/jira/browse/HADOOP-18716
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Vinay Devadiga
>Priority: Critical
>
> Compiled Hadoop - Hadoop branch 3.3.3
> mvn clean install - DskipTests=True
> Java_Home ->  points to Java-8
> maven version - 3.8.8 (Quite latest)
>  
> Ran various whole test suit on my private cloud environment -  
> Changed Java_Home to   Java-17 
>  
> mvn surefire:test 
>  
> Out of 22k tests - 2.5 k tests failed .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8

2023-10-24 Thread Dheeren Beborrtha (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779174#comment-17779174
 ] 

Dheeren Beborrtha commented on HADOOP-18716:


Any target time to fix these ? Thx.

> [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
> -
>
> Key: HADOOP-18716
> URL: https://issues.apache.org/jira/browse/HADOOP-18716
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Vinay Devadiga
>Priority: Critical
>
> Compiled Hadoop - Hadoop branch 3.3.3
> mvn clean install - DskipTests=True
> Java_Home ->  points to Java-8
> maven version - 3.8.8 (Quite latest)
>  
> Ran various whole test suit on my private cloud environment -  
> Changed Java_Home to   Java-17 
>  
> mvn surefire:test 
>  
> Out of 22k tests - 2.5 k tests failed .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8

2023-06-12 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17731795#comment-17731795
 ] 

Wei-Chiu Chuang commented on HADOOP-18716:
--

Please add a target version.

> [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
> -
>
> Key: HADOOP-18716
> URL: https://issues.apache.org/jira/browse/HADOOP-18716
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Vinay Devadiga
>Priority: Critical
>
> Compiled Hadoop - Hadoop branch 3.3.3
> mvn clean install - DskipTests=True
> Java_Home ->  points to Java-8
> maven version - 3.8.8 (Quite latest)
>  
> Ran various whole test suit on my private cloud environment -  
> Changed Java_Home to   Java-17 
>  
> mvn surefire:test 
>  
> Out of 22k tests - 2.5 k tests failed .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8

2023-05-06 Thread Ayush Saxena (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720220#comment-17720220
 ] 

Ayush Saxena commented on HADOOP-18716:
---

Thanx [~vinay._.devadiga] for the pointers. Let me find some time and raise a 
PR with these findings. In case you find some time feel free to takeover. 

> [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
> -
>
> Key: HADOOP-18716
> URL: https://issues.apache.org/jira/browse/HADOOP-18716
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Vinay Devadiga
>Priority: Critical
>
> Compiled Hadoop - Hadoop branch 3.3.3
> mvn clean install - DskipTests=True
> Java_Home ->  points to Java-8
> maven version - 3.8.8 (Quite latest)
>  
> Ran various whole test suit on my private cloud environment -  
> Changed Java_Home to   Java-17 
>  
> mvn surefire:test 
>  
> Out of 22k tests - 2.5 k tests failed .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8

2023-05-05 Thread Vinay Devadiga (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720086#comment-17720086
 ] 

Vinay Devadiga commented on HADOOP-18716:
-

Hi [~ayushtkn] i dont have band width to raise PR , but here are most of the 
fixes explanation here are there tested it out . Any body can raise PR .

 
h3. Appendix
 #1.The first major error which we encountered was the following stack trace -

java.lang.NoClassDefFoundError: Could not initialize class 
com.google.inject.internal.cglib.core.$ReflectUtils
at 
com.google.inject.internal.cglib.reflect.$FastClassEmitter.(FastClassEmitter.java:67)
at 
com.google.inject.internal.cglib.reflect.$FastClass$Generator.generateClass(FastClass.java:72)

This error seems to be the lack of support of the current guice version 4.0 for 
jre 17 . so after upgrading the guice version to 5.1.0 reduced the {*}errors by 
the 300{*}.
*Ref:*
[https://github.com/google/guice/issues/1536] 
[https://github.com/google/guice/pull/1529/files]

--4.0
++5.1.0

--

2 . The second major error which we encountered the child jvms spawned by the 
maven do not have major add-opens which were causing various failures . 

```
 1. java.lang.NoClassDefFoundError: Could not initialize class 
  
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts,TestJerseyClient
 
 2. java.io.IOException: 0.0.0.0:43529: Unexpected HTTP response: 
code=500 != 200, op=MKDIRS, 
 message=Server Error
 4. org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
 6. java.lang.Exception: Unexpected exception,  
  expected but 
  was
  7. org.apache.hadoop.service.ServiceStateException: 
   org.apache.hadoop.yarn.exceptions.YarnException: Failed to 
initialize queues
  8. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
java.lang.ExceptionInInitializerError
```
and many more

To fix this we use the arg line child property of 

-Xmx2048m 
-XX:+HeapDumpOnOutOfMemoryError

We add the following - add-opens in the arg line and in the child projects 

--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED 
--add-opens=java.base/java.io=ALL-UNNAMED 
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED 
--add-opens=java.base/java.lang=ALL-UNNAMED 
--add-opens=java.base/java.math=ALL-UNNAMED 
--add-opens=java.base/java.util=ALL-UNNAMED 
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED 
--add-opens=java.base/java.net=ALL-UNNAMED 
--add-opens=java.base/java.text=ALL-UNNAMED 
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED 
--add-opens=java.base/java.nio=ALL-UNNAMED

This fix brought down the errors by 2000. So the current count is {*}reduced to 
500{*}.
---

3. There were various mockito exceptions of these following stack trace 

java.lang.NullPointerException: Cannot enter synchronized block because 
"this.lock" is null

Mockito cannot mock this class: class com.sun.jersey.api.client.ClientResponse.

If you're not sure why you're getting this error, please report to the mailing 
list.


Java : 17
JVM vendor name : Amazon.com Inc.
JVM vendor version : 17.0.6+10-LTS
JVM name : OpenJDK 64-Bit Server VM
JVM version : 17.0.6+10-LTS
JVM info : mixed mode, sharing
OS name : Linux
OS version : 4.14.309-231.529.amzn2.x86_64


You are seeing this disclaimer because Mockito is configured to create inlined 
mocks.
You can learn about inline mocks and their limitations under item #39 of the 
Mockito class javadoc.

Underlying exception : org.mockito.exceptions.base.MockitoException: Could not 
modify all classes [class com.sun.jersey.api.client.ClientResponse, class 
java.lang.Object]

etc .These errors got fix after updating the mockito-core version and its scope 


org.mockito
mockito-core
2.28.2
3.11.2
provided

Other mockito exceptions are failing with following stack trace - 

Caused by: java.lang.UnsatisfiedLinkError: 
jdk.internal.misc.Unsafe.registerNatives()V
at jdk.internal.misc.Unsafe.registerNatives(Native Method)
at jdk.internal.misc.Unsafe.(Unsafe.java:57)
at java.base/java.lang.Class.forName0(Native Method)

This is surely because we have not access to {{Unsafe}} . 

Ref: [https://github.com/jboss-javassist/javassist/issues/227]
[https://github.com/mockito/mockito/issues/1207]
[. 
https://github.com/mockito/mockito/issues/2164|https://github.com/mockito/mockito/issues/2164]

This brought down *error failures to 200* .
 # There were some other minor errors like for [MetricsInvariantChecker 

[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8

2023-04-24 Thread Ayush Saxena (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715693#comment-17715693
 ] 

Ayush Saxena commented on HADOOP-18716:
---

[~vinay._.devadiga] should raise a PR to upgrade Guice. I can validate locally. 

We don't use github actions for builds. The most you can plan just for your PR 
may be somewhere here install & change to Java17 
[https://github.com/apache/hadoop/blob/trunk/dev-support/docker/Dockerfile#L61]

> [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
> -
>
> Key: HADOOP-18716
> URL: https://issues.apache.org/jira/browse/HADOOP-18716
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Vinay Devadiga
>Priority: Critical
>
> Compiled Hadoop - Hadoop branch 3.3.3
> mvn clean install - DskipTests=True
> Java_Home ->  points to Java-8
> maven version - 3.8.8 (Quite latest)
>  
> Ran various whole test suit on my private cloud environment -  
> Changed Java_Home to   Java-17 
>  
> mvn surefire:test 
>  
> Out of 22k tests - 2.5 k tests failed .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8

2023-04-24 Thread Vinay Devadiga (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715689#comment-17715689
 ] 

Vinay Devadiga commented on HADOOP-18716:
-

Another one -

 

{color:#16191f}ava.lang.NoClassDefFoundError: Could not initialize class 
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts,TestJerseyClient
 org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices [ERROR] 
testTaskAttemptsSlash(org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts)
 Time elapsed: 0.001 s <<< ERROR! 
{color}{color:#00}java.lang.NoClassDefFoundError: Could not initialize 
class org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttemptsat 
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
 Method)at 
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)at
 
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at
 
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)at
 java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)at 
org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:250)at
 
org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:260)at
 
org.junit.runners.BlockJUnit4ClassRunner$2.runReflectiveCall(BlockJUnit4ClassRunner.java:309)at
 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)at
 
org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:306)at
 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)at
 org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63){color}

> [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
> -
>
> Key: HADOOP-18716
> URL: https://issues.apache.org/jira/browse/HADOOP-18716
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Vinay Devadiga
>Priority: Critical
>
> Compiled Hadoop - Hadoop branch 3.3.3
> mvn clean install - DskipTests=True
> Java_Home ->  points to Java-8
> maven version - 3.8.8 (Quite latest)
>  
> Ran various whole test suit on my private cloud environment -  
> Changed Java_Home to   Java-17 
>  
> mvn surefire:test 
>  
> Out of 22k tests - 2.5 k tests failed .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8

2023-04-24 Thread Vinay Devadiga (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715687#comment-17715687
 ] 

Vinay Devadiga commented on HADOOP-18716:
-

Here is the most common error stack trace .

 

{color:#16191f}java.lang.NoClassDefFoundError: Could not initialize class 
com.google.inject.internal.cglib.core.$ReflectUtils - {color}
|java.lang.NoClassDefFoundError: Could not initialize class 
com.google.inject.internal.cglib.core.$ReflectUtils|
|at 
com.google.inject.internal.cglib.reflect.$FastClassEmitter.(FastClassEmitter.java:67)|
|at 
com.google.inject.internal.cglib.reflect.$FastClass$Generator.generateClass(FastClass.java:72)|
|at 
com.google.inject.internal.cglib.core.$DefaultGeneratorStrategy.generate(DefaultGeneratorStrategy.java:25)|
|at 
com.google.inject.internal.cglib.core.$AbstractClassGenerator.create(AbstractClassGenerator.java:216)|
|at 
com.google.inject.internal.cglib.reflect.$FastClass$Generator.create(FastClass.java:64)|
|at com.google.inject.internal.BytecodeGen.newFastClass(BytecodeGen.java:204)|
|at 
com.google.inject.internal.ProviderMethod$FastClassProviderMethod.(ProviderMethod.java:256)|
|at com.google.inject.internal.ProviderMethod.create(ProviderMethod.java:71)|
|at 
com.google.inject.internal.ProviderMethodsModule.createProviderMethod(ProviderMethodsModule.java:275)|
|at 
com.google.inject.internal.ProviderMethodsModule.getProviderMethods(ProviderMethodsModule.java:144)|
|at 
com.google.inject.internal.ProviderMethodsModule.configure(ProviderMethodsModule.java:123)|
|at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)|
|at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:349)|
|at com.google.inject.AbstractModule.install(AbstractModule.java:122)|
|at com.google.inject.servlet.ServletModule.configure(ServletModule.java:52)|
|at com.google.inject.AbstractModule.configure(AbstractModule.java:62)|
|at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)|
|at com.google.inject.spi.Elements.getElements(Elements.java:110)|
|at 
com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)|
|at 
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)|
|at com.google.inject.Guice.createInjector(Guice.java:96)|

 

{color:#16191f}It seems there is no support for guice 4 for java 17. need to 
upgrade to guice 5 Test name 
-{*}org.apache.hadoop.yarn.webapp.view.TestInfoBlock{*} 
{color}[https://github.com/google/guice/issues/1536]
[https://github.com/google/guice/pull/1529/files]
[https://bugs.openjdk.org/browse/JDK-8267178]
[https://groups.google.com/g/google-guice/c/0inCjcpWUwc]{color:#00}. {color}

> [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
> -
>
> Key: HADOOP-18716
> URL: https://issues.apache.org/jira/browse/HADOOP-18716
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Vinay Devadiga
>Priority: Critical
>
> Compiled Hadoop - Hadoop branch 3.3.3
> mvn clean install - DskipTests=True
> Java_Home ->  points to Java-8
> maven version - 3.8.8 (Quite latest)
>  
> Ran various whole test suit on my private cloud environment -  
> Changed Java_Home to   Java-17 
>  
> mvn surefire:test 
>  
> Out of 22k tests - 2.5 k tests failed .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8

2023-04-24 Thread Vinay Devadiga (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715686#comment-17715686
 ] 

Vinay Devadiga commented on HADOOP-18716:
-

Yes [~ayushtkn]  , by the way is there , way to run unit test through github 
actions . Where I can change the java environment .

> [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
> -
>
> Key: HADOOP-18716
> URL: https://issues.apache.org/jira/browse/HADOOP-18716
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Vinay Devadiga
>Priority: Critical
>
> Compiled Hadoop - Hadoop branch 3.3.3
> mvn clean install - DskipTests=True
> Java_Home ->  points to Java-8
> maven version - 3.8.8 (Quite latest)
>  
> Ran various whole test suit on my private cloud environment -  
> Changed Java_Home to   Java-17 
>  
> mvn surefire:test 
>  
> Out of 22k tests - 2.5 k tests failed .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-18716) [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8

2023-04-24 Thread Ayush Saxena (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715648#comment-17715648
 ] 

Ayush Saxena commented on HADOOP-18716:
---

the branch targeted to trunk, we plan to chase JDK-17 for trunk only mostly. 
let me know if you plan to fix stuff, I can add you as contributor & assign or 
extend some help

> [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8
> -
>
> Key: HADOOP-18716
> URL: https://issues.apache.org/jira/browse/HADOOP-18716
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Vinay Devadiga
>Priority: Blocker
>
> Compiled Hadoop - Hadoop branch 3.3.3
> mvn clean install - DskipTests=True
> Java_Home ->  points to Java-8
> maven version - 3.8.8 (Quite latest)
>  
> Ran various whole test suit on my private cloud environment -  
> Changed Java_Home to   Java-17 
>  
> mvn surefire:test 
>  
> Out of 22k tests - 2.5 k tests failed .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org