[
https://issues.apache.org/jira/browse/HADOOP-18174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mateusz Łyczek updated HADOOP-18174:
------------------------------------
Description:
In our project we are using hadoop-client library and everything works fine
while running inside containers with official OpenJDK base image.
But for optimisation purposes we also use ibm-semeru-runtimes base images
([https://www.ibm.com/support/pages/semeru-runtimes-release-notes] ). To be
specific we use *open-17.0.1_12-jre* version of this image and encountered the
following problem.
Our application is throwing an exception while using hadoop-client to upload
files:
{code:java}
failure to login: javax.security.auth.login.LoginException: No LoginModule
found for com.ibm.security.auth.module.JAASLoginModule{code}
After a little investigation I found that this login module is selected by
Hadoop only when it detects that it is being run on IBM Java (see
[https://github.com/apache/hadoop/blob/672e380c4f6ffcb0a6fee6d8263166e16b4323c2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L434]
)
{code:java}
private static String getOSLoginModuleName() {
if (IBM_JAVA) {
return "com.ibm.security.auth.module.JAASLoginModule";
} else {
return windows ? "com.sun.security.auth.module.NTLoginModule"
: "com.sun.security.auth.module.UnixLoginModule";
}
} {code}
and IBM Java is detected base on *java.vendor* system property value (see
[https://github.com/apache/hadoop/blob/672e380c4f6ffcb0a6fee6d8263166e16b4323c2/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/util/PlatformName.java#L50]
)
{code:java}
/**
* The java vendor name used in this platform.
*/
public static final String JAVA_VENDOR_NAME =
System.getProperty("java.vendor");
/**
* A public static variable to indicate the current java vendor is
* IBM java or not.
*/
public static final boolean IBM_JAVA = JAVA_VENDOR_NAME.contains("IBM");
{code}
I checked inside the ibm-semeru-runtimes based version of our docker container
and the *java.vendor* system property is set to the following value:
{code:java}
java.vendor: IBM Corporation {code}
but, as the documentation for IBM Semeru runtimes images says, it contains
OpenJDK class libraries with Eclipse OpenJ9 JVM. I confirmed it by running
{*}java -version{*}:
{code:java}
openjdk version "17.0.1" 2021-10-19
IBM Semeru Runtime Open Edition 17.0.1.0 (build 17.0.1+12)
Eclipse OpenJ9 VM 17.0.1.0 (build openj9-0.29.1, JRE 17 Linux amd64-64-Bit
Compressed References 20211207_75 (JIT enabled, AOT enabled)
OpenJ9 - 7d055dfcb
OMR - e30892e2b
JCL - fc67fbe50a0 based on jdk-17.0.1+12) {code}
therefore there is no {{com.ibm.security.auth.module.JAASLoginModule}} class
present.
Therefore I would like to ask if there is any other way of detecting IBM java
instead of checking the *java.vendor* system property, that would be more
accurate and allow using hadoop-client on OpenJDK provided by IBM?
I tried to think about something to suggest - the first thing that came to my
mind was to check if one of the classes from IBM packages actually exists by
trying to load it but I don't know the other usages of the *IBM_JAVA* variable
in hadoop-client, to be sure that it's a good idea for you.
was:
In our project we are using hadoop-client library and everything works fine
while running inside containers with official OpenJDK base image.
But for optimisation purposes we also use ibm-semeru-runtimes base images
([https://www.ibm.com/support/pages/semeru-runtimes-release-notes] ). To be
specific we use *open-17.0.1_12-jre* version of this image and encountered the
following problem.
Our application is throwing an exception while using hadoop-client to upload
files:
{code:java}
failure to login: javax.security.auth.login.LoginException: No LoginModule
found for com.ibm.security.auth.module.JAASLoginModule{code}
After a little investigation I found that this login module is selected by
Hadoop only when it detects that it is being run on IBM Java (see
[https://github.com/apache/hadoop/blob/672e380c4f6ffcb0a6fee6d8263166e16b4323c2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L434]
)
{code:java}
private static String getOSLoginModuleName() {
if (IBM_JAVA) {
return "com.ibm.security.auth.module.JAASLoginModule";
} else {
return windows ? "com.sun.security.auth.module.NTLoginModule"
: "com.sun.security.auth.module.UnixLoginModule";
}
} {code}
and IBM Java is detected base on *java.vendor* system property value (see
[https://github.com/apache/hadoop/blob/672e380c4f6ffcb0a6fee6d8263166e16b4323c2/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/util/PlatformName.java#L50]
)
{code:java}
/**
* The java vendor name used in this platform.
*/
public static final String JAVA_VENDOR_NAME =
System.getProperty("java.vendor");
/**
* A public static variable to indicate the current java vendor is
* IBM java or not.
*/
public static final boolean IBM_JAVA = JAVA_VENDOR_NAME.contains("IBM");
{code}
I checked inside the ibm-semeru-runtimes based version of our docker container
and the *java.vendor* system property is set to the following value:
{code:java}
java.vendor: IBM Corporation {code}
but, as the documentation for IBM Semeru runtimes images says, it contains
OpenJDK class libraries with Eclipse OpenJ9 JVM. I confirmed it by running
{*}java -version{*}:
{code:java}
openjdk version "17.0.1" 2021-10-19
IBM Semeru Runtime Open Edition 17.0.1.0 (build 17.0.1+12)
Eclipse OpenJ9 VM 17.0.1.0 (build openj9-0.29.1, JRE 17 Linux amd64-64-Bit
Compressed References 20211207_75 (JIT enabled, AOT enabled)
OpenJ9 - 7d055dfcb
OMR - e30892e2b
JCL - fc67fbe50a0 based on jdk-17.0.1+12) {code}
therefore there is no {{com.ibm.security.auth.module.JAASLoginModule}} class
present.
Therefore I would like to ask if there is any other way of detecting IBM java
instead of checking the *java.vendor* system property, that would be more
accurate?
I tried to think about something to suggest - the first thing that came to my
mind was to check if one of the classes from IBM packages actually exists by
trying to load it but I don't know the other usages of the *IBM_JAVA* variable
in hadoop-client, to be sure that it's a good idea for you.
> IBM Java detected while running on OpenJDK class library
> --------------------------------------------------------
>
> Key: HADOOP-18174
> URL: https://issues.apache.org/jira/browse/HADOOP-18174
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.3.2
> Reporter: Mateusz Łyczek
> Priority: Major
>
> In our project we are using hadoop-client library and everything works fine
> while running inside containers with official OpenJDK base image.
> But for optimisation purposes we also use ibm-semeru-runtimes base images
> ([https://www.ibm.com/support/pages/semeru-runtimes-release-notes] ). To be
> specific we use *open-17.0.1_12-jre* version of this image and encountered
> the following problem.
> Our application is throwing an exception while using hadoop-client to upload
> files:
> {code:java}
> failure to login: javax.security.auth.login.LoginException: No LoginModule
> found for com.ibm.security.auth.module.JAASLoginModule{code}
>
> After a little investigation I found that this login module is selected by
> Hadoop only when it detects that it is being run on IBM Java (see
> [https://github.com/apache/hadoop/blob/672e380c4f6ffcb0a6fee6d8263166e16b4323c2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L434]
> )
> {code:java}
> private static String getOSLoginModuleName() {
> if (IBM_JAVA) {
> return "com.ibm.security.auth.module.JAASLoginModule";
> } else {
> return windows ? "com.sun.security.auth.module.NTLoginModule"
> : "com.sun.security.auth.module.UnixLoginModule";
> }
> } {code}
>
> and IBM Java is detected base on *java.vendor* system property value (see
> [https://github.com/apache/hadoop/blob/672e380c4f6ffcb0a6fee6d8263166e16b4323c2/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/util/PlatformName.java#L50]
> )
> {code:java}
> /**
> * The java vendor name used in this platform.
> */
> public static final String JAVA_VENDOR_NAME =
> System.getProperty("java.vendor");
> /**
> * A public static variable to indicate the current java vendor is
> * IBM java or not.
> */
> public static final boolean IBM_JAVA = JAVA_VENDOR_NAME.contains("IBM");
> {code}
>
> I checked inside the ibm-semeru-runtimes based version of our docker
> container and the *java.vendor* system property is set to the following value:
> {code:java}
> java.vendor: IBM Corporation {code}
> but, as the documentation for IBM Semeru runtimes images says, it contains
> OpenJDK class libraries with Eclipse OpenJ9 JVM. I confirmed it by running
> {*}java -version{*}:
>
> {code:java}
> openjdk version "17.0.1" 2021-10-19
> IBM Semeru Runtime Open Edition 17.0.1.0 (build 17.0.1+12)
> Eclipse OpenJ9 VM 17.0.1.0 (build openj9-0.29.1, JRE 17 Linux amd64-64-Bit
> Compressed References 20211207_75 (JIT enabled, AOT enabled)
> OpenJ9 - 7d055dfcb
> OMR - e30892e2b
> JCL - fc67fbe50a0 based on jdk-17.0.1+12) {code}
> therefore there is no {{com.ibm.security.auth.module.JAASLoginModule}} class
> present.
>
> Therefore I would like to ask if there is any other way of detecting IBM java
> instead of checking the *java.vendor* system property, that would be more
> accurate and allow using hadoop-client on OpenJDK provided by IBM?
> I tried to think about something to suggest - the first thing that came to my
> mind was to check if one of the classes from IBM packages actually exists by
> trying to load it but I don't know the other usages of the *IBM_JAVA*
> variable in hadoop-client, to be sure that it's a good idea for you.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
