This is an automated email from the ASF dual-hosted git repository.
nicholasjiang pushed a commit to branch branch-0.5
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/branch-0.5 by this push:
new 0b964b3d0 [CELEBORN-1682] Add java tools.jar into classpath for JVM
quake
0b964b3d0 is described below
commit 0b964b3d0988dac40ceee0bc50baac1ce381df91
Author: Wang, Fei <[email protected]>
AuthorDate: Mon Nov 4 11:05:10 2024 +0800
[CELEBORN-1682] Add java tools.jar into classpath for JVM quake
### What changes were proposed in this pull request?
Add java tools.jar into classpath for JVM quake.
### Why are the changes needed?
Meet below issue with `celeborn.worker.jvmQuake.enabled=true`, see
https://github.com/apache/celeborn/pull/2061
```
24/11/03 15:51:08,453 ERROR [main] Worker: Initialize worker failed.
java.lang.NoClassDefFoundError: sun/jvmstat/monitor/HostIdentifier
at
org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.monitoredVm$lzycompute(JVMQuake.scala:180)
at
org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.monitoredVm(JVMQuake.scala:179)
at
org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.ygcExitTimeMonitor$lzycompute(JVMQuake.scala:185)
at
org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.ygcExitTimeMonitor(JVMQuake.scala:184)
at
org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.org$apache$celeborn$service$deploy$worker$monitor$JVMQuake$$getLastExitTime(JVMQuake.scala:192)
at
org.apache.celeborn.service.deploy.worker.monitor.JVMQuake.start(JVMQuake.scala:66)
at
org.apache.celeborn.service.deploy.worker.Worker.<init>(Worker.scala:360)
at
org.apache.celeborn.service.deploy.worker.Worker$.main(Worker.scala:1041)
at org.apache.celeborn.service.deploy.worker.Worker.main(Worker.scala)
Caused by: java.lang.ClassNotFoundException:
sun.jvmstat.monitor.HostIdentifier
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 9 more
```
Related code:
https://github.com/apache/celeborn/blob/c12e8881ab5b03e39a8b53d038a283351bf5906c/project/JDKTools.scala#L58-L75
Similar issue: https://github.com/vladimirvivien/jmx-cli/issues/4
After copy the `tools.jar` into worker-jars, the issue got resolved.
It is better that to involve the `tools.jar` automatically without copy.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
<img width="1202" alt="image"
src="https://github.com/user-attachments/assets/af8f6c0d-9123-4a73-93b5-69836c5f826d";>
Closes #2874 from turboFei/jdk_tools.
Authored-by: Wang, Fei <[email protected]>
Signed-off-by: SteNicholas <[email protected]>
(cherry picked from commit d4044c51525caf054a90badce8f02ff75d8a41ee)
Signed-off-by: SteNicholas <[email protected]>
---
bin/celeborn-class | 2 +-
sbin/load-celeborn-env.sh | 12 ++++++++++++
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/bin/celeborn-class b/bin/celeborn-class
index 5bb3791ff..ab05ae698 100755
--- a/bin/celeborn-class
+++ b/bin/celeborn-class
@@ -69,7 +69,7 @@ if [ ! -d "$CELEBORN_JARS_DIR" ]; then
echo "You need to build CELEBORN with the target \"package\" before running
this program." 1>&2
exit 1
else
- CELEBORN_CLASSPATH="$CELEBORN_CONF_DIR:$HADOOP_CONF_DIR:$CELEBORN_JARS_DIR/*"
+
CELEBORN_CLASSPATH="$CELEBORN_CONF_DIR:$HADOOP_CONF_DIR:$CELEBORN_JARS_DIR/*:$JAVA_TOOLS_JAR"
fi
# Turn off posix mode since it does not allow process substitution
diff --git a/sbin/load-celeborn-env.sh b/sbin/load-celeborn-env.sh
index e44633120..e18ce2942 100755
--- a/sbin/load-celeborn-env.sh
+++ b/sbin/load-celeborn-env.sh
@@ -44,12 +44,24 @@ if [ -n "${JAVA_HOME}" ]; then
else
if [ "$(command -v java)" ]; then
export JAVA="java"
+ JAVA_HOME=$(java -XshowSettings:properties -version 2>&1 > /dev/null |
grep 'java.home' | awk '{print $3}')
else
echo "JAVA_HOME is not set" >&2
exit 1
fi
fi
+# Find the java tools.jar
+if [ -f "${JAVA_HOME}/lib/tools.jar" ]; then
+ export JAVA_TOOLS_JAR="${JAVA_HOME}/lib/tools.jar"
+else
+ if [ -f "${JAVA_HOME}/../lib/tools.jar" ]; then
+ export JAVA_TOOLS_JAR="${JAVA_HOME}/../lib/tools.jar"
+ else
+ echo "WARNING: cannot locate tools.jar. Expected to find it in either
${JAVA_HOME}/lib/tools.jar or ${JAVA_HOME}/../lib/tools.jar"
+ fi
+fi
+
# Get log directory
if [ "$CELEBORN_LOG_DIR" = "" ]; then
export CELEBORN_LOG_DIR="${CELEBORN_HOME}/logs"
