This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d8e402db2aa7 [SPARK-47000][CORE] Use `getTotalMemorySize` in
`WorkerArguments`
d8e402db2aa7 is described below
commit d8e402db2aa71835d087f84173463c7346c7176b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Feb 7 14:32:41 2024 -0800
[SPARK-47000][CORE] Use `getTotalMemorySize` in `WorkerArguments`
### What changes were proposed in this pull request?
This PR aims to use `getTotalMemorySize` instead of deprecated
`getTotalPhysicalMemorySize` (OpenJDK) or `getTotalPhysicalMemory` (IBM Java)
in `WorkerArguments`.
### Why are the changes needed?
`getTotalPhysicalMemorySize` is deprecated at Java 14 in OpenJDK.
-
https://docs.oracle.com/en/java/javase/17/docs/api/jdk.management/com/sun/management/OperatingSystemMXBean.html#getTotalPhysicalMemorySize()
`getTotalPhysicalMemory` is deprecated since 1.8 in IBM.
-
https://eclipse.dev/openj9/docs/api/jdk17/jdk.management/com/ibm/lang/management/OperatingSystemMXBean.html#getTotalPhysicalMemory()
`getTotalMemorySize` is recommended in both environments for Apache Spark
4.0.0 because the minimum Java version is 17.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs with the existing test cases.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #45060 from dongjoon-hyun/SPARK-47000.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../org/apache/spark/deploy/worker/WorkerArguments.scala | 13 +++----------
1 file changed, 3 insertions(+), 10 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala
b/core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala
index 42f684c0a197..94a27e1a3e6d 100644
--- a/core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala
@@ -148,20 +148,13 @@ private[worker] class WorkerArguments(args:
Array[String], conf: SparkConf) {
}
def inferDefaultMemory(): Int = {
- val ibmVendor = System.getProperty("java.vendor").contains("IBM")
var totalMb = 0
try {
// scalastyle:off classforname
val bean = ManagementFactory.getOperatingSystemMXBean()
- if (ibmVendor) {
- val beanClass =
Class.forName("com.ibm.lang.management.OperatingSystemMXBean")
- val method = beanClass.getDeclaredMethod("getTotalPhysicalMemory")
- totalMb = (method.invoke(bean).asInstanceOf[Long] / 1024 / 1024).toInt
- } else {
- val beanClass =
Class.forName("com.sun.management.OperatingSystemMXBean")
- val method = beanClass.getDeclaredMethod("getTotalPhysicalMemorySize")
- totalMb = (method.invoke(bean).asInstanceOf[Long] / 1024 / 1024).toInt
- }
+ val beanClass = Class.forName("com.sun.management.OperatingSystemMXBean")
+ val method = beanClass.getDeclaredMethod("getTotalMemorySize")
+ totalMb = (method.invoke(bean).asInstanceOf[Long] / 1024 / 1024).toInt
// scalastyle:on classforname
} catch {
case e: Exception =>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
