This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new d8e402db2aa7 [SPARK-47000][CORE] Use `getTotalMemorySize` in `WorkerArguments` d8e402db2aa7 is described below commit d8e402db2aa71835d087f84173463c7346c7176b Author: Dongjoon Hyun <dh...@apple.com> AuthorDate: Wed Feb 7 14:32:41 2024 -0800 [SPARK-47000][CORE] Use `getTotalMemorySize` in `WorkerArguments` ### What changes were proposed in this pull request? This PR aims to use `getTotalMemorySize` instead of deprecated `getTotalPhysicalMemorySize` (OpenJDK) or `getTotalPhysicalMemory` (IBM Java) in `WorkerArguments`. ### Why are the changes needed? `getTotalPhysicalMemorySize` is deprecated at Java 14 in OpenJDK. - https://docs.oracle.com/en/java/javase/17/docs/api/jdk.management/com/sun/management/OperatingSystemMXBean.html#getTotalPhysicalMemorySize() `getTotalPhysicalMemory` is deprecated since 1.8 in IBM. - https://eclipse.dev/openj9/docs/api/jdk17/jdk.management/com/ibm/lang/management/OperatingSystemMXBean.html#getTotalPhysicalMemory() `getTotalMemorySize` is recommended in both environments for Apache Spark 4.0.0 because the minimum Java version is 17. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs with the existing test cases. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45060 from dongjoon-hyun/SPARK-47000. Authored-by: Dongjoon Hyun <dh...@apple.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- .../org/apache/spark/deploy/worker/WorkerArguments.scala | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala b/core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala index 42f684c0a197..94a27e1a3e6d 100644 --- a/core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala +++ b/core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala @@ -148,20 +148,13 @@ private[worker] class WorkerArguments(args: Array[String], conf: SparkConf) { } def inferDefaultMemory(): Int = { - val ibmVendor = System.getProperty("java.vendor").contains("IBM") var totalMb = 0 try { // scalastyle:off classforname val bean = ManagementFactory.getOperatingSystemMXBean() - if (ibmVendor) { - val beanClass = Class.forName("com.ibm.lang.management.OperatingSystemMXBean") - val method = beanClass.getDeclaredMethod("getTotalPhysicalMemory") - totalMb = (method.invoke(bean).asInstanceOf[Long] / 1024 / 1024).toInt - } else { - val beanClass = Class.forName("com.sun.management.OperatingSystemMXBean") - val method = beanClass.getDeclaredMethod("getTotalPhysicalMemorySize") - totalMb = (method.invoke(bean).asInstanceOf[Long] / 1024 / 1024).toInt - } + val beanClass = Class.forName("com.sun.management.OperatingSystemMXBean") + val method = beanClass.getDeclaredMethod("getTotalMemorySize") + totalMb = (method.invoke(bean).asInstanceOf[Long] / 1024 / 1024).toInt // scalastyle:on classforname } catch { case e: Exception => --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org