This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 8a7fa43 [SPARK-36195][BUILD] Set MaxMetaspaceSize JVM option to 2g
8a7fa43 is described below
commit 8a7fa439fad5fd13b29fe919ce178908cbbe816c
Author: Dongjoon Hyun <dongj...@apache.org>
AuthorDate: Sun Jul 18 10:15:15 2021 -0700
[SPARK-36195][BUILD] Set MaxMetaspaceSize JVM option to 2g
This PR aims to set `MaxMetaspaceSize` to `2g` because it's increasing the
native memory consumption unlimitedly by default. The unlimited increasing
memory causes GitHub Action flakiness. The value I observed during `hive`
module test was over 1.8G and growing.
-
https://docs.oracle.com/javase/10/gctuning/other-considerations.htm#JSGCT-GUID-BFB89453-60C0-42AC-81CA-87D59B0ACE2E
> Starting with JDK 8, the permanent generation was removed and the class
metadata is allocated in native memory. The amount of native memory that can be
used for class metadata is by default unlimited. Use the option
-XX:MaxMetaspaceSize to put an upper limit on the amount of native memory used
for class metadata.
In addition, I increased the following memory limit to 4g consistently from
two places.
```xml
- <jvmArg>-Xms2048m</jvmArg>
- <jvmArg>-Xmx2048m</jvmArg>
+ <jvmArg>-Xms4g</jvmArg>
+ <jvmArg>-Xmx4g</jvmArg>
```
```scala
- javaOptions += "-Xmx3g",
+ javaOptions ++= "-Xmx4g -XX:MaxMetaspaceSize=2g".split(" ").toSeq,
```
This will reduce the flakiness in CI environment by limiting the memory
usage explicitly.
When we limit it with `1g`, Hive module fails with `OOM` like the following.
```
java.lang.OutOfMemoryError: Metaspace
Error: Exception in thread "dispatcher-event-loop-110"
java.lang.OutOfMemoryError: Metaspace
```
No.
Pass the CIs.
Closes #33405 from dongjoon-hyun/SPARK-36195.
Lead-authored-by: Dongjoon Hyun <dongj...@apache.org>
Co-authored-by: Kyle Bendickson <kbendick...@apple.com>
Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
(cherry picked from commit d7df7a805fcbdf2435df1e78abd9899d3ca10dd2)
Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
---
pom.xml | 5 +++--
project/SparkBuild.scala | 4 ++--
2 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/pom.xml b/pom.xml
index a7e3a73..1fb7c5a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -2517,6 +2517,7 @@
<jvmArgs>
<jvmArg>-Xms1024m</jvmArg>
<jvmArg>-Xmx1024m</jvmArg>
+ <jvmArg>-XX:MaxMetaspaceSize=2g</jvmArg>
<jvmArg>-XX:ReservedCodeCacheSize=${CodeCacheSize}</jvmArg>
</jvmArgs>
<javacArgs>
@@ -2565,7 +2566,7 @@
<include>**/*Suite.java</include>
</includes>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
- <argLine>-ea -Xmx4g -Xss4m
-XX:ReservedCodeCacheSize=${CodeCacheSize}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
<environmentVariables>
<!--
Setting SPARK_DIST_CLASSPATH is a simple way to make sure any
child processes
@@ -2615,7 +2616,7 @@
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>SparkTestSuite.txt</filereports>
- <argLine>-ea -Xmx4g -Xss4m
-XX:ReservedCodeCacheSize=${CodeCacheSize}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
<stderr/>
<environmentVariables>
<!--
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index fa2d775..3fb690b 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1061,9 +1061,9 @@ object TestSettings {
.map { case (k,v) => s"-D$k=$v" }.toSeq,
javaOptions in Test += "-ea",
// SPARK-29282 This is for consistency between JDK8 and JDK11.
- javaOptions in Test ++= "-Xmx4g -Xss4m -XX:+UseParallelGC
-XX:-UseDynamicNumberOfGCThreads"
+ javaOptions in Test ++= "-Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:+UseParallelGC -XX:-UseDynamicNumberOfGCThreads"
.split(" ").toSeq,
- javaOptions += "-Xmx3g",
+ javaOptions ++= "-Xmx4g -XX:MaxMetaspaceSize=2g".split(" ").toSeq,
// Exclude tags defined in a system property
testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest,
sys.props.get("test.exclude.tags").map { tags =>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
