srowen commented on a change in pull request #34286:
URL: https://github.com/apache/spark/pull/34286#discussion_r729123563
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSparkSubmitSuite.scala
##########
@@ -40,15 +40,26 @@ class WholeStageCodegenSparkSubmitSuite extends
SparkSubmitTestUtils
val unusedJar = TestUtils.createJarWithClasses(Seq.empty)
// HotSpot JVM specific: Set up a local cluster with the driver/executor
using mismatched
- // settings of UseCompressedOops JVM option.
+ // settings of UseCompressedClassPointers JVM option.
val argsForSparkSubmit = Seq(
"--class",
WholeStageCodegenSparkSubmitSuite.getClass.getName.stripSuffix("$"),
"--master", "local-cluster[1,1,1024]",
"--driver-memory", "1g",
"--conf", "spark.ui.enabled=false",
"--conf", "spark.master.rest.enabled=false",
- "--conf", "spark.driver.extraJavaOptions=-XX:-UseCompressedOops",
- "--conf", "spark.executor.extraJavaOptions=-XX:+UseCompressedOops",
+ // SPARK-37008: The results of `Platform.BYTE_ARRAY_OFFSET` using
different Java versions
+ // and different args as follows table:
+ // +------------------------------+--------+---------+
+ // | |Java 8 |Java 17 |
+ // +------------------------------+--------+---------+
+ // |-XX:-UseCompressedOops | 24 | 16 |
+ // |-XX:+UseCompressedOops | 16 | 16 |
+ // |-XX:-UseCompressedClassPointers| 24 | 24 |
+ // |-XX:+UseCompressedClassPointers| 16 | 16 |
+ // +-------------------------------+-------+---------+
+ // So SPARK-37008 replace `UseCompressedOops` with
`UseCompressedClassPointers`.
+ "--conf",
"spark.driver.extraJavaOptions=-XX:-UseCompressedClassPointers",
Review comment:
So, this is only necessary to make the test pass, because it's quite
sensitive to the size in memory, but spark works otherwise without this flag
right?
Can you set this flag before Java 17 - would the JVM accept it? I just
hadn't seen it before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
