srowen commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r727677263
##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>SparkTestSuite.txt</filereports>
- <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
Review comment:
It was left at 5g - did that not work for the test?
##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>SparkTestSuite.txt</filereports>
- <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
Review comment:
Hm. So we need more memory, or else figure out why more memory is
needed. This change might have caused other flakiness; if that's not the case
we'll just increase memory again. Otherwise I think we have to 'fix' this test.
In Java 9+ we changed some of the code that handles DirectByteBuffer to not
do some things with reflection if the classes were not opened up with command
line flags. I wonder if that's a difference, but, I think these flags now open
up those classes, so would have expected this works like Java 8 again (unless
more has to be opened up).
But maybe just needs a test change.
##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>SparkTestSuite.txt</filereports>
- <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
Review comment:
Hm, I'd be OK just removing the test case "SPARK-36464: size returns
correct positive number even with over 2GB data". What it tests is a clean fix,
and this is a huge amount of mem to allocate to test.
##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>SparkTestSuite.txt</filereports>
- <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
Review comment:
That could be fine. Actually could we just allocate an array 1000 times
smaller and write it 1001 times? that kind of thing. That seems like it would
also achieve the same effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
