I'm still not sure if you are trying to enable it or disable it, and what
the issue is?
There is no logic in Spark that sets or disables this flag that I can see.
On Mon, Mar 23, 2020 at 9:27 AM angers.zhu <angers....@gmail.com> wrote:
> Hi Sean,
>
> Yea, I set -XX:+UseCompressedOops in driver(you can see in command
> line) and these days, we have more user and I set
> spark.driver.memory to 64g, in Non-default VM flags it should be
> +XX:-UseCompressdOops , but it’s still
> +XX:-UseCompressdOops.
>
> I have find the reason , in SparkSubmitCommandBuilder.buildSparkSubmitCommand,
> have logic like below
>
> if (isClientMode) {
> // Figuring out where the memory value come from is a little tricky due to
> precedence.
> // Precedence is observed in the following order:
> // - explicit configuration (setConf()), which also covers --driver-memory
> cli argument.
> // - properties file.
> // - SPARK_DRIVER_MEMORY env variable
> // - SPARK_MEM env variable
> // - default value (1g)
> // Take Thrift Server as daemon
> String tsMemory =
> isThriftServer(mainClass) ? System.getenv("SPARK_DAEMON_MEMORY") : null;
> String memory = firstNonEmpty(tsMemory,
> config.get(SparkLauncher.DRIVER_MEMORY),
> System.getenv("SPARK_DRIVER_MEMORY"), System.getenv("SPARK_MEM"),
> DEFAULT_MEM);
> cmd.add("-Xmx" + memory);
> addOptionString(cmd, driverDefaultJavaOptions);
> addOptionString(cmd, driverExtraJavaOptions);
> mergeEnvPathList(env, getLibPathEnvName(),
> config.get(SparkLauncher.DRIVER_EXTRA_LIBRARY_PATH));
> }
>
>
> For Spark Thrift Server, use SPARK_DAEMON_MEMORY First, it’s really
> reasonable, I am confused, if spark.driver.memory is bigger then 32g
> And SPARK_DAEMON_MEMORY is less then 32g, UseCompressedOops will also be
> open, it’s right?
>
> If we need to modify this logic for case >32g.
>
>
> By the way, I meet problem like
> https://issues.apache.org/jira/browse/SPARK-27097, caused by these
> strange case.
>
> Thanks
>
>
> angers.zhu
> angers....@gmail.com
>
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>
> On 03/23/2020 21:43，Sean Owen<sro...@gmail.com> <sro...@gmail.com> wrote：
>
> I don't think Spark sets UseCompressedOops in any defaults; are you
> setting it?
> It can't be used with heaps >= 32GB. It doesn't seem to cause an error if
> you set it with large heaps, just a warning.
> What's the problem?
>
> On Mon, Mar 23, 2020 at 6:21 AM angers.zhu <angers....@gmail.com> wrote:
>
>> Hi developers,
>>
>> These day I meet a strange problem and I can’t find why
>>
>> When I start a spark thrift server with spark.driver.memory 64g, then
>> use jdk8/bin/jinfo pid to see vm flags got below information,
>> In 64g vm, UseCompressedOops should be closed by default, why spark
>> thrift server is -XX: +UseCompressedOops
>>
>> Non-default VM flags: -XX:CICompilerCount=15 -XX:-CMSClassUnloadingEnabled
>> -XX:CMSFullGCsBeforeCompaction=0 -XX:CMSInitiatingOccupancyFraction=75
>> -XX:+CMSParallelRemarkEnabled -XX:-ClassUnloading -XX:+DisableExplicitGC
>> -XX:ErrorFile=null -XX:-ExplicitGCInvokesConcurrentAndUnloadsClasses
>> -XX:InitialHeapSize=2116026368 -XX:+ManagementServer
>> -XX:MaxDirectMemorySize=8589934592 -XX:MaxHeapSize=6442450944
>> -XX:MaxNewSize=2147483648 -XX:MaxTenuringThreshold=6
>> -XX:MinHeapDeltaBytes=196608 -XX:NewSize=705298432 -XX:OldPLABSize=16
>> -XX:OldSize=1410727936 -XX:+PrintGC -XX:+PrintGCDateStamps
>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution
>> -XX:-TraceClassUnloading -XX:+UseCMSCompactAtFullCollection
>> -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
>> -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
>> -XX:+UseFastUnorderedTimeStamps -XX:+UseParNewGCCommand line: -Xmx6g
>> -Djava.library.path=/home/hadoop/hadoop/lib/native
>> -Djavax.security.auth.useSubjectCredsOnly=false
>> -Dcom.sun.management.jmxremote.port=9021
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Dcom.sun.management.jmxremote.ssl=false -XX:MaxPermSize=1024m
>> -XX:PermSize=256m -XX:MaxDirectMemorySize=8192m -XX:-TraceClassUnloading
>> -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
>> -XX:+CMSClassUnloadingEnabled -XX:+UseCMSCompactAtFullCollection
>> -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSParallelRemarkEnabled
>> -XX:+DisableExplicitGC -XX:+PrintTenuringDistribution
>> -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=75
>> -Xnoclassgc -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>>
>>
>> Since I am not a professor in VM, hope for some help
>>
>>
>> angers.zhu
>> angers....@gmail.com
>>
>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>>
>>
