Re: Spark Thrift Server java vm problem need help

[angers . zhu]

Hi Sean,

Yea,  I set  -XX:+UseCompressedOops in driver(you can see in command line) and these days, we have more user and I set

spark.driver.memory to 64g, in Non-default VM flags it should be +XX:-UseCompressdOops , but it’s still 

+XX:-UseCompressdOops.

I have find the reason , in SparkSubmitCommandBuilder.buildSparkSubmitCommand, have logic like below

if (isClientMode) {
  // Figuring out where the memory value come from is a little tricky due to precedence.
  // Precedence is observed in the following order:
  // - explicit configuration (setConf()), which also covers --driver-memory cli argument.
  // - properties file.
  // - SPARK_DRIVER_MEMORY env variable
  // - SPARK_MEM env variable
  // - default value (1g)
  // Take Thrift Server as daemon
  String tsMemory =
    isThriftServer(mainClass) ? System.getenv("SPARK_DAEMON_MEMORY") : null;
  String memory = firstNonEmpty(tsMemory, config.get(SparkLauncher.DRIVER_MEMORY),
    System.getenv("SPARK_DRIVER_MEMORY"), System.getenv("SPARK_MEM"), DEFAULT_MEM);
  cmd.add("-Xmx" + memory);
  addOptionString(cmd, driverDefaultJavaOptions);
  addOptionString(cmd, driverExtraJavaOptions);
  mergeEnvPathList(env, getLibPathEnvName(),
    config.get(SparkLauncher.DRIVER_EXTRA_LIBRARY_PATH));
}

For Spark Thrift Server, use SPARK_DAEMON_MEMORY First, it’s really reasonable, I am confused, if spark.driver.memory is bigger then 32g

And SPARK_DAEMON_MEMORY is less then 32g, UseCompressedOops will also be open, it’s right?

If we need to modify this logic for case >32g.

By the way, I meet problem like https://issues.apache.org/jira/browse/SPARK-27097, caused by these strange case.

Thanks

	angers.zhu
angers....@gmail.com

签名由 网易邮箱大师 定制

On 03/23/2020 21:43，Sean Owen<sro...@gmail.com> wrote：

I don't think Spark sets UseCompressedOops in any defaults; are you setting it?

It can't be used with heaps >= 32GB. It doesn't seem to cause an error if you set it with large heaps, just a warning.

What's the problem?

On Mon, Mar 23, 2020 at 6:21 AM angers.zhu <angers....@gmail.com> wrote:

    




    

    

        Hi developers,

 These day I meet a strange problem and I can’t find why

When I start a spark thrift server with  spark.driver.memory 64g, then use jdk8/bin/jinfo pid to see vm flags got below information,
In 64g vm, UseCompressedOops should be closed by default, why spark thrift server is -XX: +UseCompressedOops

Non-default VM flags: -XX:CICompilerCount=15 -XX:-CMSClassUnloadingEnabled -XX:CMSFullGCsBeforeCompaction=0 -XX:CMSInitiatingOccupancyFraction=75 -XX:+CMSParallelRemarkEnabled -XX:-ClassUnloading -XX:+DisableExplicitGC -XX:ErrorFile=null -XX:-ExplicitGCInvokesConcurrentAndUnloadsClasses -XX:InitialHeapSize=2116026368 -XX:+ManagementServer -XX:MaxDirectMemorySize=8589934592 -XX:MaxHeapSize=6442450944 -XX:MaxNewSize=2147483648 -XX:MaxTenuringThreshold=6 -XX:MinHeapDeltaBytes=196608 -XX:NewSize=705298432 -XX:OldPLABSize=16 -XX:OldSize=1410727936 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:-TraceClassUnloading -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseFastUnorderedTimeStamps -XX:+UseParNewGC
Command line:  -Xmx6g -Djava.library.path=/home/hadoop/hadoop/lib/native -Djavax.security.auth.useSubjectCredsOnly=false -Dcom.sun.management.jmxremote.port=9021 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:MaxPermSize=1024m -XX:PermSize=256m -XX:MaxDirectMemorySize=8192m -XX:-TraceClassUnloading -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:+PrintTenuringDistribution -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=75 -Xnoclassgc -XX:+PrintGCDetails -XX:+PrintGCDateStamps 

Since I am not a professor in VM, hope for some help


    

        
            

        
    
    

      
    

        

                    
            

                

                    

                            

                                
                            
                            

                                
angers.zhu
                            
                    
                        

                            

                                    
angers....@gmail.com
                            
                        
                
            
        
        
    
    

        签名由
        网易邮箱大师
        定制