Hi, community!
I notice a change about the memory module of yarn container between spark-2.3.0 and spark-3.2.1 when requesting containers from yarn. org.apache.spark.deploy.yarn.Client.java # verifyClusterResources ``` spark-2.3.0 val executorMem = executorMemory + executorMemoryOverhead ``` ``` spark-3.2.1 val executorMem = executorMemory + executorOffHeapMemory + executorMemoryOverhead + pysparkWorkerMemory ``` And i have these questions: 1. in spark-2.3.0 and spark-3.2.1, what is memoryOverhead and where is it used? 2. what is the difference between memoryOverhead and off-heap memory, native memory, direct memory? There is no such concept in apache flink, is it an unique concept of spark? 3. in spark-2.3.0, i think that memoryOverhead contains all non-heap memory, including off-heap / native / direct. Do i think wrong? Thanks for your any replies. Best Regards!
