@Sean Owen, Thanks for your reply.
I put the wrong link to the blog post. Here is the correct link <https://www.altiscale.com/blog/tips-and-tricks-for-running-spark-on-hadoop-part-4-memory-settings/> which describes Spark Memory settings on Yarn. I guess they have misused the terms Spark driver/BlockManager, and explained memory usage of driver falsely. 1) Then does that mean if nothing specified, then Spark will use defaults specified in Spark config site <http://spark.apache.org/docs/latest/running-on-yarn.html> ? 2) Let me clarify, if i understood it correctly: (due to Yarn restrictions) *Yarn-cluster mode*: SparkAppMaster+Driver Memory < Yarn container max size allocation SparkExecutor Memory < Yarn container max size allocation *Yarn-client mode* (assume Spark Job is launched from the map task): Driver memory is independent of any Yarn properties, only limited by machines memory. SparkAppMaster Memory < Yarn container max size allocation SparkExecutor Memory < Yarn container max size allocation Did i get it correctly ? 3) Any resource for Spark components memory calculations for Yarn cluster ? (other than this site which describes default config values http://spark.apache.org/docs/latest/running-on-yarn.html ) Thanks. On Sat, Nov 12, 2016 at 12:24 AM, Sean Owen <so...@cloudera.com> wrote: If you're pointing at the 336MB, then it's not really related any of the items you cite here. This is the memory managed internally by MemoryStore. The blog post refers to the legacy memory manager. You can see a bit of how it works in the code, but this is the sum of the on-heap and off-heap memory it can manage. See the memory config docs, however, to understand what user-facing settings you can make; you don't really need to workk about this value. mapreduce settings are irrelevant to Spark. Spark doesn't pay attention to the YARN settings, but YARN does. It enforces them, yes. It is not exempt from YARN. 896MB is correct there. yarn-client mode does not ignore driver properties, no. On Sat, Nov 12, 2016 at 2:18 AM Elkhan Dadashov <elkhan8...@gmail.com> wrote: Hi, Spark website <http://spark.apache.org/docs/latest/running-on-yarn.html> indicates default spark properties as like this: I did not override any properties in spark-defaults.conf file, but when I launch Spark in YarnClient mode: spark.driver.memory 1g spark.yarn.am.memory 512m spark.yarn.am.memoryOverhead : max(spark.yarn.am.memory * 0.10, 384m) spark.yarn.driver.memoryOverhead : max(spark.driver.memory * 0.10, 384m) I launch Spark job via SparkLauncher#startApplication() in *Yarn-client mode from the Map task of Hadoop job*. *My cluster settings*: yarn.scheduler.minimum-allocation-mb 256 yarn.scheduler.maximum-allocation-mb 2048 yarn.app.mapreduce.am.resource.mb 512 mapreduce.map.memory.mb 640 mapreduce.map.java.opts -Xmx400m yarn.app.mapreduce.am.command-opts -Xmx448m *Logs of Spark job*: INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (2048 MB per container) INFO Client: Will allocate *AM container*, with 896 MB memory including 384 MB overhead INFO MemoryStore: MemoryStore started with capacity 366.3 MB ./application_1478727394310_0005/container_1478727394310_0005_01_000002/stderr:INFO: 16/11/09 14:18:42 INFO BlockManagerMasterEndpoint: Registering block manager <machine-ip>:57246 with *366.3* MB RAM, BlockManagerId(driver, <machine-ip>, 57246) *Questions*: 1) How is driver memory calculated ? How did Spark decide for 366 MB for driver based on properties described above ? I thought the memory allocation is based on this formula ( https://www.altiscale.com/blog/spark-on-hadoop/ ): "Runtime.getRuntime.maxMemory * memoryFraction * safetyFraction ,where memoryFraction=0.6, and safetyFraction=0.9. This is 1024MB x 0.6 x 0.9 = 552.96MB. However, 552.96MB is a little larger than the value as shown in the log. This is because of the runtime overhead imposed by Scala, which is usually around 3-7%, more or less. If you do the calculation using 982MB x 0.6 x 0.9, (982MB being approximately 4% of 1024) then you will derive the number 530.28MB, which is what is indicated in the log file after rounding up to 530.30MB." 2) If Spark job is launched from the Map task via SparkLauncher#startApplication() will driver memory respect (mapreduce.map.memory.mb and mapreduce.map.java.opts) OR (yarn.scheduler.maximum-allocation-mb) when launching Spark Job as child process ? The confusion is, as SparkSubmit is a new JVM process - because it is launched as child process of the map task, and it does not depend on Yarn configs. But not obeying any limits (if this is the case), will make things tricky on NodeManager reporting back memory usage. 3) Is this correct formula for calculating AM memory ? For AM it matches to this formula calculation ( https://www.altiscale.com/blog/spark-on-hadoop/ ):how much memory to allocate to the AM: amMemory + amMemoryOverhead amMemoryOverhead is set to 384MB via spark.yarn.driver.memoryOverhead. args.amMemory is fixed at 512MB by Spark when it’s running in yarn-client mode. Adding 384MB of overhead to 512MB provides the 896MB figure requested by Spark. 4) For Spark Yarn-client mode, are all spark.driver properties ignored, and only spark.yarn.am properties used ? Thanks. -- Best regards, Elkhan Dadashov
