Satish,
I agree - that was my impression too. However I am seeing a smaller set of storage memory used on a given executor than the amount of memory required for my broadcast variables. I am wondering if the statistics in the ui are incorrect or if the broadcasts are simply not a part of that storage memory fraction. Bryan Jeffrey Get Outlook for Android On Wed, Jun 21, 2017 at 6:48 PM -0400, "satish lalam" <satish.la...@gmail.com> wrote: My understanding is - it from storageFraction. Here cached blocks are immune to eviction - so both persisted RDDs and broadcast variables sit here. Ref On Wed, Jun 21, 2017 at 1:43 PM, Bryan Jeffrey <bryan.jeff...@gmail.com> wrote: Hello. Question: Do broadcast variables stored on executors count as part of 'storage memory' or other memory? A little bit more detail: I understand that we have two knobs to control memory allocation:- spark.memory.fraction- spark.memory.storageFraction My understanding is that spark.memory.storageFraction controls the amount of memory allocated for cached RDDs. spark.memory.fraction controls how much memory is allocated to Spark operations (task serialization, operations, etc.), w/ the remainder reserved for user data structures, Spark internal metadata, etc. This includes the storage memory for cached RDDs. You end up with executor memory that looks like the following:All memory: 0-100Spark memory: 0-75RDD Storage: 0-37Other Spark: 38-75Other Reserved: 76-100 Where do broadcast variables fall into the mix? Regards, Bryan Jeffrey
