My understanding is - it from storageFraction. Here cached blocks are immune to eviction - so both persisted RDDs and broadcast variables sit here. Ref <https://image.slidesharecdn.com/sparkinternalsworkshoplatest-160303190243/95/apache-spark-in-depth-core-concepts-architecture-internals-20-638.jpg?cb=1457597704>
On Wed, Jun 21, 2017 at 1:43 PM, Bryan Jeffrey <bryan.jeff...@gmail.com> wrote: > Hello. > > Question: Do broadcast variables stored on executors count as part of > 'storage memory' or other memory? > > A little bit more detail: > > I understand that we have two knobs to control memory allocation: > - spark.memory.fraction > - spark.memory.storageFraction > > My understanding is that spark.memory.storageFraction controls the amount > of memory allocated for cached RDDs. spark.memory.fraction controls how > much memory is allocated to Spark operations (task serialization, > operations, etc.), w/ the remainder reserved for user data structures, > Spark internal metadata, etc. This includes the storage memory for cached > RDDs. > > You end up with executor memory that looks like the following: > All memory: 0-100 > Spark memory: 0-75 > RDD Storage: 0-37 > Other Spark: 38-75 > Other Reserved: 76-100 > > Where do broadcast variables fall into the mix? > > Regards, > > Bryan Jeffrey >
