Hey all, Just wondering if anyone has had issues with this or if it is expected that the semantic around the memory management is different here.
Thanks -Pat On Tue, Apr 19, 2016 at 9:32 AM, Patrick Woody <patrick.woo...@gmail.com> wrote: > Hey all, > > I had a question about the MemoryStore for the BlockManager with the > unified memory manager v.s. the legacy mode. > > In the unified format, I would expect the max size of the MemoryStore to be > > <total max memory> * <spark.memory.fraction> * > <spark.memory.storageFraction> > > in the same way that when using the StaticMemoryManager it is > > <total max memory> * <spark.storage.memoryFraction> * > <spark.storage.safetyFraction>. > > Instead it appears to be > > (<total max memory> * <spark.memory.fraction>) - > onHeapExecutionMemoryPool.memoryUsed > > I would expect onHeapExecutionMemoryPool.memoryUsed to be around size 0 at > the time of initialization, so this may be overcommitting memory to the > MemoryStore. Does this line up with expectations? > > For context, I have a very large SQL query that does a significant amount > of broadcast joins (~100) so I need my driver to be able to toss those into > the DiskStore without going into GC hell. Since my MemoryStore seems to not > be bounded by the storage pool, this fills up my heap and causes my > application to OOM. Simply reducing spark.memory.fraction alleviates the > problem, but I'd love to understand if that is actually the correct fix > here rather than simply lowering the storageFraction. > > Thanks! > -Pat > > >
