Hi, Thanks for the very quick reply! If I see the metrics "storage memory", always at 0, does that mean that the memory is neither used for caching or computing?
Thanks, Michel <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Garanti sans virus. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> Le ven. 20 mars 2020 à 14:45, Jack Kolokasis <koloka...@ics.forth.gr> a écrit : > Hello Michel, > > Spark seperates executors memory using an adaptive boundary between > storage and execution memory. If there is no caching and execution > memory needs more space, then it will use a portion of the storage memory. > > If your program does not use caching then you can reduce storage memory. > > Iacovos > > On 20/3/20 4:40 μ.μ., msumbul wrote: > > Hello, > > > > Im asking mysef the exact meaning of the setting of > > spark.memory.storageFraction. > > The documentation mention: > > > > "Amount of storage memory immune to eviction, expressed as a fraction of > the > > size of the region set aside by spark.memory.fraction. The higher this > is, > > the less working memory may be available to execution and tasks may > spill to > > disk more often" > > > > Does that mean that if there is no caching that part of the memory will > not > > be used at all? > > In the spark UI, in the tab "Executor", I can see that the "storage > memory" > > is always zero. Does that mean that that part of the memory is never > used at > > all and I can reduce it or never used for storage specifically? > > > > Thanks in advance for your help, > > Michel > > > > > > > > -- > > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
