Hello, I was reading the Garbage Collection Tuning guide here: Tuning - Spark 3.1.2 Documentation (apache.org)<https://spark.apache.org/docs/3.1.2/tuning.html#garbage-collection-tuning>, specifically section on "Advanced GC Tuning". It is stated that if OldGen region is getting full, it is recommended to lower spark.memory.fraction. I am wondering if this would lower the overall amount of memory available for both storage and execution, slowing down execution. Isn't it better to lower spark.memory.storageFraction instead? In this case there is less memory available for caching objects, while execution is not being affected. Please see below the copy of the passage I am referring to:
* "In the GC stats that are printed, if the OldGen is close to being full, reduce the amount of memory used for caching by lowering spark.memory.fraction; it is better to cache fewer objects than to slow down task execution. Alternatively, consider decreasing the size of the Young generation. This means lowering -Xmn if you've set it as above. If not, try changing the value of the JVM's NewRatio parameter. Many JVMs default this to 2, meaning that the Old generation occupies 2/3 of the heap. It should be large enough such that this fraction exceeds spark.memory.fraction." I would greatly appreciate if you could clarify it for me. Thank you in advance. Best regards, Oleksandr (Alex) Kuznetsov DC Solution Specialist | Consulting/Strategy and Analytics Deloitte Consulting LLP 1001 Heathrow Park Ln, Lake Mary, FL 32746 Mobile: 281-384-1331 olkuznet...@deloitte.com<mailto:mayv...@deloitte.com> | www.deloitte.com<http://www.deloitte.com/> This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and any disclosure, copying, or distribution of this message, or the taking of any action based on it, by you is strictly prohibited. Deloitte refers to a Deloitte member firm, one of its related entities, or Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is a separate legal entity and a member of DTTL. DTTL does not provide services to clients. Please see www.deloitte.com/about to learn more. v.E.1
