I recently came across this (haven’t tried it out yet) but maybe it can help guide you to identify the root cause.
https://github.com/groupon/sparklint From: Vitaliy Pisarev <[email protected]> Date: Thursday, November 15, 2018 at 10:08 AM To: user <[email protected]> Cc: David Markovitz <[email protected]> Subject: How to address seemingly low core utilization on a spark workload? I have a workload that runs on a cluster of 300 cores. Below is a plot of the amount of active tasks over time during the execution of this workload: What I deduce is that there are substantial intervals where the cores are heavily under-utilised. What actions can I take to: Increase the efficiency (== core utilisation) of the cluster? Understand the root causes behind the drops in core utilisation?
--------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
