Re: Spark cluster tuning recommendation

Hi, Have you see a slide in spark summit 2016? https://spark-summit.org/2016/events/top-5-mistakes-when-writing-spark-applications/ This is a good slide for your capacity planning. // maropu On Tue, Jul 12, 2016 at 2:31 PM, Yash Sharma wrote: > I would say use the dynamic

Re: Spark cluster tuning recommendation

I would say use the dynamic allocation rather than number of executors. Provide some executor memory which you would like. Deciding the values requires couple of test runs and checking what works best for you. You could try something like - --driver-memory 1G \ --executor-memory 2G \

Re: Spark cluster tuning recommendation

That configuration looks bad. With only two cores in use and 1GB used by the app. Few points- 1. Please oversubscribe those CPUs to at-least twice the amount of cores you have to start-with and then tune if it freezes 2. Allocate all of the CPU cores and memory to your running app (I assume it is

Spark cluster tuning recommendation

I am trying a run terasort in spark , for a 7 node cluster with only 10g of data and executors get lost with GC overhead limit exceeded error. This is what my cluster looks like - - *Alive Workers:* 7 - *Cores in use:* 28 Total, 2 Used - *Memory in use:* 56.0 GB Total, 1024.0 MB Used