Hi All, I have a java program which submits a spark job to a standalone spark cluster (2 nodes; 10 cores (6+4); 12GB (8+4)). This is being called by another java program through ExecutorService and invokes it multiple times with different set of arguments and parameters. I have set spark memory usage to 3GB (tried with 2GB and 1GB also)
When I run this for 4 parallel jobs - first job finishes successfully, second and third jobs start executing in parallel and never complete and fourth job is waiting in queue. I've tried using "Fair Scheduler" but didn't notice any change in the behavior. Also in the spark job submission program I'm calling SparkContext.stop at the end of execution. Some times all jobs fail with status as "Exited". Please let me know what is going wrong and how to overcome the issue? ~Sarath