Hello, I have quite a weird behaviour that I can't quite wrap my head around. I am running Spark on a Hadoop YARN cluster. I have Spark configured in such a way that it utilizes all free vcores in the cluster (setting max vcores per executor and number of executors to use all vcores in cluster).
Once oozie launcher job and spark AM claim their job, there should be free resources for 8 spark executor nodes but in spark UI I only see 7 active nodes (there should be two spark nodes per one hadoop host). I have checked what containers are running on each hadoop nodes and discovered that one node is in deed running more spark containers than is reported in spark UI. This behaviour is very strange to me and I have no idea what to make of it or how to debug it. Any thoughts? Thanks. -- Jan Sterba https://twitter.com/honzasterba | http://flickr.com/honzasterba | http://500px.com/honzasterba --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
