In my production setup spark is always taking 40 seconds between these steps like a fixed counter is set. In my local lab these steps take exact 1 second. I am not able to find the exact root cause of this behaviour. My Spark application is running on Hortonworks platform in yarn client mode. Can someone guide me what is happening between these steps
18/05/04 *07:56:45* INFO spark.SparkContext: Added JAR file:/app/jobs/jobs.jar at spark://10.233.69.5:37668/jars/jobs.jar with timestamp 1525420605369 18/05/04 *07:57:26* WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 18/05/04 *07:58:12* INFO client.AHSProxy: Connecting to Application History server at gsidev001-mgt-01.thales.fr/192.168.1.11:10200 -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org