Number of executors and tasks

Hi, I am running Spark in the stand alone mode. 1) I have a file of 286MB in HDFS (block size is 64MB) and so is split into 5 blocks. When I have the file in HDFS, 5 tasks are generated and so 5 files in the output. My understanding is that there will be a separate partition for each block and

Spark on YARN - master role

Hi, In the Spark on YARN, the AM (driver) will ask the RM for resources. Once the resources are allocated by the RM, the AM will start the executors through the NM. This is my understanding. But, according to the Spark documentation (1), the `spark.yarn.applicationMaster.waitTries` properties