Hello, We have an application in which specific partitions need to be executed on specific worker machines. Thus, we would like to assign for each partition, the worker machine that will execute that partition. Moreover we would like to do it before each stage of the SPARK app, and be able to reassign the worker machine prior to every stage. Currently, we are able to do so before the first stage by overwriting getPrefferedLocation. Is there a way to change the preferred location before each stage?
Thanks and Regards, Simona Rabinovici-Cohen