Hi all, I am trying to do some actions at the Driver side in Spark while an application is running. The Driver needs to know the tasks progress before making any decision. I know that tasks progress can be accessed within each executor or task from RecordReader class by calling getProgress().
The question is, how can I let the Driver call or have an access to getProgress() method of each task? I thought about using broadcast and accumulator variables, but I don't know how the Driver would distinguish between different tasks. Note that I am not looking for results displayed in Spark UI. Any help is appreciated!
