Hi, I'm trying to build automatic Kafka watermark handling in my stream apps by overriding the KafkaRDDIterator, and adding a taskcompletionlistener and updating watermarks if task was completed (the iterator has access to offsets). But I found out that there is no way to listen to a task error inside the executors. Only the driver gets the TaskReason notification, but not the task completion listener. This basically means watermarks will be updated regardless of whether the task completed successfully or not.
Is there any way to listen for the task failures on the executors? Thanks! -neelesh
