Github user CodingCat commented on the issue:
https://github.com/apache/spark/pull/19824
if you just worry about
> As I was using the StreamingListenerBatchCompleted to do some metadata
checkpointing stuff, which should be done only when the batch succeeded.
If this is your concern, you should handle whether complete is with a
failure in your own listener, check:
https://github.com/Azure/azure-event-hubs-spark/blob/fecc34de8a238049806d033d8a85d888cad75901/core/src/main/scala/org/apache/spark/streaming/eventhubs/checkpoint/ProgressTrackingListener.scala#L38
IMHO, `Complete` includes Failure&Success cases is more intuitive to the
(scala) user since we are used to .OnComplete in Scala Future, etc.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
