Hi Natalie,
in case we are using Python then listener is not available, we can use the
SPARK UI REST API's .
For that to happen we have to start the SPARK session with following
settings: spark.sql("SET spark.sql.streaming.metricsEnabled=true")
For the UI from which we can use the JSON results
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#reporting-metrics-programmatically-using-asynchronous-apis
On Tue, Sep 3, 2019, 3:26 PM Natalie Ruiz
wrote:
> Hello all,
>
>
>
> I’m a beginner, new to Spark and wanted to know if there was an equivalent
> to Spark
Hello all,
I'm a beginner, new to Spark and wanted to know if there was an equivalent to
Spark Streaming's StreamingListenerBatchCompleted in Structured Streaming? I
want to add a listener for when a batch is complete but the documentation and
examples I find are for Spark Streaming and not
