Chul Kang created S2GRAPH-242:
---------------------------------

             Summary: Reporting streaming job metrics
                 Key: S2GRAPH-242
                 URL: https://issues.apache.org/jira/browse/S2GRAPH-242
             Project: S2Graph
          Issue Type: Sub-task
          Components: s2jobs
            Reporter: Chul Kang
            Assignee: Chul Kang


Spark structured streaming provides a simple metric for each small batch, using 
them it is possible to monitor a streaming job.

[https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#monitoring-streaming-queries]

 

We can simply report metrics using drop wizard library via the metricsEnabled 
setting, but I think it would be convenient to implement a 
StreamingQueryListener so that only the necessary events are taken into the 
callback.

 

Personally, if we implement a KafkaStreamingListener and send metrics to Kafka, 
it seems to be easy to save in other storage or configure dashboards or alarms.

 

 

 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to