Chul Kang created S2GRAPH-242:

             Summary: Reporting streaming job metrics
                 Key: S2GRAPH-242
             Project: S2Graph
          Issue Type: Sub-task
          Components: s2jobs
            Reporter: Chul Kang
            Assignee: Chul Kang

Spark structured streaming provides a simple metric for each small batch, using 
them it is possible to monitor a streaming job.



We can simply report metrics using drop wizard library via the metricsEnabled 
setting, but I think it would be convenient to implement a 
StreamingQueryListener so that only the necessary events are taken into the 


Personally, if we implement a KafkaStreamingListener and send metrics to Kafka, 
it seems to be easy to save in other storage or configure dashboards or alarms.







This message was sent by Atlassian JIRA

Reply via email to