Hello I have several *Structured Streaming Spark applications*, and I want to measure three key performance metrics *per micro-batch*:
1. *Write duration* – the total time taken to write data to an output sink (e.g., a Delta Lake sink). 2. *Processing time* – the time spent performing operations like aggregations or joins within each micro-batch. 3. *Read duration* – the time spent reading from a streaming source (e.g., Kafka) while data is arriving. How can I access these metrics? I have experimented with writing a custom listener using both the SparkListener and the StreamingQueryListener classes. The most relevant metrics I’ve found are: - *For processing time*: CPU time and runtime metrics available in stageInfo.taskMetrics, which are aggregated across all tasks per stage. - *For write duration*: ShuffleWriteMetrics.writeTime, which unfortunately only reflects time spent writing shuffle data, *not* writing to external sinks. - *For read duration*: QueryProgressEvent.durationMs.latestOffset, which applies only to certain streaming sources like Kafka, and doesn't reflect the actual read duration across all sources. As a result, I haven’t found a reliable or complete set of metrics that accurately report *write, read, and processing durations* separately for each micro-batch. Can anyone provide guidance or solutions for how to extract or approximate these metrics?
