Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21469
@arunmahadevan
Added custom metrics in state store to streaming query status as well. You
can see `providerLoadedMapSize` is added to `stateOperators.customMetrics` in
below output.
I have to exclude `providerLoadedMapCountOfVersions` from the list, since
average metric is implemented a bit tricky and doesn't look like easy to
aggregate for streaming query status.
We may want to reimplement SQLMetric and subclasses to make sure everything
works correctly without any tricky approach, but that doesn't look like trivial
to address and I think this is out of scope on this PR.
```
18/06/06 22:51:23 INFO MicroBatchExecution: Streaming query made progress: {
"id" : "7564a0b7-e3b2-4d53-b246-b774ab04e586",
"runId" : "8dd34784-080c-4f86-afaf-ac089902252d",
"name" : null,
"timestamp" : "2018-06-06T13:51:15.467Z",
"batchId" : 4,
"numInputRows" : 547,
"inputRowsPerSecond" : 67.15776550030694,
"processedRowsPerSecond" : 65.94333936106088,
"durationMs" : {
"addBatch" : 7944,
"getBatch" : 1,
"getEndOffset" : 0,
"queryPlanning" : 61,
"setOffsetRange" : 5,
"triggerExecution" : 8295,
"walCommit" : 158
},
"eventTime" : {
"avg" : "2018-06-06T13:51:10.313Z",
"max" : "2018-06-06T13:51:14.250Z",
"min" : "2018-06-06T13:51:07.098Z",
"watermark" : "2018-06-06T13:50:36.676Z"
},
"stateOperators" : [ {
"numRowsTotal" : 20,
"numRowsUpdated" : 16,
"memoryUsedBytes" : 26679,
"customMetrics" : {
"providerLoadedMapSize" : 181911
}
} ],
"sources" : [ {
"description" : "KafkaV2[Subscribe[apachelogs-v2]]",
"startOffset" : {
"apachelogs-v2" : {
"2" : 489056,
"4" : 489053,
"1" : 489055,
"3" : 489051,
"0" : 489053
}
},
"endOffset" : {
"apachelogs-v2" : {
"2" : 489056,
"4" : 489053,
"1" : 489055,
"3" : 489051,
"0" : 489053
}
},
"numInputRows" : 547,
"inputRowsPerSecond" : 67.15776550030694,
"processedRowsPerSecond" : 65.94333936106088
} ],
"sink" : {
"description" :
"org.apache.spark.sql.execution.streaming.ConsoleSinkProvider@60999714"
}
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
