You can use the monitoring APIs of Structured Streaming to get metrics. See http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#monitoring-streaming-queries
On Tue, Jan 17, 2017 at 5:01 PM, Heji Kim <hster.investiga...@gmail.com> wrote: > Hello. We are trying to migrate and performance test the kafka sink for > structured streaming in 2.1. Obviously we miss the beautiful Streaming > Statistics ui tab and we are trying to figure out the most reasonable way > to monitor event processing rates and lag time. > > 1. Are the SourceStatus and SinkStatus mentioned in the "Metrics in > Structured Streaming" design doc > <https://docs.google.com/document/d/1NIdcGuR1B3WIe8t7VxLrt58TJB4DtipWEbj5I_mzJys/edit#> > available with the json or jmx metrics? I can't seem to find any of the > structured streaming metrics in either sink. > > 2. RDD Streaming metrics provides total counts such as > > totalProcessedRecords, totalReceivedRecords > > Is there anything similar in Structured Streaming? > > 3. Are there any plans for supporting something like the Streaming > Statistics tab for structured streaming in the web UI? Or if we use > structured streaming, should we be expected to integrate our own reporting > server like ganglia? > > Any help is greatly appreciated. > > Thanks, > Heji > > > >
