Hello. We are trying to migrate and performance test the kafka sink for structured streaming in 2.1. Obviously we miss the beautiful Streaming Statistics ui tab and we are trying to figure out the most reasonable way to monitor event processing rates and lag time.
1. Are the SourceStatus and SinkStatus mentioned in the "Metrics in Structured Streaming" design doc <https://docs.google.com/document/d/1NIdcGuR1B3WIe8t7VxLrt58TJB4DtipWEbj5I_mzJys/edit#> available with the json or jmx metrics? I can't seem to find any of the structured streaming metrics in either sink. 2. RDD Streaming metrics provides total counts such as totalProcessedRecords, totalReceivedRecords Is there anything similar in Structured Streaming? 3. Are there any plans for supporting something like the Streaming Statistics tab for structured streaming in the web UI? Or if we use structured streaming, should we be expected to integrate our own reporting server like ganglia? Any help is greatly appreciated. Thanks, Heji
