Also, you can get information about the last progress made (input rates, etc.) from StreamingQuery.lastProgress, StreamingQuery.recentProgress, and using StreamingQueryListener. Its all documented - https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#monitoring-streaming-queries
On Wed, Jun 20, 2018 at 6:06 PM, Tathagata Das <tathagata.das1...@gmail.com> wrote: > Structured Streaming does not maintain a queue of batch like DStream. > DStreams used to cut off batches at a fixed interval and put in a queue, > and a different thread processed queued batches. In contrast, Structured > Streaming simply cuts off and immediately processes a batch after the > previous batch finishes. So the question about queue size and lag does not > apply to Structured Streaming. > > That said, there is no UI for Structured Streaming. You can see the sql > plans for each micro-batch in the SQL tab. > > > > > > On Wed, Jun 20, 2018 at 12:12 PM, SRK <swethakasire...@gmail.com> wrote: > >> hi, >> >> How do we get information like lag and queued up batches in Structured >> streaming? Following api does not seem to give any info about lag and >> queued up batches similar to DStreams. >> >> https://spark.apache.org/docs/2.2.1/api/java/org/apache/spar >> k/streaming/scheduler/BatchInfo.html >> >> Thanks! >> >> >> >> >> -- >> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> >
