There're sections in SS programming guide which exactly answer these questions:
http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#managing-streaming-queries http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#monitoring-streaming-queries Also, for Kafka data source, there's a 3rd party project (DISCLAIMER: I'm the author) to help you commit the offset to Kafka with the specific group ID. https://github.com/HeartSaVioR/spark-sql-kafka-offset-committer After then, you can also leverage the Kafka ecosystem to monitor the progress in point of Kafka's view, especially the gap between highest offset and committed offset. Hope this helps. Thanks, Jungtaek Lim (HeartSaVioR) On Mon, Jul 6, 2020 at 2:53 AM Gabor Somogyi <gabor.g.somo...@gmail.com> wrote: > In 3.0 the community just added it. > > On Sun, 5 Jul 2020, 14:28 KhajaAsmath Mohammed, <mdkhajaasm...@gmail.com> > wrote: > >> Hi, >> >> We are trying to move our existing code from spark dstreams to structured >> streaming for one of the old application which we built few years ago. >> >> Structured streaming job doesn’t have streaming tab in sparkui. Is there >> a way to monitor the job submitted by us in structured streaming ? Since >> the job runs for every trigger, how can we kill the job and restart if >> needed. >> >> Any suggestions on this please >> >> Thanks, >> Asmath >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >>
