When you say "the application remained alive", do you mean the StreamingQuery stayed alive, or the whole process stayed alive? The StreamingQuery should be terminated immediately. And the stream execution threads are all daemon threads, so it should not affect the termination of the application whether the queries are active or not. May be something else is keeping the application alive?
On Tue, Aug 29, 2017 at 2:09 AM, Yuval Itzchakov <yuva...@gmail.com> wrote: > I wasn't sure if this would be a proper bug or not. > > Today, the behavior of Structured Streaming is such that if a source fails > with an exception, the `StreamExecution` class halts reading further from > the source, but the application is remained alive. For applications where > the sole purpose is to transform data from a non static source (such as > Kafka), this is rather useless and might be surprising. > > For example, if you have a simple monitor which checks whether the > application is alive or not, you'll still get reports that the application > is alive and running, but actually it isn't consuming anything from the > source and is logically dead. > > Should this be the behavior? I think that perhaps there should be a > configuration that asks whether to completely shutdown the application on > source failure. > > What do you guys think? > > -- > Best Regards, > Yuval Itzchakov. >
