This is how Spark runner handle this: https://github.com/apache/beam/blob/3e6f7b77add44b2b7b1a0ef4afd631642b7d0b59/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java#L173
> On 21 Sep 2020, at 18:17, Luke Cwik <[email protected]> wrote: > > +user <mailto:[email protected]> > > On Mon, Sep 21, 2020 at 9:16 AM Luke Cwik <[email protected] > <mailto:[email protected]>> wrote: > You need the "sources" to stop and advance the watermark to infinity and have > that propagate through the entire pipeline. There are propoosals for pipeline > drain[1] and also for snapshot and update[2] for Apache Beam. We would love > contributions in this space. > > Max shared some more details about how Flink users typically do this[3], does > that apply to Spark? > > 1: > https://docs.google.com/document/d/1NExwHlj-2q2WUGhSO4jTu8XGhDPmm3cllSN8IMmWci8 > > <https://docs.google.com/document/d/1NExwHlj-2q2WUGhSO4jTu8XGhDPmm3cllSN8IMmWci8> > 2: > https://docs.google.com/document/d/1UWhnYPgui0gUYOsuGcCjLuoOUlGA4QaY91n8p3wz9MY > > <https://docs.google.com/document/d/1UWhnYPgui0gUYOsuGcCjLuoOUlGA4QaY91n8p3wz9MY> > 3: > https://lists.apache.org/thread.html/864eb7b4e7192706074059eef1e116146382552fa885dd6054ef4988%40%3Cuser.beam.apache.org%3E > > <https://lists.apache.org/thread.html/864eb7b4e7192706074059eef1e116146382552fa885dd6054ef4988%40%3Cuser.beam.apache.org%3E> > On Mon, Sep 21, 2020 at 7:43 AM Sunny, Mani Kolbe <[email protected] > <mailto:[email protected]>> wrote: > Forgot to mention, we are using spark runner. > > > > From: Sunny, Mani Kolbe <[email protected]> > Sent: Monday, September 21, 2020 12:33 PM > To: [email protected] <mailto:[email protected]> > Subject: How to gracefully stop a beam application > > > > CAUTION: This email originated from outside of D&B. Please do not click links > or open attachments unless you recognize the sender and know the content is > safe. > > > > Hello Beam community, > > > > When you are running a Beam application in full stream mode, it is > continuously running. What is the recommended way to stop it gracefully for > say maintenance/upgrades etc? When I say gracefully, I mean (1) without data > loss and (2) application existing with exit 0 code. > > > > Regards, > > Mani >
