Hi Luke, Thanks a lot for your reply. Since it allows to share one SDK harness between multiple executable stages, the control service termination may occur much later than the completion of an executable stage. This is the main reason I prefer runners to control the teardown of DoFns.
Regarding to "SDK harnesses can terminate instances any time they want and start new instances anytime as well.", personally I think it's not conflict with the proposed Approach 1 as the SDK harness could decide what to do when receiving the teardown request. It could do nothing if the DoFns has already been teared down and could also tear down the DoFns if needed. What do you think? Best, Jincheng Luke Cwik <lc...@google.com> 于2019年10月22日周二 上午2:05写道： > Approach 2 is currently the suggested approach[1] for DoFn's to shutdown. > Note that SDK harnesses can terminate instances any time they want and > start new instances anytime as well. > > Why do you want to expose this logic so that Runners could control it? > > 1: > https://docs.google.com/document/d/1n6s3BOxOPct3uF4UgbbI9O9rpdiKWFH9R6mtVmR7xp0/edit# > > On Mon, Oct 21, 2019 at 4:27 AM jincheng sun <sunjincheng...@gmail.com> > wrote: > >> Hi, >> I found that in `SdkHarness` do not stop the `SdkWorker` when finish. >> We should add the logic for stop the `SdkWorker` in `SdkHarness`. More >> detail can be found [1]. >> >> There are two approaches to solve this issue: >> >> Approach 1: We can add a Fn API for teardown purpose and the runner will >> teardown a specific bundle descriptor via this teardown Fn API during >> disposing. >> Approach 2: The control service termination could be seen as a signal and >> once SDK harness receives this signal, the teardown of the bundle >> descriptor will be performed. >> >> More detail can be found in [2]. >> >> As the Approach 2, SDK harness could be shared between multiple >> executable stages. The control service termination only occurs when all the >> executable stages sharing the same SDK harness finished. This means that >> the teardown of DoFns may not be executed immediately after an executable >> stage is finished. >> >> So, I prefer Approach 1. Welcome any feedback :) >> >> Best, >> Jincheng >> >> [1] >> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/sdk_worker.py >> [2] >> https://docs.google.com/document/d/1sCgy9VQPf9zVXKRquK8P6N4x7aB62GEO8ozkujRSHZg/edit?usp=sharing >> >
