[
https://issues.apache.org/jira/browse/BEAM-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17123986#comment-17123986
]
Kenneth Knowles commented on BEAM-3736:
---------------------------------------
I would start by taking a look at the places where SetUp and TearDown are
invoked:
- https://github.com/apache/beam/search?q=invokeSetup&unscoped_q=invokeSetup
-
https://github.com/apache/beam/search?q=invokeTearDown&unscoped_q=invokeTearDown
The two implementations I am somewhat familiar with are the DirectRunner and
Dataflow, at:
-
https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DoFnLifecycleManager.java
-
https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DoFnInstanceManagers.java
These each keep a function available for a given class. The function can be
re-used across bundles and it calls SetUp only when making a new one and
TearDown when discarding.
> Add SetUp() and TearDown() for CombineFns
> -----------------------------------------
>
> Key: BEAM-3736
> URL: https://issues.apache.org/jira/browse/BEAM-3736
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, sdk-py-core
> Reporter: Chuan Yu Foo
> Priority: P2
>
> I have a CombineFn that has a large amount of state that needs to be loaded
> once before it can add_input or merge_combiners (for example, the CombineFn
> might load up a large lookup table used for combining).
> Right now, to initialise this state, for each of the methods, I check if the
> state has already been initialised, and if not, I initialise it. It would be
> nice if CombineFn provided a SetUp() method that is called once to initialise
> this state (and a corresponding TearDown() method to clean up this state if
> necessary).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
