+1 I think this will prove itself even more important with Per-Key workflows where DoFn's might communicate constantly with external services - DB, Elasticsearch, etc.
On Mon, Jul 18, 2016 at 5:44 PM Maximilian Michels <[email protected]> wrote: > Hoping it becomes usual as soon as we have this useful addition :) > > On Mon, Jul 18, 2016 at 1:53 PM, Aljoscha Krettek <[email protected]> > wrote: > > > Did you mean "usual" or "useful"? ;-) > > > > On Mon, 18 Jul 2016 at 12:42 Maximilian Michels <[email protected]> wrote: > > > > > +1 for setup() and teardown() methods. Very usual for proper > > initialization > > > and cleanup of DoFn related data structures. > > > > > > On Wed, Jun 29, 2016 at 9:34 PM, Aljoscha Krettek <[email protected] > > > > > wrote: > > > > > > > +1 I think some people might already mistake the > > > > startBundle()/finishBundle() methods for what the new methods are > > > supposed > > > > to be > > > > > > > > On Tue, 28 Jun 2016 at 19:38 Raghu Angadi <[email protected] > > > > > > wrote: > > > > > > > > > This is terrific! > > > > > Thanks for the proposal. > > > > > > > > > > On Tue, Jun 28, 2016 at 9:06 AM, Thomas Groh > > <[email protected] > > > > > > > > > wrote: > > > > > > > > > > > Hey Everyone: > > > > > > > > > > > > We've recently started to be permitted to reuse DoFn instances in > > > > > Beam[1]. > > > > > > Beyond the efficiency gains from not having to deserialize new > DoFn > > > > > > instances for every bundle, DoFn reuse also provides the ability > to > > > > > > minimize expensive setup work done per-bundle, which hasn't > > formerly > > > > been > > > > > > possible. Additionally, it has also enabled more failure cases, > > where > > > > > > element-based state leaks improperly across bundles. > > > > > > > > > > > > I've written a document proposing that two methods are added to > the > > > API > > > > > of > > > > > > DoFn, setup and teardown, which both provides hooks for users to > > > write > > > > > > efficient DoFns, as well as signals that DoFns will be reused. > > > > > > > > > > > > The document is located at > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1LLQqggSePURt3XavKBGV7SZJYQ4NW8yCu63lBchzMRk/edit?ts=5771458f# > > > > > > and committers have edit access > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Thomas > > > > > > > > > > > > [1] https://github.com/apache/incubator-beam/pull/419 > > > > > > > > > > > > > > > > > > > > >
