[ https://issues.apache.org/jira/browse/BEAM-681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmet Altay updated BEAM-681: ----------------------------- Labels: sdk-consistency (was: ) > DoFns should be serialized at apply time and deserialized when executing > ------------------------------------------------------------------------ > > Key: BEAM-681 > URL: https://issues.apache.org/jira/browse/BEAM-681 > Project: Beam > Issue Type: Improvement > Components: sdk-py > Reporter: Ben Chambers > Assignee: Frances Perry > Labels: sdk-consistency > > 1. Serializing DoFns at application time ensures that any modifications of > fields within the DoFn after application do not accidentally pollute the > execution. This mirrors the approach taken in Java to provide an > approximation of lexical-closure (eg., you only need to know the state of the > DoFn at the time it was applied, not afterwards, to understand its behavior). > 2. Based on 1, the DIrectRunner should also be deserializing DoFns before > running them, which should also detect other classes of errors such as using > the pipeline object (which is not pickleable) within the DoFn -- This message was sent by Atlassian JIRA (v6.3.4#6332)