[
https://issues.apache.org/jira/browse/BEAM-738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chamikara Jayalath resolved BEAM-738.
-------------------------------------
Resolution: Fixed
Fix Version/s: Not applicable
> Update sourcetestutils to fail when BoundedSources maintain state
> -----------------------------------------------------------------
>
> Key: BEAM-738
> URL: https://issues.apache.org/jira/browse/BEAM-738
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py
> Reporter: Chamikara Jayalath
> Assignee: Chamikara Jayalath
> Fix For: Not applicable
>
>
> Maintaining state in BoundedSource implementations is problematic and can
> lead to hard to debug errors. For example (1) pickling errors (2) errors due
> to a runner reusing a BoundedSource object with state.
> We can try to prevent users from adding state to BoundedSource
> implementations in following two ways.
> (1) Clearly mention in BoundedSource API that objects should not maintain
> transient state.
> (2) Update sourcetesutils to catch source objects that maintain local state.
> (2) can be done by adding a check that verifies that a source produces
> expected output in the presence of a re-entrant read.
> i = s.read_records()
> i.next()
> i.next()
> read the whole thing from s.read_records()
> i.next() some more
> Verify that 'i' produced correct output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
