Chamikara Jayalath resolved BEAM-738.
       Resolution: Fixed
    Fix Version/s: Not applicable

> Update sourcetestutils to fail when BoundedSources maintain state
> -----------------------------------------------------------------
>                 Key: BEAM-738
>                 URL: https://issues.apache.org/jira/browse/BEAM-738
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-py
>            Reporter: Chamikara Jayalath
>            Assignee: Chamikara Jayalath
>             Fix For: Not applicable
> Maintaining state in BoundedSource implementations is problematic and can 
> lead to hard to debug errors. For example (1) pickling errors (2) errors due 
> to a runner reusing a BoundedSource object with state.
> We can try to prevent users from adding state to BoundedSource 
> implementations in following two ways.
> (1) Clearly mention in BoundedSource API that objects should not maintain 
> transient state.
> (2) Update sourcetesutils to catch source objects that maintain local state.
> (2) can be done by adding a check that verifies that a source produces 
> expected output in the presence of a re-entrant read.
> i = s.read_records()
> i.next()
> i.next()
> read the whole thing from s.read_records()
> i.next() some more
> Verify that 'i' produced correct output.

This message was sent by Atlassian JIRA

Reply via email to