Charles Chen created BEAM-3537: ---------------------------------- Summary: Remove DirectRunner-specific internal PValue cache, allow more general eager in-process pipeline execution Key: BEAM-3537 URL: https://issues.apache.org/jira/browse/BEAM-3537 Project: Beam Issue Type: Bug Components: sdk-py-core Affects Versions: 2.2.0 Reporter: Charles Chen Assignee: Charles Chen
Currently, the Python SDK supports an eager execution mode. For example, a list can be directly passed into a PTransform to obtain its result: {{result = [1, 2, 3] | MyPTransform()}} To support this use, the Python DirectRunner has an option to cache its intermediate results into a PValueCache. The above line, when run, implicitly creates an ephemeral pipeline and runs it with the DirectRunner. This, however, adds a lot of complexity to the DirectRunner, and is not generalizable to other in-process Python runners (like the in-process Python FnApiRunner). To improve this, we should remove this DirectRunner-specific implementation and add functionality that allows all in-process Python runners to be run in eager mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)