Chad Dombrova created BEAM-8088:
-----------------------------------
Summary: PCollection boundedness should be tracked and propagated
Key: BEAM-8088
URL: https://issues.apache.org/jira/browse/BEAM-8088
Project: Beam
Issue Type: Improvement
Components: sdk-py-core
Reporter: Chad Dombrova
As far as I can tell Python does not care about boundedness of PCollections
even in streaming mode, but external transforms _do_. In my ongoing effort to
get PubsubIO external transforms working I discovered that I could not generate
an unbounded write.
My pipeline looks like this:
{code:python}
(
pipe
| 'PubSubInflow' >>
external.pubsub.ReadFromPubSub(subscription=subscription, with_attributes=True)
| 'PubSubOutflow' >> external.pubsub.WriteToPubSub(topic=OUTPUT_TOPIC,
with_attributes=True)
)
{code}
The PCollections returned from the external Read are Unbounded, as expected,
but python is responsible for creating the intermediate PCollection, which is
always Bounded, and thus external Write is always Bounded.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
