Hi Sam,

Thanks for clarifying the accessor to output when building a pipeline.

Internally, we have AppliedPTransform, where the output is always a
dictionary:
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pipeline.py#L770
And it seems to me that with key 'None', the output will be the main output.

Ning.

On Thu, Nov 7, 2019 at 1:39 PM Sam Rohde <sro...@google.com> wrote:

> Hi All,
>
> In the Python SDK there are three ways of representing the output of a
> PTransform with multiple PCollections:
>
>    - dictionary: PCollection tag --> PCollection
>    - tuple: index --> PCollection
>    - DoOutputsTuple: tag, index, or field name --> PCollection
>
> I find this inconsistent way of accessing multiple outputs to be
> confusing. Say that you have an arbitrary PTransform with multiple outputs.
> How do you know how to access an individual output without looking at the
> source code? *You can't!* Remember there are three representations of
> multiple outputs. So, you need to look at the output type and determine
> what the output actually is.
>
> What purpose does it serve to have three different ways of representing a
> single concept of multiple output PCollections?
>
> My proposal is to have a single representation analogous to Java's
> PCollectionTuple. With this new type you will able to access PCollections
> by tag with the "[ ]" operator or by field name. It should also up-convert
> returned tuples, dicts, and DoOutputsTuples from composites into this new
> type.
>
> Full example:
>
> class SomeCustomComposite(PTransform):
>   def expand(self, pcoll):
>     def my_multi_do_fn(x):
>       if isinstance(x, int):
>         yield pvalue.TaggedOutput('number', x)
>       if isinstance(x, str):
>         yield pvalue.TaggedOutput('string', x)
>
>     def printer(x):
>       print(x)
>       yield x
>
>     outputs = pcoll | beam.ParDo(my_multi_do_fn).with_outputs()*    return 
> pvalue.PTuple({
>         'number': output.number | beam.ParDo(printer),
>         'string': output.string | beam.ParDo(printer)
>     })*
>
> p = beam.Pipeline()
> *main = p | SomeCustomComposite()*
>
> # Access PCollection by field name.
> numbers = *main.number* | beam.ParDo(...)
>
> # Access PCollection by tag.
> strings = *main['string']* | beam.ParDo(...)
>
> What do you think? Does this clear up the confusion of using multiple
> output PCollections in Python?
>
> Regards,
> Sam
>

Reply via email to