Looks like your code is not grammatically correct. It's impossible to chain
pcollection and pcollection with a vertical bar. Each chain should start
with an initial pipeline or pcollection and generates another pcollection
via a given ptransform.
If you modify your code like below, it should work:
import apache_beam as beam
def m(x, u):
print(u)
return x
p = beam.Pipeline()
data_beam = beam.Create(['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c'])
chain_1 = p | data_beam | beam.combiners.Count.PerElement()
chain_2 = chain_1 | beam.Map(lambda x: x[0]) | beam.combiners.ToList()
chain_total = chain_1 | beam.Map(m, beam.pvalue.AsSingleton(chain_2))
chain_total | beam.Map(print)
p.run()
Thanks,
On Tue, Apr 16, 2019 at 12:28 PM Paul, Austin <austin.p...@wyss.harvard.edu>
wrote:
> I encountered this error when writing my own Beam pipeline, and was
> frustrated at the lack of documentation/support as to what was wrong.
> Here's what I think the issue is (apologies if it is totally misguided):
>
> I find it helpful to think of PCollections as nouns and Transforms as
> verbs. Hopefully this adds clarity, but if it doesn't feel free to
> substitute the proper jargon in the explanation below.
> Applying a verb (or multiple verbs) to a noun will yield another noun.
> Applying verbs to verbs will just get you another verb.
> The only noun you start out with is p, your initial Pipeline object.
> You can use this p with verbs to create other nouns, as you do with
> chain1, which is a noun.
> chain2 on the otherhand is a verb applied to a verb, so it's just a verb.
> Since there's no actual content there, it can't be turned into a side
> input, since only nouns can do that.
>
> Chain_total is a noun. Did you mean to pass that instead?
>
> Best,
> Austin
>
>
>
> -----Original Message-----
> From: bruno.filipe.silva.dias@james.finance
> <bruno.filipe.silva.dias@james.finance>
> Sent: Tuesday, April 16, 2019 12:36 PM
> To: user@beam.apache.org
> Subject: Re: (python) Using ToList output as input for AsSingleton or
> AsList in Apache Beam
>
>
>
> On 2019/04/16 16:32:46, Bruno Dias <bruno.filipe.silva.dias@james.finance>
> wrote:
> > I'm getting an unexpected error when I try to use the output of
> > beam.combiners.ToList as the input of beam.pvalue.AsSingleton or
> > beam.pvalue.AsList in order to experiment with side inputs. I was able
> > to use single numbers (e.g.: the mean of a list) as a side input but,
> > for lists and dictionaries, I'm getting exceptions. For
> > beam.pvalue.AsSingleton, I get this:
> > https://www.pastiebin.com/5cb475355dedf
> >
> > For beam.pvalue.AsList, I get https://www.pastiebin.com/5cb475f0e7f96.
> >
> > This is the code I'm running: https://www.pastiebin.com/5cb4760eb9abe.
> >
> > Replace beam.pvalue.AsSingleton with beam.pvalue.AsList to get the
> > other error. I'm using Apache Beam python SDK version 2.11.0.
> >
>
>
> It seems the "." at the right of each link got concatenated with the link.
> You need to copy the 2nd and 3rd links to the address bar. Sorry about that.
>
