Re: Reading all elements of a PCollection after running beam go pipeline
Great! On Fri, May 4, 2018 at 4:37 PM 8 Gianfortoni <8...@tokentransit.com> wrote: > Thanks for the workaround! That should work for me. > > On Fri, May 4, 2018, 1:51 PM Henning Rohdewrote: > >> Hey there, >> >> Until side input is fully supported, you can use GBK with a fixed key to >> get all elements in a single bundle (assuming global windowing). That is >> how textio.Write works internally to produce a single file currently: >> >> https://github.com/apache/beam/blob/90ba47114997e75a3a7daa1e8db92768d05e5432/sdks/go/pkg/beam/io/textio/textio.go#L144 >> >> >> >> Thanks, >> Henning >> >> >> On Fri, May 4, 2018 at 11:41 AM 8 Gianfortoni <8...@tokentransit.com> wrote: >> >>> Hi dev team, >>> >>> I would like to be able to read the entire results of a PCollection >>> serially after running beam. In other frameworks this is fairly >>> straightforward, but I don't understand how one might do this with the Beam >>> Go SDK. >>> >>> I guess I can read in a file that I write, but I want to be able to read >>> the elements in struct format. I looked through a bunch of examples but >>> don't see any that do this. >>> >>> Thanks, >>> 8 >>> >>
Re: Reading all elements of a PCollection after running beam go pipeline
Thanks for the workaround! That should work for me. On Fri, May 4, 2018, 1:51 PM Henning Rohdewrote: > Hey there, > > Until side input is fully supported, you can use GBK with a fixed key to > get all elements in a single bundle (assuming global windowing). That is > how textio.Write works internally to produce a single file currently: > > https://github.com/apache/beam/blob/90ba47114997e75a3a7daa1e8db92768d05e5432/sdks/go/pkg/beam/io/textio/textio.go#L144 > > > > Thanks, > Henning > > > On Fri, May 4, 2018 at 11:41 AM 8 Gianfortoni <8...@tokentransit.com> wrote: > >> Hi dev team, >> >> I would like to be able to read the entire results of a PCollection >> serially after running beam. In other frameworks this is fairly >> straightforward, but I don't understand how one might do this with the Beam >> Go SDK. >> >> I guess I can read in a file that I write, but I want to be able to read >> the elements in struct format. I looked through a bunch of examples but >> don't see any that do this. >> >> Thanks, >> 8 >> >
Re: Reading all elements of a PCollection after running beam go pipeline
Hey there, Until side input is fully supported, you can use GBK with a fixed key to get all elements in a single bundle (assuming global windowing). That is how textio.Write works internally to produce a single file currently: https://github.com/apache/beam/blob/90ba47114997e75a3a7daa1e8db92768d05e5432/sdks/go/pkg/beam/io/textio/textio.go#L144 Thanks, Henning On Fri, May 4, 2018 at 11:41 AM 8 Gianfortoni <8...@tokentransit.com> wrote: > Hi dev team, > > I would like to be able to read the entire results of a PCollection > serially after running beam. In other frameworks this is fairly > straightforward, but I don't understand how one might do this with the Beam > Go SDK. > > I guess I can read in a file that I write, but I want to be able to read > the elements in struct format. I looked through a bunch of examples but > don't see any that do this. > > Thanks, > 8 >
Reading all elements of a PCollection after running beam go pipeline
Hi dev team, I would like to be able to read the entire results of a PCollection serially after running beam. In other frameworks this is fairly straightforward, but I don't understand how one might do this with the Beam Go SDK. I guess I can read in a file that I write, but I want to be able to read the elements in struct format. I looked through a bunch of examples but don't see any that do this. Thanks, 8