Would the following work though? I could be misunderstanding the situation:
transform = p.apply(some transform) transform.apply(write) transform.apply(combine).apply(something on combined result) p.run() Cheers, Gwilym On 12 June 2017 at 02:36, Lukasz Cwik <[email protected]> wrote: > Unfortunately you can't Combine Writes since they return PDone (a terminal > node) during pipeline construction. > > On Sun, Jun 11, 2017 at 3:23 PM, Gwilym Evans < > [email protected]> wrote: > >> I'm not 100% sure as I haven't tried it, but, Combining comes to mind as >> a possible way of doing this, assuming your data is finite >> >> https://beam.apache.org/documentation/programming-guide/# >> transforms-combine >> >> You could take the PCollection result of 2 and simultaneously apply the >> Write and the Combine, using the singular result of the Combine to trigger >> the remaining steps >> >> Hope that helps, I'm still learning >> >> -Gwilym >> >> >> On 11 June 2017 at 16:50, Morand, Sebastien <[email protected]> >> wrote: >> >>> No, I can't, the pipeline is created within a cron, which is limited to >>> 10 minutes. >>> >>> *Sébastien MORAND* >>> Team Lead Solution Architect >>> Technology & Operations / Digital Factory >>> Veolia - Group Information Systems & Technology (IS&T) >>> Cell.: +33 7 52 66 20 81 / Direct: +33 1 85 57 71 08 >>> <+33%201%2085%2057%2071%2008> >>> Bureau 0144C (Ouest) >>> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France >>> *www.veolia.com <http://www.veolia.com>* >>> <http://www.veolia.com> >>> <https://www.facebook.com/veoliaenvironment/> >>> <https://www.youtube.com/user/veoliaenvironnement> >>> <https://www.linkedin.com/company/veolia-environnement> >>> <https://twitter.com/veolia> >>> >>> On 11 June 2017 at 18:21, Eugene Kirpichov <[email protected]> wrote: >>> >>>> Hmm can you simply do this in your main program after the pipeline >>>> finishes? >>>> >>>> p.run().waitUntilFinish(); >>>> ... Send report ... >>>> >>>> On Sun, Jun 11, 2017, 1:50 AM Morand, Sebastien < >>>> [email protected]> wrote: >>>> >>>>> Yes this use case can be treated by using parallel operation. >>>>> >>>>> I have a 2nd one, I would like to send a report at the end of the >>>>> pipeline when the last line has been written in bigquery: number of lines >>>>> treated, number of lines ignored (from another part of the pipeline using >>>>> graph as you described), number of files at the begining, and so on. >>>>> >>>>> This report could be: >>>>> >>>>> 1. Write a pub/sub >>>>> 2. Send an email >>>>> 3. Call an url with parameters >>>>> >>>>> Is this possible? >>>>> >>>>> Regards, >>>>> >>>>> >>>>> >>>>> *Sébastien MORAND* >>>>> Team Lead Solution Architect >>>>> Technology & Operations / Digital Factory >>>>> Veolia - Group Information Systems & Technology (IS&T) >>>>> Cell.: +33 7 52 66 20 81 / Direct: +33 1 85 57 71 08 >>>>> <+33%201%2085%2057%2071%2008> >>>>> Bureau 0144C (Ouest) >>>>> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France >>>>> *www.veolia.com <http://www.veolia.com>* >>>>> <http://www.veolia.com> >>>>> <https://www.facebook.com/veoliaenvironment/> >>>>> <https://www.youtube.com/user/veoliaenvironnement> >>>>> <https://www.linkedin.com/company/veolia-environnement> >>>>> <https://twitter.com/veolia> >>>>> >>>>> On 11 June 2017 at 04:14, Eugene Kirpichov <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi! >>>>>> It sounds like you want to write data to BigQuery and then load the >>>>>> same data back from BigQuery? Why? I'm particularly confused by your >>>>>> comment "nothing left in the PCollection" - writing a collection to >>>>>> BigQuery doesn't remove data from the collection, a PCollection is just a >>>>>> logical description of a dataset, not a mutable container. Transforms are >>>>>> like mathematical functions - they don't change their inputs, they only >>>>>> compute their outputs. >>>>>> >>>>>> Perhaps that you're assuming that Beam pipelines can only be a strict >>>>>> linear sequence of transforms? That is not the case - pipelines are an >>>>>> arbitrary graph, you can use a collection multiple times, i.e. apply >>>>>> multiple transforms to it. E.g. you can both write the collection to >>>>>> bigquery (step 3) and apply some other transform to the same collection >>>>>> (step 5). >>>>>> >>>>>> Assuming you use Java: >>>>>> PCollection<Foo> foos = p.apply(TextIO.read().from(...)).apply(...some >>>>>> transform...); >>>>>> foos.apply(BigQueryIO.write().to(...)); >>>>>> PCollection<Bar> bars = foos.apply(...some other transform...); >>>>>> bars.apply(BigQueryIO.write().to(...)); >>>>>> >>>>>> Let me know if this helps. >>>>>> >>>>>> On Sat, Jun 10, 2017 at 3:42 PM Morand, Sebastien < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Is there any way to add some step after a Write, because Write >>>>>>> return un PDone, so I can't do anything, but I would like actually do >>>>>>> something. >>>>>>> >>>>>>> Example : >>>>>>> >>>>>>> 1. Load data from gcs >>>>>>> 2. Some transform >>>>>>> 3. Write data into bigquery >>>>>>> => Nothing left in the pcollection, but when 3 is over => >>>>>>> 4. Load data from bigquery >>>>>>> 5. Some other transform >>>>>>> 6. Write data into bigquery >>>>>>> >>>>>>> Any way to do that? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> *Sébastien MORAND* >>>>>>> Team Lead Solution Architect >>>>>>> Technology & Operations / Digital Factory >>>>>>> Veolia - Group Information Systems & Technology (IS&T) >>>>>>> Cell.: +33 7 52 66 20 81 / Direct: +33 1 85 57 71 08 >>>>>>> <+33%201%2085%2057%2071%2008> >>>>>>> Bureau 0144C (Ouest) >>>>>>> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France >>>>>>> *www.veolia.com <http://www.veolia.com>* >>>>>>> <http://www.veolia.com> >>>>>>> <https://www.facebook.com/veoliaenvironment/> >>>>>>> <https://www.youtube.com/user/veoliaenvironnement> >>>>>>> <https://www.linkedin.com/company/veolia-environnement> >>>>>>> <https://twitter.com/veolia> >>>>>>> >>>>>>> >>>>>>> ------------------------------------------------------------ >>>>>>> -------------------------------- >>>>>>> This e-mail transmission (message and any attached files) may >>>>>>> contain information that is proprietary, privileged and/or confidential >>>>>>> to >>>>>>> Veolia Environnement and/or its affiliates and is intended exclusively >>>>>>> for >>>>>>> the person(s) to whom it is addressed. If you are not the intended >>>>>>> recipient, please notify the sender by return e-mail and delete all >>>>>>> copies >>>>>>> of this e-mail, including all attachments. Unless expressly authorized, >>>>>>> any >>>>>>> use, disclosure, publication, retransmission or dissemination of this >>>>>>> e-mail and/or of its attachments is strictly prohibited. >>>>>>> >>>>>>> Ce message electronique et ses fichiers attaches sont strictement >>>>>>> confidentiels et peuvent contenir des elements dont Veolia Environnement >>>>>>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc >>>>>>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce >>>>>>> message par erreur, merci de le retourner a son emetteur et de le >>>>>>> detruire >>>>>>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la >>>>>>> publication, la distribution, ou la reproduction non expressement >>>>>>> autorisees de ce message et de ses pieces attachees sont interdites. >>>>>>> ------------------------------------------------------------ >>>>>>> -------------------------------- >>>>>>> >>>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------ >>>>> -------------------------------- >>>>> This e-mail transmission (message and any attached files) may contain >>>>> information that is proprietary, privileged and/or confidential to Veolia >>>>> Environnement and/or its affiliates and is intended exclusively for the >>>>> person(s) to whom it is addressed. If you are not the intended recipient, >>>>> please notify the sender by return e-mail and delete all copies of this >>>>> e-mail, including all attachments. Unless expressly authorized, any use, >>>>> disclosure, publication, retransmission or dissemination of this e-mail >>>>> and/or of its attachments is strictly prohibited. >>>>> >>>>> Ce message electronique et ses fichiers attaches sont strictement >>>>> confidentiels et peuvent contenir des elements dont Veolia Environnement >>>>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc >>>>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce >>>>> message par erreur, merci de le retourner a son emetteur et de le detruire >>>>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la >>>>> publication, la distribution, ou la reproduction non expressement >>>>> autorisees de ce message et de ses pieces attachees sont interdites. >>>>> ------------------------------------------------------------ >>>>> -------------------------------- >>>>> >>>> >>> >>> >>> ------------------------------------------------------------ >>> -------------------------------- >>> This e-mail transmission (message and any attached files) may contain >>> information that is proprietary, privileged and/or confidential to Veolia >>> Environnement and/or its affiliates and is intended exclusively for the >>> person(s) to whom it is addressed. If you are not the intended recipient, >>> please notify the sender by return e-mail and delete all copies of this >>> e-mail, including all attachments. Unless expressly authorized, any use, >>> disclosure, publication, retransmission or dissemination of this e-mail >>> and/or of its attachments is strictly prohibited. >>> >>> Ce message electronique et ses fichiers attaches sont strictement >>> confidentiels et peuvent contenir des elements dont Veolia Environnement >>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc >>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce >>> message par erreur, merci de le retourner a son emetteur et de le detruire >>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la >>> publication, la distribution, ou la reproduction non expressement >>> autorisees de ce message et de ses pieces attachees sont interdites. >>> ------------------------------------------------------------ >>> -------------------------------- >>> >> >> >
