This has been a common problem I've run into with lots of built-in IOs, I've generally submitted PRs for them to add support for emitting something once writed are completed.
On Wed, Mar 24, 2021 at 1:04 PM Vincent Marquez <vincent.marq...@gmail.com> wrote: > > *~Vincent* > > > On Wed, Mar 24, 2021 at 10:01 AM Reuven Lax <re...@google.com> wrote: > >> Does that work if cassandra returns a PDone? >> > > No, it doesn't work. I wrote my own CassandraIO.Write that is a > PTransform<PCollection<A>, PCollection<A>> instead. > > I'm just asking if there's a better way of doing this because I'm having > to do this with multiple types of Writers, and don't want to have to hand > roll my own Write for each IO type I need this pattern for. > > >> >> On Wed, Mar 24, 2021 at 10:00 AM Chamikara Jayalath <chamik...@google.com> >> wrote: >> >>> If you want to wait for all records are written (per window) to >>> Cassandra before writing that window to PubSub, you should be able to use >>> the Wait transform: >>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Wait.java >>> >>> Thanks, >>> Cham >>> >>> On Wed, Mar 24, 2021 at 9:49 AM Alexey Romanenko < >>> aromanenko....@gmail.com> wrote: >>> >>>> Do you want to wait for ALL records are written for Cassandra and then >>>> write all successfully written records to PubSub or it should be performed >>>> "record by record"? >>>> >>>> On 24 Mar 2021, at 04:58, Vincent Marquez <vincent.marq...@gmail.com> >>>> wrote: >>>> >>>> I have a common use case where my pipeline looks like this: >>>> CassandraIO.readAll -> Aggregate -> CassandraIO.write -> PubSubIO.write >>>> >>>> I do NOT want my pipeline to look like the following: >>>> >>>> CassandraIO.readAll -> Aggregate -> CassandraIO.write >>>> | >>>> -> >>>> PubsubIO.write >>>> >>>> Because I need to ensure that only items written to Pubsub have >>>> successfully finished a (quorum) write. >>>> >>>> Since CassandraIO.write is a PTransform<A, PDone> I can't actually use >>>> it here so I often roll my own 'writer', but maybe there is a recommended >>>> way of doing this? >>>> >>>> Thanks in advance for any help. >>>> >>>> *~Vincent* >>>> >>>> >>>>
