Thanks for the quick response. I'll keep digging. On Wed, May 11, 2016 at 3:14 PM Dan Halperin <[email protected]> wrote:
> Hi Bobby, > > You are correct that unbounded sinks are currently implemented as > PTransforms that interact with an external service in a ParDo. Even bounded > sinks are implemented this way -- look at the internals of the Write > transform -- the sink class itself is just a convenience wrapper for one > common pattern. > > Thanks, > Dan > > On Wed, May 11, 2016 at 1:00 PM, Bobby Evans <[email protected]> wrote: > > > I have been trying to get my head wrapped around some of the internals of > > beam so I can come up with an architecture/plan for STORM-1757 > > <https://issues.apache.org/jira/browse/STORM-1757> / BEAM-9 > > <https://issues.apache.org/jira/browse/BEAM-9>. > > > > I see that there are Sources and Sinks. Sources can be unbounded, but > > there appears to be no equivalent to an unbounded Sink. What I do find > are > > things like WriteToBigQuery which despite some internal complexity ends > up > > being an idempotent transform producing a PDone. Is this the intended > way > > for data to be output from a streaming DAG? > > > > I will likely have more questions as I dig more into state checkpointing, > > etc. > > > > Thanks, > > > > Bobby Evans > > >
