I have been trying to get my head wrapped around some of the internals of
beam so I can come up with an architecture/plan for STORM-1757
<https://issues.apache.org/jira/browse/STORM-1757> / BEAM-9
<https://issues.apache.org/jira/browse/BEAM-9>.

I see that there are Sources and Sinks.  Sources can be unbounded, but
there appears to be no equivalent to an unbounded Sink.  What I do find are
things like WriteToBigQuery which despite some internal complexity ends up
being an idempotent transform producing a PDone.  Is this the intended way
for data to be output from a streaming DAG?

I will likely have more questions as I dig more into state checkpointing,
etc.

Thanks,

Bobby Evans

Reply via email to