Thanks - the general ideas seem solid, I added some questions/comments as well.
On Fri, Nov 10, 2023 at 1:32 PM Robert Bradshaw via dev <dev@beam.apache.org> wrote: > Thanks. I added some comments to the doc and open PR. > > On Wed, Nov 8, 2023 at 12:44 PM John Casey via dev <dev@beam.apache.org> > wrote: > > > > Hi All, > > > > I've written up a design for adding DLQs to existing Beam IOs. It's been > through a round of reviews with some Dataflow folks at Google, but I'd > appreciate any comments the rest of Beam have around how to refine the > design. > > > > TL;DR: Make it easy for a user to configure IOs to route bad data to an > alternate sink instead of crashing the pipeline or having the record be > retried indefinitely. > > > > > https://docs.google.com/document/d/1NGeCk6tOqF-TiGEAV7ixd_vhIiWz9sHPlCa1P_77Ajs/edit?usp=sharing > > > > Thanks! > > > > John >
